Terminal icon LAUNCH EXTERNAL PROCESS, My Secret Shame

Like I don't know how many 4D developers, I've underused LAUNCH EXTERNAL PROCESS. It's not because I'm afraid of the command line. I'm not. It's because I've found it frustrating to construct statements that work correctly with LAUNCH EXTERNAL PROCESS. It seems like I can readily build instructions that work perfectly in Terminal, only to see them crash and burn with LAUNCH EXTERNAL PROCESS. I like to call this kind of problem a "4D bug"....but that fools no one.

I recently wrote a piece on PhantomJS & 4D and and finally took the time to sort through my LAUNCH EXTERNAL PROCESS problems on OS X. Oh. 4D bug solved! It turns out I've been a victim of the paradox of the active user. Put simply, "I'm too busy to save time!" After briefly and systematically looking at what was going wrong, I discovered that LAUNCH EXTERNAL PROCESS works fine on OS X and that it's pretty easy to use.

Rather than clutter the PhantomJS & 4D piece with remedial information on OS X paths, POSIX, and permissions, I've done this separate write up. My own problems came down to quoting and that's addressed below. I've also included other relevant information for people that haven't used or integrated 4D with the OS X command line much before. You'll find that a lot of the samples discussed here speak about PhantomJS specifically but the overall principles apply more generally. The main topics are listed here:

Sample Database

The LEP_Code database (400 KB Zip) includes all of the code described here and some additional utilities besides. There are no visual demos or code examples included, just some handy utility code. If you already have the PhantomJS & 4D demo, you probably won't need the LEP_Code demo.

System_LaunchExternalProcess

LAUNCH EXTERNAL PROCESS behaves slightly differently between Windows and OS X and has some helpful optional. The System_LaunchExternalProcess method simplifies calling LAUNCH EXTERNAL PROCESS while allowing access to the command's discretionary features. In its simplest form, all you need to do is pass the wrapper a command for execution:

System_LaunchExternalProcess ($command_text)

In its longest form, you can specify if the Windows console window should be hidden (hidden by default), if LAUNCH EXTERNAL PROCESS should wait for stdout and stderr to be populated, and retrieve the contents of one or both of these results:

System_LaunchExternalProcess ($command_text;$hide_console;$call_synchronously;->$stdout;->$stderr)

Tell OS X Where to Look for Executables

Depending on what and how you install command line tools under OS X, the OS may or may not find them. For example, the PDFtk Server toolkit uses a package installer that attempts to register its location with the OS automatically. Once an executable is known to the OS, you can run it by its short name from any directory in the system. Built in commands like curl, netstat, traceroute and so on work this way. If you ever want to find the path to a command on OS X, run the which command, like so:

OSXMachine:~ FruitBat$ which traceroute
/usr/sbin/traceroute

The traceroute command is located in the /usr/sbin/ directory.

If command line installation doesn't update the OS, you may find that commands don't work when you pass in code using LAUNCH EXTERNAL PROCESS. PhantomJS is a good example here because it uses drag-and-drop installation and no installer. Below is a PhantomJS example that converts a Wikipedia page about Musk Lorikeets into a local PDF file:

phantomjs rasterize.js 'https://en.wikipedia.org/wiki/Musk_Lorikeet' Musk_Lorikeet.pdf

If you download PhantomJS and the rasterize.js script to your machine and run the example listed above, chances aren't you'll get an error similar to the one below:

-bash: phantomjs: command not found

The error message is right because the operating system has no idea where the PhantomJS binary is located. There are at least three ways to make the OS figure out where programs are stored:

  1. Update the PATH variable relevant to your shell to include the directory that holds the PhantomJS executable.
  2. Move the program into a directory that the OS automatically scans for executables and make sure that the OS updates it's list of known binaries.
  3. Pass full paths to the command line.

Changing the PATH variable is an appealing option on a development machine or server. If you're typing directly into Terminal, long command paths are tedious at best. On the other hand, full paths are a great approach when you're using LAUNCH EXTERNAL PROCESS. In fact, this may be the only viable alternative for code running on machines where you are unable to update the command line environment. As an example, below is the earlier PhantomJS example again with fully expanded paths (whitespace added for legibility):

"/Users/FruitBat/Documents/PhantomJS_and_4D/Database/Resources/PhantomJS/Mac_OS/phantomjs"
"/Users/FruitBat/Documents/PhantomJS_and_4D/Database/Resources/PhantomJS/Scripts/rasterize.js"
"https:/en.wikipedia.org/wiki/Musk_Lorikeet"
"/Users/FruitBat/Desktop/Musk_Lorikeet.pdf" Letter

That would all be too much to type, but it doesn't matter with LAUNCH EXTERNAL PROCESS. The only time you'll ever need to see raw commands is while developing or debugging LAUNCH EXTERNAL PROCESS calls so it doesn't matter if the paths are short or long. To get an idea of what constructing a complete call looks like, look at the Example_MuskiesToPDF example below (also found in the PhantomJS & 4D demo):

C_TEXT($command_text)
$command_text:=""
$command_text:=$command_text+Char(Double quote)+PhantomJS_GetCommandCLIPath+Char(Double quote)
$command_text:=$command_text+" "
$command_text:=$command_text+Char(Double quote)+PhantomJS_GetScriptCLIPath+Char(Double quote)
$command_text:=$command_text+" "
$command_text:=$command_text+Char(Double quote)+"https://en.wikipedia.org/wiki/Musk_Lorikeet"+Char(Double quote)
$command_text:=$command_text+" "
$command_text:=$command_text+Char(Double quote)+PhantomJS_GetDirectoryCLIPath+"Musk_Lorikeet.pdf"+Char(Double quote)
$command_text:=$command_text+" Letter"

// If you want to grab the command and try it out in Terminal, set a breakpoint
// here and then drag the curser into the SET TEXT TO PASTEBOARD assignment below.
If (False)
  SET TEXT TO PASTEBOARD ($command_text)
End if

If ($command_text#"")
  C_TEXT($stdin;$stdout;$stderr)
  LAUNCH EXTERNAL PROCESS ($command_text;$stdin;$stdout;$stderr)
End if

In the PhantomJS example code above, the functions ending with CLIPath return "Command Line Interface formatted Paths." The LEP_Code sample database includes several general-purpose path conversion functions, including File_ConvertPathToCLIPath and File_ConvertPathToPosix. We'll look at some path syntax and formatting issues next.

OS , 4D, Terminal, Paths and POSIX

Short version: Wrap paths in quotes for the OS X command line.

Back in the original Mac OS, the native folder separator is a colon. Under the OS X UNIX version of the Mac OS, the folder separator is the / character. The command line requires UNIX (POSIX) paths, not paths with colons. Additionally, many special character allowed in Mac file names must be escaped for use on the command line. Notably, spaces and virtually all common punctuation marks cannot be used in paths without special handling. If you ever want to see how a path name should be formatted for use on the command line, find a file in the Desktop and then drop it into a Terminal window. As an example,the path to a document named Hello World.pdf in the Desktop shows up like this in Terminal:

/Users/FruitBat/Desktop/Hello\ World.pdf

Tip: Terminal's integration with the Desktop is one of OS X's great hidden features. Not only does the system automatically escape paths correctly, you can drop items in as you build up a command line. For example, if you want to switch directories type cd and a space and then drop in a folder. Hit return and now your command line working directory is reset to the folder you dragged into Terminal. Likewise, type open and a space and then drop in something from the desktop to open the folder, document or application. The example below shows how simple it is to open something from the Terminal:

open /Applications/4D_V13/13.4/4D.app

The File_OpenUsingTerminal method from the LEP_Code database handles files, documents, applications and folders, even if there are special characters in the path:

  // File_OpenUsingTerminal
  // Not tested on Windows - it's an OS X thing.

C_TEXT($0;$error_name)
C_TEXT($1;$file_path) // Mac OS X path.

$file_path:=$1

$error_name:=""

Case of
  : (System_OnWindows )
  $error_name:="File_MethodNotSupportedOnWindows"

  : ($file_path="")
  $error_name:="File_PathIsEmpty"

  : (Test path name($file_path)#Is a document)
  $error_name:="File_PathDoesNotExist"
End case

If ($error_name="")
  C_TEXT($file_posix)
  $file_posix:=File_ConvertPathToPosix ($file_path)
  $file_posix:=Text_WrapInDoubleQuotes ($file_posix)

  C_TEXT($command_text)
  $command_text:="open "+$file_posix

  If (False)
    SET TEXT TO PASTEBOARD($command_text)
  End if

  System_LaunchExternalProcess ($command_text) // Ignoring errors.

End if

$0:=$error_name

4D includes commands to translate paths in various ways, including Convert path system to POSIX. This command's optional * parameter activates an encoding option. Unfortunately on OS X, URL encoding is used. This behavior is documented so I don't know if it's a bug or not, but the results are unusable at the command line. This is easy to understand because URL encoding escapes characters with a % which is itself a reserved syntactic character. With Terminal commands, special characters are escaped using a leading slash \. Mostly. The / character is converted to an escaped colon. A quick example shows how it all works...and how hard it makes file names and paths to read. First, a picture of a directory of sample documents:

Some hard paths to escape

Now have a look at the table below comparing the name of the file at the Desktop, in Terminal and as converted by Convert path system to POSIX with the optional * parameter. For clarity, only final document names are shown, not full paths.

Finder Terminal Convert path system to POSIX (*)
Hello World.html Hello\ World.html Hello%20World.html
Hello/World.html Hello\:World.html Hello:World.html
Hello\World.html Hello\\World.html Hello%5CWorld.html
Hello%World.html Hello\%World.html Hello%25World.html

The example database include a function named File_ConvertPathToPosix that correctly escapes spaces in OS X path names and a wrapper named File_ConvertPathToCLIPath that is meant to work cross-platform. File_ConvertPathToPosix's only special behavior is to correctly escape spaces on OS X. If you want to pursue this approach, the routine needs to be extended to escape additional restricted characters. A simpler option is to skip encoding and enclose full POSIX paths in single or double quotes. This technique is pretty painless:

  1. Get a regular OS X path with colons from Select document or any other source.
  2. Convert the path with Convert path system to POSIX without using the optional * parameter.
  3. Wrap the converted path with single or double quotes.

Now paths look much like you would expect at the finder except that they're in quotes and have / instead of : marks, like in an example we saw earlier (whitespace added for legibility):

"/Users/FruitBat/Documents/PhantomJS_and_4D/Database/Resources/PhantomJS/OS_X/phantomjs"
"/Users/FruitBat/Documents/PhantomJS_and_4D/Database/Resources/PhantomJS/Scripts/rasterize.js"
"https:/en.wikipedia.org/wiki/Musk_Lorikeet"
"/Users/FruitBat/Documents/Desktop/Musk_Lorikeet.pdf" Letter

The code in the example database includes routines named Text_WrapInDoubleQuotes and Text_WrapInDoubleQuotes as conveniences. Under OS , the command line accepts single or double quotes. Under Windows, the command line accepts double quotes. Special escaping or nested quoting may be needed under Windows for LAUNCH EXTERNAL PROCESS code that is sent to cmd.exe as an argument.

Check Permissions Under 4D Remote

The last tip has nothing to do with paths but can be the root cause of LAUNCH EXTERNAL PROCESS calls failing. If you're deploying to 4D Remote (4D Client) machines running OS X via one of the folders that 4D Server automatically copies, you may find that your executable doesn't work. Why? Because the permissions on the binary are lowered and lose their right to run. Specifically under OS X, the permissions on a file such as PhantomJS are lowered from 755 to 644 and need to be raised again. The PhantomJS_ResetPermissions routine in the PhantomJS & 4D demo sorts this out automatically at startup. The routine, listed below, automates call to the chmod command that you can reuse in similar situations:

If (PhantomJS_Exists )
    C_TEXT($command_text)
    $command_text:="chmod 755 "+Char(Double quote)+PhantomJS_GetCommandPosix+Char(Double quote)

    System_LaunchExternalProcess ($command_text)

End if

Let me Know

I always love hearing from people with ideas, suggestions and corrections. Feel free to send me tips or questions and I'll reply, as my schedule allows. I'm also a coder for hire if you need a hand.