Wufoo & 4D Home

What is Wufoo?

Wufoo is a service for creating and handling Web forms. That may not sound like a lot on the face of it, but it offers a huge range of benefits for application developers, 4D or otherwise.

• Wufoo provides a killer on-line form editor. Honestly, it's an absolute pleasure to use.
• Your back end database is down? No problem. Form entries are stored on Wufoo's servers until you're ready to collect them.
• Optionally, new form entries can be forwarded in real-time to one or more destinations, including email, SMS, or your backend Web server + database.
• Easy integration with other API-oriented services, like MailChimp.
• A nice, clear API, outstanding documentation and first-rate technical support.
• A goodly range of customization options for both form behavior and appearance. For example, fields can be shown or hidden dynamically based on form entries and the appearance of virtually any element can be customized.
• A wide range of simple and complex field types, including support for file uploads.

Wufoo's feature set is broad, deep, and well documented. After reading this post, you should come away with several ideas about how Wufoo can simplify Web-based data collection. This page doesn't provide a ready-to-go set of code for integrating Wufoo into your system, but does include a lot of relevant (and sometimes little-known) details about working with Wufoo and it's main form APIs. Feel free to write with questions.

Building a Wufoo Form

The best way to get a feeling for Wufoo itself is to sign up for a free account and jump into the form editor. I'll wait, come back when you're ready...

Hopefully, you've taken a couple of minutes to try out Wufoo for yourself, or at least taken some of the video tour. If so, you've already seen Wufoo's fantastic drag-and-drop form editor in action. In just a couple of minutes you can pick a form template or build a form from scratch, such as the contact form below:

Anyone can build and great looking and functional form without messing around in a text or HTML editor, tinkering with CSS, or writing any JavaScript or backend code. Aaaaaah! That's a sigh of contentment that anyone who has messed around in HTML and CSS for hours on end should recognize. The picture below shows the form in a browser:

Here's the form again with data:

Full disclosure: The custom banner logo and slightly widened first and last name entry fields require a few minutes of work customizing the theme. This optional task doesn't take long. Theme customization isn't discussed further here.

Tip: Using Wufoo for Requirements Gathering and Collaborative Development

The Wufoo form editor is a powerful and unobtrusive requirements gathering tool, in some situations. Instead of just interviewing clients or spending time on prototypes or coding, sit down with them and build a form. Some users are even happy to build their own forms to show you what they want. There are some big advantages to giving customer's a hands-on role in the design process:

• You get better information. Instead of interpreting what people say, you can see what they mean. Better communication means less rework.
• You get stronger buy-in. People get hooked into things that they make and invest more in their success.
• Your program wins a friend. People tend to be possessive of "their" program...and nothing makes the system theirs like having a direct hand in its creation.

These benefits are a combination of practical and political, and that's all to the good..

Wufoo Form Features

Wufoo offers a range of features and options including integrated payments, forwarding entries via email, SMS, or an HTTP POST, feeding back entries in the response page, delegating response page handling to an external server, showing and hiding fields dynamically during entry, and a whole lot more. We'll take a look at some of these features here. The Form Settings page also serves as a good entry point to talking about the Wufoo API:

If you notice around the middle of the left side there's a link labeled "Template Tags" that brings up an additional page, as in the fragment below:

The API Key listed at the top of the page is a unique signature for your Wufoo account that applies to all of your form definitions and other data. The API key is a password so protect yours. The list of fields below are the HTML form field names and labels for the form. For example, the "Email" field is named Field3 internally, listed as 3 in the table above. When you receive the form data from Wufoo, the email field is sent as Field3, the first name as Field10, and so on. The fragments below show what the data looks like when it reaches your system:

JSON:

"Field2": "5415415411",
"Field3": "Dan@west.com",
"Field10": "Dan",
"Field11": "West"

HTTP POST:

Field2=5415415411&Field3=Dan%40west.com&Field10=Dan&Field11=West

Back on the Wufoo side, you can use field references to enhance the response page shown after a form is submitted. Look at the "Templating Options" link the Form Settings screen in Wufoo for details. As a quick example, here's the setup for echoing the first and last name values on the form acknowledgement:

Below is what the acknowledgement page looks like:

Working with the Wufoo API

The list of fields shown earlier is all you need to use the Wufoo form editor. But what if you want to pull entries stored on Wufoo into your database? What if you want to pull and parse the form definitions themselves? Wufoo's API allows for both of these scenarios, and several more. Wufoo's API is nicely designed, well-documented, and supports most every requirement I could think of for day-to-day use. We'll take a look at some of the API calls below.

The Entries API

When forms are submitted, the entry is saved in Wufoo. (With Wufoo, there's no assumption that you've got backend or database programming skills or resources.) You can view entries right from with the Wufoo Web interface:

Entries can be exported in a variety of formats from the Web:

The Web interface is fine for small jobs or when you're just getting acquainted with the data. With the Wufoo Entries API you can seamlessly integrate Wufoo and your main system.

Once you start storing entries on Wufoo and in your own backend, it's important to keep track of which entries you've already downloaded and processed. Wufoo supports this by including a unique ID value with each entry. For any particular form, the entry ID starts at 1 and goes up by one forever. If you delete an entry from Wufoo, the ID is lost and not reused. Below is how an entry looks when pulled over the API (entry ID highlighted for emphasis):

{     "Entries": [
        {
            "CreatedBy": "public",
            "DateCreated": "2013-11-04 18:08:52",
            "Status": null,
            "Field10": "Dan",
            "EntryId": "2",
            "UpdatedBy": null,
            "Field11": "West",
            "Field2": "5415415411",
            "DateUpdated": "",
            "Currency": null,
            "Field3": "Dan@west.com",
            "PurchaseTotal": null,
            "Field9": "United States",
            "MerchantType": null,
            "Field8": "",
            "Field17": "Yes, you can send me SMS",
            "Field13": "I am interested in learning more about your full range of products.",
            "Field4": "123 Main Street",
            "Field5": "",
            "Field6": "Maintown",
            "Field7": "CA",
            "TransactionId": null,
            "IP": "124.169.148.252",
            "LastPage": "1",
            "CompleteSubmission": "1"
        }
    ]
}

The following code fragment gives you an idea of what's involved in pulling down an entry:

Here's what the code's doing in outline form:

1. Build a URL specifying the form and entry number of numbers to pull.
2. Pass the URL to 4D's HTTP Get command, one one of its alternatives.
3. Parse the resulting JSON.
4. Extract the ID number of the entry.
5. Save the entry locally, if it hasn't been saved before.

Once the data is stored locally, the next step is to process it. What to do in this step depends entirely on what data the form collects. In the case above, we would likely save a [Contact] record. If the form handles RSVPs for a graduation ceremony, we would probably link a [Student] or their guests to a graduation [Event] record. Really, you can do just about anything with Web forms.

You only need to call JSON Release for:

JSON Parse Text/Blob/Document
JSON New Object/Array (only if it is the root object, so not for child objects)
JSON Retain (an advanced command for advanced uses, see the documentation)

JSON Get Object/Array do not need to be matched with a release. You are only obtaining a reference to an object that is a child of another object.

Getting Form Submissions in Real Time with a WebHook

When form entries are submitted, they're saved by Wufoo. This feature is a Very Good Thing. Why? Because then if your backend is down, nothing is lost. Ever. As mentioned, each entry for a specific form is given a unique ID. The number series for the IDs starts at one and goes up from there. If you ever delete entries, the entry ID is not reused. This is all helpful because you can use the Wufoo Entries API to pull any form submissions that you're missing in your database. But what if you want entries to pass directly into your backend? That's a pretty good idea for a couple of reasons:

• There are limits on how many API calls you can make in a day.
• Sometimes it's better to get results into your database in real time.

Then again, it really is great that form entries keep working even when your backend is down for maintenance. With this in mind, I recommend using a WebHook to get as many entires as possible in real-time and also implementing the Wufoo Entry API to scan for any submissions that didn't make it through. So long as you track which entry IDs have been handled already, you don't need to worry about double-processing the same request.

To get started with real-time processing, you configure a WebHook using the Notifications section of the Form Manager.

Besides being saved to the Wufoo servers, new entries can be forwarded via email, SMS and/or one or more external programs:

Wufoo provides built-in links for several external service APIs, like MailChimp and Twitter, or you can create a custom WebHook. When you want to pass data to 4D or some other custom system, you define a custom WebHook.

The picture below shows a custom WebHook for the contact form:

A few quick points about WebHooks before getting into detail:

• You can send the same entry to multiple destinations.
• The entry is still saved on the Wufoo servers.
• If the target server is unavailable or returns an error, Wufoo will retry the send a few times and then give up. (Wufoo offers to show you an error log, when appropriate.)

For 4D developers, it's Very Good News that Wufoo saves the entries. If you need to restart your database for an update or some other reason, Wufoo is still able to capture entries and your site seems to be working without any interruption. This also points out what it's important to be able to pull entries via the API in case anything didn't make it through the WebHook system. (If it sounds like I'm repeating myself about this advantage to combining Wufoo & 4D, it's because I am repeating myself. It's a great, great architecture to have.)

Note: A WebHook sends a copy of an entry, it doesn't make an external server responsible for handling the form submission. All your 4D system needs to do is return a valid HTTP response to Wufoo to avoid setting a spurious integration error. If you would like to have your own system handle form confirmation, see the Form Settings page in the Form Manager.

The URL indicates where each completed entry should be sent and may, optionally, include an extra handshake field to help identify the form. The specified Web server gets the form contents as a standard HTTP POST along with the URL. Below is an example of what the POST payload looks like:

Field10=Dan&Field11=West&Field3=Dan%40west.com&Field2=5415415411&Field17=Yes%2C+you+can+send+me+SMS&Field4=123+Main+Street&Field5=&Field6=Maintown&Field7=CA&Field8=&Field9=United+States&Field13=I+am+interested+in+learning+more+about+your+full+range+of+products.&CreatedBy=public&DateCreated=2013-11-04+20%3A08%3A52&EntryId=2&IP=124.169.148.252&HandshakeKey=ContactUsForm

These are the same contents as we saw earlier in JSON format, only formatted differently. A simple strategy for dealing with the data arriving in different formats is:

1. Write a format-specific parser for JSON and HTTP POST.
2. Keep the parsing code distinct from the data processing code.

Since entries in JSON and send as HTTP POST values are both structured as name-value pairs, the natural solution is to parse everything into a parallel set of text arrays and then use accessor methods to read the array values. Below are some method names to help flesh out this idea:

• WufooEntry_ParseJSON
• WufooEntry_ParseHTTPPost
• WufooEntry_GetFormField
• WufooEntry_GetFormFieldExists

NoteUnless you send along full form meta-data with each submission, WebHooks do not include the form's unique ID. Wufoo's approach is to provide a handshake field to help you match up submissions with forms. You can also write a custom URL for each form to simplify identifying which form data you're handling.

The Forms and Fields APIs

To integrate data from a Wufoo form submission into your database, you need to know the structure of the form and its fields. The Wufoo Forms API and Wufoo Fields API provide all of the details that you need, albeit in a somewhat complex format. The JSON below describes the fields of the contact form we've been looking at:

That's a lot of detail for a pretty simple form, but all of it's necessary and most of it won't slow you down. Things get a bit more complex with some of Wufoo's fancy pants field types. Wufoo has a range of complex field types for names, addresses, file uploads and so on. These compound fields make it easy to lay out sophisticated and attractive forms, but the underlying data descriptors get pretty involved. As a quick snapshot, below is a picture of a 4D table structure that holds Wufoo forms, fields, and entries. The [Wufoo_Field_Choice] list table is logically but not physically linked child table to the [Wufoo_Field] table.

Before getting overwhelmed by the potential complexity, keep in mind that the incoming entry data is in simple name-value pairs. The only big task you need to face is mapping the Wufoo field names, like Field10 to your local database names, like $first_name or [Contact]First_Name.

Field Mapping: Write a Code Generator

Wouldn't it be nice if there were a way to automate mapping Wufoo form entries to internal targets? Well, when you've got a programming language that's integrated with a database, it's not too hard. Even if you only generate the basic field extraction and assignment code, that's already a big chunk of the work required to copy incoming data to 4D records. To make this clearer, look at the screenshot below of a Wufoo form's raw JSON definition in a 4D form:

This next screenshot shows the Wufoo field definitions parsed into a listbox for easy display:

On this screen, you can see the connection between the fields shown in a browser and the entry data you get via HTTP or the Entries API. For example, you can see that the Contact Name field is of the complex type shortname and that its child fields are First and Last. Without going into too much detail, incoming data for this field maps like this:

If you remember, these same details are visible up at Wufoo in the Form Editor:

Note: The API ID is numeric but the name is Field + field ID, such as Field10.

The last step is to specify where you want incoming data to go, entered manually into the Assign to Hint column pictured below:

Even in this crude form, it's now easy to automatically generate data handling code that assigns incoming data to local targets. Since the Wufoo field names are meaningless on their own, it's handy to make your generator add comments. Below is an example of the kind of code you can produce.

The generated code pulls the incoming data out of Wufoo and assigns it to local variables. Even at this limited level of functionality, there are some big wins to the generated code system:

• No name mistakes. Wufoo's form field names are unique but meaningless. It is too easy to make an assignment mistake when writing this sort of code by hand.
• Good comments. Notice how the comments indicate the field's label and, for nested field, some information about the parent field. This helps a lot when you're reading the code. Personally, I find parser code impossible to follow without sample inputs and comments like these.
• Simplicity. With a dialog and code generator in place, building stub code is quick and easy. Even if the code is only getting 80% of the work done, it's doing it in 1% of the time..

The manufactured code shown earlier is real, but written to match a specific project's requirements and style. You are likely to have different requirements and an alternative style. If you do write a Wufoo field mapper and code generator, consider how perfect you want it to be. Aiming for a 100% complete code generator would be pretty tough. Getting the basic field mapping part down, as shown above, takes a reasonable level of effort. From there, it's easy enough to hand-code the logic for loading and saving the data, as appropriate.

Automating Wufoo Interactions

With all of the different options for data delivery, APIs for data and meta-data retrieval, it's worth thinking about how to automate your system efficiently. There are many approaches to Wufoo integration and the best approach will depend on your requirements, resources, and style. As an example, in one Wufoo-4D system, the automated components break down like this:

The dialog pictured above shows three background processes that handle some aspect of the Wufoo connection:

• WufooEntry_Processor runs in the background locally to process any entries that have been pulled from Wufoo bot not yet handled. No network calls are required for this as the data is already stored locally .
• WufooEntry_Reader calls the Entries API on a regular schedule to copy down new entires. Note that you're limited to 5,000 API calls a day, which goes faster than you might imagine. I've got the WufooEntry_Reader set to run every 30-60 minutes to avoid hammering the Wufoo servers
• WufooForm_Reader calls the Forms and Fields APIs to update the form definitions locally. If something is new or changed, it's noted for review. In my case, I pull the definitions once a day, which is more than enough in most cases. Checking for unknown modifications is helpful on projects where more than one person can edit the forms in Wufoo. Otherwise, one person can change a form and only later be surprised to learn that the new field data hasn't been extracted.

The WufooEntry_Processor deserves a bit more comment. When you pull in data from Wufoo or get a submission via a WebHook, you can process the entry on the spot or save it to a table for later handling. I like to save entries for review and to avoid processing the same EntryID. From there, it's largely a matter of system architecture and personal preference if you want to process values immediately or not.

Gratuitous Screenshots

Below are a couple of spare screenshots to give you an idea of how Wufoo data can look inside a 4D system. First, a list of form entries:

The sheet view below shows a parsed form entry:

This page displays the raw JSON for an entry:

Missing Features and Rough Spots

Below is a grab bag of negatives and rough spots I found while working with Wufoo. Many of these are fairly small points but, chances are, others will run into the same issues that I did.

• Forms cannot be started, saved, and resume. Therefore, an end user has to complete an entire form or else lose all of their work. This is Very Annoying. The best strategy is to keep each form a manageable size. In the plus column incomplete entries are saved on Wufoo and you can determine how much was completed before the user gave up.
• The JSON structure for the form definition API is pretty involved and not clearly documented in one place.
• Form entries posted via a WebHook that include the system field CompleteSubmission set the value to 0, meaning "false". This is incorrect. The CompleteSubmission field is meaningless in a WebHook because a hook only ever fires when a form is completed. Therefore, the CompleteSubmission needs to be overridden locally to 1.
• The EntryID number sequence can get holes in it and the Entries API only tells you how many entries have been saved for a form. You can end up with 100 entries on the server where the highest ID is 118, for example. This makes it harder to scan efficiently from the backend. A decent solution would be an API call that pulls an array of stored EntryID values for a form...but no such call exists.

Like I said, there are some pretty fine points in the list ;-)

Wufoo Versus its Competitors

Wufoo has plenty of competitors, including Formstack, JotForm, and Gravity Forms. If you've got some experience with one of Wufoo's competitors and are willing to share your thoughts, please drop me a line.

Let me Know

I'd love to hear from other people integrating Wufoo into their 4D apps. Feel free to send me questions and I'll reply, as my schedule allows. I'm also a coder for hire if you need something more extensive done.