Text Size

Chris Carpenter's blog

Setting the default date to GMT in a CFWheels ColdFusion App

I've been working on an app in ColdFusion for a client that needs to use dates/times across multiple countries, and thought I'd post this in case anyone else needs it. We are using CFWheels and I needed a way to ensure that the dates the framework automatically creates for the createdat and updatedat fields are in GMT time. There are a couple of  ways to do this, the first being to set your ColdFusion JVM settings to include a default timezone using an argument like "-Duser.timezone=GMT" in the JVM arguments fields of the CF Administrator. However, this will then set it for all apps on the ColdFusion server. If you only need it for the one CFWheels app, you can override the following method in your Model.cfc class:

1
2
3
4
5
6
7
<!--- Override the default timestamping with the GMT time --->
<cffunction access="public" name="$timestampProperty" output="false" returntype="void">
        <cfargument name="property" required="true" type="string">
                <cfscript>
                        this[arguments.property] = DateConvert(&quot;local2utc&quot;, now());
                </cfscript>
</cffunction>

This will ensure that all date values saved to the database will useUTC/GMT time.  

Dynamic Element Text in Doculicious

Version 3.5 of Doculicious, our online tool to help anyone create fillable pdf forms, was released a few weeks ago, and in addition to some performance enhancements a great new feature was added which we are calling "Dynamic Element Text". Basically, this lets you display the text value from a field in your form elsewhere in the final PDF. So if you ask for someones First Name and Last Name on page 1 of the form, you can then show those values later in your final PDF, without having to ask the user to fill them in on every page you need them. Details on how to use this feature can be found here on the Documentation site.

We also made many performance enhancements in this release, which will give improvements across most functions of Doculicious, but especially the entries page when viewing report of your data and for deleting data from your entries or trash. The PDF upload conversion tool has been tweaked to create nicer looking templates from uploaded PDF files. Previously the text could look a little jaggy, but now it is very smooth and looks much nicer. 

Enhanced page movement features in Doculicious

We've had a lot of requests from users for more advanced functions for adding pages to templates in Doculicious, our tool to help anyone make web generated PDF files, so we thought we'd add these in the latest release. Previously you could only add new pages at the end of a template, but now we've added the ability to add pages anywhere, as well as move pages around to anywhere within the template too. The new interface looks like this:

 

The page column lists all the pages in your template. You can click on this link to change to that page for editing. Clicking on the red (x) will delete the page. Be careful with this feature, as you cannot get a deleted page back. To move pages you can choose the new page number in the dropdown. So to make page 3 the first page, you sould choose "to Page 1" in the drop down next to page 3 and then click the move button. This would move page 3 to page 1, and push page 1 and 2 down so that they would become page 2 and 3.

How To - Sell your PDF files by integrating Doculicious and PayPal

We've had a lot of requests for payment integration with Doculicious, and this is something we've thought about adding. One issue is that Doculicious is a little different than other form builders - a lot of the time the people who contact us want to sell the output of the form itself and not use the form as just an order form. People are selling Visa applications, Government forms, Award certificates and a range of other forms using Doculicious. Integrated payment within Doculicious will be a while away, so I thought the next best thing was to write a comprehensive tutorial on how to integrate Doculicious and PayPal.

This tutorial will explain how you can sell a PDF form on your website using Doculicious and PayPal, but the concepts here can easily be used to gather payment for any other type of order. After going through the tutorial, you will be able to create a process that is something like this:

  1. Customer fills in your Doculicious form from your website;
  2. When complete, they are redirected to a sales page where they can purchase the completed form for $10;
  3. They click through to PayPal and enter their payment details;
  4. They return to a Thank you page on your site;
  5. PayPal sends you an "Instant Payment Notification" informing you of the payment success or failure;
  6. On success, you send the customer a link to the completed PDF form.

This is going to be a fairly long post, so I've split it into sections and included a table of contents to make it easier to navigate. I'm going to go through the whole process - from how to set up a PayPal sandbox account for testing, to using the Doculicious API to retreive a submitted entry. So feel free to skip bits you don't need.

Table of contents

  1. Setting up a PayPal testing environment
  2. Creating a PayPal "Buy Now" button
  3. Create and configure the Doculicious form
  4. Set up your PayPal purchase form
  5. Create the PayPal IPN page
Setting up a PayPal testing environment

There are a couple of ways to test your PayPal buttons and integrations. The first, and easiest, is to use your normal business account and take advantage of the 60 day refund option that PayPal provides. This lets you cancel a transaction and all the fees will be refunded. PayPal lets you have 2 accounts - one business and one personal. So you just use your personal account to purchase from your business account and then refund after the transaction. Easy, but messy. After a small while it gets pretty annoying to keep refunding.

The second way is to use the PayPal testing environment ("the PayPal Sandbox") where you can create multiple buyer and seller accounts to test your buttons. This is much cleaner, and is pretty easy to set up.

First, go to the PayPal Developer web site and sign up for an account. Use a different email address than your other PayPal accounts. Once you are signed up, log in.

The PayPal sandbox let's you pretend to be a buyer and a seller, and create fake accounts with essentially play money and credit cards. For our purposes, we are just going to create one buyer and one seller account. Click on the "Create a preconfgured account" link.

You will then be shown a form like the following one. Select the account type as "buyer", choose an email address and make a note of the password. Make sure you add some money to the account balance.

Once created, go and do the same again, but make this one a "Seller" account. When you're done, you should have 2 accounts in your sandbox, like so:

We can now use these accounts to simulate our purchases, and also create buttons for testing without worrying about breaking our production setup. It's a good idea to write down both the email addresses and the passwords you noted down when creating them. Do this in notepad so you can access them easily later.

Remember, you need to be logged into the PayPal sandbox to use the mock buyer and seller accounts.

Whenever you make a purchase, you need to use the full buyer email address created for you. When you sell, you need to use the seller email address. Paypal also let you see all email communications. We won't be using that too much, but it is useful to know.

In the next section you need to create the button for selling your PDF, so enter the sandbox site using your pretend seller account. Click on the radio button next to the seller, and press on the "Enter Sandbox Test Site" button. A new window will display where you can log in using the password noted earlier. You are now inside the PayPal sandbox, and should recognize this as similar to your normal account screen.

Creating a PayPal "Buy Now" button

To sell our Doculicious PDF form, we are going to use a PayPal "Buy Now" button. You do this within the PayPal site by clicking on the "Merchant Services" tabk at the top, then click on the "Buy Now Button" link near the top. There are 3 steps to create your button:

Step 1 - Choose the button type and enter details

We need the Template UID here because later it is used when we call the Doculicious API.

  • The Item Name can be anything you want. 
  • The Item ID needs to be the Unique ID of your Doculicious Template. You can find this from the Doculicious Dashbord by clicking on the orange "api" link, and looking for the text labeled "Template UID".
  • Enter a Price
  • Enter the Currency
  • Continue on to Step 2

Step 2 - Track inventory, profit and loss

This step is pretty small, just make sure the check box to save the button at PayPal is clicked. You don't need to do this, but it makes it easier to edit later. When done, continue to Step 3

/p>

Step 3 - Customize advanced features

This step lets you add some advanced features, one of which is what we use to have PayPal notifiy us when a purchase is made. This is called the "Instant Payment Notification" and we can set up the URL that PayPal will notify here. There are also a few other things we can turn off, as they are not really needed, such as shipping address and customer message. If you would like this information, feel free to keep them turned on.

The IPN is how PayPal notify us of a successful payment. We don't send the completed PDF until we receive this, and it's valid!

  • Enter a URL for when customers cancel their checkout. This should be a page on your website with some information on how they can continue their purchase.
  • Enter a URL for then customers finish their checkout. This should be a page on your website that informs the customer of what they just purchased and when you will deliver it. We will go into more detail about this later when we create these pages.
  • In Advanced Variables, make sure the checkbox is clicked, and add the following line:

    notify_url=https://www.YOURWEBSITE.com/paypal/ipn.php

    (You can use HTTP here instead of HTTPS if you dont have an SSL certificate for your site, but you should really get one for security purposes.)

  • When done, click on the Create button at the bottom.

The next page will display the code that you need to put onto your website for this button to work. Make sure you copy and paste this into a text file for later. We need to make a couple of small changes before this goes onto your site. These will be outlined in a later section of the tutorial.

Create and configure Your Doculicious form

Click here to learn how to build a Doculicious form

If you are reading this tutorial, you most likely have a Doculicious form that you want to sell or integrate with a Payment gateway. If not, you can use one of the examples from the "Template Gallery" or create a new form first. You don't need anything fancy to follow this tutorial, you could even just create a basic test form to get everything working first. Once your form is created within Doculicious, you need to activate it which will automatically add it to your Doculicious Dashboard.

The Dashboard is where you can change the settings for your form. Below we explain what changes to make and why they are made. Below this is a quick summary (for those that don't like to read :)

Disable PDF download

You don't want the user to be able to download the PDF file once they have completed the form, so checking this will disable that feature.

Clicking on Disable PDF download will automatically check the Save Entries box. 

Optionally set notifications

When Save Entries is turned on, the system will allow you to set notifications. On premium accounts, you can also attach the completed PDF to the notification email, or use custom notifications to send the PDF to different email addresses depending on choices made on the form. This is a good time to set any notifications you may want to receieve. Don't use this to send the user their PDF file, as they wouldn't have paid at this point in time.

Turn on Redirect 

Please note that Redirect is a premium feature, and will not be available on free accounts.

After your user completed the Doculicious form, you want to redirect them to your payment page. So you need to check the Redirect box and enter the URL to your payment page. We will edit this page at a later step of this tutorial to add the PayPal button you created earlier, so now you can just put in the URL where you are going to put that page. It may be a good idea to create that page now so that you can test the redirect working. 

Summary

Ok, fo all you non-readers out there ... this is the quick summary of what we just did to configure the Doculicious form for PayPal integration:

  1. Turned on Disable PDF Download
  2. Turned on Redirect
  3. Entered a Redirect URL to a page with the PayPal button on it
  4. Turned on Save Entries (Optionally, added notifications.)
  5. Clicked Save
Set up your PayPal purchase form

We now need to create the page that people are redirected to after they complete your Doculicious form. This is pretty easy, and the main thing you need on this page is the PayPal button code that you copied earlier. However, we are also going to add another field which contains our unique entry ID of the submitted form. 

So, open up the page you set as the Redirect URL from the earlier step, and past in your PayPal button code. It should look similar to this:

1
2
3
4
5
6
<form action="https://www.sandbox.paypal.com/cgi-bin/webscr" method="post">
    <input type="hidden" name="cmd" value="_s-xclick">
    <input type="hidden" name="hosted_button_id" value="957FC8G74KTR4">
    <input type="image" src="https://www.sandbox.paypal.com/WEBSCR-640-20110429-1/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
    <img alt="" border="0" src="https://www.sandbox.paypal.com/WEBSCR-640-20110429-1/en_US/i/scr/pixel.gif" width="1" height="1">
</form>

Now we need to add a small bit of PHP code before the PayPal button code. This gets the Unique ID of the entry just made so that we can send this to PayPal.

1
2
3
4
5
6
7
8
9
10
11
12
13
<?php 
    //Check that the doculicious URI is set - It should be
    if(isset($_GET['dclcs_entry_uri']))
    {
        $uri = $_GET['dclcs_entry_uri'];
 
        //Put all the parts of the URI into an array
        $uri_parts = explode("/", $uri);
 
        //Get the last item in the array. This is the uniqeID of the submitted form 
        $document_id = array_pop($uri_parts);
    }
?>

Ok, we have the $document_id variable with the unique ID in it, so we can add a hidden field to the PayPal button code that contains this using the custom PayPal variable, which will ultimately be passed to the IPN code:

1
<input type="hidden" name="custom" value="<?php echo($document_id);?>">;

Add this line in between 2 of the hidden fields of the PayPal button.

This page will just have a PayPal button and nothing else, so you should add some more text above the button explaining things to your customer. Something like:

Thank you for using our service. Please click the button below to pay for your form using PayPal. Once confirmed we will send through a link where you can download the completed PDF file.

The completed code for the PayPal form page can be viewed here.

Create the PayPal IPN page

Ok, this is the final step - creating the PayPal IPN page. This is what links everything together to provide the PDF to your customer, and tells you that the payment was successful. You will remember that we provided PayPal the address of this page when creating our custom button, and it was something like this:

1
notify_url=https://www.YOURWEBSITE.com/paypal/ipn.php

You now need to create or edit that page and put in the PHP code which will talk to PayPal to verify the payment, then talk to Doculicious to get the completed PDF and finally email your customer with the details on how they can download the file. I will go through the file line by line explaining what we are doing. Top help you follow along, you can download an example containing all the code here.

In the first part of the IPN page we declare some default variables that are used later on in the script. This is where you customize the script with your details. The inline comments will explain what all the settings are for.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
//CONFIGURATION OF DEFAULT VARIABLES
//Your PayPal Seller email address (Change to your PayPal sandbox email for testing)
$paypal_email = "chris@digitalcarpenter.com.au";
 
//The currency you use for your PayPal buttons
$currency = "USD";
 
//The cost of your form. Make sure you add the cents
$cost = "10.00";
 
//The email address used to send you notification emails
$from_email = "email@YOURDOMAIN.com";
 
//The email address used to receive notification emails on success or failure
$notification_email = "notifyme@YOURDOMAIN.com";
 
//Your API access ID and Security key from the API link on the dashboard 
$api_access_id = "xxxxxxxxxxxxxxxx";
$api_security_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
 
//Send this back to PayPal to validate the fields
$req = 'cmd=_notify-validate';

 

The following lines are to verify with PayPal the values that are being sent. PayPal is doing a POST to this IPN page with all the details of your transaction. What you are doing here is sending those details back so that PayPal can verify you have the right information. This is to help ensure that people can't post to your IPN page and tell you about fake transactions.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
//Add each field to a value we will post back to PayPal
foreach ($_POST as $key => $value) 
{
        $value = urlencode(stripslashes($value));
        $req .= "&$key=$value";
}
 
//Post back to PayPal
$header .= "POST /cgi-bin/webscr HTTP/1.0\r\n";
$header .= "Content-Type: application/x-www-form-urlencoded\r\n";
$header .= "Content-Length: " . strlen($req) . "\r\n\r\n";
 
//PayPal Production
$fp = fsockopen ('ssl://www.paypal.com', 443, $errno, $errstr, 30);

 

Next we check if there was a failure communicating with the PayPal servers, and if there was send an email to the notification address you defined earlier with the details:

 

1
2
3
4
5
6
7
8
9
10
11
if (!$fp) 
{
 
        //Send an email on failure with the error message (errstr) from paypal
        $from = "From: " . $from_email;
        $to = $notification_email;
        $subject = "HTTP ERROR";
        $body = $errstr;
 
        mail($to, $subject, $body, $from);      
}

 

If we got a result from PayPal, we then check if it is "VERIFIED", and if so get the POSTed values that PayPal has sent us:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
else
{
    fputs ($fp, $header . $req);
    while (!feof($fp)) {
        $res = fgets ($fp, 1024);
        if (strcmp ($res, "VERIFIED") == 0) 
        {
            //The custom field contains the unique Doculicious Document ID of the completed form and the item_number contains the Template ID.
            $item_number = $_POST['item_number'];
            $document_uid = $_POST['custom'];
                        
            $item_name = $_POST['item_name'];
            $payment_status = $_POST['payment_status'];
            $payment_amount = $_POST['mc_gross']; 
            $payment_currency = $_POST['mc_currency'];
            $txn_id = $_POST['txn_id'];
            $receiver_email = $_POST['receiver_email'];
            $payer_email = $_POST['payer_email'];

 

We need to check if some values are what we expect them to be. Payment Status should be "Completed", the price should match what we have configured, as well as the currency and reciever email address:

 

1
2
3
4
if (($payment_status == 'Completed') &&      //check the payment_status is Completed
    ($receiver_email == $paypal_email) &&    //check that receiver_email is your Primary PayPal email
    ($payment_amount == $cost ) &&           //check they payed what they should have
    ($payment_currency == $currency))        //check that payment_amount/payment_currency are correct <span _fck_bookmark="1" style="display: none; "> </span>

Once we confirm that all that is good, we can now use the Doculicious API to get the PDF and save it to our website somewhere. To do this, we are going to use CURL. First we need to set the parameters for this:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
//The full authorization string - needed for the header. API access and security codes are set at the top of the page
$auth = "authorization:DCS " . $api_access_id . ":" . $api_security_key;
 
//This is the URL to get completed the completed PDF for a specific entry. 
$resource_uri = "https://api.doculicious.com/entries/" . $item_number . "/" . $document_uid . ".pdf";
 
//Initialize curl and set URL and other appropriate options
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $resource_uri);
 
//Send the authorization details in the header
curl_setopt($ch, CURLOPT_HTTPHEADER, array($auth));
 
//This tells curl to return the actual result of the API call (ie, the PDF or XML error message). Returns False if HTTPS get fails.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE );
 
//SSL Settings
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
 
//Execute the API call
$output = curl_exec($ch);

The last line of the above snippet of code executed the cUrl request, the next bit of code will process the response we receive from Doculicious. First we check that there were not any cUrl errors, then we get information about the cUrl response so we can check if Doculicious sent us what we wanted.

The 200 result means everything went OK. If Doculicious couldn't find the PDF we asked it for, we would have gotten a 404 response, and the code below would have printed out an error message.

If everything is OK, we save the file to our website, and email the customer with a URL to where they can download it.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
if(!curl_errno($ch))
{
        $info = curl_getinfo($ch);
 
        //200 means success.
        if($info['http_code'] == 200)
        {       
                //Save the PDF file to our website
                $my_file = "/YOUR/WEB/ROOT/customer-files/" . $document_uid . ".pdf";
                $fh = fopen($my_file, 'wb') or die("can't open file");
                fwrite($fh, $output);
                fclose($fh);
 
                //Create the URL where the customer can download their PDF
                $download_url = "http://www.YOURDOMAIN.com/files/" . $document_uid . ".pdf";
                
                //Send an email to the purchaser with a link to download the PDF file
                $from = "From: " . $from_email;
                $to = $payer_email;
                $subject = "Your PDF form is ready.";
                $body =  'Thank you for your purchase. Please use the link below to download your PDF file:' . PHP_EOL . PHP_EOL .  
                                 $download_url . PHP_EOL . PHP_EOL .  
                                 'Regards,' . PHP_EOL .
                                 'YOUR NAME' . PHP_EOL;
       
                mail($to, $subject, $body, $from); 
        }
        else //there was an error, so output the content
        {
                header("Content-type: " . $info['content_type']);
                echo $output;
        };
}

The rest of the file contains some error handling, basically sending notifications when things go wrong.

The completed code for the Doculicious PayPal IPN can be seen here.

3D printing feels like the future

3D Lion Dog model

I've been interested in 3D modelling and animation since the late 90's after installing a demo of Lightwave 3D I found on the front of a magazine. I think I had an old Pentium 2 or 3 back then, and remember spending hours dragging pixels around the screen to create all types of crazy 3D models, like dragons and knights in shining armour. It began as a hobby, but soon became my key to entering the IT industry. I began building 3D architectural movies for CD-ROM and then moved into web development, but now most of my time is spent programming, setting up servers or playing around with JavaScript, CSS and HTML. I still have a soft spot for 3D modelling, and keep my version of Lightwave up-to-date every few years as I find something I want to model.

3D Lion Dog PrintThe idea that you could physically print out a 3D model is surprising to many people, but It's been around for a while in many industries for rapid prototyping, metal casting and design visualisation. Personally, I've been looking at 3D printers and waiting for the price to come down on the ones with great detail, or for the quality to rise on the ones that are relatively cheap. But thanks to the internet and some enterprising companies, I don't have to wait for this any longer, and it's really exciting. 

Shapeways, i.materialise  and Ponoko are 3 companies that are leading the way in a new manufacturing revolution. All of them let you upload your 3D design and have it printed in a multitude of different materials from textured plastics to glass and metal. The costs are surprisingly cheap, the quality is surprisingly good and it feels like the future has arrived.

On the right is a photo of my first printed model that I've just recieved from Shapeways. Above is a rendering of the 3D model the print is of. It's a Lion Dog from the Forbidden City in Beijing which I made a few years ago from photos taken while on holiday. To make something on the computer, putting hundreds of hours of work at all hours of the morning, and then to now be able to print it out and hold it in your hand is amazing and exciting. To think that in the future this technology will only get better, cheaper and more widespread is awe inspiring. The ability for anyone to build pretty much anything without the need for large manufacturing processes or factories will change so many things.

Many people are writing and talking about how this could change the world and others are leading the way creating 3D printers for everybody. Soon this will be everyday, and it won't just be art or 3D geeks printing out their hobbies, it will be people printing replacement parts for all types of things, from cars to kitchen gadgets. All those things you can't buy anymore, or break and are impossible to get parts for ... soon you will just print out a new bit and be good to go. The future is so exciting.

 

 

 

 

Doculicious 3.1 - Custom emails, validation and more

Another release of our web form and PDF builder - Doculicious 3.1. It's late and I'm tired, so not going to write much, but I created a new doculicious form that explains it all. The biggest change is the Custom Emails. Now you can use fields in your form to determine who gets an email with the PDF attached. Either let people type in their email address (doculicious will validate it too) or use a drop down list to determine who gets an email (or both. Yes, you can set 2 custom email options for each form). Great for when you need to send the form to different people within your company depending on some chosen field.

So, here's the sample form. If you want to see the final PDF, make sure you put in your email address and click one of the new submit buttons (which are another great feature . use your own images and turn off our default submit buttons if you don't like them). Enough talk. Sleep.

Automatically create web forms from PDF forms

I love this photo ... it sort of sums up how I feel after releasing the latest version of our online PDF web form tool, Doculicious.com: happy, a little delirious and looking to relax on the beach after a hard bit of work. Sydney's getting cold now though, so I'll probably just stay in and program some more cool features.

Anyway, this latest release has something which has been on the drawing board for a while. The ability to upload a PDF file with fields and have it converted into a web form has been requested by many of our users, and I'm happy to say that our first version of this is ready to go. Our goal with this release was to enable someone with a bunch of PDF forms to upload them and have them automatically converted to web forms. This means that if you have a PDF file with fields, you will have a working web form in minutes after uploading it to Doculicious. If the original file doesn't have fields, the system will still help, but there will be some manual work to do. I'll give some tips on re-creating a form like this very quickly, so don't worry ... if your PDF files don't have fields, this new feature will still cut the time it takes to create a form by at least half.

PDF's are messy

PDF files are not nice ordered documents that make it easy to extract text. However, they print very well, and will pretty much look the same when sent to any printer. This gave us a clue as to how to implement the form conversion. We take the uploaded PDF and generate screenshots of each page by printing to an image file. We then create a new Doculicous template and use the images of each page as a background for that page. If the PDF has form fields, we re-create each one in the correct place, ensuring that drop down lists are filled with the right options, checkboxes and radio buttons are created and using the right groups and that some of the same formatting is applied, such as text alignment. Doculicious has a great feature that will scale images to load faster when viewed on the web, but will use a larger image when output as PDF. This lets converted forms stay small for the web form, but still print nice when downloaded.  Ok, enough background, here is the detail.

Converting your PDF to a web form

Doculicious makes this really simple. Just follow these steps:

Step 1 - Go to "Your templates" and click on the "Upload form" button

This will show the "Choose file" or "Browse" button , depending on your web browser. 

Step 2 - Click to choose a file to upload

You can choose a PDF file, postscript file or image files. Only PDF have form fields, but postscript can have multiple pages, and can be useful to convert a MS Word form into a web form. Images can be used to create single page templates, and are great if your source form is a scan of a paper form.

Step 3 - Wait for the PDF to Web Form conversion to complete

It should only take a few seconds, but PDF's with many pages could take a while. When completed, refresh the page to see your new template at the top.

Step 4 - Web form goodness! You're ready to rumble

That's it. If your PDF form had fields, they should be converted properly into Doculicious elements, so all you should need to do now is load up the template and check that everything looks ok, then press publish and activate it on your dashboard. Some things you may want to consider is turning on number validation for numeric only fields and checking that each form element has a nice name or your CSV downloads may be hard to decipher. Also, you may find that some elements may not line up perfectly with the background image. In my experience text boxes all work fine, but radio buttons and check boxes may be off by a few pixels. A small nudge using the arrow keys will fix this up in no time.

My PDF doesn't have fields, what do i do?

Don't worry, atleast 50% of your form is complete. You just need to add the text boxes and move them into position. Luckily you have a nice background image that will help you do this, and Doculicious' keyboard shortcuts will help you get this done fast. We'll soon be posting a video capture of how to do this really quick, but for now make sure you use the "c" for clone keyboard shortcut. Most text fields on a PDF form are the same width, border style, font style etc. You just need to create one text field, put it in place, and then press "c" select the new element and place in the right position, then press "c" and move the new one down to the right position, then press "c" ... you get the idea. Forms that use to take 60 minutes to re-create can be done in about 10 minutes or less!

Doculicious update 3.0

This weekend we completed testing on the new version of  Doculicious, our PDF web form creator and have released it to the production site. This version is the 3rd major revision of the site, and includes a few visible changes, but also a lot of internal changes that are required for the continuing updates that will be released this year.

The visible updates released this weekend are a PDF form upload and conversion function, which allows users to upload an existing PDF form with fields, and have it converted into a Doculicious web form, ready to be used for gathering information almost straight away (depending on the original form :) I'm hugely excited about this feature, and have been playing with it for days. It's great to get a PDF file with a bunch of fields and have it automatically converted into a web form using Doculicious. It feels almost magical ... (even though I know the amount of work that went into making it happen). I'll be writing a post just on this feature with some details on how you can use it to convert your existing PDF forms to be web enabled.

The second update includes a new validation framework which saw alot of work going on in the background to allow future additions. The visible part of this release is Numeric validation for text fields and List options for text fields, allowing drop down list entry.

We plan a bunch of releases this year, and here's a teaser of some of the things we have on the drawing board:

  1. Nicer editing within the Template Workshop with things like snap to grid and multiple selection and editing of elements.
  2. Numeric calculations on text elements.
  3. Updates to the form display, including submission buttons and page controls at the bottom of forms.
  4. Ability for users to come back and edit forms before submission via the use of a password and unique URL for thier form entry.
  5. Routing of email notifications based on a list field within the template.
  6. File uploads as data within a template.

I'll try to expand on these in future blog posts.

New website and Blog

We've just launched our new website for Digital Carpenter Pty Ltd. The old site ended up being more of a blog and had less information about the company and what we do, so this change was very much in need. This blog will also become the main channel for announcing new features for our online fillable PDF generator, Doculicous. I've got alot plans for a couple of big releases to add some much needed and asked for functionality to Doculicious, and am pretty excited to get back into it.  

Decreasing AJAX web app loading times

When developing Doculicious, one of my main concerns was how fast the heavy JavaScript parts would load and run in the browser. Doculicious uses a fairly customised version of the Dojo Toolkit, and the raw files that were being loaded for some pages added up to over 550KB - made up of about 50 individual files, all being loaded by Dojo with its packaging system. This was fine for development and great for debugging, but I knew I had to do something before this went into production. In this post I'll talk about some of the things we use in Doculicious to make our JavaScript smaller and load faster, but still easy to develop on. Hopefully it should be general enough to help anyone who has a site using JavaScript, but we use Dojo so is a bit focused on its packaging system which includes multiple files at run-time. If your app doesn't use Dojo but has multiple JavaScript files, you'll probably still get some benefit out of this.

There are 2 main ways to speed up a JavaScript application - Make the JavaScript load faster into the browser, and make it run quicker. There's a great article at IBM developerWorks called AJAX performance analysis that goes into both of these. The first section talks about the tools to use in Web application development - Firebug and YSlow, which are both required tools on any development box I use. The next goes into reducing the network transfer time, and this is where I'll focus on what we did with doculicious.

Reducing File Size and HTTP requests

The Dojo Toolkit uses a packaging system similar to Java, and it's a great way to separate out all the files and keep them organised and easy to develop. We've created our own package for doculicious and one of our JavaScript heavy pages may require up to 50 individual files to be included. Because of current browser connection limits, this is way too much, and it's much better to limit the number of JavaScript files to one or two instead of loading many. What I wanted to do for doculicious was to create individual files for each JS heavy page, with each file including just the JavaScript code it needed. This individual file could then be used with the dojo.js file so that all the required JavaScript for a page would be in 2 files. Because Dojo automatically includes JS files depending on which widgets and functions are used, I needed to define which ones were used on which page so that I could join them all together. Firebug is great for this, and using the Net tab quickly lets you see all the included JS files:

My initial goal was to create a range of "base" files that could be concatenated together in an Ant script to create a JavaScript file for any doculicious page. Using Firebug to find the common files across each of our pages, I made a list of all the Dojo specific files that were needed, and also of our custom doculicious files. At this point I realised that each of our pages used very similar Dojo functions and widgets, so I made the decision to manually join these together into one file that could be included across all the files I needed. This is probably not the best thing to do if you are using the latest version of Dojo or another packaging system, as when changes are made it will be harder to integrate them, but our base Dojo is version 0.4.3 and I do not plan on upgrading it. We've only found a couple of bugs over the past 18 months, so I'm comfortable that this base is stable.

Concatenating your JavaScript into a single file is great for production code, but you still need to develop on it. So when doing this we need to keep our individual files available for development, while having a simple process to make the production code. I use Eclipse with the MyEclipse plugin as my IDE. The JavaScript project is structured like below, with the js folder being included in the web project so that it is deployed along with it:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<span style="font-weight: bold;">JavaScript Project</span>
&nbsp;&nbsp; - <span style="font-weight: bold;">buildscripts</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">build</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">functions</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">widgets</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">lib</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - build.xml
&nbsp;&nbsp; -<span style="font-weight: bold;"> js</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">digitalcarpenter</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">system</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Command.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - DocumentState.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - State.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - ...
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">widgets
</span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Canvas.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Elements.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - ...
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - <span style="font-weight: bold;">dojo</span>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - dojo_functions_widgets_minimal.js
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - dojo.js

Underneath the dojo folder is the dojo_functions_widgets_minimal.js file which contains all the dojo functions and widgets that our pages need. The digitalcarpenter folder is our base package, and contains all our individual JavaScript files, including widgets and system classes. Under the buildscripts folder is our build file and a build folder to hold temp files. The lib folder holds some tools we will use later. The ant script is pretty basic, here's what it does:

  1. At the beginning we define the paths that are used and the files we are going to create.
  2. We then build the directory structure for the build
  3. Then in copy we move all the files we are going to use into the build/functions and build/widgets. Taking care to not move stuff that isn't needed, such as testing files, images and css.
  4. Next we define a regular expression replace to remove all lines with "dojo.require" on them. This line is the way Dojo knows which files to include, and it's not needed in our final file because all the code is together. In fact things break if these are left in. Depending on which JS library you use, you may not need to do something similar when cancatenating your files together.
  5. The next area is where we actually create our specific file. First we copy the base file to our working directory, then we concat all the individual files together, appending them to this file. Order may be important. Some of the widgets in our code include other widgets, so they need to be added before-hand.
  6. The last step is calling ./lib/custom_rhino.jar. This is the Dojo ShrinkSafe application which removes comments and shrinks down the code. I highly recommend that you use this. ShrinkSafe has worked great for all of my JS, and I've never had errors. Because it parses the JavaScript it does a great job. It will even tell me if I've left some horrible bug in the code.
  7. At the end is the create and clean methods you use to run the whole thing.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
        <?xml version="1.0" encoding="ISO-8859-1"?>
<project name="DigitalCarpenter_JavaScript" default="create"
        basedir=".">
        <description>
                Creates the javascript files needed for Doculicious.
        </description>
        <!-- set global properties for this build -->
        <property name="build" location="build" />
        <property name="widgets" location="${build}/widgets" />
        <property name="functions" location="${build}/functions" />
        <property name="lib" location="lib" />
        <property name="dojo_src" location="${basedir}/../js/dojo" />
        <property name="dc_src" location="${basedir}/../js/digitalcarpenter" />
        
        <property name="dojo_base" value="dojo_base" />
        <property name="dojo_minimal" value="dojo_functions_widgets_minimal" />
        <property name="base_workshop_dojo_file" value="workshop_dojo" />
        <property name="document_workshop_js" value="document_workshop" />
 
        <target name="init">
                <!-- Create the build directory structure used to create the JS files -->
                <mkdir dir="${build}" />
                <mkdir dir="${widgets}" />
                <mkdir dir="${functions}" />
        </target>
        
        <target name="copy" depends="init">
                <copy file="${dojo_src}/${base_workshop_dojo_file}.js" todir="${build}" />
                <copy file="${dojo_src}/${dojo_minimal}.js" todir="${build}" />
                <copy todir="${functions}">
                        <fileset dir="${dc_src}/system" />
                </copy>
                <copy todir="${widgets}" includeEmptyDirs="false">
                        <fileset dir="${dc_src}/widget">
                                <exclude name="<u>_package_</u>.js" />
                                <exclude name="ButtonField.js" />
                                <exclude name="InlineEditor.js" />
                                <exclude name="RichEditor.js" />
                                <exclude name="TestWidget.js" />
                                <exclude name="Popup.js" />
                                <exclude name="templates/*" />
                                <exclude name="templates/buttons/*" />
                                <exclude name="templates/images/*" />
                        </fileset>
                </copy>
        </target>
        
        <target name="remove_dojorequires" depends="copy">
                <replaceregexp match='dojo.require' replace="//dojo.require" byline="true">
                        <fileset dir="${widgets}"/>
                </replaceregexp>
                
                <replaceregexp match='dojo.require' replace="//dojo.require" byline="true">
                        <fileset dir="${functions}"/>
                </replaceregexp>
        </target>
 
        <target name="document_workshop" depends="remove_dojorequires">
                <copy file="${build}/${base_workshop_dojo_file}.js" 
                               tofile="${build}/${document_workshop_js}.js" />
                <!-- NOTE: The order of the below files is important -
                            later code expects the previous code to be available -->
                <concat destfile="${build}/${document_workshop_js}.js" append="true">
                <filelist id="dw_functions" dir="${functions}">
                            <file name="Utils.js"/>
                            <file name="SHA1.js"/>
                            <file name="Constants.js"/>
                            <file name="Command.js"/>
                            <file name="DocState.js"/>
                        </filelist>
                        <filelist id="dw_widgets" dir="${widgets}">
                                <!-- file name="Resizer.js"/ -->
                            <file name="ImagePicker.js"/>
                            <file name="FloatingImagePicker.js"/>
                            <file name="TextField.js"/>
                            <file name="Element.js"/>
                            <file name="Canvas.js"/>
                            <file name="ColorPalette.js"/>
                            <file name="FontSizePicker.js"/>
                            <file name="FontPicker.js"/>
                            <file name="RichTextToolbar.js"/>
                            <file name="Dialog.js"/>
                        </filelist>
                </concat>
                
                <antcall target="-rhino-compress">
                        <param name="srcFile" value="${build}/${document_workshop_js}" />
                </antcall>
                <!-- Copy the completed file to the js/dojo directory - 
                           will overwrite the one already there -->
                <copy overwrite="true" 
                               file="${build}/${document_workshop_js}.js" 
                               tofile="${dojo_src}/${document_workshop_js}.js" />
        </target>
        
        <target name="-rhino-compress" unless="nostrip">
                <copy overwrite="true" file="${srcFile}.js" tofile="${srcFile}.uncompressed.js" />
                <java jar="./lib/custom_rhino.jar" 
                              failonerror="true" 
                              fork="true" 
                              logerror="true" 
                              output="${srcFile}.js">
                        <arg value="-strict" />
                        <arg value="-opt"/>
                        <arg value="-1" />
                        <arg value="-c" />
                        <arg value="${srcFile}.uncompressed.js" />
                </java>
        </target>
        
        <target name="create" depends="clean, init, copy, remove_dojorequires, document_workshop" 
                          description="creates the javascript files">
                <echo message="All done! :)" />
        </target>
 
        <target name="clean" description="clean up">
                <!-- Clean up the files used -->
                <delete dir="${build}" />
                <delete dir="${widgets}" />
                <delete dir="${functions}" />
        </target>
</project>

Cool, we've now got our Ant file ready to create the individual files. The example above creates only one file. Our production one creates a couple of different files. Because this allows you to pick and choose individual files you can customise the generated JS very specifically. Doculicious has two main "applications". One is the front end web form and the other is the template/style creator. They use similar functions and widgets, but doing it this way I can save about 30kb on the uncompressed files by only including the stuff each page needs.

Now when I'm developing on my code I can just comment out the single file call at the top of the HTML page and uncomment the dojo.require lines to use the individual files, allowing us to program on nicely formatted files that have comments and line breaks. When everything's tested and ready for prod we just re-comment the dojo.requires, uncomment the individual line, run ant create and increment the jsVersion variable so that clients get the new version.

1
2
3
4
5
6
7
8
<!-- Comment out the following line to test the individual js files
<script type="text/javascript" src="/js/dojo/document_workshop.js?v${jsVersion }"></script>              
-->
<script type="text/javascript">
        <!-- Uncomment these to test the individual js files-->
        dojo.require("digitalcarpenter.widget.Element");
        dojo.require("digitalcarpenter.widget.Canvas");
        dojo.require("digitalcarpenter.system.DocState");

Now our production code is all nicely together in one file. We still have commented and formatted code to develop on, and running it all through ShrinkSafe has also lowered the file size. Our uncompressed file has gone from 370KB to 262KB, but most importantly we've gone from about 50 individual file calls to 2. The file size still seems large, but with mod_deflate on Apache2 it comes down to 68KB and with aggressive caching, clients should only need to download it when we change it.