Cloudinary Blog

Webhooks, upload notifications and background image processing

by Tal Lev-Ami
Web applications and web APIs are usually programmed to perform requests in a synchronous manner. A common web-based access flow will start with a requester accessing a remote service. The remote service in turn will process the request and quickly return the result. 
 
But sometimes a web application will need to perform actions asynchronously in the background. 
 
Consider, for example, a computing intensive web request. Such a request might take a very long time to finish. UI wise, you might not want your users to actively wait so long for this command to finish. Technically, such a long running HTTP request might even time-out completely. 
 
In this case, you will probably want to employ a different access flow. In this flow, when the requester accesses the remote service, the remote service will return immediately with a 'pending' result. The remote service will then continue to perform the action in the background. Finally, when the remote service is finished, it will notify the requester that the required action was completed. 
 
The same concepts apply to Cloudinary's image management APIs. 
 
So far, Cloudinarys APIs were synchronous. For example, if you wanted an image transformed, you would have to wait synchronously until the transformation was finished.
 
In this blog post we wanted to introduce Cloudinary's new notifications and background processing capabilities that further extend the image management solution available when using Cloudinary.
 

Webhooks and upload notifications

Many mobile and web applications upload images and files to Cloudinary. Such uploading can be performed either from these services' backend or directly from their users' browsers or mobile applications. In such cases you might want to make sure your server is notified when an upload is completed. 
 
You can do that by setting the notification_url parameter of Cloudinary's upload API to any valid HTTP (or HTTPS) public URL.
 
For example, in PHP:
$result = \Cloudinary\Uploader::upload("sample.jpg", 
   array("notification_url" => "http://mysite/my_notification_endpoint"));
When the upload is completed, an HTTP POST request will be sent to the notification URL you provided. The post data will contain all upload result details as returned by our upload API (public ID, URLs, image dimensions, semantic data if requested and more). The request will also contain a signature of the sent data so you can verify the sender.
 
Here's a sample POST request with a JSON of the upload result:
POST /my_notification_endpoint HTTP/1.1
X-Cld-Timestamp: 1368881627
X-Cld-Signature: 29a383e289bc569310a8ca9899019a3167b4909e
Content-Type: application/json

{"public_id":"djhoeaqcynvogt9xzbn9","version":1368881626,"width":864,"height":576,"format":"jpg","resource_type":"image","created_at":"2013-05-18T12:53:46Z","bytes":120253,"type":"upload","url":"http://res.cloudinary.com/1233456ab/image/upload/v1368881626/djhoeaqcynvogt9xzbn9.jpg","secure_url":"https://cloudinary-a.akamaihd.net/1233456ab/image/upload/v1368881626/djhoeaqcynvogt9xzbn9.jpg"}
By the way, you can use RequestBin to test notifications received during development.
 
The signature is calculated as follows:     
SHA1-Hex-Digest("{data}{timestamp}{api_secret}")
Your server application can use this notification to update your model and database regarding the uploaded images or to cleanup unused uploaded files.
 
Here's another example in Ruby on Rails for generating a file input field that performs direct uploading from the browser using a jQuery plugin.
<%= cl_image_upload_tag(:picture, 
      :notification_url => "http://mysite/my_notification_endpoint") %>

Eager transformations in the background

By default, when you access Cloudinary delivery URLs, transformed images are generated dynamically if they don't already exist, according to the transformation parameters. You can also use Cloudinary's eager transformations, in which case multiple transformed versions of an image are created eagerly while uploading so they are ready for fast delivery even for the first download request.
 
Sometimes, you might need to create a single time consuming transformation or many eager transformations of an uploaded image. Creating such transformations might take seconds, especially if the original image is very large. In such cases, you probably don't want your user to wait for the process to complete. 
 
Using Cloudinary's asynchronous eager transformations, you can now define multiple eager transformations while uploading. The upload request will return immediately, the transformations will be performed in the background by Cloudinary and you will get notified (if you want) when the transformations are ready.
 
You can enable asynchronous eager transformations by setting the new eager_async upload API parameter to true. In addition, you can set the optional eager_notification_url to receive an HTTP POST request when the transformation generation is completed.
 
The following example in Python (or Django) creates multiple transformations eagerly in the background.
cloudinary.uploader.upload("sample.jpg", 
   eager=[dict(crop="fill", width="150", height="100"), 
          dict(effect="sepia", width="0.5", crop="scale")], 
   eager_async=True, eager_notification_url="http://mysite/my_notification_endpoint")
Here's an example of how the notification post request might look like:
POST /my_notification_endpoint HTTP/1.1
X-Cld-Timestamp: 1368883736
X-Cld-Signature: 09ef8c16f6e0ff85f5345d4813973de8f746448d
Content-Type: application/json

{"batch_id":"c0ed79e610f0636d15a1353a0a866933","notification_type":"eager","eager":[{"width":150,"height":100,"url":"http://res.cloudinary.com/1233456ab/image/upload/c_fill,h_100,w_150/v1368883730/noapoaq60varndxnh2or.jpg","secure_url":"https://cloudinary-a.akamaihd.net/1233456ab/image/upload/c_fill,h_100,w_150/v1368883730/noapoaq60varndxnh2or.jpg"},{"width":432,"height":288,"url":"http://res.cloudinary.com/1233456ab/image/upload/c_scale,e_sepia,w_0.5/v1368883730/noapoaq60varndxnh2or.jpg","secure_url":"https://cloudinary-a.akamaihd.net/1233456ab/image/upload/c_scale,e_sepia,w_0.5/v1368883730/noapoaq60varndxnh2or.jpg"}]}


Background sprites, animated GIFs and PDFs handling 

Cloudinary supports generating sprites: merging multiple images into a single large image for a more efficient browser loading using a generated dynamic CSS. 
 
Sometimes you might want to merge tens or even hundreds of images into a single sprite. 
 
Cloudinary's image manipulation are fast and efficient, but transforming and merging so many images might take more that your API client request can wait. In this case an HTTP request timeout might occur.
 
You can now tell Cloudinary to perform such sprite generations in the background by setting the async API parameter to true. In addition, you can set the notification_url parameter for receiving a webhook request when the process is completed (as explained above).
 
The following Ruby example requests for a sprite generation to be performed in the background:
Cloudinary::Uploader.generate_sprite("faces", :width => 150, :height => 100, 
   :crop => :fill, :async => true, 
   :notification_url => "http://mysite/my_notification_endpoint")
 => {"status"=>"processing", "batch_id"=>"163a919cde8f9317d6ee0fbfe6375dc4"} 
The API returns immediately with the response below while the later notification will include the final generated sprite details:
POST /my_notification_endpoint HTTP/1.1
X-Cld-Timestamp: 1368882586
X-Cld-Signature: d91bccba5f1bb38905fc6a9284914e8959fb4a7b
Content-Type: application/json

{"batch_id":"f97c6bf03074d9641f1a9c9b6bcebef7","css_url":"http://res.cloudinary.com/1233456ab/image/sprite/c_fill,h_100,w_150/v1368882583/faces.css","image_url":"http://res.cloudinary.com/1233456ab/image/sprite/c_fill,h_100,w_150/v1368882583/faces.png","secure_css_url":"https://cloudinary-a.akamaihd.net/1233456ab/image/sprite/c_fill,h_100,w_150/v1368882583/faces.css","json_url":"http://res.cloudinary.com/1233456ab/image/sprite/c_fill,h_100,w_150/v1368882583/faces.json","version":1368882583,"image_infos":{"face_center_eer8ss":{"width":150,"height":100,"x":0,"y":0},"face_left_b7ikxf":{"width":150,"height":100,"x":152,"y":0},"face_top_usjwvk":{"width":150,"height":100,"x":0,"y":102},"front_face_cywiib":{"width":150,"height":100,"x":152,"y":102}},"notification_type":"sprite"}
You can now create an animated GIF or a multi-page PDF document in a similar fashion. 
 
You can also "explode" a PDF in the background into multiple separated images representing all of its pages. All this is done using Cloudinary's 'multi' and 'explode' methods by setting the async parameter to true and optionally setting a notification_url. Here are some examples in Ruby:
Cloudinary::Uploader.multi("faces", :format => 'gif', :width => 150, :height => 100, 
   :crop => :scale, :async => true, 
   :notification_url => "http://mysite/my_notification_endpoint")
Cloudinary::Uploader.explode("multi_page_pdf", :format => 'png', :width => 200, 
   :height => 300, :crop => :scale, :page => :all, :async => true, 
   :notification_url => "http://mysite/my_notification_endpoint")

Summary 

As always, our goal with Cloudinary is to solve all of your web and mobile app's image management and image manipulation aspects for you and let you focus on your core business logic.
 
As a developer, you are probably familiar with the complexities involved in managing background processing and push notifications yourself. We believe that Cloudinary’s new webhooks and the ability to perform computing intensive image manipulations asynchronously in the background will prove as very useful tools in your image management tool-belt.
 
We're anxious to hear your feedback about these new features. Make sure you drop us a line in the comment thread below!
 

Recent Blog Posts

How mindbodygreen reduce costs & save development time

mindbodygreen is a digital media brand dedicated to health and wellness. The company’s website bring together leaders in the wellness world, best-selling authors, fitness experts, trusted healers, pioneering doctors, top chefs, celebrities and top-tier journalists to provide readers fresh, expert content and tools to help them live healthy, happy lives. The company was founded in 2009 and is headquartered in Brooklyn, N.Y. 
Read more

Why JPEG is like a photocopier

by Jon Sneyers
Why JPEG is like a photocopier (generation loss)

If you make a copy of a copy of a copy, the quality will deteriorate with every ‘generation’. This problem is called ‘generation loss’. It is not difficult to understand why this happens with actual copier machines. Scanning and printing are not perfect, being based on noisy sensors and physical paper and ink, and the resulting noise will tend to accumulate.

Read more
Apply conditions to dynamically manipulate images.

It's great to have the capability to manipulate images on the fly by using dynamic URLs to customize the images to fit the graphic design of your site or mobile application. However, what if you want to manipulate an image depending on a specific image characteristic (like its width or aspect ratio) or its contents (does it contain a face?). What you need is a way to apply a transformation to an image only if a specific condition is met. Take for example a situation where you have allocated space on your page for a user uploaded image with a width and height of 200 pixels. Furthermore, if the image contains a face you would like to zoom in and focus on the face itself, otherwise you would like to fit the entire image into the available space:

Read more
Moviepilot focus on core business and reduce costs
 
Moviepilot Inc. is a media company that hosts a website and social publishing platform designed to bring together movie and TV fans, and help studios trigger, fuel and amplify positive conversations about entertainment topics. The site invites fans to create content, such as writing articles and producing videos, which are then curated on the site and distributed to more than 30 million fans worldwide. The site boasts more than 20 million unique visitors per month, with 87 percent of visitors accessing the site on mobile devices. The company is headquartered in Venice Beach, Calif., and has an office Berlin, Germany, where it was originally founded.
Read more
Powerful facial detection for image manipulation

Many of the photos displayed on the internet these days are of people. If your website or mobile application displays photos that include people, you will want to make sure that their faces are included in the delivered images when cropping and manipulating them to fit your graphic design and responsive layout. You may even want to further manipulate an image according to the faces present, for example, adding a harlequin mask overlay on all of their eyes, where each mask is adjusted to the correct size and orientation (although not a typical use case, it's a cool example of using advanced facial attribute detection):

Read more