Cloudinary Blog

Beyond face detection - smart cropping in the cloud using Imagga and Cloudinary

by Nadav Soferman
Beyond face detection - smart cropping with Imagga & Cloudinary

It’s a common challenge in many mobile and web applications: how do you allow users to upload their own images, while automatically adapting these images to a fixed graphic design?

A classic example is a user uploading a profile picture, but instead of providing a headshot (which is what we really need from them), they upload a picture of their entire body with additional objects in the background. Obviously this image will need to be cropped to the size of the profile picture, while focusing on the user’s face.

One way to achieve this is to allow the user to crop the image themselves as part of the upload process. But even so, there is usually a need to use the image in several different contexts, in different sizes and positions, and the user’s cropping can’t help us produce all these different versions.

A more advanced approach is face detection - there are new software tools and APIs that can recognize the face in the image, and crop the image automatically, focusing on the human face. Actually, face-detection based cropping is one of the more popular features of Cloudinary’s cloud-based image management solution and API.

But now we’d like to take  it one step forward: What if you want to automatically crop thousands of images, focusing on an object that is not a human face? For example, e-commerce products, food, animals, or anything else that is more difficult to detect algorithmically.

Take these three images of food plates - many images like these are uploaded by users of food sites, such as Cloudinary customers Yummi, Cibando and EatWith:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

The website needs to automatically generate large, small and thumbnail versions of these photos, smartly cropped to show the main portion of the food plates. There's definitely a need for smart cropping based on the most appealing part of a photo, no matter what kind of photo it is. This is an even bigger technical challenge than cropping based on face detection.

And that’s exactly what Imagga have done with their smart cropping technology - they built a system that can detect the most appealing part of any image and focus on it automatically. We have integrated Imagga into Cloudinary’s image management solution, and you can try it out as part of our free tier.

Read on to see how Cloudinary makes it possible to automatically crop complex images to focus on the most appealing part of the photo, and automatically generate beautiful thumbnails, with just one line of code.

Photo scaling and smart thumbnail generation

Let's continue the example of the food site which uses Cloudinary and Imagga to automatically generate thumbnails of plates of food. The site’s graphic design requires thumbnails of 200x90. Again, here are the demo images we showed above:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

Cloudinary has a crop mode called fill, which allows you to crop an image to certain dimensions, while automatically filling the space with the result of the crop. You can specify a gravity which tells Cloudinary in which direction the focus of the crop should be.

Here is a dynamic URL and code which crops one of the images above using fill mode, and sets gravity to north:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:fill, :gravity=>:north)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"fill", "gravity"=>"north"))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="fill", gravity="north")
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("fill").gravity("north")).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("fill").Gravity("north")).BuildImageTag("pasta.jpg")

The code above generates a dynamic URL, and when users access this URL, the image is manipulated and delivered on the fly. You can create this URL directly, or use Cloudinary’s client libraries to do the same thing with one line of code, in all popular languages and frameworks including PHP, Ruby on Rails and Node.js. Click on the tabs in the code samples in this article to view the code in your language or framework of choice.

The results are:

Cashew chicken north gravity Vegetable soup north gravity Pasta north gravity

As you can see the results are still far from optimal. Especially the photo of the soup does not show the soup plate at all… We can try the center gravity as well. This time, the results are better, but still not good enough.

Cashew chicken center gravity Vegetable soup center gravity Pasta center gravity

The photos below were generated using the south gravity. Again, this is not good enough for a professional food site.

Cashew chicken south gravity Vegetable soup south gravity Pasta south gravity

Thankfully, we have the Imagga Add-on. By setting the crop parameter in the URL to imagga_scale and specifying dimensions of 200x90, better-looking thumbnails are dynamically generated:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:imagga_scale, :sign_url=>true)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"imagga_scale", "sign_url"=>true))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="imagga_scale", sign_url=True)
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("imagga_scale")).signed(true).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("imagga_scale")).Signed(true).BuildImageTag("pasta.jpg")

Cashew chicken cropped with Imagga Vegetable soup cropped with Imagga Pasta cropped with Imagga

The images above, generated by Imagga’s smart cropping technology, are automatically focused on the most important part of the original image.

Note: In the URL we showed above, which resulted in the Imagga manipulation, there was a special signature before the image manipulation parameters - /s--MwkAA0qt--/. This code is required by Cloudinary to activate the Imagga add on. To see how to add it to the URL, see the documentation. You can also use Imagga in Cloudinary without this signature by performing Eager Transformations.

Smart cropping - focusing on what’s interesting in user images

User-uploaded photos often have a main object which is the most relevant, and a background which is less interesting and in many cases should not be shown on the site. For example, in the following photo the main object is the kitten, and there is a green grass surrounding the kitten.

Ruby:
cl_image_tag("kitten.jpg")
PHP:
cl_image_tag("kitten.jpg")
Python:
CloudinaryImage("kitten.jpg").image()
Node.js:
cloudinary.image("kitten.jpg")
Java:
cloudinary.url().imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("kitten.jpg")
Uploaded kitten photo

While the photo with the green background is very nice, when displaying the highlights of a photo album or a certain news feed, you might wish to emphasize the kitten itself.

By setting the crop parameter in the URL to imagga_crop without specifying width or height, the Imagga add-on smartly crops the image to keep the relevant part visible, while removing less relevant elements.

The following URL and sample code can be used to embed an HTML image tag with this photo automatically cropped by Imagga:

Ruby:
cl_image_tag("kitten.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("kitten.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("kitten.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("kitten.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("kitten.jpg")
Smartly cropped kitten photo

While the original image is 850x565, the cropped version is 462x441. In the image above we also scaled it down to a width of 200 pixels to better fit in this post.

Another example, this time of a fat cat. We have the same issue in this photo: a blurred green background surrounding the cat.

Ruby:
cl_image_tag("fat_cat.jpg")
PHP:
cl_image_tag("fat_cat.jpg")
Python:
CloudinaryImage("fat_cat.jpg").image()
Node.js:
cloudinary.image("fat_cat.jpg")
Java:
cloudinary.url().imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("fat_cat.jpg")
Uploaded fat cat photo

Adding c_imagga_crop to the URL automatically crops the photo to focus on the cat, using the Imagga Add-on.

Ruby:
cl_image_tag("fat_cat.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("fat_cat.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("fat_cat.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("fat_cat.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("fat_cat.jpg")
Smartly cropped fat cat photo

It’s cropped - now what?

Of course, smart cropping is usually not the end of the story, there are usually additional manipulations you’ll need to perform to exactly fit the look and feel of the site.

So - while we’re at it, let’s take the same code we used above to tell Cloudinary to smartly crop the photo, and modify it slightly to perform even more effects. Here's a simple example: let’s say our graphic design requires the cat be resized to 200x200 pixels, with white space padding and a border around it.

We can do this simply by using the pad crop mode:

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:border=>"1px_solid_rgb:aaa", :width=>200, :height=>200, :crop=>:pad}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("border"=>"1px_solid_rgb:aaa", "width"=>200, "height"=>200, "crop"=>"pad")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"border": "1px_solid_rgb:aaa", "width": 200, "height": 200, "crop": "pad"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .border("1px_solid_rgb:aaa").width(200).height(200).crop("pad")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Border("1px_solid_rgb:aaa").Width(200).Height(200).Crop("pad")).Signed(true).BuildImageTag("fat_cat.jpg")
Padding to 200x200 of the smartly cropped photo

Here’s the same code with a few more parameters tacked on that make the image circular, add a green border, increase color saturation, apply a sharpen effect, and add a logo watermark, while modifying its brightness and size. All these image manipulations are done on the fly by Cloudinary when a user first accesses the image.

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).Signed(true).BuildImageTag("fat_cat.jpg")
Further image manipulation of the smartly cropped photo

One last illustration: Apply the same image manipulation without smartly cropping the original photo. Quite a difference, isn't it?

Ruby:
cl_image_tag("fat_cat.jpg", :transformation=>[
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("transformation"=>array(
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(transformation=[
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).BuildImageTag("fat_cat.jpg")
Same manipulation of the original photo

A final note - when you use the above URLs to generate images on-the-fly in Cloudinary, both the source and the resulting images are stored in the cloud, delivered to users using a fast CDN with advanced caching, and automatically optimized to reduce file size. So when you use Cloudinary to perform these smart image manipulations, you also get optimal storage and delivery of images to users around the world.

Summary

Smart cropping is a must for any website or mobile application that involves user-uploaded images. Cloudinary's face-detection based cropping, together with Imagga's smart cropping, allows you to perform this cropping automatically with only a single line of code - and while making sure that images are stored and delivered in an optimal manner.  

Want to try it out for yourself? Sign up for a free Cloudinary account and also grab the Imagga Crop and Scale Add-on. Additional documentation of the Imagga Add-on is available here.

We would be happy to hear any feedback or suggestion you may have.

Recent Blog Posts

Taking the labor out of baby books

by Nicole Amsler
How Baby's Firsts save development time with Cloudinary
Cloudinary helps Baby’s Firsts App deliver images quickly and preserve consistency from app to print. Baby’s Firsts is a free iPhone app that helps parents collect photos, instantly share photos via Facebook, Twitter and Flickr, and produce photo albums of their baby’s first year. Using more than 300 creative, developmentally timed reminders to capture key moments and milestones, the easy-to-navigate app enables parents to store photos in the cloud and create customized photo pages that are then transformed into heirloom-quality, printed baby books. 
Read more

Lossy compression for optimizing animated GIFs

by Meir Feinberg
How to optimize animated GIFs with lossy compression

Animated GIFs keep getting more and more popular, but they are generally very big files with slow loading times and high bandwidth costs, while the format itself is quite old and not optimized for modern video clips. As developers, you need to allow users to upload their animated GIF files, but you also need to deliver them optimized, which can be a complex, time consuming process.

Read more
Building and scaling a cloud service for developers

Last month I was invited to speak at Daho.am, Munich's developers conference. This conference was organized by Stylight, a very successful fashion technology startup.  Stylight signed up for a free Cloudinary account about 3 years ago and similarly to Cloudinary back then, Stylight were quite a young startup. Since then both companies have grown impressively together and Stylight are now a premium customer of Cloudinary, managing hundreds of millions of images.

Read more
10 Startups managing images in the cloud, Part 6
Cloudinary’s customers can range in size, be found in a variety of verticals and boast multiple use cases. Large and small, our customers are our backbone and we like to support them by highlighting their stories on our blog.
 
Check out the ten featured startups below who are Cloudinary’s customers and utilizing the service as their image management system of choice.
Read more
Automatic visual image enhancement for web apps

Various factors can have an effect on the visual quality of photos captured by a wide variety of digital cameras. Technical limitations of cameras, coupled with changing conditions in which users take photos, results in a wide range of visual quality. Camera-related limitations arise from a combination of poor optics, noisy sensors, and the modest capabilities of mobile camera phones that are used to take photos in conditions that range from bright daylight to indoor scenes with incandescent light or even dark night scenes.

Read more