Cloudinary Blog

Beyond face detection - smart cropping in the cloud using Imagga and Cloudinary

by Nadav Soferman
Beyond face detection - smart cropping with Imagga & Cloudinary

It’s a common challenge in many mobile and web applications: how do you allow users to upload their own images, while automatically adapting these images to a fixed graphic design?

A classic example is a user uploading a profile picture, but instead of providing a headshot (which is what we really need from them), they upload a picture of their entire body with additional objects in the background. Obviously this image will need to be cropped to the size of the profile picture, while focusing on the user’s face.

One way to achieve this is to allow the user to crop the image themselves as part of the upload process. But even so, there is usually a need to use the image in several different contexts, in different sizes and positions, and the user’s cropping can’t help us produce all these different versions.

A more advanced approach is face detection - there are new software tools and APIs that can recognize the face in the image, and crop the image automatically, focusing on the human face. Actually, face-detection based cropping is one of the more popular features of Cloudinary’s cloud-based image management solution and API.

But now we’d like to take  it one step forward: What if you want to automatically crop thousands of images, focusing on an object that is not a human face? For example, e-commerce products, food, animals, or anything else that is more difficult to detect algorithmically.

Take these three images of food plates - many images like these are uploaded by users of food sites, such as Cloudinary customers Yummi, Cibando and EatWith:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

The website needs to automatically generate large, small and thumbnail versions of these photos, smartly cropped to show the main portion of the food plates. There's definitely a need for smart cropping based on the most appealing part of a photo, no matter what kind of photo it is. This is an even bigger technical challenge than cropping based on face detection.

And that’s exactly what Imagga have done with their smart cropping technology - they built a system that can detect the most appealing part of any image and focus on it automatically. We have integrated Imagga into Cloudinary’s image management solution, and you can try it out as part of our free tier.

Read on to see how Cloudinary makes it possible to automatically crop complex images to focus on the most appealing part of the photo, and automatically generate beautiful thumbnails, with just one line of code.

Photo scaling and smart thumbnail generation

Let's continue the example of the food site which uses Cloudinary and Imagga to automatically generate thumbnails of plates of food. The site’s graphic design requires thumbnails of 200x90. Again, here are the demo images we showed above:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

Cloudinary has a crop mode called fill, which allows you to crop an image to certain dimensions, while automatically filling the space with the result of the crop. You can specify a gravity which tells Cloudinary in which direction the focus of the crop should be.

Here is a dynamic URL and code which crops one of the images above using fill mode, and sets gravity to north:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:fill, :gravity=>:north)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"fill", "gravity"=>"north"))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="fill", gravity="north")
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("fill").gravity("north")).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("fill").Gravity("north")).BuildImageTag("pasta.jpg")

The code above generates a dynamic URL, and when users access this URL, the image is manipulated and delivered on the fly. You can create this URL directly, or use Cloudinary’s client libraries to do the same thing with one line of code, in all popular languages and frameworks including PHP, Ruby on Rails and Node.js. Click on the tabs in the code samples in this article to view the code in your language or framework of choice.

The results are:

Cashew chicken north gravity Vegetable soup north gravity Pasta north gravity

As you can see the results are still far from optimal. Especially the photo of the soup does not show the soup plate at all… We can try the center gravity as well. This time, the results are better, but still not good enough.

Cashew chicken center gravity Vegetable soup center gravity Pasta center gravity

The photos below were generated using the south gravity. Again, this is not good enough for a professional food site.

Cashew chicken south gravity Vegetable soup south gravity Pasta south gravity

Thankfully, we have the Imagga Add-on. By setting the crop parameter in the URL to imagga_scale and specifying dimensions of 200x90, better-looking thumbnails are dynamically generated:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:imagga_scale, :sign_url=>true)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"imagga_scale", "sign_url"=>true))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="imagga_scale", sign_url=True)
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("imagga_scale")).signed(true).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("imagga_scale")).Signed(true).BuildImageTag("pasta.jpg")

Cashew chicken cropped with Imagga Vegetable soup cropped with Imagga Pasta cropped with Imagga

The images above, generated by Imagga’s smart cropping technology, are automatically focused on the most important part of the original image.

Note: In the URL we showed above, which resulted in the Imagga manipulation, there was a special signature before the image manipulation parameters - /s--MwkAA0qt--/. This code is required by Cloudinary to activate the Imagga add on. To see how to add it to the URL, see the documentation. You can also use Imagga in Cloudinary without this signature by performing Eager Transformations.

Smart cropping - focusing on what’s interesting in user images

User-uploaded photos often have a main object which is the most relevant, and a background which is less interesting and in many cases should not be shown on the site. For example, in the following photo the main object is the kitten, and there is a green grass surrounding the kitten.

Ruby:
cl_image_tag("kitten.jpg")
PHP:
cl_image_tag("kitten.jpg")
Python:
CloudinaryImage("kitten.jpg").image()
Node.js:
cloudinary.image("kitten.jpg")
Java:
cloudinary.url().imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("kitten.jpg")
Uploaded kitten photo

While the photo with the green background is very nice, when displaying the highlights of a photo album or a certain news feed, you might wish to emphasize the kitten itself.

By setting the crop parameter in the URL to imagga_crop without specifying width or height, the Imagga add-on smartly crops the image to keep the relevant part visible, while removing less relevant elements.

The following URL and sample code can be used to embed an HTML image tag with this photo automatically cropped by Imagga:

Ruby:
cl_image_tag("kitten.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("kitten.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("kitten.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("kitten.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("kitten.jpg")
Smartly cropped kitten photo

While the original image is 850x565, the cropped version is 462x441. In the image above we also scaled it down to a width of 200 pixels to better fit in this post.

Another example, this time of a fat cat. We have the same issue in this photo: a blurred green background surrounding the cat.

Ruby:
cl_image_tag("fat_cat.jpg")
PHP:
cl_image_tag("fat_cat.jpg")
Python:
CloudinaryImage("fat_cat.jpg").image()
Node.js:
cloudinary.image("fat_cat.jpg")
Java:
cloudinary.url().imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("fat_cat.jpg")
Uploaded fat cat photo

Adding c_imagga_crop to the URL automatically crops the photo to focus on the cat, using the Imagga Add-on.

Ruby:
cl_image_tag("fat_cat.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("fat_cat.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("fat_cat.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("fat_cat.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("fat_cat.jpg")
Smartly cropped fat cat photo

It’s cropped - now what?

Of course, smart cropping is usually not the end of the story, there are usually additional manipulations you’ll need to perform to exactly fit the look and feel of the site.

So - while we’re at it, let’s take the same code we used above to tell Cloudinary to smartly crop the photo, and modify it slightly to perform even more effects. Here's a simple example: let’s say our graphic design requires the cat be resized to 200x200 pixels, with white space padding and a border around it.

We can do this simply by using the pad crop mode:

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:border=>"1px_solid_rgb:aaa", :width=>200, :height=>200, :crop=>:pad}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("border"=>"1px_solid_rgb:aaa", "width"=>200, "height"=>200, "crop"=>"pad")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"border": "1px_solid_rgb:aaa", "width": 200, "height": 200, "crop": "pad"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .border("1px_solid_rgb:aaa").width(200).height(200).crop("pad")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Border("1px_solid_rgb:aaa").Width(200).Height(200).Crop("pad")).Signed(true).BuildImageTag("fat_cat.jpg")
Padding to 200x200 of the smartly cropped photo

Here’s the same code with a few more parameters tacked on that make the image circular, add a green border, increase color saturation, apply a sharpen effect, and add a logo watermark, while modifying its brightness and size. All these image manipulations are done on the fly by Cloudinary when a user first accesses the image.

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).Signed(true).BuildImageTag("fat_cat.jpg")
Further image manipulation of the smartly cropped photo

One last illustration: Apply the same image manipulation without smartly cropping the original photo. Quite a difference, isn't it?

Ruby:
cl_image_tag("fat_cat.jpg", :transformation=>[
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("transformation"=>array(
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(transformation=[
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).BuildImageTag("fat_cat.jpg")
Same manipulation of the original photo

A final note - when you use the above URLs to generate images on-the-fly in Cloudinary, both the source and the resulting images are stored in the cloud, delivered to users using a fast CDN with advanced caching, and automatically optimized to reduce file size. So when you use Cloudinary to perform these smart image manipulations, you also get optimal storage and delivery of images to users around the world.

Summary

Smart cropping is a must for any website or mobile application that involves user-uploaded images. Cloudinary's face-detection based cropping, together with Imagga's smart cropping, allows you to perform this cropping automatically with only a single line of code - and while making sure that images are stored and delivered in an optimal manner.  

Want to try it out for yourself? Sign up for a free Cloudinary account and also grab the Imagga Crop and Scale Add-on. Additional documentation of the Imagga Add-on is available here.

We would be happy to hear any feedback or suggestion you may have.

blog comments powered by Disqus

Recent Blog Posts

How Answers.com manage millions of images

by Orly Bogler
How Answers.com utilizes Cloudinary to manage millions of images

When was the last time you've asked Google about your favorite band, movie star, or personal hobby? I can only assume that one of the first results that came up was from Answers.com. Nearly everyone knows this website, which is on the Quantcast Top 10 most visited sites in the world.

Read more
Control the zoom level with automatic image cropping

Many websites now offer their users the ability to upload images and profile pictures, making it a challenge for web designers to maintain a certain graphic design and style when subsequently displaying these images. The profile pictures may need to be smartly cropped to focus on the faces, with some sites that prefer close-ups of faces and others that prefer including more background when displaying images of people.

Read more
Introducing cloud based service for video management

They say that a picture is worth a thousand words. For modern websites, a video surely takes the visual impact to a whole new level.

Nowadays, people enjoy the amazing capability of shooting videos with smartphone cameras that easily fit in their back pockets and are accessible nearly everywhere. Modern web applications have an opportunity to dramatically increase their visual impact by showcasing these videos online. Between news reports, user shared video snippets, explainer videos and ad campaigns, we see more and more videos appearing daily in our visited websites.

Read more
Automatic and accurate red eye removal with Cloudinary

Red eye often happens due to the use of flash in low light conditions as the light hits the eye very quickly and into the retina. It then bounces off of the back of the eye and emits a red color due to the blood vessels there. Although more professional modern cameras and flashes generally prevent this from happening, red eye may still occur with simpler, smaller cameras (including smartphones). There are various software solutions for red eye removal available on mobile devices and desktops, some of which require manual processing to get good results.

Read more
How to detect and prevent malware infected user uploads

Social networking sites allow users to upload images or other types of files that are immediately available to other users via news feeds or notifications. In some cases, attackers can directly spread infected files, but more commonly, they leverage the viral effect and the fact that users are simply unaware that their files are infected through sharing and collaborating with others. As a site owner or application developer, it is your responsibility to protect users and prevent these situations from occurring. Fortunately, Cloudinary makes this easier with its Metascan add-on.

Read more