Cloudinary Blog

Beyond face detection - smart cropping in the cloud using Imagga and Cloudinary

by Nadav Soferman
Beyond face detection - smart cropping with Imagga & Cloudinary

It’s a common challenge in many mobile and web applications: how do you allow users to upload their own images, while automatically adapting these images to a fixed graphic design?

A classic example is a user uploading a profile picture, but instead of providing a headshot (which is what we really need from them), they upload a picture of their entire body with additional objects in the background. Obviously this image will need to be cropped to the size of the profile picture, while focusing on the user’s face.

One way to achieve this is to allow the user to crop the image themselves as part of the upload process. But even so, there is usually a need to use the image in several different contexts, in different sizes and positions, and the user’s cropping can’t help us produce all these different versions.

A more advanced approach is face detection - there are new software tools and APIs that can recognize the face in the image, and crop the image automatically, focusing on the human face. Actually, face-detection based cropping is one of the more popular features of Cloudinary’s cloud-based image management solution and API.

But now we’d like to take  it one step forward: What if you want to automatically crop thousands of images, focusing on an object that is not a human face? For example, e-commerce products, food, animals, or anything else that is more difficult to detect algorithmically.

Take these three images of food plates - many images like these are uploaded by users of food sites, such as Cloudinary customers Yummi, Cibando and EatWith:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

The website needs to automatically generate large, small and thumbnail versions of these photos, smartly cropped to show the main portion of the food plates. There's definitely a need for smart cropping based on the most appealing part of a photo, no matter what kind of photo it is. This is an even bigger technical challenge than cropping based on face detection.

And that’s exactly what Imagga have done with their smart cropping technology - they built a system that can detect the most appealing part of any image and focus on it automatically. We have integrated Imagga into Cloudinary’s image management solution, and you can try it out as part of our free tier.

Read on to see how Cloudinary makes it possible to automatically crop complex images to focus on the most appealing part of the photo, and automatically generate beautiful thumbnails, with just one line of code.

Photo scaling and smart thumbnail generation

Let's continue the example of the food site which uses Cloudinary and Imagga to automatically generate thumbnails of plates of food. The site’s graphic design requires thumbnails of 200x90. Again, here are the demo images we showed above:

Cashew chicken original photo Vegetable soup original photo Pasta original photo

Cloudinary has a crop mode called fill, which allows you to crop an image to certain dimensions, while automatically filling the space with the result of the crop. You can specify a gravity which tells Cloudinary in which direction the focus of the crop should be.

Here is a dynamic URL and code which crops one of the images above using fill mode, and sets gravity to north:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:fill, :gravity=>:north)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"fill", "gravity"=>"north"))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="fill", gravity="north")
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("fill").gravity("north")).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "fill", gravity: "north"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("fill").Gravity("north")).BuildImageTag("pasta.jpg")

The code above generates a dynamic URL, and when users access this URL, the image is manipulated and delivered on the fly. You can create this URL directly, or use Cloudinary’s client libraries to do the same thing with one line of code, in all popular languages and frameworks including PHP, Ruby on Rails and Node.js. Click on the tabs in the code samples in this article to view the code in your language or framework of choice.

The results are:

Cashew chicken north gravity Vegetable soup north gravity Pasta north gravity

As you can see the results are still far from optimal. Especially the photo of the soup does not show the soup plate at all… We can try the center gravity as well. This time, the results are better, but still not good enough.

Cashew chicken center gravity Vegetable soup center gravity Pasta center gravity

The photos below were generated using the south gravity. Again, this is not good enough for a professional food site.

Cashew chicken south gravity Vegetable soup south gravity Pasta south gravity

Thankfully, we have the Imagga Add-on. By setting the crop parameter in the URL to imagga_scale and specifying dimensions of 200x90, better-looking thumbnails are dynamically generated:

Ruby:
cl_image_tag("pasta.jpg", :width=>200, :height=>90, :crop=>:imagga_scale, :sign_url=>true)
PHP:
cl_image_tag("pasta.jpg", array("width"=>200, "height"=>90, "crop"=>"imagga_scale", "sign_url"=>true))
Python:
CloudinaryImage("pasta.jpg").image(width=200, height=90, crop="imagga_scale", sign_url=True)
Node.js:
cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(90).crop("imagga_scale")).signed(true).imageTag("pasta.jpg")
jQuery:
$.cloudinary.image("pasta.jpg", {width: 200, height: 90, crop: "imagga_scale"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(90).Crop("imagga_scale")).Signed(true).BuildImageTag("pasta.jpg")

Cashew chicken cropped with Imagga Vegetable soup cropped with Imagga Pasta cropped with Imagga

The images above, generated by Imagga’s smart cropping technology, are automatically focused on the most important part of the original image.

Note: In the URL we showed above, which resulted in the Imagga manipulation, there was a special signature before the image manipulation parameters - /s--MwkAA0qt--/. This code is required by Cloudinary to activate the Imagga add on. To see how to add it to the URL, see the documentation. You can also use Imagga in Cloudinary without this signature by performing Eager Transformations.

Smart cropping - focusing on what’s interesting in user images

User-uploaded photos often have a main object which is the most relevant, and a background which is less interesting and in many cases should not be shown on the site. For example, in the following photo the main object is the kitten, and there is a green grass surrounding the kitten.

Ruby:
cl_image_tag("kitten.jpg")
PHP:
cl_image_tag("kitten.jpg")
Python:
CloudinaryImage("kitten.jpg").image()
Node.js:
cloudinary.image("kitten.jpg")
Java:
cloudinary.url().imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("kitten.jpg")
Uploaded kitten photo

While the photo with the green background is very nice, when displaying the highlights of a photo album or a certain news feed, you might wish to emphasize the kitten itself.

By setting the crop parameter in the URL to imagga_crop without specifying width or height, the Imagga add-on smartly crops the image to keep the relevant part visible, while removing less relevant elements.

The following URL and sample code can be used to embed an HTML image tag with this photo automatically cropped by Imagga:

Ruby:
cl_image_tag("kitten.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("kitten.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("kitten.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("kitten.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("kitten.jpg")
jQuery:
$.cloudinary.image("kitten.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("kitten.jpg")
Smartly cropped kitten photo

While the original image is 850x565, the cropped version is 462x441. In the image above we also scaled it down to a width of 200 pixels to better fit in this post.

Another example, this time of a fat cat. We have the same issue in this photo: a blurred green background surrounding the cat.

Ruby:
cl_image_tag("fat_cat.jpg")
PHP:
cl_image_tag("fat_cat.jpg")
Python:
CloudinaryImage("fat_cat.jpg").image()
Node.js:
cloudinary.image("fat_cat.jpg")
Java:
cloudinary.url().imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("fat_cat.jpg")
Uploaded fat cat photo

Adding c_imagga_crop to the URL automatically crops the photo to focus on the cat, using the Imagga Add-on.

Ruby:
cl_image_tag("fat_cat.jpg", :crop=>:imagga_crop, :sign_url=>true)
PHP:
cl_image_tag("fat_cat.jpg", array("crop"=>"imagga_crop", "sign_url"=>true))
Python:
CloudinaryImage("fat_cat.jpg").image(crop="imagga_crop", sign_url=True)
Node.js:
cloudinary.image("fat_cat.jpg", {crop: "imagga_crop", sign_url: true})
Java:
cloudinary.url().transformation(new Transformation().crop("imagga_crop")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {crop: "imagga_crop"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Crop("imagga_crop")).Signed(true).BuildImageTag("fat_cat.jpg")
Smartly cropped fat cat photo

It’s cropped - now what?

Of course, smart cropping is usually not the end of the story, there are usually additional manipulations you’ll need to perform to exactly fit the look and feel of the site.

So - while we’re at it, let’s take the same code we used above to tell Cloudinary to smartly crop the photo, and modify it slightly to perform even more effects. Here's a simple example: let’s say our graphic design requires the cat be resized to 200x200 pixels, with white space padding and a border around it.

We can do this simply by using the pad crop mode:

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:border=>"1px_solid_rgb:aaa", :width=>200, :height=>200, :crop=>:pad}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("border"=>"1px_solid_rgb:aaa", "width"=>200, "height"=>200, "crop"=>"pad")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"border": "1px_solid_rgb:aaa", "width": 200, "height": 200, "crop": "pad"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .border("1px_solid_rgb:aaa").width(200).height(200).crop("pad")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {border: "1px_solid_rgb:aaa", width: 200, height: 200, crop: "pad"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Border("1px_solid_rgb:aaa").Width(200).Height(200).Crop("pad")).Signed(true).BuildImageTag("fat_cat.jpg")
Padding to 200x200 of the smartly cropped photo

Here’s the same code with a few more parameters tacked on that make the image circular, add a green border, increase color saturation, apply a sharpen effect, and add a logo watermark, while modifying its brightness and size. All these image manipulations are done on the fly by Cloudinary when a user first accesses the image.

Ruby:
cl_image_tag("fat_cat.jpg", :sign_url=>true, :transformation=>[
  {:crop=>:imagga_crop},
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("sign_url"=>true, "transformation"=>array(
  array("crop"=>"imagga_crop"),
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(sign_url=True, transformation=[
  {"crop": "imagga_crop"},
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {sign_url: true, transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .crop("imagga_crop").chain()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).signed(true).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {crop: "imagga_crop"},
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Crop("imagga_crop").Chain()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).Signed(true).BuildImageTag("fat_cat.jpg")
Further image manipulation of the smartly cropped photo

One last illustration: Apply the same image manipulation without smartly cropping the original photo. Quite a difference, isn't it?

Ruby:
cl_image_tag("fat_cat.jpg", :transformation=>[
  {:width=>200, :crop=>:scale},
  {:radius=>"max", :border=>"5px_solid_rgb:19340b"},
  {:effect=>"saturation:100"},
  {:effect=>"sharpen"},
  {:opacity=>25, :overlay=>"cloudinary_icon", :width=>0.4, :crop=>:scale, :flags=>:relative, :effect=>"brightness:-90"}
  ])
PHP:
cl_image_tag("fat_cat.jpg", array("transformation"=>array(
  array("width"=>200, "crop"=>"scale"),
  array("radius"=>"max", "border"=>"5px_solid_rgb:19340b"),
  array("effect"=>"saturation:100"),
  array("effect"=>"sharpen"),
  array("opacity"=>25, "overlay"=>"cloudinary_icon", "width"=>0.4, "crop"=>"scale", "flags"=>"relative", "effect"=>"brightness:-90")
  )))
Python:
CloudinaryImage("fat_cat.jpg").image(transformation=[
  {"width": 200, "crop": "scale"},
  {"radius": "max", "border": "5px_solid_rgb:19340b"},
  {"effect": "saturation:100"},
  {"effect": "sharpen"},
  {"opacity": 25, "overlay": "cloudinary_icon", "width": 0.4, "crop": "scale", "flags": "relative", "effect": "brightness:-90"}
  ])
Node.js:
cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(200).crop("scale").chain()
  .radius("max").border("5px_solid_rgb:19340b").chain()
  .effect("saturation:100").chain()
  .effect("sharpen").chain()
  .opacity(25).overlay("cloudinary_icon").width(0.4).crop("scale").flags("relative").effect("brightness:-90")).imageTag("fat_cat.jpg")
jQuery:
$.cloudinary.image("fat_cat.jpg", {transformation: [
  {width: 200, crop: "scale"},
  {radius: "max", border: "5px_solid_rgb:19340b"},
  {effect: "saturation:100"},
  {effect: "sharpen"},
  {opacity: 25, overlay: "cloudinary_icon", width: 0.4, crop: "scale", flags: "relative", effect: "brightness:-90"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(200).Crop("scale").Chain()
  .Radius("max").Border("5px_solid_rgb:19340b").Chain()
  .Effect("saturation:100").Chain()
  .Effect("sharpen").Chain()
  .Opacity(25).Overlay("cloudinary_icon").Width(0.4).Crop("scale").Flags("relative").Effect("brightness:-90")).BuildImageTag("fat_cat.jpg")
Same manipulation of the original photo

A final note - when you use the above URLs to generate images on-the-fly in Cloudinary, both the source and the resulting images are stored in the cloud, delivered to users using a fast CDN with advanced caching, and automatically optimized to reduce file size. So when you use Cloudinary to perform these smart image manipulations, you also get optimal storage and delivery of images to users around the world.

Summary

Smart cropping is a must for any website or mobile application that involves user-uploaded images. Cloudinary's face-detection based cropping, together with Imagga's smart cropping, allows you to perform this cropping automatically with only a single line of code - and while making sure that images are stored and delivered in an optimal manner.  

Want to try it out for yourself? Sign up for a free Cloudinary account and also grab the Imagga Crop and Scale Add-on. Additional documentation of the Imagga Add-on is available here.

We would be happy to hear any feedback or suggestion you may have.

Recent Blog Posts

Introducing responsive image breakpoints solutions

The number of different devices available and their potential screen resolutions keep increasing, and to support this wide range of resolutions and devices, responsive website design is now the standard. A website's markup must adapt itself to look perfect on all the different devices and in various resolutions, pixel densities and mobile device orientations. Managing, manipulating and delivering images, is one of the main challenges of responsive design that web developers face.

Read more
Different methods to deliver your images via HTTPS

Are you delivering your site via HTTPS or considering it? You're not alone - in 2015 the number of sites running on HTTPS almost doubled. Both consumers and web developers are now much more aware of the value of the humble green lock displayed in the address bar. The benefits of using HTTPS extend beyond the customer’s safety, to SEO boosts, and advanced functionalities that are only available when delivering via HTTPS, such as HTTP/2 and WebRTC.

Read more

How to get media to load faster on your website

by Kasia Kramnik
Tips and tricks to help your website's media load faster

This is a guest post by Kasia Kramnik, Content Marketing Manager at Netguru, a full stack development house and one of Cloudinary's Consulting partners.

Take a look at your website. Are you happy with the way it looks? I bet you are, and that’s really awesome. Keep in mind though, there is one thing you can’t actually see, but you need to experience: the load speed. Sometimes the most important element is invisible to the eye. In this article you’ll find tips on perfecting the invisible as well: loading your site and media with visibly better results.

Read more

Happy New Year and a hat trick

by Nadav Soferman
Happy New Year and a hat trick

As the end of 2015 approaches, we wanted to share a quick summary of Cloudinary’s accomplishment this year and some of our plans for next year. We couldn't possibly do this without including an image manipulation example! That's our hat trick in the title :-)

Read more