Cloudinary Blog

Smart automatic image cropping: Maybe you CAN always get what you want

Smart and Automatic Content-Aware Image Cropping

The Rolling Stones claim, “You can’t always get what you want”.

And when your application needs to crop hundreds or thousands of images to specific sizes within a strictly defined UI design, that frustrating Rolling Stones phrase may be ringing in your ears.

But maybe it doesn’t have to.

There’s no feasible way to crop so many images manually, especially for user-generated content that must be displayed immediately. What you really need is a way to automatically crop all those images to fit your design (or even multiple, responsive designs for different devices). But you need to feel confident that the main area of interest will remain in the final cropped images, regardless of the varied sizes, content, and layout of the original images.

You may already have heard that Cloudinary provides a bunch of automation algorithms that can intelligently adjust your images on-the-fly to automatically deliver the best cropping, quality, and optimization for each image and device. (If you missed it, you can check out Introducing smart cropping, intelligent quality selection and automated responsive images.)


How to Optimize for Page Load Speed


In this article, we’ll do a deep-dive into the smart cropping element of this solution.

In a nutshell, it’s a content-aware algorithm that helps you achieve the best cropping result, regardless of the required dimensions, the main subject, or the positioning and layout of the elements in the picture.

For example, if a user uploaded the portrait-oriented bugs-in-a-flower photo shown on the left, and your app used default (centered) cropping to convert it to a small square image, you would end up with the unfortunate result shown on the right...

Bugs in a flower photo - Original

Bugs in a flower photo - default, centered cropping

But if you simply set the gravity parameter to auto (g_auto in URLs), the algorithm automatically finds the most important elements of the picture on-the-fly, and you get exactly what you want!

Ruby:
Copy to clipboard
cl_image_tag("yellow_flower_2bugs.jpg", :gravity=>"auto", :height=>150, :width=>150, :crop=>"fill")
PHP:
Copy to clipboard
cl_image_tag("yellow_flower_2bugs.jpg", array("gravity"=>"auto", "height"=>150, "width"=>150, "crop"=>"fill"))
Python:
Copy to clipboard
CloudinaryImage("yellow_flower_2bugs.jpg").image(gravity="auto", height=150, width=150, crop="fill")
Node.js:
Copy to clipboard
cloudinary.image("yellow_flower_2bugs.jpg", {gravity: "auto", height: 150, width: 150, crop: "fill"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().gravity("auto").height(150).width(150).crop("fill")).imageTag("yellow_flower_2bugs.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('yellow_flower_2bugs.jpg', {gravity: "auto", height: 150, width: 150, crop: "fill"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("yellow_flower_2bugs.jpg", {gravity: "auto", height: 150, width: 150, crop: "fill"})
React:
Copy to clipboard
<Image publicId="yellow_flower_2bugs.jpg" >
  <Transformation gravity="auto" height="150" width="150" crop="fill" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="yellow_flower_2bugs.jpg" >
  <cld-transformation gravity="auto" height="150" width="150" crop="fill" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="yellow_flower_2bugs.jpg" >
  <cl-transformation gravity="auto" height="150" width="150" crop="fill">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("auto").Height(150).Width(150).Crop("fill")).BuildImageTag("yellow_flower_2bugs.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().gravity("auto").height(150).width(150).crop("fill")).generate("yellow_flower_2bugs.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setGravity("auto").setHeight(150).setWidth(150).setCrop("fill")).generate("yellow_flower_2bugs.jpg")!, cloudinary: cloudinary)
Bugs in a flower photo - g_auto cropping

Responsive layouts, art direction, and cropping - A Web developer’s challenge:

The art direction for modern websites often requires displaying the same image in different places, shapes, and sizes.

For example, a site’s home page might show square image thumbnails that link to selected articles. Each of those images might be shown as a large landscape picture on the top of the relevant article page, while a zoomed-in excerpt from the same picture might be shown lower down in the article. Other sizes might be used next to search results or in a collage.

Add to this the fact that each of the images probably needs to be presented in totally different sizes and aspect ratios when shown on the responsive mobile version of the site. Plus, larger images may also need to resize responsively as a browser is resized.

When you multiply all these considerations times the many, many images on a site, the number of unique croppings required becomes somewhat mind boggling. For example, imagine each color below is a unique photo in your site. This gives a tiny glimpse into how many different crops could be required for each image (color) and how many crops overall:

diagram - so many unique croppings

The same image is often shown in varying shapes and sizes
throughout your site and on different devices

And what if the graphic designer shows up one morning with plans to change the look and feel of your site, and you have to change all the shapes and sizes displayed in each and every scenario? I feel a headache coming on…

Luckily, you can use Cloudinary’s transformation capabilities to take a single, high quality original image and crop it on-the-fly to different sizes and aspect ratios, so that you don’t have to crop and re-crop all those images manually. Phew!

But a problem arises if you can’t predict the layout of each image in advance, and how to crop them for each of the required dimensions without losing (or even slicing right through) the most important content. This becomes near impossible when a lot of your images come from user-generated content.

Face-detection based cropping (supported by Cloudinary) works nicely if you expect images to be portrait photographs. But what if your site is showing images of tourist sites, or customers’ favorite meals at local restaurants, or sports action shots?

There’s no way to guess in advance whether to focus on the top, middle, or one side of the pictures. And you can’t even assume that any people in the picture are the main or only region of interest.

Take this photo of a couple sitting in front of a mountain scene:

Couple and mountain scene - original

If you try generating default thumbnails, using facial recognition, or just request a standard crop using center, some of your displayed images might end up like these:

mountain scene Mountain scene
300 X 250 moderate landscape 200 X 300 portrait 200 X 200 thumbnail

Yikes!

Content-Aware Automated Cropping - A Web developer's answer

All you need to do to prevent all of the messy scenarios shown above, is to include the g_auto image manipulation parameter in your delivery URL.

Behind this simple parameter setting is a complex algorithm that evaluates the image pixel by pixel and assigns each area of interest a priority value. If there are faces in an image, they will automatically (unless you specify otherwise) earn a high priority, but other prominent areas of your picture also win high priority points.

g_auto algorithm simulation

In the end, the algorithm chooses the part of the picture containing the most prominent areas in the photo, within the constraints of the other cropping parameters you specify, much the same as a real person would do… replacing the completely infeasible alternative of having a real person sitting 24 hours a day to manually analyze and crop every single one of your professional graphics as well as all user-uploaded content.

And the whole thing happens in the cloud, on-the-fly, so that the resulting cropped image is quickly distributed across the CDN and to your users.

Remember those three awful image crops from earlier in this article? Check out how they look when we add g_auto to the URL:

mountain scene mountain scene
300 X 250 moderate landscape 200 X 300 portrait 200 X 200 thumbnail

 

So far, we’ve mostly seen examples of how g_auto works when using the fill cropping mode. You can also use g_auto with the thumb cropping mode to zoom in more on the main subjects while still retaining the most important content.

Let’s try this with our flower and bugs picture from the beginning of this article.

  • If we use the default (center) gravity, we cut off the best part.
  • If we try another area of focus, such as south, which may be useful for some images, we get useless content for this one.
  • But with g_auto, we get a nicely zoomed view of our flower and bugs, right in the center.
150 X 150 thumb with default
(center) gravity
150 X 150 thumb with g_south 150 X 150 thumb with g_auto

If you want to exert some extra influence on what the algorithm thinks is important, you can use one of the special auto gravity parameters, for example to tell it to give higher or lower priority to faces, or to impact the aggressiveness of the zoom when using g_auto with thumbnail cropping. For details, have a look at the Automatic cropping documentation.

Of course no algorithm is perfect one-hundred percent of the time, and what’s important in an image can sometimes be subjective. So for these special exceptions, you can save custom cropping coordinates with a particular image. You can set these coordinates in an upload command, using the Admin API, or using the Media Library UI. Any specified coordinates will override the g_auto algorithm, or you can optionally set your transformation to still use the g_auto algorithm, but to give a higher priority to your specified coordinates.

At Cloudinary, we love to learn and get better, so when you use custom coordinate overrides, we use that as input that can potentially improve our algorithm in the future.

Cropping for Responsive Design

We mentioned above that one of the many reasons you may need to smart crop images relates to resizing in response to browser size.

To get the best image crops when your responsive art direction requires different aspect ratios for different screen sizes, just use automatic image cropping (g_auto) together with your fill or thumb crops in the Cloudinary URLs that you supply in your <srcset> and <picture> tags. The responsive images and art-directed responsive images posts, teach you just how to use these tags.

You can even take a step further and add the w_auto or dpr_auto values to your URL, which use Google Client Hints to automatically determine the required width of an image based on the browser's viewport-width or the layout width.

Another option is to use the free Responsive Breakpoints Generator or the Cloudinary API to help you find the optimal responsive image dimensions in each case.

Find out more in the responsive breakpoints generation blog post.

The Bottom Line

So, what’s the bottom line of all this? What’s the main point?
It’s simply that the main point is not always on the bottom line… or the top or the middle for that matter.

With Cloudinary’s content-aware automatic image cropping option, you no longer have to bet on a majority-rule strategy, always focusing on the center or top of all images, nor do you have to get a graphic artist to crop every image manually. Instead, a sophisticated algorithm running in the cloud evaluates each image on-the-fly, and then generates and delivers the image that fits the specified dimensions and is most likely to contain the main focus of the image, enabling you, your graphic artist, and your customers to ‘get what you want’!

This smart automated cropping feature is one element within the full Cloudinary end-to-end offering, which includes programmatic image upload, cloud storage, powerful administration, fast CDN delivery, and a comprehensive image manipulation SDK that enables developers to incorporate these functionalities in their Web and mobile applications with simple code one-liners.

All of the above is available with all Cloudinary's plans, including the free plan.
Open a free account and try it yourself.

Want to learn more about image editing?

Have a look at these articles:

Recent Blog Posts

Maya Shavin: How I Built My Website

Besides working as a senior front-end developer at Cloudinary, I'm also a content creator, a blogger, and an open-source developer. Follow me at @mayashavin and on mayashavin.com.

In the beginning, my website, mayashavin.com, was mainly for showcasing the status of my development projects and keeping me organized with my speaking schedule. Initially, I built it with Vue.js, later on switching to Nuxt.js (aka Nuxt) for a higher SEO score, and deployed it with Netlify. After some time, I added a blog section with Netlify CMS as the content management system (CMS). Everything was fine until I added more content and features, which led to a significant decline in the site’s performance. Also, the site design needed a modern look. So, I gave the site a makeover.

Read more
Automation Frees Up PetRescue’s Staff to Help Pets Find Their Forever Homes

As we spend more time at home, many of us are adopting pets for the joy, companionship and a surprising range of health benefits. In Australia, where our nonprofit customer PetRescue is located, there’s a shortage of pets to adopt. Last August, the Guardian reported that dog shelters in Australia emptied and adoption fees for puppies were running as high as $AUS1800.

Read more
Cloudinary and Contentful Make Modern Content Management Easier

I am pleased to share that Cloudinary and Contentful have joined forces to further streamline the creation, processing, and delivery of online content through Cloudinary’s digital asset management (DAM) solution and advanced transformation and delivery capabilities for images and video. What’s more, the partnership delivers a headless approach to DAM. By leveraging APIs for media management tasks, marketers and developers alike benefit from an integrated stack of optimized assets for optimization and automation. As a result, page loads are fast and beautiful, and at scale—with less overhead and effort.

Read more
Introducing Cloudinary's Nuxt Module

Since its initial release in October 2016 by the Chopin brothers as a server-side framework that runs on top of Vue.js, Nuxt (aka Nuxt.js) has gained prominence in both intuitiveness and performance. The framework offers numerous built-in features based on a modular architecture, bringing ease and simplicity to web development. Not surprisingly, Nuxt.js has seen remarkable growth in adoption by the developer community along with accolades galore. At this writing, Nuxt has earned over 30K stars on GitHub and 96 active modules with over a million downloads per month. And the upward trend is ongoing.

Read more
How Quality and Quantity can go Hand in Hand

When it comes to quality versus quantity, you’ll often hear people say, “It’s the quality that counts, not the quantity”. While that’s true in many situations, there are also cases where you want both quality and quantity. You may have thousands of images on your website and you want them all to look great. This is especially important if your website allows users to upload their own content, for example, to sell their own products or services. You don't want their poor quality images to reflect badly on your brand.

Read more