Cloudinary Blog

New AI-Based Image Auto-Crop Algorithm Sticks to the Subject

AI-Based Image Auto-Crop Algorithm Sticks to the Subject

If you're the developer for an online store, a news site, a social media app, or any other website that delivers new media content on a regular basis, you are probably painfully familiar with both the challenge and importance of delivering well-cropped images.

In a nutshell, your customers expect top-quality, quick-loading photos that clearly show them what they need to see no matter what device they happen to be using at the moment. Making them happy requires delivering the same image in many different aspect ratios, and potentially cropping closer or wider on your main subject, depending on size. Hardly sounds reasonable, but if you don't meet their expectations, your customers will just go somewhere else...

(Why do I feel like I'm describing my teenage daughters? Well, to be honest, the demanding nature and short attention span of today's online customers have a lot in common with teens…)

But unlike teens, you can't afford to fall short of your users' demands. If you've got hundreds or thousands of images to deliver every day, you can't manually crop every image to the right size for every device and placement. Simple client-side scaling of your high quality images would mean completely unacceptable performance and quality. So it's clear that you must adjust the size and aspect ratio of your images programmatically, and then deliver the various image resolutions using responsive delivery code, probably via <picture> and <srcset> tags.

Webinar
How to Optimize for Page Load Speed

Then the question arises -- if you programmatically crop the same photo to significantly different aspect ratios, how can you be sure your code won't crop off the most important parts of your products, news subjects, or that adorable cat your users just have to share with the world?

If you are already familiar with Cloudinary's powerful media management and delivery capabilities, you might be saying - what's the problem? Just use Cloudinary's image auto-cropping gravity parameter (aka g_auto) and it will do the hard work for you!

And in fact, thousands of Cloudinary developers already use g_auto in their image URLs to automatically crop millions of images daily. That parameter applies a sophisticated algorithm that analyzes the pixels in an image and prioritizes the most salient areas of each image on-the-fly. The analysis gives priority to skin tones, edge detection, color contrasts, any detected faces, and more, in order to determine the most important areas to keep when it crops.

But what if Cloudinary's already powerful image auto-crop functionality could be even smarter?

Introducing Subject Detection Image Auto Cropping

Cloudinary's new and improved deep-learning-based g_autotransformation parameter goes beyond the saliency analysis described above, adding the capability to actually detect the subjects in an image that would be most likely to capture a person's attention.

To accomplish this, our new image auto-cropping deep learning mechanism has been (and continues to be) fed with tens of thousands of images and corresponding human input that together teach our machines to predict the important regions in images, no matter their subject and layout. This involves advanced computations performed by GPU-based hardware clusters that process millions of crop requests on the fly. The result is completely unique auto-cropping functionality that provides very impressive outputs.

Remember those products, news images, and user-generated content (aka cats) we talked about above?

Look what would happen to these mobile-camera photographs if we were to crop them to a square using the standard center cropping that other programs often apply, and how the new and improved g_auto with subject-detection cropping comes to the rescue in each of these crops:

Product - original original Product image with center cropping center crop Product image with subject detection cropping g_auto crop
News image - original original News image with center cropping center crop News image with subject detection cropping g_auto crop
Cat image - original original Cat image with center cropping center crop Cat image with subject detection cropping g_auto crop

How do you apply this cool cropping method on your delivered images? After uploading your original image to your Cloudinary account, just specify auto as the gravity (area to keep) in your on-the-fly delivery URL, along with a crop mode such as fill, lfill, or crop, and of-course an aspect ratio or width/height combination that's different than the original.

For example, here's how you'd deliver that nicely cropped cat above as a 500px square (1.0 aspect ratio or ar_1 in the delivery URL):

Ruby:
Copy to clipboard
cl_image_tag("docs/cat_yellow_leaves2.jpg", :gravity=>"auto", :aspect_ratio=>"1", :width=>500, :crop=>"fill")
PHP:
Copy to clipboard
cl_image_tag("docs/cat_yellow_leaves2.jpg", array("gravity"=>"auto", "aspect_ratio"=>"1", "width"=>500, "crop"=>"fill"))
Python:
Copy to clipboard
CloudinaryImage("docs/cat_yellow_leaves2.jpg").image(gravity="auto", aspect_ratio="1", width=500, crop="fill")
Node.js:
Copy to clipboard
cloudinary.image("docs/cat_yellow_leaves2.jpg", {gravity: "auto", aspect_ratio: "1", width: 500, crop: "fill"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().gravity("auto").aspectRatio("1").width(500).crop("fill")).imageTag("docs/cat_yellow_leaves2.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('docs/cat_yellow_leaves2.jpg', {gravity: "auto", aspectRatio: "1", width: 500, crop: "fill"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("docs/cat_yellow_leaves2.jpg", {gravity: "auto", aspect_ratio: "1", width: 500, crop: "fill"})
React:
Copy to clipboard
<Image publicId="docs/cat_yellow_leaves2.jpg" >
  <Transformation gravity="auto" aspectRatio="1" width="500" crop="fill" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="docs/cat_yellow_leaves2.jpg" >
  <cld-transformation gravity="auto" aspectRatio="1" width="500" crop="fill" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="docs/cat_yellow_leaves2.jpg" >
  <cl-transformation gravity="auto" aspect-ratio="1" width="500" crop="fill">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("auto").AspectRatio("1").Width(500).Crop("fill")).BuildImageTag("docs/cat_yellow_leaves2.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().gravity("auto").aspectRatio("1").width(500).crop("fill")).generate("docs/cat_yellow_leaves2.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setGravity("auto").setAspectRatio("1").setWidth(500).setCrop("fill")).generate("docs/cat_yellow_leaves2.jpg")!, cloudinary: cloudinary)

Classic Or Subject?

This latest addition to Cloudinary's growing set of machine learning capabilities analyzes the image as a whole, rather than the pixel by pixel analysis that's applied in our classic auto-cropping feature.

In the majority of cases, the classic saliency algorithms and our new subject-detection method will provide the same or very similar results. But when processing the vast number of auto-crops that Cloudinary handles every day, there are some cases where we felt we could provide even better results. For example, the new subject-detection algorithm can be more reliable in cases where the true main subject of the photo would otherwise have to compete with elements like sunlight, faces, or other large areas of color contrast that are not actually central to the 'story' of the image.

For example, the classic auto-cropping algorithm gives increased priority to the bright contrasts of the leaves against the sky, and thus in this case, doesn't keep the girl when the aspect ratio is changed significantly. But the artificial intelligence algorithm emulates what our intuition tells us, and the automatic crop is right on target.

Original Original Original Classic auto-crop Original New auto-crop

Just a quick note here, that in order to get the best of all worlds, the default g_auto parameter now applies a combination of the subject and classic algorithms (more heavily leaning towards the subject results). But you can always proactively request either the classic or subject mechanism with any crop mode using auto:classic or auto:subject as the gravity (g_) value.

Auto-Cropping and Responsive Art Direction

The value of great image auto-cropping really comes into play when you start thinking about delivering the same image in different devices.

Organizations who need to play it safe tend to just scale down their original image and deliver the same thing regardless of the device viewport aspect ratio. But that means you really sacrifice on the detail, when on smaller screens or when the aspect ratios are significantly different from the original.

When you rotate your phone between portrait and landscape, you are switching between ~19:9 vs. 9:19, or an aspect ratio of 2.1 vs. 0.47, while your computer screen viewers are probably using a 4:3 or 16:9 screen. Do you really want to deliver the identical image in all these views?

In general, images that fill the device viewport get the most engagement. But you can only go that route if you can be sure that even if 50% or more of the image may be cropped out, you'll still keep the important parts of your image in tact.

For example, with the new g_auto, you can confidently deliver images that fill a phone's view port, even when users rotate their phones, so when viewer's rotate their phones, you can offer your viewers the more engaging option Bs rather than the tiny option As.

Original
Option A
Original
Option B
Original
Option A
Original
Option B

When the Subject, is Subject to Change…

Programmatically cropped images are an essential part of any web or mobile site. But when the subjects, and the location of those subjects within your images, are unpredictable, automatic cropping isn't enough. The cropping mechanism you use has to be smart enough to 'know' what the end users are going to want to see. With Cloudinary's subject detection auto cropping, you can confidently deliver great photos that bring your users' attention directly to the subject at hand, from responsive view to responsive view.

Oh, by the way, if your subject is not so likely to change, for example, suppose you are selling microwaves, umbrellas, vehicles, or food, you may want to take advantage of our new object-aware image cropping add-on. This add-on applies another deep learning tool that gives the highest preservation ("don't crop") priority to specific objects or categories you specify. If the specified object(s) aren't found, then (by default) the subject-detection auto-cropping algorithm described here is still applied.

The end goal? With these smart AI algorithms, you can confidently use the same cropping transformation with virtually every image you deliver, no matter the size and layout of your images, the user's device, or your graphic design.

The subject detection auto-cropping we've demonstrated in this post is available with all of Cloudinary's plans; even the free plan!

Learn more about g_auto:subject and all the automatic cropping options in our docs.

About Cloudinary

Cloudinary provides easy-to-use, cloud-based media management solutions for the world’s top brands. With offices in the US, UK and Israel, Cloudinary has quickly become the de facto solution used by developers and marketers at major companies around the world to streamline rich media management and deliver optimal end-user experiences.

For more information, visit www.cloudinary.com or follow us on Twitter

Recent Blog Posts

A New, Simple Tool for Creating Text Overlays for Images

Many Cloudinary users desire a UI for tasks like creating text overlays for images, which they then embed on webpages or download for marketing campaigns. Generating such overlays with the Cloudinary Media Library UI involves a bit of a learning curve, especially if they require multiple fonts or text lines, which even experienced users might find challenging to implement.

Read more
Transitioning JPEG-Based to JPEG XL-Based Images for Web Platforms

When the JPEG codec was being developed in the late 1980s, no standardized, lossy image-compression formats existed. JPEG became ready at exactly the right time in 1992, when the World Wide Web and digital cameras were about to become a thing. The introduction of HTML’s <img> tag in 1995 ensured the recognition of JPEG as the web format—at least for photographs. During the 1990s, digital cameras replaced analog ones and, given the limited memory capacities of that era, JPEG became the standard format for photography, especially for consumer-grade cameras.

Read more

Amplify Your Jamstack With Video

By Alex Patterson
Amplify Your Jamstack With Cloudinary Video

As defined by Amazon Web Services (AWS), Amplify is a set of products and tools with which mobile and front-end web developers can build and deploy AWS-powered, secure, and scalable full-stack apps. Also, you can efficiently configure their back ends, connect them to your app with just a few lines of code, and deploy static web apps in only three steps. Historically, because of their performance issues, managing images and videos is a daunting challenge for developers. Even though you can easily load media to an S3 bucket with AWS Amplify, transforming, compressing, and responsively delivering them is labor intensive and time consuming.

Read more
Cloudinary Helps Move James Hardie’s Experience Online

While COVID has affected most businesses, it has been particularly hard on those that sell products for the physical ‘brick and mortar’ world. One company that literally fits that bill is our Australian customer James Hardie, the largest global manufacturer of fibre cement products used in both domestic and commercial construction. These are materials that its buyers ideally want to see up close, in detail. When customers have questions, they expect personal service.

Read more