Perform Face-Detection Transformation Techniques On-the-Fly with Cloudinary

With the decreasing price-performance ratio of computing, research efforts in face detection and face recognition algorithms are rapidly expanding and new techniques for both of these are achieving greater accuracy and reduced processing time.

Face detection is a technology that identifies whether and where human faces are located in digital images. It’s commonly used in cameras, security applications, graphic applications, and a variety of other web and mobile apps. It is also a prerequisite for face recognition, which uses a database and additional algorithms to match a detected face with a specific individual.

A large percentage of the images in most websites and apps include photos of people – from profile pics to news articles to group selfies. These photos need to be displayed in sizes, shapes, and styles that match the site design. Performing automatic face cropping or otherwise programmatically modifying images that contain faces necessitates a reliable face detection technology. The reliability can be affected by the colors, or lack of colors in the image, the direction a person is facing, the tilt of the head, the size of a face, the complexity of the background, lighting, and more.

Cloudinary provides a complete image and video management solution for web and mobile app programmers. Our solution includes a built-in image face detection functionality and several face-detection features you can take advantage of in your own apps. In most cases, applying this functionality is just a question of adding a parameter or two to your image delivery URLs… and your own creativity, of-course.

Cloudinary recently updated its face detection technology. The new technology is now very efficient and precise, including many “difficult” scenarios such as side portraits, blurry faces, tricky lighting, and more.

For example, the people in these images are now easily detected despite low lighting, unusual shadows, black and white images, turned heads and closed eyes:

 

 

When you deliver an image URL that includes face-detection transformations, the faces in the original photo are detected on-the-fly, the requested transformations are performed in the cloud, and then the final image is delivered via CDN.

The rest of this blog will cover some of the many ways that developers can automatically transform proprietary and user-generated photos containing faces to fit the needs and design of a website or app.

Perhaps one of the most common uses of face detection is for cropping. A large percentage of images, especially with user-generated content, contain photos of people.

Headshots are needed for profile pictures, chat heads, and more, but the original uploaded photo may include much more than just a face, and that face may not even be the central element in the photo. Group selfies are also popular, and rarely centered, so again, we need to be very careful that even if we crop the photo to a different aspect ratio, we don’t lose any of the crew.

In short, it’s critical in most situations that when we resize and crop uploaded photos, we keep our users’ faces up front and center. There are a variety of features you can use or combine to ensure this.

When cropping photos intended for profile pictures or other headshots, you will usually want to use face as your cropping gravity (g_face in URLs). This ensures that the detected face (or largest face if there is more than one) will be the center of the cropped photo, regardless of its initial location.

If in conjunction with face gravity, you use thumb as your crop method, you will get a crop that’s as tight as possible (given the cropping dimensions) to the detected face coordinates.

You can further adjust your thumbnail cropping by setting a zoom value, to zoom in or out from the detected face coordinates. And then of-course you can add any other transformations you want.

Here’s what the delivery code looks like for that last photo: Loading code examples

Everything mentioned above works equally well for multiple faces. Just use faces gravity instead of face, and your cropping will be based on a rectangle that includes all detected faces.

All of this assumes you can reasonably expect that the uploaded image is going to contain one or multiple faces as the main subject.

Sometimes the main subject of an image isn’t necessarily expected to be a person, but you still need to ensure you aren’t cutting off people or other important content of the image. In this case, you can take advantage of the auto gravity feature. With this automated cropping method, the most prominent elements of the picture determine the cropping coordinates. Unless otherwise specified, any detected faces get a high priority in this cropping decision, but other prominent elements are also included.

You can learn lots more about auto gravity in this blog article and this documentation.

There may be situations where you’d like to keep the people in a photograph anonymous. In these cases, you can pixelate or blur all detected faces just by adding a single parameter to your image URL.

In the example below, the faces that Cloudinary detects in the concert audience are generally those that could likely be identified by a human viewer. All of these faces are automatically blurred before the image is delivered, using the blur_faces effect.

Loading code examples

And here’s an example of an ice skating scene where a couple of childrens’ faces are clearly visible. This time, we use the pixelate option to protect the kids’ identities:

Loading code examples

Tip: For cases like these, you may want to take advantage of one of the access control options to prevent users from accessing the non-blurred or non-pixelated versions of the image.

There’s no such thing as a face detection technology that will detect 100% of the faces 100% of the time, so what if you notice that a particular face you need was not detected? No problem. You can always update the detected coordinates explicitly.

For example, when a user uploads an image to your site, you can retrieve and display the detected face coordinates and then enable your users to adjust the coordinates or add a new face via your UI. You can then pass the updated coordinates to Cloudinary via the Update method of the Admin API.

For your own images, you can also manually change the coordinates using the Media library UI.

Sometimes, just blurring or pixelating a face isn’t enough. Instead, you want to completely cover the face with another image. Or, maybe you just want to add a fun mask. Whatever the reason, it’s simple to add the same image to all faces in a photo.

To do this, you take advantage of the region_relative flag and faces gravity when you specify the overlay image. That tells Cloudinary that you want the overlay to be placed on all faces, and that the specified size of each overlay is a percentage relative to the size of each detected face.

For example, here’s a quick way to get your family dressed up in costume: Loading code examples

You probably already noticed that we gave anonymous (happy) faces to our roller coaster riders at the beginning of this article using the same technique.

And remember, the overlay you use doesn’t have to cover the whole face or even be centered. Below, we add a ranger hat to a man’s head using a relative y- coordinate to offset the location of the hat about 30% above the center of the face.

Loading code examples

If you want to check the face detection status of uploaded images programmatically, you can request to return face coordinate data when you or your users upload images, or you can request them upon demand for already uploaded images using the explicit method of the upload API, or with the list details of a single resource functionality of the Admin API. The following shows an excerpt from the JSON response of an upload call where the faces parameter was set to true.

{
  ...
  "faces":  [ [513, 19, 38, 52], [409, 26, 40, 54], [79, 31, 43, 59], [232, 32, 40, 54], [321, 33, 41, 57], [160, 37, 43, 59]
}
Code language: JavaScript (javascript)

For example, you could use this data to calculate the ideal location for a text overlay so that it won’t cover any faces on a photo.

Cloudinary enables you to create programmatically complex transformations by adding conditions to your transformations. One of the things you can base your conditions on is face_count.

For example, you can’t always be sure that a profile picture will be of a person. Some people choose to upload family portraits, scenery, or other images to represent themselves. The following images use the identical conditional transformation, where thumbnail cropping is applied if one or more faces is found, but scaled cropping is used on images without faces. We’ve also added a shadow vs black border for the alternative conditions to highlight the differences.

(The code shown here is for the nice_couple on the left, but you can click the bridge picture to see that it uses the identical transformation URL except for the image name (public ID).)

Loading code examples

You can learn more about conditional transformations here.

Just a reminder that in addition to applying all this cool stuff on images that you or your users directly upload, you can also use the same tricks with remotely fetched images, such as Facebook profile pictures.

For example, you can allow your users to select to use their Facebook profile photo as the basis for their account profile photo in your app and then apply resizing, rounding, artistic effects, or any other transformation to fit your site’s art direction, while ensuring that the detected face of the fetched photo remains the main focus of the final photo.

Here we apply cropping and several other transformations to the Facebook profile photo from the Remembering JFK page:

Loading code examples

You can do the same with images fetched from other sites, like Twitter, Google+, or from any image URL.

By now, the many potential uses for face detection techniques have probably become obvious to you, as plain as the nose on your face, you might say…. OK, I’m done with the painful face puns.

The fact is that almost every Website includes both local and user-generated content that centers around people. It’s essential that those people remain the focus of the displayed pictures regardless of how you resize, crop, add images and text overlays, etc.

Cloudinary’s face detection capabilities and updated mechanism enable you to accomplish all of these things programmatically, using simple transformation parameters, and with very reliable results.

Ready to see how far your creativity can take these face detection features? These features are available in all plans including the free plan, with no add-ins required. If you don’t have a Cloudinary account yet, you can sign up for a free one, and give it a go!