Cloudinary Blog

Facial Attribute Detection with Microsoft's Face API

Facial Attribute Detection with Microsoft's Face API

Many of the photos displayed on the internet these days are of people. If your website or mobile application displays photos that include people, you will want to make sure that their faces are included in the delivered images when cropping and manipulating them to fit your graphic design and responsive layout. You may even want to further manipulate an image according to the faces present, for example, adding a harlequin mask overlay on all of their eyes, where each mask is adjusted to the correct size and orientation (although not a typical use case, it's a cool example of using advanced facial attribute detection):

Ruby:
Copy to clipboard
cl_image_tag("cloudinary_team.jpg", :transformation=>[
  {:width=>700, :radius=>"max", :crop=>"scale"},
  {:flags=>"region_relative", :gravity=>"adv_eyes", :overlay=>"harlequinmask", :width=>1.7}
  ])
PHP:
Copy to clipboard
cl_image_tag("cloudinary_team.jpg", array("transformation"=>array(
  array("width"=>700, "radius"=>"max", "crop"=>"scale"),
  array("flags"=>"region_relative", "gravity"=>"adv_eyes", "overlay"=>"harlequinmask", "width"=>1.7)
  )))
Python:
Copy to clipboard
CloudinaryImage("cloudinary_team.jpg").image(transformation=[
  {'width': 700, 'radius': "max", 'crop': "scale"},
  {'flags': "region_relative", 'gravity': "adv_eyes", 'overlay': "harlequinmask", 'width': 1.7}
  ])
Node.js:
Copy to clipboard
cloudinary.image("cloudinary_team.jpg", {transformation: [
  {width: 700, radius: "max", crop: "scale"},
  {flags: "region_relative", gravity: "adv_eyes", overlay: "harlequinmask", width: "1.7"}
  ]})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation()
  .width(700).radius("max").crop("scale").chain()
  .flags("region_relative").gravity("adv_eyes").overlay(new Layer().publicId("harlequinmask")).width(1.7)).imageTag("cloudinary_team.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('cloudinary_team.jpg', {transformation: [
  {width: 700, radius: "max", crop: "scale"},
  {flags: "region_relative", gravity: "adv_eyes", overlay: new cloudinary.Layer().publicId("harlequinmask"), width: "1.7"}
  ]}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("cloudinary_team.jpg", {transformation: [
  {width: 700, radius: "max", crop: "scale"},
  {flags: "region_relative", gravity: "adv_eyes", overlay: new cloudinary.Layer().publicId("harlequinmask"), width: "1.7"}
  ]})
React:
Copy to clipboard
<Image publicId="cloudinary_team.jpg" >
  <Transformation width="700" radius="max" crop="scale" />
  <Transformation flags="region_relative" gravity="adv_eyes" overlay="harlequinmask" width="1.7" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="cloudinary_team.jpg" >
  <cld-transformation width="700" radius="max" crop="scale" />
  <cld-transformation flags="region_relative" gravity="adv_eyes" overlay="harlequinmask" width="1.7" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="cloudinary_team.jpg" >
  <cl-transformation width="700" radius="max" crop="scale">
  </cl-transformation>
  <cl-transformation flags="region_relative" gravity="adv_eyes" overlay="harlequinmask" width="1.7">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(700).Radius("max").Crop("scale").Chain()
  .Flags("region_relative").Gravity("adv_eyes").Overlay(new Layer().PublicId("harlequinmask")).Width(1.7)).BuildImageTag("cloudinary_team.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation()
  .width(700).radius("max").crop("scale").chain()
  .flags("region_relative").gravity("adv_eyes").overlay(new Layer().publicId("harlequinmask")).width(1.7)).generate("cloudinary_team.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation()
  .setWidth(700).setRadius("max").setCrop("scale").chain()
  .setFlags("region_relative").setGravity("adv_eyes").setOverlay("harlequinmask").setWidth(1.7)).generate("cloudinary_team.jpg")!, cloudinary: cloudinary)
Cloudinary team with masks

Face Detection is a great feature that enables the automatic modification of images according to the detected faces within an image, making it simple to intelligently crop, position, resize and transform your images appropriately.

At Cloudinary we strive to create an enriched environment that can solve our customers media asset related needs. By taking a holistic approach to image management, we create partnerships with leading companies developing image processing and media related technologies that can extend our internal features and capabilities and cater customers with more “complex” needs. Our add-ons feature services pre-integrated into Cloudinary that are tailored for developing, extending, and operating web and mobile apps.

Webinar
Marketing Without Barriers Through Dynamic Asset Management

We have recently partnered with Microsoft's Cognitive Services which provides a Face API for high precision face detection with state-of-the-art cloud-based algorithms. The Face API technology is fully integrated within our Advanced Facial Attributes Detection add-on that can do more than just detect the human faces in an image. The Advanced Facial Attribute Detection add-on can also extract meaningful advanced data about the face(s) in an image, including the exact location of facial features. This allows you even greater control over your image categorization, and to automatically use these details to smartly crop, position, rotate and overlay images based on the detected facial features.

Microsoft Cognitive Services logo

How to automatically detect facial attributes

Cloudinary supports uploading images using a cloud-based API. You can request further information while uploading the image by setting the detection parameter to adv_face when calling Cloudinary's upload API and Advanced Facial Attribute Detection is utilized to automatically extract detailed face attributes from the uploaded image. The detected faces are returned in the JSON response with rectangles (left, top, width and height) indicating the location of faces in the image in pixels, the exact position details of the eyes, mouth, eyebrows, nose and lips, as well as a series of face related attributes from each face such as pose, gender and age. See the Advanced Facial Attribute Detection documentation for more information. The code sample below uploads the lady image while requesting that the facial attributes are also returned in the JSON response:

Ruby:
Copy to clipboard
Cloudinary::Uploader.upload("lady.jpg", 
              :detection: "adv_face")
PHP:
Copy to clipboard
\Cloudinary\Uploader::upload("lady.jpg", 
              array(
               "detection": "adv_face"));
Python:
Copy to clipboard
cloudinary.uploader.upload("lady.jpg", 
              detection = "adv_face")
Node.js:
Copy to clipboard
cloudinary.uploader.upload("lady.jpg", 
              function(result) {console.log(result); }, { detection: "adv_face" });
Java:
Copy to clipboard
cloudinary.uploader().upload("lady.jpg", 
              Cloudinary.asMap("detection", "adv_face"));

Ruby:
Copy to clipboard
cl_image_tag("lady.jpg")
PHP:
Copy to clipboard
cl_image_tag("lady.jpg")
Python:
Copy to clipboard
CloudinaryImage("lady.jpg").image()
Node.js:
Copy to clipboard
cloudinary.image("lady.jpg")
Java:
Copy to clipboard
cloudinary.url().imageTag("lady.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('lady.jpg').toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("lady.jpg")
React:
Copy to clipboard
<Image publicId="lady.jpg" >

</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="lady.jpg" >

</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="lady.jpg" >

</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.BuildImageTag("lady.jpg")
Android:
Copy to clipboard
MediaManager.get().url().generate("lady.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().generate("lady.jpg")!, cloudinary: cloudinary)
Original uploaded image

The example JSON snippet below contains the results of the upload response when applying advanced facial attribute detection on the uploaded image. The response includes very detailed information regarding the face that was automatically detected in the image above:

Copy to clipboard
{
...
 "info": 
  {"detection": 
    {"adv_face": 
      {"status": "complete",
       "data": 
        [{"bounding_box": 
           {"top": 234.0, "left": 216.0, "width": 244.0, "height": 244.0},
          "attributes": 
           {"smile": 0.649,
            "head_pose": {"pitch": 0.0, "roll": -6.7, "yaw": 0.6},
            "gender": "female",
            "age": 30.3,
            "facial_hair": {"moustache": 0.0, "beard": 0.0, 
                            "sideburns": 0.0}},
          "facial_landmarks": 
           {"mouth": 
             {"left": {"x": 277.1, "y": 410.6},
              "right": {"x": 410.2, "y": 395.1},
              "under_lip": 
               {"bottom": {"x": 352.8, "y": 445.0},
                "top": {"x": 349.1, "y": 431.8}},
              "upper_lip": 
               {"bottom": {"x": 346.3, "y": 415.1},
                "top": {"x": 345.4, "y": 407.5}}},
            "eyebrow": 
             {"left_outer": {"x": 224.1, "y": 298.0},
              "left_inner": {"x": 306.6, "y": 283.4},
              "right_inner": {"x": 361.4, "y": 279.8},
              "right_outer": {"x": 428.8, "y": 272.2}},
            "eye": 
             {"left_outer": {"x": 258.5, "y": 314.1},
              "left_top": {"x": 277.0, "y": 306.2},
              "left_bottom": {"x": 277.1, "y": 315.6},
              "left_inner": {"x": 296.4, "y": 312.9},
              "right_inner": {"x": 373.4, "y": 305.2},
              "right_top": {"x": 388.0, "y": 294.5},
              "right_bottom": {"x": 389.0, "y": 306.0},
              "right_outer": {"x": 406.5, "y": 300.2},
              "left_pupil": {"x": 278.3, "y": 309.4},
              "right_pupil": {"x": 386.0, "y": 298.7}},
            "nose": 
             {"tip": {"x": 341.6, "y": 381.6},
              "root_left": {"x": 321.9, "y": 314.6},
              "root_right": {"x": 343.6, "y": 311.8},
              "left_alar_top": {"x": 312.7, "y": 359.7},
              "right_alar_top": {"x": 359.2, "y": 351.2},
              "left_alar_out_tip": {"x": 305.4, "y": 380.4},
              "right_alar_out_tip": {"x": 374.3, "y": 367.5}}}}]}}},
}

Dynamic image manipulation with facial attribute detection

Cloudinary supports on-the-fly image manipulation using simple parameters in HTTP delivery URLs. Based on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on, Cloudinary can crop your images to focus on the detected facial features, while providing a large set of image transformation and cropping options when using a Cloudinary delivery URL. To focus an automatic crop on the detected faces, simply set the crop parameter to thumb, fill or crop and the gravity parameter to adv_faces (set gravity to adv_face for focusing on the single largest detected face in the image). The resulting images are dynamically generated on-the-fly and the result is delivered via a fast CDN.

For example, to deliver a 300x300 thumbnail of the lady image shown above:

Ruby:
Copy to clipboard
cl_image_tag("lady.jpg", :gravity=>"adv_face", :height=>300, :width=>300, :crop=>"thumb")
PHP:
Copy to clipboard
cl_image_tag("lady.jpg", array("gravity"=>"adv_face", "height"=>300, "width"=>300, "crop"=>"thumb"))
Python:
Copy to clipboard
CloudinaryImage("lady.jpg").image(gravity="adv_face", height=300, width=300, crop="thumb")
Node.js:
Copy to clipboard
cloudinary.image("lady.jpg", {gravity: "adv_face", height: 300, width: 300, crop: "thumb"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().gravity("adv_face").height(300).width(300).crop("thumb")).imageTag("lady.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('lady.jpg', {gravity: "adv_face", height: 300, width: 300, crop: "thumb"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("lady.jpg", {gravity: "adv_face", height: 300, width: 300, crop: "thumb"})
React:
Copy to clipboard
<Image publicId="lady.jpg" >
  <Transformation gravity="adv_face" height="300" width="300" crop="thumb" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="lady.jpg" >
  <cld-transformation gravity="adv_face" height="300" width="300" crop="thumb" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="lady.jpg" >
  <cl-transformation gravity="adv_face" height="300" width="300" crop="thumb">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("adv_face").Height(300).Width(300).Crop("thumb")).BuildImageTag("lady.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().gravity("adv_face").height(300).width(300).crop("thumb")).generate("lady.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setGravity("adv_face").setHeight(300).setWidth(300).setCrop("thumb")).generate("lady.jpg")!, cloudinary: cloudinary)
150x150 thumbnail of lady.jpg

Cloudinary can also dynamically crop your images based on the position of detected eyes. Simply set the gravity parameter to adv_eyes (g_adv_eyes for URLs) to center the image on the detected eyes. The example below delivers a 200x60 thumbnail centered on the eyes:

Ruby:
Copy to clipboard
cl_image_tag("lady.jpg", :gravity=>"adv_eyes", :width=>200, :height=>60, :crop=>"thumb")
PHP:
Copy to clipboard
cl_image_tag("lady.jpg", array("gravity"=>"adv_eyes", "width"=>200, "height"=>60, "crop"=>"thumb"))
Python:
Copy to clipboard
CloudinaryImage("lady.jpg").image(gravity="adv_eyes", width=200, height=60, crop="thumb")
Node.js:
Copy to clipboard
cloudinary.image("lady.jpg", {gravity: "adv_eyes", width: 200, height: 60, crop: "thumb"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().gravity("adv_eyes").width(200).height(60).crop("thumb")).imageTag("lady.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('lady.jpg', {gravity: "adv_eyes", width: 200, height: 60, crop: "thumb"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("lady.jpg", {gravity: "adv_eyes", width: 200, height: 60, crop: "thumb"})
React:
Copy to clipboard
<Image publicId="lady.jpg" >
  <Transformation gravity="adv_eyes" width="200" height="60" crop="thumb" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="lady.jpg" >
  <cld-transformation gravity="adv_eyes" width="200" height="60" crop="thumb" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="lady.jpg" >
  <cl-transformation gravity="adv_eyes" width="200" height="60" crop="thumb">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Gravity("adv_eyes").Width(200).Height(60).Crop("thumb")).BuildImageTag("lady.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().gravity("adv_eyes").width(200).height(60).crop("thumb")).generate("lady.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setGravity("adv_eyes").setWidth(200).setHeight(60).setCrop("thumb")).generate("lady.jpg")!, cloudinary: cloudinary)
200x60 thumbnail centered on eyes

Thanks to the detailed information on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on, Cloudinary can add overlays while taking into account the pose of the face, and automatically scale and rotate the overlay accordingly.

Ruby:
Copy to clipboard
cl_image_tag("HarlequinMask.jpg", :width=>150, :crop=>"scale")
PHP:
Copy to clipboard
cl_image_tag("HarlequinMask.jpg", array("width"=>150, "crop"=>"scale"))
Python:
Copy to clipboard
CloudinaryImage("HarlequinMask.jpg").image(width=150, crop="scale")
Node.js:
Copy to clipboard
cloudinary.image("HarlequinMask.jpg", {width: 150, crop: "scale"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().width(150).crop("scale")).imageTag("HarlequinMask.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('HarlequinMask.jpg', {width: 150, crop: "scale"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("HarlequinMask.jpg", {width: 150, crop: "scale"})
React:
Copy to clipboard
<Image publicId="HarlequinMask.jpg" >
  <Transformation width="150" crop="scale" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="HarlequinMask.jpg" >
  <cld-transformation width="150" crop="scale" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="HarlequinMask.jpg" >
  <cl-transformation width="150" crop="scale">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(150).Crop("scale")).BuildImageTag("HarlequinMask.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().width(150).crop("scale")).generate("HarlequinMask.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setWidth(150).setCrop("scale")).generate("HarlequinMask.jpg")!, cloudinary: cloudinary)
Harlequin mask

For example, in order to automatically overlay the above image of a harlequin mask scaled to 170% relative to the detected eyes in the main image:

Ruby:
Copy to clipboard
cl_image_tag("lady.jpg", :flags=>"region_relative", :gravity=>"adv_eyes", :overlay=>"HarlequinMask", :width=>1.7, :crop=>"scale")
PHP:
Copy to clipboard
cl_image_tag("lady.jpg", array("flags"=>"region_relative", "gravity"=>"adv_eyes", "overlay"=>"HarlequinMask", "width"=>1.7, "crop"=>"scale"))
Python:
Copy to clipboard
CloudinaryImage("lady.jpg").image(flags="region_relative", gravity="adv_eyes", overlay="HarlequinMask", width=1.7, crop="scale")
Node.js:
Copy to clipboard
cloudinary.image("lady.jpg", {flags: "region_relative", gravity: "adv_eyes", overlay: "HarlequinMask", width: "1.7", crop: "scale"})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation().flags("region_relative").gravity("adv_eyes").overlay(new Layer().publicId("HarlequinMask")).width(1.7).crop("scale")).imageTag("lady.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('lady.jpg', {flags: "region_relative", gravity: "adv_eyes", overlay: new cloudinary.Layer().publicId("HarlequinMask"), width: "1.7", crop: "scale"}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("lady.jpg", {flags: "region_relative", gravity: "adv_eyes", overlay: new cloudinary.Layer().publicId("HarlequinMask"), width: "1.7", crop: "scale"})
React:
Copy to clipboard
<Image publicId="lady.jpg" >
  <Transformation flags="region_relative" gravity="adv_eyes" overlay="HarlequinMask" width="1.7" crop="scale" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="lady.jpg" >
  <cld-transformation flags="region_relative" gravity="adv_eyes" overlay="HarlequinMask" width="1.7" crop="scale" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="lady.jpg" >
  <cl-transformation flags="region_relative" gravity="adv_eyes" overlay="HarlequinMask" width="1.7" crop="scale">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation().Flags("region_relative").Gravity("adv_eyes").Overlay(new Layer().PublicId("HarlequinMask")).Width(1.7).Crop("scale")).BuildImageTag("lady.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation().flags("region_relative").gravity("adv_eyes").overlay(new Layer().publicId("HarlequinMask")).width(1.7).crop("scale")).generate("lady.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation().setFlags("region_relative").setGravity("adv_eyes").setOverlay("HarlequinMask").setWidth(1.7).setCrop("scale")).generate("lady.jpg")!, cloudinary: cloudinary)
Harlequin masked face

See the Advanced Facial Attribute Detection documentation for more information on the features available with the add-on and how to use them.

Summary

The Advanced Facial Attribute Detection add-on powered by Cloudinary and the Face API of Microsoft's Cognitive Services provides a high precision mechanism that can analyze images and create the best crop for most sites as well as automatically add the nice artistic effects of exact overlay placing. The integration within Cloudinary's pipeline is seamless and dynamic using simple manipulation URLs.

We are excited to enter into this partnership with Microsoft's Cognitive Services so give the add-on a try. The Advanced Facial Attribute Detection add-on is available to all our free and paid plans.

Recent Blog Posts

Transitioning JPEG-Based to JPEG XL-Based Images for Web Platforms

When the JPEG codec was being developed in the late 1980s, no standardized, lossy image-compression formats existed. JPEG became ready at exactly the right time in 1992, when the World Wide Web and digital cameras were about to become a thing. The introduction of HTML’s <img> tag in 1995 ensured the recognition of JPEG as the web format—at least for photographs. During the 1990s, digital cameras replaced analog ones and, given the limited memory capacities of that era, JPEG became the standard format for photography, especially for consumer-grade cameras.

Read more

Amplify Your Jamstack With Video

By Alex Patterson
Amplify Your Jamstack With Cloudinary Video

As defined by Amazon Web Services (AWS), Amplify is a set of products and tools with which mobile and front-end web developers can build and deploy AWS-powered, secure, and scalable full-stack apps. Also, you can efficiently configure their back ends, connect them to your app with just a few lines of code, and deploy static web apps in only three steps. Historically, because of their performance issues, managing images and videos is a daunting challenge for developers. Even though you can easily load media to an S3 bucket with AWS Amplify, transforming, compressing, and responsively delivering them is labor intensive and time consuming.

Read more
Cloudinary Helps Move James Hardie’s Experience Online

While COVID has affected most businesses, it has been particularly hard on those that sell products for the physical ‘brick and mortar’ world. One company that literally fits that bill is our Australian customer James Hardie, the largest global manufacturer of fibre cement products used in both domestic and commercial construction. These are materials that its buyers ideally want to see up close, in detail. When customers have questions, they expect personal service.

Read more
How to Build an Enhanced Gravatar Service, Part 2

Part 1 of this post defines the capabilities of an enhanced Gravatar service, which I named Clavatar, and describes the following initial steps for building it:

This post, part 2 of the series, explains how to make Clavatar work like Gravatar and to develop Clavatar’s capabilities of enabling requests for various versions of the images related to user accounts.

Read more