Amazon Rekognition Celebrity Detection

Overview

Cloudinary is a cloud-based service that provides an end-to-end image and video management solution including uploads, storage, manipulations, optimizations and delivery. Cloudinary offers a very rich set of image manipulation and analysis capabilities and allows you to extract the semantic data from uploaded images: photo metadata (Exif & IPTC) including location and camera details, coordinates of automatically detected faces, color histogram and predominant colors. In addition, Cloudinary allows you to assign multiple tags to images for listing and managing your media library.

Amazon Rekognition is a service that makes it easy to add image analysis to your applications. Cloudinary provides an add-on for Amazon Rekognition's automatic celebrity tagging and face detection capabilities, fully integrated into Cloudinary's image management and manipulation pipeline. When using the Amazon Rekognition Celebrity Detection add-on, your images are automatically tagged according to the celebrities detected in each image.

Amazon Rekognition can recognize thousands of celebrities in a wide range of categories, such as entertainment and media, sports, business, and politics, a process that would take huge amounts of time and resources if performed manually. Amazon Rekognition's deep learning-based image recognition allows you to search, verify and organize all your images, with outstanding performance in terms of the time needed to process large photo volumes with a very high precision rate.

Celebrity recognition and categorization

Take a look at the following photo of The Rolling Stones:

Ruby:
cl_image_tag("rolling_stones.jpg")
PHP:
cl_image_tag("rolling_stones.jpg")
Python:
CloudinaryImage("rolling_stones.jpg").image()
Node.js:
cloudinary.image("rolling_stones.jpg")
Java:
cloudinary.url().imageTag("rolling_stones.jpg")
JS:
cl.imageTag('rolling_stones.jpg').toHtml();
jQuery:
$.cloudinary.image("rolling_stones.jpg")
React:
<Image publicId="rolling_stones.jpg" >

</Image>
Angular:
<cl-image public-id="rolling_stones.jpg" >

</cl-image>
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("rolling_stones.jpg")
Android:
MediaManager.get().url().generate("rolling_stones.jpg")
rolling_stones

By setting the detection parameter to aws_rek_face when calling Cloudinary's upload, explicit, or update methods, Amazon Rekognition is used to automatically detect celebrity faces in the uploaded or specified existing image. For example:

Ruby:
Cloudinary::Uploader.upload("rolling_stones.jpg", 
  :detection => "aws_rek_face")
PHP:
\Cloudinary\Uploader::upload("rolling_stones.jpg", 
  array("detection" => "aws_rek_face"));
Python:
cloudinary.uploader.upload("rolling_stones.jpg",
  detection = "aws_rek_face")
Node.js:
cloudinary.v2.uploader.upload("rolling_stones.jpg", 
  {detection: "aws_rek_face" }, 
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("rolling_stones.jpg", ObjectUtils.asMap(
  "detection", "aws_rek_face"));

The upload API response includes the celebrities identified by the Amazon Rekognition Celebrity Detection add-on. As can be seen in the response snippet below, all four of the celebrities were automatically detected in the uploaded photo. The match_confidence score is a percentage value that represents the confidence level of the detected celebrity, where 100 means 100% confidence.

The response also includes information about unrecognized faces detected in the image if relevant. Additionally, the results provide detailed data on the face's location, pose, orientation within the image, and facial landmarks.

{
...
"info": 
  {"detection": 
    {"aws_rek_face": 
      {"status": "complete",
       "data": 
        {"celebrity_faces": 
          [{ 
            "name": "Ronnie Wood",
            "match_confidence": 100.0
            "face":
             {"bounding_box":
               {"width"...
              },
              "landmarks":
               [{"type": "eyeLeft",
                 "x": 
               }],
            ...
           },
                 ...
            "name": "Charlie Watts",
            "match_confidence": 100.0},
                 ...
            "name": "Mick Jagger",
            "match_confidence": 100.0},
                 ...
            "name": "Keith Richards",
            "match_confidence": 98.0}],
         "unrecognized_faces": [],
         ...
}}}}}

Automatically adding tags to images

Automatically categorizing your images is a useful way to organize your Cloudinary media library. By providing the auto_tagging parameter to an upload or explicit call alongside the aws_rek_face value for the detection parameter, images are automatically assigned resource tags based on the Amazon Rekognition detected celebrities. The value of the auto_tagging parameter is the minimum confidence score of a detected celebrity that should be used for the automatically assigned resource tag. The value of the auto_tagging parameter is given as a decimal value between 0 and 1.0, where 1.0 represents 100%. Assigning these resource tags allows you to list and search images in your media library using Cloudinary's API and Web interface.

The following code example automatically tags an uploaded image with all detected celebrities that have a match_confidence score of at least 70% (auto_tagging = 0.7).
(You can use the same syntax to automatically assign tags to any image already stored in your account by replacing the upload method in the example below with explicit).

Ruby:
Cloudinary::Uploader.upload("rolling_stones.jpg", 
  :detection => "aws_rek_face", :auto_tagging => 0.7)
PHP:
\Cloudinary\Uploader::upload("rolling_stones.jpg", 
  array("detection" => "aws_rek_face", "auto_tagging" => 0.7));
Python:
cloudinary.uploader.upload("rolling_stones.jpg",
  detection = "aws_rek_face", auto_tagging = 0.7)
Node.js:
cloudinary.v2.uploader.upload("rolling_stones.jpg", 
  {detection: "aws_rek_face", auto_tagging: 0.7},
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("rolling_stones.jpg", ObjectUtils.asMap(
  "detection", "aws_rek_face", "auto_tagging", "0.7"));

The response of the upload API call above returns the detected categories as well as the assigned tags. In this case, all the faces in the image were recognized with a high enough score to be used as tags.

{ 
  ...    
  "tags" => [ "Ronnie Wood", "Charlie Watts", "Mick Jagger", "Keith Richards" ]   
  ...

You can also use the update method of the Admin API to apply Amazon Rekognition to already uploaded images, based on their public IDs, and then automatically tag them according to the detected celebrities.

For example, the following image was uploaded to Cloudinary with the golfer public ID.

Ruby:
cl_image_tag("golfer.jpg")
PHP:
cl_image_tag("golfer.jpg")
Python:
CloudinaryImage("golfer.jpg").image()
Node.js:
cloudinary.image("golfer.jpg")
Java:
cloudinary.url().imageTag("golfer.jpg")
JS:
cl.imageTag('golfer.jpg').toHtml();
jQuery:
$.cloudinary.image("golfer.jpg")
React:
<Image publicId="golfer.jpg" >

</Image>
Angular:
<cl-image public-id="golfer.jpg" >

</cl-image>
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("golfer.jpg")
Android:
MediaManager.get().url().generate("golfer.jpg")
golfer.jpg

The following code sample uses Cloudinary's update method to apply Amazon Rekognition's celebrity detection to the golfer uploaded image, and then automatically assign resource tags based on the celebrities detected with at least a 90% confidence level.

Ruby:
Cloudinary::Api.update("golfer", 
  :detection => "aws_rek_face", :auto_tagging => 0.9)
PHP:
\Cloudinary\Api::update("golfer", 
  array("detection" => "aws_rek_face", "auto_tagging" => 0.9));
Python:
cloudinary.api.update("golfer",
  detection = "aws_rek_face", auto_tagging = 0.9)
Node.js:
cloudinary.v2.api.update("golfer", 
  {detection: "aws_rek_face", auto_tagging: 0.9},
  function(error, result){console.log(result);});
Java:
cloudinary.api().update("golfer", ObjectUtils.asMap(
  "detection", "aws_rek_face", "auto_tagging", 0.9));

The response of the update API call includes the detected celebrities, and automatically assigned tags. As you can see in the response snippet below, 'Tiger Woods' was recognized in the image with a match_confidence score of over 90%, which led to his name being auto-assigned as a tag.

{
  ...
  "tags": [ "Tiger Woods" ]
  "info": {
    "categorization": {
      "aws_rek_face": {
        "status": "complete",
        "data": [
          { "name": "Tiger Woods", "match_confidence": 100,
         ...
}

You can use the Admin API's resource method to return the details of a resource, including the celebrities that you already extracted using the upload, explicit, or update methods.

Ruby:
Cloudinary::Api.resource("golfer")
PHP:
\Cloudinary\Api::resource("golfer");
Python:
cloudinary.api.resource("golfer")
Node.js:
cloudinary.v2.api.resource("golfer", 
  function(error, result){console.log(result);});
Java:
cloudinary.api().resource("golfer", ObjectUtils.emptyMap());