Amazon Rekognition Auto Tagging

Overview

Cloudinary is a cloud-based service that provides an end-to-end image management solution including uploads, storage, manipulations, optimizations and delivery. Cloudinary offers a very rich set of image manipulation and analysis capabilities and allows you to extract the semantic data from uploaded images: photo metadata (Exif & IPTC) including location and camera details, coordinates of automatically detected faces, color histogram and predominant colors. In addition, Cloudinary allows you to assign multiple tags to images for listing and managing your media library.

Amazon Rekognition is a service that makes it easy to add image analysis to your applications. Cloudinary provides an add-on for Amazon Rekognition's automatic image tagging capabilities, fully integrated into Cloudinary's image management and manipulation pipeline. Amazon Rekognition uses deep neural network models to automatically identify scenes and suggests tags, a process that would take huge amounts of time and resources if performed manually. Amazon Rekognition's deep learning-based image recognition allows you to search, verify and organize all your images and results in outstanding performance in terms of the time needed to process large photo volumes with a very high precision rate.

With the Amazon Rekognition Auto Tagging add-on, you can extend Cloudinary's powerful semantic data extraction and image tagging features. When using the Amazon Rekognition Auto Tagging add-on, your images are automatically tagged according to the categories detected in each image.

Image recognition and categorization

Take a look at the following photo, depicting a guitarist on stage:

Ruby:
cl_image_tag("guitarist.jpg")
PHP:
cl_image_tag("guitarist.jpg")
Python:
CloudinaryImage("guitarist.jpg").image()
Node.js:
cloudinary.image("guitarist.jpg")
Java:
cloudinary.url().imageTag("guitarist.jpg")
JS:
cl.imageTag('guitarist.jpg').toHtml();
jQuery:
$.cloudinary.image("guitarist.jpg")
React:
<Image publicId="guitarist.jpg" >

</Image>
Angular:
<cl-image public-id="guitarist.jpg" >

</cl-image>
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("guitarist.jpg")
Android:
MediaManager.get().url().generate("guitarist.jpg")
guitarist.jpg

By setting the categorization parameter to aws_rek_tagging when calling Cloudinary's upload, explicit, or update methods, Amazon Rekognition is used to automatically classify the scenes of the uploaded or specified existing image. For example:

Ruby:
Cloudinary::Uploader.upload("guitarist.jpg", 
  :categorization => "aws_rek_tagging")
PHP:
\Cloudinary\Uploader::upload("guitarist.jpg", 
  array("categorization" => "aws_rek_tagging"));
Python:
cloudinary.uploader.upload("guitarist.jpg",
  categorization = "aws_rek_tagging")
Node.js:
cloudinary.v2.uploader.upload("guitarist.jpg", 
  {categorization: "aws_rek_tagging"},
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("guitarist.jpg", ObjectUtils.asMap(
  "categorization", "aws_rek_tagging"));

The upload API response includes the automatic categorization identified by the Amazon Rekognition Auto Tagging add-on. As can be seen in the response snippet below, various categories were automatically detected in the uploaded photo. The confidence score is a numerical value that represents the confidence level of the detected category, where 1.0 means 100% confidence.

{
...
"info": {
    "categorization": {
      "aws_rek_tagging": {
        "status": "complete",
        "data": [
         [{"tag": "People", "confidence": 0.9922},
          {"tag": "Person", "confidence": 0.9922},
          {"tag": "Human", "confidence": 0.9921},
          {"tag": "Electric Guitar", "confidence": 0.7046},
          {"tag": "Guitar", "confidence": 0.7046},
          {"tag": "Musical Instrument", "confidence": 0.7046},
          {"tag": "Musician", "confidence": 0.5754},
          {"tag": "Performer", "confidence": 0.5754}]}}}}

Automatically adding tags to images

Automatically categorizing your images is a useful way to organize your Cloudinary media library, and by also providing the auto_tagging parameter to an upload or explicit call, images are automatically assigned resource tags based on the Amazon Rekognition detected scene categories. The value of the auto_tagging parameter is the minimum confidence score of a detected category that should be automatically used as an assigned resource tag. Assigning these resource tags allows you to list and search images in your media library using Cloudinary's API and Web interface.

The following code example automatically tags an uploaded image with all detected categories that have a score higher than 0.7.
(You can use the same syntax to automatically assign tags to any image already stored in your account by replacing the upload method in the example below with explicit).

Ruby:
Cloudinary::Uploader.upload("guitarist.jpg", 
  :categorization => "aws_rek_tagging", :auto_tagging => 0.7)
PHP:
\Cloudinary\Uploader::upload("guitarist.jpg", 
  array("categorization" => "aws_rek_tagging", "auto_tagging" => 0.7));
Python:
cloudinary.uploader.upload("guitarist.jpg",
  categorization = "aws_rek_tagging", auto_tagging = 0.7)
Node.js:
cloudinary.v2.uploader.upload("guitarist.jpg", 
  {categorization: "aws_rek_tagging", auto_tagging: 0.7},
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("guitarist.jpg", ObjectUtils.asMap(
  "categorization", "aws_rek_tagging", "auto_tagging", "0.7"));

The response of the upload API call above returns the detected categories as well as the assigned tags. In this case, only the ''People', 'Person', 'Human', 'Electric Guitar', 'Guitar', and 'Musical Instrument' categories have a high enough score to be used as tags.

{ 
  ...    
  "tags" => [ "People", "Person", "Human", "Electric Guitar", "Guitar", "Musical Instrument" ]   
  ...

You can also use the update method of the Admin API to apply Amazon Rekognition to already uploaded images, based on their public IDs, and then automatically tag them according to the detected categories.

For example, the following image was uploaded to Cloudinary with the koala5 public ID.

Ruby:
cl_image_tag("koala5.jpg")
PHP:
cl_image_tag("koala5.jpg")
Python:
CloudinaryImage("koala5.jpg").image()
Node.js:
cloudinary.image("koala5.jpg")
Java:
cloudinary.url().imageTag("koala5.jpg")
JS:
cl.imageTag('koala5.jpg').toHtml();
jQuery:
$.cloudinary.image("koala5.jpg")
React:
<Image publicId="koala5.jpg" >

</Image>
Angular:
<cl-image public-id="koala5.jpg" >

</cl-image>
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("koala5.jpg")
Android:
MediaManager.get().url().generate("koala5.jpg")
koala5.jpg

The following code sample uses Cloudinary's update method to apply Amazon Rekognition's automatic image tagging and categorization to the koala5 uploaded image, and then automatically assign resource tags based on the categories detected with over a 90% confidence level.

Ruby:
Cloudinary::Api.update("koala5", 
  :categorization => "aws_rek_tagging", :auto_tagging => 0.9)
PHP:
\Cloudinary\Api::update("koala5", 
  array("categorization" => "aws_rek_tagging", "auto_tagging" => 0.9));
Python:
cloudinary.api.update("koala5",
  categorization = "aws_rek_tagging", auto_tagging = 0.9)
Node.js:
cloudinary.v2.api.update("koala5", 
  {categorization: "aws_rek_tagging", auto_tagging: 0.9},
  function(error, result){console.log(result);});
Java:
cloudinary.api().update("koala5", ObjectUtils.asMap(
  "categorization", "aws_rek_tagging", "auto_tagging", 0.9));

The response of the update API call includes the detected categories, and automatically assigned tags. As you can see in the response snippet below, the 'Animal', 'Koala' and 'Mammal' scene categories were detected with a confidence score of over 90%, which led to them being auto-assigned as tags.

{
  ...
  "tags": [ "Animal", "Koala", "Mammal" ]
  "info": {
    "categorization": {
      "aws_rek_tagging": {
        "status": "complete",
        "data": [
          { "tag": "Animal", "confidence": 0.9933 },
          { "tag": "Koala", "confidence": 0.9933 },
          { "tag": "Mammal", "confidence": 0.9933 },
         ...
}

You can use the Admin API's resource method to return the details of a resource, including the the scene categories that you already extracted using the upload, explicit, or update methods.

Ruby:
Cloudinary::Api.resource("koala5")
PHP:
\Cloudinary\Api::resource("koala5");
Python:
cloudinary.api.resource("koala5")
Node.js:
cloudinary.v2.api.resource("koala5", 
  function(error, result){console.log(result);});
Java:
cloudinary.api().resource("koala5", ObjectUtils.emptyMap());