Cloudinary Blog

How to automatically tag images with Amazon Rekognition

Analyze and auto tag images with Amazon Rekognition

Knowledge is power. And if you allow your users to upload images, you also probably want to better understand what their images contain. Whether a photo is of a building, people, animals, celebrities, or a product, image processing and analysis can assist in further comprehension. The benefits of this knowledge can go beyond "merely" categorizing your content and making your image library searchable: drawing insights from user generated content can be very useful! What better way to learn more about your users than to analyze the images they upload and find out what they care about and then have the ability to display relevant content to them according to their interests or even match them with other users that share similar interests.

Analyzing the contents of a photograph would be a huge time sink if performed manually on a lot of images. Enter Amazon Rekognition, a service that uses deep neural network models to detect and tag thousands of people, objects and scenes in your images, and lets you easily build powerful visual search and discovery into your applications.

Webinar
Marketing Without Barriers Through Dynamic Asset Management

Cloudinary has integrated two Amazon Rekognition add-ons into its image management and manipulation pipeline: the Amazon Rekognition Auto Tagging add-on and the Amazon Rekognition Celebrity Detection add-on.

Customers are increasingly looking for innovative ways to generate more data around their media assets to derive actionable business insights. By using Amazon Rekognition, Cloudinary is bringing this technology closer to its clients so they can build their own unique experience.

- Ranju Das, Director of Amazon Rekognition, Amazon Web Services, Inc.

Amazon Rekognition: Categorization

The Amazon Rekognition Auto Tagging add-on can automatically identify what's in a photograph and returns a list of detected categories and the confidence score of each category. The add-on is very simple to use: just set the categorization parameter of Cloudinary's image upload API to aws_rek_tagging while uploading a new image or updating an existing image and the response will include a list of identified tags for the image.

For example, uploading the following picture of a puppy to Cloudinary and requesting categorization:

Ruby:
Copy to clipboard
Cloudinary::Uploader.upload("cute_puppy.jpg", 
  :categorization => "aws_rek_tagging")
PHP:
Copy to clipboard
\Cloudinary\Uploader::upload("cute_puppy.jpg", 
  array("categorization" => "aws_rek_tagging"));
Python:
Copy to clipboard
cloudinary.uploader.upload("cute_puppy.jpg",
  categorization = "aws_rek_tagging")
Node.js:
Copy to clipboard
cloudinary.v2.uploader.upload("cute_puppy.jpg", 
  {categorization: "aws_rek_tagging"},
  function(error, result){console.log(result);});
Java:
Copy to clipboard
cloudinary.uploader().upload("cute_puppy.jpg", ObjectUtils.asMap(
  "categorization", "aws_rek_tagging"));

Ruby:
Copy to clipboard
cl_image_tag("cute_puppy.jpg")
PHP:
Copy to clipboard
cl_image_tag("cute_puppy.jpg")
Python:
Copy to clipboard
CloudinaryImage("cute_puppy.jpg").image()
Node.js:
Copy to clipboard
cloudinary.image("cute_puppy.jpg")
Java:
Copy to clipboard
cloudinary.url().imageTag("cute_puppy.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('cute_puppy.jpg').toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("cute_puppy.jpg")
React:
Copy to clipboard
<Image publicId="cute_puppy.jpg" >

</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="cute_puppy.jpg" >

</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="cute_puppy.jpg" >

</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.BuildImageTag("cute_puppy.jpg")
Android:
Copy to clipboard
MediaManager.get().url().generate("cute_puppy.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().generate("cute_puppy.jpg")!, cloudinary: cloudinary)
image of a cute puppy

The upload response includes the various categories that were automatically detected in the uploaded photo together with the confidence level of the detected category. So Amazon Rekognition is 91% sure that the picture contains a puppy and is 82% sure that the breed is Newfoundland.

Copy to clipboard
...
"info": {
  "categorization": {
    "aws_rek_tagging": {
      "status": "complete", 
      "data": [
        {"tag": "Animal", "confidence": 0.9117},
        {"tag": "Canine", "confidence": 0.9117}, 
        {"tag": "Dog", "confidence": 0.9117}, 
        {"tag": "Mammal", "confidence": 0.9117}, 
        {"tag": "Pet", "confidence": 0.9117}, 
        {"tag": "Puppy", "confidence": 0.9117}, 
        {"tag": "Newfoundland", "confidence": 0.8253}, 
        {"tag": "Husky", "confidence": 0.596}, 
        {"tag": "Adorable", "confidence": 0.5791} 
...

For more information on the Amazon Rekognition Auto Tagging add-on, see the documentation.

Amazon Rekognition: Celebrity Detection

The Amazon Rekognition Celebrity Detection add-on can automatically recognize thousands of celebrities in a wide range of categories, such as entertainment and media, sports, business, and politics. This add-on is also very simple to use: just set the detection parameter of Cloudinary's image upload API to aws_rek_face while uploading a new image or updating an existing image, and the response will include a list of identified celebrities for the image.

For example, uploading the following picture to Cloudinary and requesting detection:

Ruby:
Copy to clipboard
Cloudinary::Uploader.upload("brangelina.jpg", 
  :detection => "aws_rek_face")
PHP:
Copy to clipboard
\Cloudinary\Uploader::upload("brangelina.jpg", 
  array("detection" => "aws_rek_face"));
Python:
Copy to clipboard
cloudinary.uploader.upload("brangelina.jpg",
  detection = "aws_rek_face")
Node.js:
Copy to clipboard
cloudinary.v2.uploader.upload("brangelina.jpg", 
  {detection: "aws_rek_face"},
  function(error, result){console.log(result);});
Java:
Copy to clipboard
cloudinary.uploader().upload("brangelina.jpg", ObjectUtils.asMap(
  "detection", "aws_rek_face"));

Ruby:
Copy to clipboard
cl_image_tag("brangelina.jpg")
PHP:
Copy to clipboard
cl_image_tag("brangelina.jpg")
Python:
Copy to clipboard
CloudinaryImage("brangelina.jpg").image()
Node.js:
Copy to clipboard
cloudinary.image("brangelina.jpg")
Java:
Copy to clipboard
cloudinary.url().imageTag("brangelina.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('brangelina.jpg').toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("brangelina.jpg")
React:
Copy to clipboard
<Image publicId="brangelina.jpg" >

</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="brangelina.jpg" >

</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="brangelina.jpg" >

</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.BuildImageTag("brangelina.jpg")
Android:
Copy to clipboard
MediaManager.get().url().generate("brangelina.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().generate("brangelina.jpg")!, cloudinary: cloudinary)
brangelina

The upload response includes the various celebrities that were automatically detected in the uploaded photo together with the confidence level of the detected celebrity. So Amazon Rekognition is 100% sure that this is a picture of Brad Pitt and Angelina Jolie. The response also includes information about unrecognized faces detected in the image if relevant, and the results provide detailed data on each face's location, pose, orientation within the image, and facial landmarks.

Copy to clipboard
{
...
"info": 
  {"detection": 
    {"aws_rek_face": 
      {"status": "complete",
       "data": 
        {"celebrity_faces": 
          [{ 
            "name": "Angelina Jolie",
            "match_confidence": 100.0
            "face":
             {"bounding_box":
               {"width"...
              },
              "landmarks":
               [{"type": "eyeLeft",
                 "x": ...
               }],
            ...
           },
                 ...
            "name": "Brad Pitt",
            "match_confidence": 100.0},
                 ...
        "unrecognized_faces": [ ],
         ...
}}}}}

For more information on the Amazon Rekognition Celebrity Detection add-on, see the documentation.

Automatically tagging images

Cloudinary allows you to tag uploaded images, where each image can be assigned one or more tags. You can also leverage the Amazon Rekognition add-ons to automatically assign tags to your images based on a minimum confidence score of identified categories/celebrities.

Simply add the auto_tagging parameter to the upload call and the image is automatically assigned resource tags based on the Amazon Rekognition detected categories/celebrities. The value of the auto_tagging parameter is the minimum confidence score of a detected celebrity that should be used for the automatically assigned resource tag. The value of the auto_tagging parameter is given as a decimal value between 0 and 1, where 1 represents 100%. Assigning these resource tags allows you to list and search images in your media library using Cloudinary's API and Web interface.

Note that the automatic tagging feature can be used with either of the Amazon Rekognition add-ons or with both together. The following example automatically tags an uploaded image with all detected categories and celebrities that have a confidence score of at least 70% (auto_tagging = 0.7).

Ruby:
Copy to clipboard
Cloudinary::Uploader.upload("steve.jpg", 
  :detection => "aws_rek_face", 
  :categorization => "aws_rek_tagging",
  :auto_tagging => 0.7)
PHP:
Copy to clipboard
\Cloudinary\Uploader::upload("steve.jpg", 
  array(
    "detection" => "aws_rek_face", 
    "categorization" => "aws_rek_tagging", 
    "auto_tagging" => 0.7));
Python:
Copy to clipboard
cloudinary.uploader.upload("steve.jpg",
  detection = "aws_rek_face",
  categorization = "aws_rek_tagging",
  auto_tagging = 0.7)
Node.js:
Copy to clipboard
cloudinary.v2.uploader.upload("steve.jpg", {
  detection: "aws_rek_face", 
  categorization: "aws_rek_tagging", 
  auto_tagging: 0.7},
  function(error, result){console.log(result);});
Java:
Copy to clipboard
cloudinary.uploader().upload("steve.jpg", ObjectUtils.asMap(
  "detection", "aws_rek_face", 
  "categorization", "aws_rek_tagging",
  "auto_tagging", "0.7"));

Ruby:
Copy to clipboard
cl_image_tag("steve.jpg")
PHP:
Copy to clipboard
cl_image_tag("steve.jpg")
Python:
Copy to clipboard
CloudinaryImage("steve.jpg").image()
Node.js:
Copy to clipboard
cloudinary.image("steve.jpg")
Java:
Copy to clipboard
cloudinary.url().imageTag("steve.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('steve.jpg').toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("steve.jpg")
React:
Copy to clipboard
<Image publicId="steve.jpg" >

</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="steve.jpg" >

</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="steve.jpg" >

</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.BuildImageTag("steve.jpg")
Android:
Copy to clipboard
MediaManager.get().url().generate("steve.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().generate("steve.jpg")!, cloudinary: cloudinary)
Steve Jobs

The response of the upload API call above returns the detected categories, the detected celebrities, and the automatically assigned tags.

Copy to clipboard
{
   ...
  "tags": ["people","person","human","electronics","iphone","mobile phone","phone","Steve Jobs"]
  "info": {
    "detection": {
      "aws_rek_face": {
        "status": "complete",
        "data": [{
           "name": "Steve Jobs", 
           "match_confidence": 99.99,
            "face": {
              "bounding_box": {
                "width"...
             },
             "landmarks": [{
               "type": "eyeLeft",
                 "x": ...
               }],
            ...
           },
        "unrecognized_faces": [ ],
         ...
     }
     "categorization": {
       "aws_rek_tagging": {
         "status": "complete", 
         "data": [{
           "tag": "People", "confidence": 0.9925},
           "tag": "Person", "confidence": 0.9925},
           "tag": "Human", "confidence": 0.9924},
           "tag": "Electronics", "confidence": 0.7918},
           "tag": "Iphone", "confidence": 0.7918},
           "tag": "Mobile Phone", "confidence": 0.7918},
           "tag": "Phone", "confidence": 0.7918},
           "tag": "Computer", "confidence": 0.6756},
           "tag": "Tablet Computer", "confidence": 0.6756},
           "tag": "Face", "confidence": 0.5396},
           "tag": "Selfie", "confidence": 0.5396},
           "tag": "Cell Phone", "confidence": 0.5067}]}
      ...  
}

Summary

Understanding the images that your users upload can provide you with great insights, allow you to take actions such as creating dynamic content pages, and match content to user preferences. Cloudinary’s service together with the fully integrated Amazon Rekognition Auto Tagging and Amazon Rekognition Celebrity Detection add-ons provides developers with the powerful ability to enhance their image content management as well as increase their online users’ engagement and conversion.

celebrity detection

auto tagging

The add-ons are available with all Cloudinary plans, and both add-ons have a free tier for you to try out. If you don't have a Cloudinary account yet, sign up for a free account here.

(Hugh Laurie photo credit to Fido)

Recent Blog Posts

Cloudinary and Contentful Make Modern Content Management Easier

I am pleased to share that Cloudinary and Contentful have joined forces to further streamline the creation, processing, and delivery of online content through Cloudinary’s digital asset management (DAM) solution and advanced transformation and delivery capabilities for images and video. What’s more, the partnership delivers a headless approach to DAM. By leveraging APIs for media management tasks, marketers and developers alike benefit from an integrated stack of optimized assets for optimization and automation. As a result, page loads are fast and beautiful, and at scale—with less overhead and effort.

Read more
Introducing Cloudinary's Nuxt Module

Since its initial release in October 2016 by the Chopin brothers as a server-side framework that runs on top of Vue.js, Nuxt (aka Nuxt.js) has gained prominence in both intuitiveness and performance. The framework offers numerous built-in features based on a modular architecture, bringing ease and simplicity to web development. Not surprisingly, Nuxt.js has seen remarkable growth in adoption by the developer community along with accolades galore. At this writing, Nuxt has earned over 30K stars on GitHub and 96 active modules with over a million downloads per month. And the upward trend is ongoing.

Read more
How Quality and Quantity can go Hand in Hand

When it comes to quality versus quantity, you’ll often hear people say, “It’s the quality that counts, not the quantity”. While that’s true in many situations, there are also cases where you want both quality and quantity. You may have thousands of images on your website and you want them all to look great. This is especially important if your website allows users to upload their own content, for example, to sell their own products or services. You don't want their poor quality images to reflect badly on your brand.

Read more
Product Videos 101: What Makes Them Great?

A product’s benefits and usage, including its value proposition, features, and instructive details, are best demonstrated through video. Product-video types vary, depending on the funnel, channel, and audience, the most popular ones being demos, reviews, installation, and how-tos.

Read more