Amazon Rekognition AI Moderation

Overview

Cloudinary is a cloud-based service that provides an end-to-end image management solution including uploads, storage, manipulations, optimizations and delivery. Cloudinary offers a very rich set of image manipulation and analysis capabilities and allows you to upload images to the cloud, manipulate them on-the-fly and deliver them to your users optimized and cached via a fast CDN.

Amazon Rekognition is a service that makes it easy to add image analysis to your applications. Cloudinary provides an add-on for Amazon Rekognition's image moderation service based on Deep Learning algorithms, fully integrated into Cloudinary's image management and manipulation pipeline.

With Amazon Rekognition's AI Moderation add-on, you can extend Cloudinary's powerful cloud-based image media library and delivery capabilities with automatic artificial intelligence based moderation of your photos. Protect your users from explicit and suggestive adult content in your user uploaded images, making sure that no offensive photos are displayed to your web and mobile viewers.

Automatic image moderation flow

The following list describes the flow of uploading and displaying moderated images using Cloudinary and the Amazon Rekognition's AI Moderation add-on:

  • Your users upload an image to Cloudinary through your application.
  • The uploaded image is sent to Amazon Rekognition for moderation.
  • The image is either approved or rejected by Amazon Rekognition.
  • An optional notification callback is sent to your application with the image moderation result.
  • A rejected image is moved to a secondary backup repository, consuming additional storage.
  • Moderated images can be listed programmatically using Cloudinary's Admin API or interactively using our online Media Library Web interface.
  • You can manually override the automatic moderation using the Admin API or the Media Library.

Moderation categorization

Amazon Rekognition assigns a moderation confidence score (0 - 100) indicating the chances that an image belongs to an offensive content category. The categories are:

Top-Level Category Child Category
Explicit Nudity (explicit_nudity) Nudity (nudity)
Graphic Male Nudity (graphic_male_nudity)
Graphic Female Nudity (graphic_female_nudity)
Sexual Activity (sexual_activity)
Partial Nudity (partial_nudity)
Suggestive (suggestive) Female Swimwear or Underwear (female_underwear)
Male Swimwear or Underwear (male_underwear)
Revealing Clothes (revealing_clothes)


The default moderation confidence level to reject an image is 0.5 for all categories, unless specifically overridden (see explanation and examples below). All images classified by Amazon Rekognition with a value greater than the moderation confidence level (in any of the categories) are classified as "rejected". Otherwise, their status is set to “approved”.

Request image moderation

In order to request moderation while uploading an image, with default moderation confidence levels, simply set the moderation upload API parameter to aws_rek:

Ruby:
Cloudinary::Uploader.upload("local_file.jpg", 
  :moderation => "aws_rek")
PHP:
\Cloudinary\Uploader::upload("local_file.jpg", 
  array("moderation" => "aws_rek"));
Python:
cloudinary.uploader.upload("local_file.jpg",
  moderation = "aws_rek")
Node.js:
cloudinary.v2.uploader.upload("local_file.jpg", 
  {moderation: "aws_rek"},
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("local_file.jpg", 
  ObjectUtils.asMap("moderation", "aws_rek"));

You can also override the default moderation confidence level of 0.5 on a per category basis by including the category name and new value as part of the 'moderation' parameter value, separated by colons. You can override multiple categories by adding them in series together with their new value, all separated by colons.

Notes:

  • Overriding the default moderation confidence value of a top-level category will also set all its child categories to the same value.
  • To exclude a category from the moderation check, set the category's value to ignore.

For example, to request moderation with the moderation confidence level set to 0.85 for the 'Female Swimwear or Underwear' child category, 0.7 for the 'Explicit Nudity' top-level category (overriding the default for all its child categories as well) and exclude the 'Revealing Clothes' category:

Ruby:
Cloudinary::Uploader.upload("local_file.jpg", 
  :moderation => "aws_rek:female_underwear:0.85:explicit_nudity:0.7:revealing_clothes:ignore")
PHP:
\Cloudinary\Uploader::upload("local_file.jpg", 
  array("moderation" => "aws_rek:female_underwear:0.85:explicit_nudity:0.7:revealing_clothes:ignore"));
Python:
cloudinary.uploader.upload("local_file.jpg",
  moderation = "aws_rek:female_underwear:0.85:explicit_nudity:0.7:revealing_clothes:ignore")
Node.js:
cloudinary.v2.uploader.upload("local_file.jpg", 
  {moderation: "aws_rek:female_underwear:0.85:explicit_nudity:0.7:revealing_clothes:ignore"},
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("local_file.jpg", 
  ObjectUtils.asMap("moderation", "aws_rek:female_underwear:0.85:explicit_nudity:0.7:revealing_clothes:ignore"));

Moderation response

The following snippet shows an example of a response to an upload API call, indicating the results of the request for moderation, and shows that the moderation result has put the image in 'rejected' status.

{
...
 "moderation": 
  [{"response": 
     {"moderation_labels": 
       [{"confidence": 94.9907455444336,
         "name": "Suggestive",
         "parent_name": ""},
        {"confidence": 94.9907455444336,
         "name": "Female Swimwear Or Underwear",
         "parent_name": "Suggestive"}]},
    "status": "rejected",
    "kind": "aws_rek",
    "updated_at": "2017-08-03T08:26:58Z"}],
 ...
}

Image moderation listing

Cloudinary's Admin API can be used to list all moderated images. You can list either the approved or the rejected images by specifying the second parameter of the resources_by_moderation API method. For example to list all rejected images:

Ruby:
Cloudinary::Api.resources_by_moderation("aws_rek", "rejected")
PHP:
$api = new \Cloudinary\Api();
      $api->resources_by_moderation("aws_rek", "rejected");
Python:
cloudinary.api.resources_by_moderation("aws_rek", "rejected")
Node.js:
cloudinary.api.resources_by_moderation(aws_rek, 'rejected', 
        function(result)  { console.log(result) });
Java:
cloudinary.api().resourcesByModeration("aws_rek", "rejected", 
        ObjectUtils.emptyMap());
{
 "resources": 
  [{
    "public_id": "q7vcvrfjm9mj4bfp3qc8",
    "format": "jpg",
    "version": 1393794403,
    "resource_type": "image",
    "type": "upload",
    "created_at": "2017-05-04T41:02:23Z",
    "bytes": 120253,
    "width": 864,
    "height": 576,
    "backup": true,
    "url": "http://res.cloudinary.com/demo/image/upload/v1393794403/q7vcvrfjm9mj4bfp3qc8.jpg",
    "secure_url": "https://res.cloudinary.com/demo/image/upload/v1393794403/q7vcvrfjm9mj4bfp3qc8.jpg"
  },
  {
    "public_id": "zp4fgdbabhlwwa7bxu84",
    "format": "jpg",
    ...
  }
  ...
 ]
}

Manual override

As the automatic image moderation of the Amazon Rekognition AI Moderation add-on is based on a decision made by an advanced algorithm, in some cases you may want to manually override the moderation decision and either approve a previously rejected image or reject an approved one.

Overriding moderation via the Media Library

One way to manually override the moderation result is using Cloudinary's Media Library Web interface. As you can see in the following screenshot, you can click on the Moderation Queue link, select Amazon Rekognition and then the specific queue of images to list (approved or rejected). In the following example, we list the queue of images rejected by Amazon Rekognition. You can then click on the green Approve button to revert the decision and recover the original rejected image.

rejected

As the following screenshot illustrates, the same goes for the list of approved images. You can click on the red Reject button to revert the decision and prevent a certain image from being publicly available to your users.

approved

Overriding moderation via the Admin API

Alternatively to using the Media Library interface, you can use Cloudinary's Admin API to manually override the moderation result. The following example uses the update API method while specifying a public ID of a moderated image and setting the moderation_status parameter to the rejected status.

Ruby:
Cloudinary::Api.update("hwepb67oxzh4lrigssld", 
        :moderation_status => "rejected")
PHP:
$api = new \Cloudinary\Api();
      $api->update("hwepb67oxzh4lrigssld", 
        array("moderation_status" => "rejected"));
Python:
cloudinary.api.update("hwepb67oxzh4lrigssld",
        moderation_status = "rejected")
Node.js:
cloudinary.api.update("hwepb67oxzh4lrigssld", 
        function(result) { console.log(result); }, 
        { moderation_status: "rejected" });
Java:
cloudinary.api().update("hwepb67oxzh4lrigssld", 
        ObjectUtils.asMap("moderation_status", "rejected"));