Automatic image tagging

The automatic image tagging behavior of the Cloudinary AI Content Analysis add-on can be invoked on uploading an image, or by updating an image that's already stored in your product environment. Using the specified model, it analyzes the image, identifies categories and objects, and suggests tags that could be applied to the image.

Object and category detection

Take a look at the following photo of a woman dressed fashionably for winter:

By setting the detection parameter to the name of the model (and optionally the version, e.g. cld-fashion_v3) you want to invoke when calling Cloudinary's upload or update methods, the add-on automatically analyzes the content of the uploaded or specified existing image. For example, invoking the cld-fashion detection model while uploading winter_fashion.jpg:

The upload API response includes the categories and objects automatically identified by the model you requested. As can be seen in the response snippet below, a hat and a specific type of outerwear are automatically detected in the uploaded photo. Depending on the capabilities of each model, different information is returned. In the example below, a confidence score, bounding box and in some cases, attributes, are returned for each detected object. The confidence score is a numerical value representing the certainty of a correct detection, where 1.0 means 100% confidence. The bounding-box parameter shows the location of the object in the image, as an array: [x-coordinate of top left corner, y-coordinate of top left corner, width of box, height of box]. Bounding-box information is used in the object detection demo.

Adding tags to images

By providing the auto_tagging parameter to an upload or update request, images are automatically assigned tags based on the detected content. The value of the auto_tagging parameter is the minimum confidence score of a detected category or object that should be automatically used as an assigned tag. You can also set auto_tagging to default, which uses the model's default confidence.

The following code example automatically tags an uploaded image with all detected categories that have a confidence score higher than 0.6.

The response to the upload request returns the detected categories as well as the assigned tags for categories meeting the minimum confidence score of 0.6:

You can also use the update method to apply auto-tagging to images already stored in your product environment.

The following example uses Cloudinary's update method on the puppy image in the product environment, to detect objects and categories in the LVIS model. Tags are automatically assigned based on the objects and categories detected with over a 90% confidence level.

You can use the Admin API's resource_by_tag method to return all resources with a certain tag, for example hat:

You can also use the search method or the Media Library advanced search to find images with certain tags.

Asynchronous handling

As automatic image tagging may not be immediate, it is good practice to use asynchronous handling for these calls.

To make the call asynchronous, set the async parameter of the upload method to true. To be notified when the processing is complete, you can either set the notification_url parameter of the upload method (as in the example below) or the global webhook Notification URL in the Upload page of your Cloudinary Console Settings.

The response to an asynchronous upload call looks similar to this:

When the processing is finished, the complete upload response is sent to the notification URL that you specified.