Cloudinary Blog

How to automatically tag and categorize photos according to their content

How to automatically tag and categorize photos by their content

Update - December 2015: The add-on described in this post is no longer available since ReKognition terminated their services. However, all features described here are still available via a different and even better add-on by Imagga. See Automatic image categorization and tagging with Imagga and the Imagga Auto Tagging Add-on documentation.

Drawing insights from user generated content can be very useful. If you allow users to upload images, you might want to better understand what their images contain. Whether a photo is of a landscape, people, animals, or nightlife, image processing and analysis can assist in further comprehension.

ReKognition scene categorization

Cloudinary takes care of the entire image management pipeline, from uploading, via dynamic image manipulation, to fast CDN delivery. You can use Cloudinary with our add-on for automatic scene categorization provided by ReKognition, a visual recognition solution developed by Orbe.us.

ReKognition logo The ReKognition scene categorization add-on analyzes scenes within photos and allows you to automatically classify your website’s images into a long list of potential categories.

Improve user experience, engagement and conversion

By automatically recognizing what images contain, developers can easily create pages or site sections based on images’ content (e.g. photos of animals, travel, special events, etc.). In addition, if your site supports a search option, you can have the search results include relevant images based on their visual content instead of just by their file names or user-assigned tags. These benefits can improve user experience on your site and increase engagement.

When you identify image content, you then know what images your users upload or search for and can target them with related content in order to enhance engagement and conversion. For example, if a user uploaded or searched for cat images, you can target him/her with actions, such as presenting pictures of other pets or any cats’ related content. Knowing the content of images on your site can have a direct impact on your online business.

How-to automatically categorize your images

Web and mobile developers can use Cloudinary's API to process images while uploading or process previously uploaded images.

For example, the image below was uploaded to Cloudinary using the code below. You can see that the image’s content is clearly of a dance party, so let’s see how it is categorized with the ReKognition add-on. Simply set the categorization upload parameter to rekognition_scene in order to perform the image’s content analysis and get the detected categories:

Cloudinary::Uploader.upload("my_dance_photo.jpg",
          :categorization => 'rekognition_scene', 
          :public_id => "dance_party")

Ruby:
cl_image_tag("dance_party.jpg")
PHP:
cl_image_tag("dance_party.jpg")
Python:
CloudinaryImage("dance_party.jpg").image()
Node.js:
cloudinary.image("dance_party.jpg")
Java:
cloudinary.url().imageTag("dance_party.jpg");
JS:
cloudinary.imageTag('dance_party.jpg').toHtml();
jQuery:
$.cloudinary.image("dance_party.jpg")
React:
<Image publicId="dance_party.jpg" >

</Image>
Angular:
<cl-image public-id="dance_party.jpg" >

</cl-image>
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("dance_party.jpg")
Android:
MediaManager.get().url().generate("dance_party.jpg");
iOS:
imageView.cldSetImage(cloudinary.createUrl().generate("dance_party.jpg")!, cloudinary: cloudinary)
Sample dance party uploaded photo

Cloudinary’s upload API's result shows the uploaded image's detected categories as well as their level of accuracy, provided as a score between 0 and 1, where 1 is 100% accurate. As you can see in the example below, the ReKognition add-on detected several categories for the image above, that were each given a score. The best category match was 'dance', with a 0.719 accuracy score.  

    {"public_id"=>"dance_party",
     "url"=>
      "https://res.cloudinary.com/demo/image/upload/v1417446163/dance_party.jpg",
     ...
     "info"=>
      {"categorization"=>
        {"rekognition_scene"=>
          {"status"=>"complete",
           "data"=>
            {"matches"=>
              [{"tag"=>"dance", "score"=>0.719},
               {"tag"=>"blonde", "score"=>0.3055},
               {"tag"=>"model", "score"=>0.24},
               {"tag"=>"night_club", "score"=>0.1859},
               {"tag"=>"hair", "score"=>0.1449}]}}}},
    }

Image categorization data can be stored on your side in order to organize your content and target user interest. You can then take actions such as linking the categorization data to users that upload categorically related images. Once the information is captured, a data analysis can take place generating insights, including users’ image preferences. Based on the information and analysis provided, you will be able to take proactive actions to enhance user engagement and conversion.

Automatic tagging with ReKognition scene categorization

Cloudinary allows you to tag uploaded images. Each image can be assigned one or more tags. This feature automatically assigns tags based on categories that are detected by the ReKognition add-on.

You can see below that we set the auto_tagging parameter to 0.15 for the dance party example image above, which means that auto tagging will use the detected categories above this threshold. You can set the threshold to a higher level if you want to be more strict regarding the detected categories you want to tag. As for our example, the tagging results were 'blonde', 'dance', 'model', and 'nightclub':

Cloudinary::Uploader.upload("my_dance_photo.jpg", 
      :categorization => 'rekognition_scene', 
      :auto_tagging => 0.15, 
      :public_id => "dance_party")

Results in:

    "tags" => ["blonde", "dance", "model", "night_club"]

With automatic tagging using ReKognition’s scene categorization add-on, you will be able to use Cloudinary's admin API in order to list and search for all images with specific tags.

Summary

Understanding user generated content can provide you with great insights, allow you to take actions such as creating dynamic content pages, and match content to user preferences. Cloudinary’s service together with the fully integrated ReKognition scene categorization add-on provides developers with the powerful ability to enhance their image content management as well as increase their online users’ engagement and conversion.

ReKognition add-on

The ReKognition scene categorization add-on is available with all Cloudinary plans, You can try it out with the add-on’s free tier. If you don't have a Cloudinary account yet, sign up for a free account here.

Update - December 2015: The add-on described in this post is no longer available since ReKognition terminated their services. However, all features described here are still available via a different and even better add-on by Imagga. See Automatic image categorization and tagging with Imagga and the Imagga Auto Tagging Add-on documentation.

Recent Blog Posts

CoreMedia Adds Cloudinary to its CoreMedia Studio Platform

Today we’re pleased to announce a new technology partnership with CoreMedia, a leading Content Experience Platform provider. CoreMedia users can now leverage Cloudinary’s web-based digital asset management (DAM) solution to organize, search, manage and optimize their media assets, including images and videos, and to orchestrate, preview and deliver digital experiences consistently and optimized across all channels and browsers. The official press release is available here.

Read more
Facial-Surveillance System for Restricted Zones

In Africa, where Internet access and bandwidth are limited, it’s not cost-effective or feasible to establish and maintain a connectivity for security and surveillance applications. That challenge makes it almost impossible to build a service that detects, with facial-recognition technology, if someone entering a building is authorized to do so. To meet the final-year research requirement for my undergraduate studies, I developed a facial-surveillance system. Armed with a background in computer vision, I decided to push the limits and see if I could build a surveillance system that does not require recording long video footage.

Read more
Complex Networks Case Study

Complex Networks has been using Cloudinary since 2014 to manage and optimize images across seven websites and two mobile apps, making editorial workflow more efficient, improving page performance and load time, and increasing user engagement. Cloudinary was instrumental in enabling Complex Networks to redesign its web properties. Without the flexibility that Cloudinary offers to both creative and development teams, it would not have been possible for Complex Networks to achieve such a fast time to market.

Read more
Automate Placeholder Generation and Accelerate Page Loads

If you run a Google search on LQIP you’ll see very few relevant articles, very little guidance, and definitely no Wikipedia articles. In this post, we’ll discuss some of the feedback on LQIP we have gathered from the community and suggest and open for conversation a few approaches based on the built-in capabilities of the Cloudinary service. Specifically, we’ll explain what LQIP are, where they are best used, and how you can leverage them to accelerate page loads and optimize user experience.

Read more
Best Practices for Optimizing Web Page Speed

If you're like most consumers today, you engage more with pictures or videos on a website than text. The stats don't lie - four times as many visitors would rather watch a video about a product than read about it, and sites with compelling images average twice as many views as text-heavy ones.

Read more
A day of fun with Girls Who Code and Cloudinary

During both my computer science studies and work in the tech field, there have not been a lot of women present. While our ranks have grown, women still make up only a small percentage. In many ways, I think the traditionally male-dominated world can be intimidating to women and girls who may be interested in pursuing these types of tech careers.

Read more