Google Automatic Video Tagging

Overview

Cloudinary is a cloud-based service that provides an end-to-end image and video management solution including uploads, storage, manipulations, optimizations and delivery. Cloudinary offers a rich set of manipulation and analysis capabilities and allows you to assign multiple tags to resources for listing and managing your media library.

The Google Automatic Video Tagging add-on integrates Google's automatic video tagging capabilities with Cloudinary's complete video management and manipulation pipeline. Google analyzes video data to automatically identify scenes and suggest tags; a process that would take huge amounts of time and resources if performed manually. Google's technology includes state-of-the-art machine learning approaches that allow it to be trained in the recognition of various visual objects and concepts. When this algorithm is combined with Google's large data sets containing the visual characteristics of videos and their associated tags, the result is outstanding performance in terms of the time needed to process large videos with a high precision rate.

With the Google video tagging add-on, you can extend Cloudinary's powerful semantic data extraction and tagging features, so that your videos can be automatically tagged according to the automatically detected categories in each video.

Video recognition and categorization

Take a look at the following video of a turtle:

Ruby:
cl_video_tag("turtle")
PHP:
cl_video_tag("turtle")
Python:
CloudinaryVideo("turtle").video()
Node.js:
cloudinary.video("turtle")
Java:
cloudinary.url().videoTag("turtle")
JS:
cl.videoTag('turtle').toHtml();
jQuery:
$.cloudinary.video("turtle")
React:
<Video publicId="turtle" >

</Video>
Angular:
<cl-video public-id="turtle" >

</cl-video>
.Net:
cloudinary.Api.UrlVideoUp.BuildVideoTag("turtle")
Android:
MediaManager.get().url().resourceType("video").generate("turtle.mp4")

By setting the categorization parameter to google_video_tagging when calling Cloudinary's upload, explicit, or update method, Google is used to automatically classify the scenes of the uploaded or specified existing video. For example:

Ruby:
Cloudinary::Uploader.upload("turtle.mp4", 
  :resource_type => :video, :categorization => "google_video_tagging")
PHP:
\Cloudinary\Uploader::upload("turtle.mp4", 
  array("resource_type" => "video", "categorization" => "google_video_tagging"));
Python:
cloudinary.uploader.upload("turtle.mp4",
  resource_type = "video", categorization = "google_video_tagging")
Node.js:
cloudinary.uploader.upload("turtle.mp4", 
  function(result) { console.log(result); }, 
  { resource_type: "video", categorization: "google_video_tagging" });
Java:
cloudinary.uploader().upload("turtle.mp4", ObjectUtils.asMap(
  "resource_type", "video", "categorization", "google_video_tagging"));

The video analysis and categorization is performed asynchronously after the method call is completed.

Note: The amount of time it takes for analysis and categorization of the video depends on the size and length of the video file itself. You can include a notification_url parameter in your request to get a notification to the requested URL when the categorization is ready.

The response of the upload method indicates that the process is in pending status.

{ 
  ...
  "info": { "categorization" => 
    { "google_video_tagging": { "status": "pending" }}
  },
  ...
}

Once the categorization process completes, the information is returned to Cloudinary and stored with your video. The details of the analysis and categorization are also sent to the notification_url if this option was included with your method call. For example:

{
...
"info_kind": {
   "google_video_tagging": {
      "info_status": "complete",
      "info_data": [
         [{"tag": "turtle", 
          "start_time_offset": 0.0, 
          "end_time_offset": 13.47364, 
          "confidence": 0.9922},
          {"tag": "turtle", 
          "start_time_offset": -1, 
          "end_time_offset": -1, 
          "confidence": 0.9922},
          {"tag": "animal", 
          "start_time_offset": 0.0, 
          "end_time_offset": 13.47364, 
          "confidence": 0.9715},
           
          ]
...

The information includes the automatic categorization identified by the Google Automatic Video Tagging add-on. As can be seen in the example snippet above, various categories were automatically detected in the uploaded video. Each category is listed together with the start and end times of the relevant video segment (an offset time of -1 means the category represents the entire video). The confidence score is a numerical value that represents the confidence level of the detected category, where 1.0 means 100% confidence.

Adding resource tags to videos

Automatically categorizing your videos is a useful way to organize your Cloudinary media library. By providing the auto_tagging parameter in an upload, explicit, or update call for any video where google_video_tagging was run, the video is automatically assigned resource tags based on the detected scene categories. The value of the auto_tagging parameter is the minimum confidence score of a detected category that should be automatically used as an assigned resource tag. Assigning these resource tags allows you to list and search videos in your media library using Cloudinary's API or Web interface.

The following code example automatically tags an uploaded video with all detected categories that have a confidence score higher than 0.4.
(You can use the same syntax to automatically assign tags to any video already stored in your account by replacing the upload method in the example below with explicit.)

Ruby:
Cloudinary::Uploader.upload("turtle.mp4", 
   :resource_type => :video, :categorization => "google_video_tagging", :auto_tagging => 0.4)
PHP:
\Cloudinary\Uploader::upload("turtle.mp4", 
  array("categorization" => "google_video_tagging", "auto_tagging" => 0.4));
Python:
cloudinary.uploader.upload("turtle.mp4",
  categorization = "google_video_tagging", auto_tagging = 0.4)
Node.js:
cloudinary.uploader.upload("turtle.mp4", 
  function(result) { console.log(result); }, 
  { categorization: "google_video_tagging", auto_tagging: 0.4 });
Java:
cloudinary.uploader().upload("turtle.mp4", ObjectUtils.asMap(
  "categorization", "google_video_tagging", "auto_tagging", "0.4"));

You can also use the update method to apply Google auto tagging to already uploaded videos, based on their public IDs, and then automatically tag them according to the detected categories.

For example, the following video was uploaded to Cloudinary with the 'horses' public ID:

Ruby:
cl_video_tag("horses")
PHP:
cl_video_tag("horses")
Python:
CloudinaryVideo("horses").video()
Node.js:
cloudinary.video("horses")
Java:
cloudinary.url().videoTag("horses")
JS:
cl.videoTag('horses').toHtml();
jQuery:
$.cloudinary.video("horses")
React:
<Video publicId="horses" >

</Video>
Angular:
<cl-video public-id="horses" >

</cl-video>
.Net:
cloudinary.Api.UrlVideoUp.BuildVideoTag("horses")
Android:
MediaManager.get().url().resourceType("video").generate("horses.mp4")

The following code sample uses Cloudinary's update method to apply Google's automatic video tagging and categorization to the sample uploaded video, and then automatically assign resource tags based on the categories detected with over a 60% confidence level.

Ruby:
Cloudinary::Api.update("sample", 
   :resource_type => :video, :categorization => "google_video_tagging", :auto_tagging => 0.6)
PHP:
\Cloudinary\Api::update("sample", 
  array("categorization" => "google_video_tagging", "auto_tagging" => 0.6));
Python:
cloudinary.api.update("sample",
  categorization = "google_video_tagging", auto_tagging = 0.6)
Node.js:
cloudinary.api.update("sample", 
  function(result) { console.log(result); }, 
  { categorization: "google_video_tagging", auto_tagging: 0.6 });
Java:
cloudinary.api().update("sample", ObjectUtils.asMap(
  "categorization", "google_video_tagging", "auto_tagging", 0.6));

Tip: Whether or not you included a notification_url to get a response from the analysis, you can always use the Admin API's resource method to return the details of a resource, including the scene categories that you already extracted using the upload, explicit, or update methods.