Cloudinary Blog

A Deep Dive Into Video Auto-Tagging

Auto-Tag Videos With Cloudinary: A Tutorial

Video is one of the best ways for capturing audience attention. Product demonstrations, property tours, tutorials, and keynotes all make great content—provided that they’re readily searchable by your team and visitors alike. Even though tagging and categorization help organize videos and make them easy to find, the labeling process is labor intensive, taking time away from key tasks like creating new videos.

The solution? Automation. Through machine learning (ML) algorithms, Cloudinary’s video-tagging feature automatically detects video content. Video tagging extends image classification to image sequences, capable of detecting types of scenes (cities, mountains, the seaside) and objects (people, cars), and even inferring whether the content is appropriate for public viewing. Additionally, with video auto-tagging supplemented by detailed labels, you can moderate videos and automatically determine the minimum age for viewers.

Video auto-tagging benefits a wide range of apps. For example:

  • For a basketball game, auto-tagging can identify which player has the ball and for how long. The team can then make use of that information in optimizing its game strategy.

  • For personal protective equipment, such as face masks or hand covers, auto-tagging can help you pinpoint and recognize human emotions, automatically detecting if the people in a video are happy or sad, for example.

Those capabilities might be highly challenging to implement, however. Gratifyingly, many major IT companies have recognized the need, offering various cloud services with AI-powered image and video analysis. Nonetheless, because each provider has its own programming interfaces or SDKs, you must use several completely different toolsets, complicating the workflow.

Cloudinary’s convenient API simplifies video tagging through automation. You can decide which underlying tagging engine to use through the API and combine various services to further improve the tagging performance. This tutorial cites several use cases to which video tagging adds value, and steps you through an example that explains how to use Cloudinary’s API with the JavaScript client library. To follow this tutorial, you just need basic familiarity with HTML and JavaScript.

Organization of Media Library

Large companies often store hundreds or even thousands of videos in their website’s media library, which is a pain to organize. Organizing by file name or upload date is straightforward but unhelpful. Organizing through tagging, i.e., grouping videos by category, is much more effective. Instead of sorting videos into a single folder structure based on one basic property, you organize them by tag category, sorting footage by attribute, such as the featured product, location type, and color scheme.

The process of manually adding descriptions and tagging is arduous and time consuming, however. Auto-tagging categorizes videos automatically, making it easy to quickly find what you and your app users are looking for. The technology, which relies on artificial intelligence (AI) trained on large datasets, automatically recognizes and labels objects, such as cars and animals; attributes, such as color; and sentiments, such as adventure. Cloudinary, for example, combines AI from Google and other sources to automatically label the objects in your videos as you upload them, making tagging as simple and efficient as possible.

To work with the Cloudinary API, start by uploading your media assets. (Note that even though that post refers to examples on image upload, they apply to videos, too.) During the upload, you can specify the public identifier (ID) of your assets and divide them into folders to create structured delivery URLs. Alternatively, you can group the assets in folders and subfolders, organizing the media library according to the same structure as your on-premise server.

While uploading media to Cloudinary, you can also apply transformations, e.g., on-the-fly video encoding and delivery, transcoding, resizing, and cropping, which can convert videos to animated images and even add blurred backgrounds.

User-Uploaded Content

For all that user-uploaded content enhances your website, you cannot assume that your users are as knowledgeable and skilled as your internal team or that they would mount the level of effort for tagging and descriptions. For instance, if you’re running a real-estate website, users would upload videos of their property but might not accurately tag all the elements in the video. Instead, you can rely on auto-tagging to find objects like "pool" and "palm tree," which certain house shoppers might be looking for. Those tags greatly improve the listing’s discoverability and searchability.

Another example: consider an animal shelter that posts videos of animals that are up for adoption. Instead of having to manually type in a comprehensive set of tags for each animal, the staff needs to enter only the basic information, based on which the auto-tagging capability can figure out the animal type (like cat, dog, hamster, or horse), color, and other attributes.

Any web app into which users can upload their media can benefit from the Cloudinary API, saving you the time and effort of building the capabilities yourself: upload, transformation, storage, and so on. Another alternative is to adopt serverless approaches like AWS Lambda or Azure Functions, which is easier than manual tagging. However, processing large videos with those services is either impossible (Lambda has an execution-time limit) or expensive (you pay for the execution time—the long tasks incur significantly higher costs). Ultimately, Cloudinary is the best answer for processing and tagging videos.

Auto-Tagging in Action

The demo below, which makes use of a Pexels video of a giraffe in the wild, shows you how to use Cloudinary’s video-tagging feature with the JavaScript SDK.

First, upload the video with the upload method or the Cloudinary upload widget on the Cloudinary portal. Here, you use the upload widget to access the advanced video uploader. Integrate the upload widget with your site by referencing the JavaScript. Afterwards, you can use the various widget initialization methods.

The upload widget supports signed and unsigned uploads. For this demo, use unsigned uploads, which require the upload preset. See the related documentation on how to create an unsigned preset in the Cloudinary management console.

Upload preset

When configuring the upload preset, you can also enable video auto-tagging on upload, setting your preference for how “confident” you want the algorithm to be before applying a tag.To do so, click the Media analysis and AI tab of your preset, choose a categorization engine (for example, Google Automatic Video Tagging), and set the auto-tagging threshold (0.5), which denotes 50% confidence, as shown below:

Edit Preset

Next, proceed to implementation. See the complete source code on GitHub. Follow these steps:

1. Create a simple HTML page that references the Cloudinary widget scripts and that contains a button serving as the upload widget’s opener. For example:

Copy to clipboard
<html>
<head>
    <title>Video auto-tagging</title>

    …
    <script src="https://widget.cloudinary.com/v2.0/global/all.js" type="text/javascript"></script>        
</head>

<body class="w3-black">
    <div class="w3-container">
       <h1>Video auto-tagging</h1>
       <hr />

       <button id="upload_widget_opener" class="w3-button w3-orange">
           Upload video file...
       </button>        

       <a id="info_link" href="" target="_blank">Click to get info</p>       
    </div>
</body>
</html>

Now add custom JavaScript to handle the logic. Here, you create the upload widget and then store the reference to it under the uploadWidget variable:

Copy to clipboard
var uploadWidget = cloudinary.createUploadWidget({
    cloudName: '<YOUR_CLOUD_NAME>',
    uploadPreset: 'autotagging-preset',
    sources: [
        'local'
    ],
    cropping: false,
    multiple: false,
    styles: {
    palette: {
        window: "#10173a",
            sourceBg: "#20304b",
            windowBorder: "#7171D0",
            tabIcon: "#79F7FF",
            inactiveTabIcon: "#8E9FBF",
            menuIcons: "#CCE8FF",
            link: "#72F1FF",
            action: "#5333FF",
            inProgress: "#00ffcc",
            complete: "#33ff00",
            error: "#cc3333",
            textDark: "#000000",
            textLight: "#ffffff"
        }
    }
}, (error, result) => uploadCallback(error, result));

2. Generate the above code on Cloudinary’s Upload Widget Demo page to visually configure the widget and get the corresponding configuration code.

Show the widget by calling the open method, like this:

Copy to clipboard
$('#upload_widget_opener').click(() => { 
    uploadWidget.open();        
});

3. Click the Upload video file button. The app then displays the upload widget:

upload widget

4. Implement the uploadCallback function, which is invoked by the upload widget for specific events, for example, when the asset is uploaded.

Copy to clipboard
function uploadCallback(error, result) {         
    if (!error && result && result.event === "success") {   

        // Cloudinary API info
        const apiKey = '<YOUR_API_KEY>';
        const apiSecret = '<YOUR_SECRET>';
        const baseUrl ='api.cloudinary.com/v1_1/<YOUR_CLOUD_NAME>/resources/video/upload/';

        const url = 'https://' + apiKey +':' + apiSecret 
            + '@' + baseUrl + result.info.public_id;

        $infoLink.attr("href", url);
        $infoLink.show();
    }    
}

Here, the callback creates the URL to upload resources. The URL is supplemented by basic authorization (Cloudinary’s API and secret) and is displayed as an HTML tag next to the Upload video file button (see the screenshot above).

5. Click the generated URL for details on the resource along with the video tags in JSON format, for example:

JSON

As shown, Cloudinary successfully tagged the giraffe video with the appropriate keywords, from the type of animal to the type of environment, and to helpful words like the activity “safari” and the sentiment “adventure.” With those useful tags, giraffe fans, students who are studying grasslands, and adventure-seeking travelers can all readily find and enjoy the video.

Note
Initially, the categorization section might be pending since the tagging might take some time.

Next Steps

Tools like Cloudinary save you innumerable hours of manual labor. Not only does auto-tagging enable you and your team to stay organized, it also helps your audience find videos through which they can become acquainted with your goods and services on a deeper level than with images and text alone.

Now that you know how easy it is to use Cloudinary’s upload widget to automatically tag videos, do incorporate this feature into your own app. Uploading your videos and encouraging your users to share theirs helps the audience find what they need: from adventure seekers in search of wildlife to homeowners looking for decorative trim and picture windows, and to pet lovers finding their lifelong companion.

Do give auto-tagging a try and discover how Cloudinary can enhance your app’s videos. Start by signing up for a free account along with free credits.

Recent Blog Posts

Automate the Staging Process of Videos for Social Media

Rich and engaging media helps build customer engagement and trust but can be time consuming to stage. Developers save a tremendous amount of time by preparing videos for social media with Cloudinary. That’s because Cloudinary’s interface, widgets, and application programming interface (API) transform raw media into polished content, optimizing footage and enabling effortless customization and publishing.

Read more

Top Five Web-Video Formats of 2021

By William Imoh
The Five Most Popular Web-Video Formats and Streaming Protocols

Over the past 15 years, the video industry has undergone a significant change in video formats on the web. In particular, in the early 2010s, the 3GP format, which the 3rd Generation Partnership Project (3GPP) created for 3G-enabled mobile devices, went nearly extinct. The advancement of mobile devices and cellular networks has brought about the need for pioneers to build better formats for a faster user experience.

Read more
Cloudinary Introduces Integration With SAP Commerce Cloud

We’re excited to announce Cloudinary’s integration with SAP Commerce Cloud, through which the latter’s customers can significantly boost the visual media experience on their website or app.

SAP Commerce Cloud powers some of the largest e-commerce sites (B2C, B2B, and B2B2C businesses), complete with building blocks like storefront design and order management. Reinforced with Cloudinary’s laser-sharp focus on optimizing, managing, and delivering images and videos, the new extension will enable SAP Commerce Cloud customers to create unique and engaging visual experiences effortlessly.

Read more
Personalizing Video Email for Marketing Campaigns With Cloudinary

As critical as it is to engage with shoppers in order to succeed in e-commerce, old-style, boring emails are far from being effective. In fact, they tend to be annoying because no one likes to read formulaic, generic messages that are sent en masse. For better results, rethink your email marketing campaigns and try out creative strategies.

Read more
Muted Videos and Subtitles

The bane of our existence is the lack of efficient ways for tackling the plethora of recurring tasks in our lives. One of those tasks is surfing the internet. We consume a lot of web content daily, of which a large percentage are images and videos. We’re constantly quickly scrolling through 30-second videos or checking out pictures of cute items we’d like to buy in our free time.

Read more

Building a Roommate-Matching App With Cloudinary and Jamstack

By Marcelo Ricardo de Oliveira
Building a Roommate-Matching App With Cloudinary and Jamstack

Roommate matching can be a pain—especially during the COVID pandemic when people don't want to meet in person. Matching apps like Flatmates, Roomster, and roommates.com are helpful, and if you're in the roommate-matching space, you know that great video is essential for those seeking roommates. Fortunately, Cloudinary can help.

Read more