Cloudinary Blog
Build a Facial Emotion Recognition Based Video Suggestion App

Developers are always looking for new and creative ways to deliver content that resonates with the way users feel. Often using the latest technical innovations the market has to offer such as Artificial Intelligence (AI) and Machine Learning (ML). What better way to demonstrate innovative uses of these technology in a consumer market than capturing expressions from your users and then serving content based on that expression!

Facial Emotion Recognition

In this article we are going to build an app that suggests videos to users based on their facial expressions (i.e. emotion). To do this, we will use Cloudinary’s Advanced Facial Attributes Detection Add-on and the Cloudinary Video Player.


Sign up for Cloudinary free today!


Set Up Cloudinary

Cloudinary makes adding image and video optimizations to applications a breeze. Head over to the Solutions Page to learn more about the features offered.

First, sign up for Cloudinary. Once you have created an account, you are able to use the “Advanced Facial Attributes Detection” add-on.

You can add the “Advanced Facial Attributes Detection” to your Cloudinary account here and follow the on-screen instructions.

Advanced Facial Attributes Detection Add-on

Make sure to note your API_KEY , API_SECRET and CLOUD_NAME from your developer console. This information is needed when integrating with your application.

Upload Images and Retrieve Facial Emotion Recognition from Images

Using Cloudinary to obtain the emotion from a facial image requires the Advanced Facial Attributes Detection Add-on when uploading the image. We add the adv_face as a tag to let Cloudinary know we are going to use the Advanced Facial Attributes Detection add-on to register the emotions from the detected users face.

When the image is successfully uploaded, emotions detected from the face and their confidence values are returned as part of the result. The emotion with the highest confidence is then selected.

JavaScript Sample:

Copy to clipboard
cloudinary.uploader.upload( req.body.image , function(result) {
  const emotions = result.info.detection.adv_face.data[0].attributes.emotion;
  let arr = JSON.parse(JSON.stringify(emotions));
  let visible_emotion = getMaxKey(arr); // this function gets the emotion with the highest confidence
  return res.json({
    status: true,
    emotion: visible_emotion
  })
},{ detection: "adv_face" });

You can then return the detected emotion to the frontend of your application.

Creating Video Playlists Based on emotion

Once the emotion is obtained on the front end, we want to display a set of videos to the user that correspond to the selected emotion. The Cloudinary Video Player makes creating playlists simple. You can create a video player and then populate it with videos from your media library that are tagged with the highest confidence emotion, such as ‘happiness’.

JavaScript Sample:

Copy to clipboard
let emotion = "happiness";
let cld = cloudinary.Cloudinary.new({ cloud_name: CLOUD_NAME, secure: true});b
// initialize video player
let demoplayer = cld.videoPlayer('elementID');
//  create playlist based on emotion
demoplayer.playlistByTag( emotion, { sourceParams: {angle:0}, autoAdvance: 0, repeat: true, presentUpcoming: 5});

Putting it all together, we have the sample of how it all works below:

Video Suggestion at Work

Conclusion

In this post, we have shown how to use the facial emotion recognition add-on from Cloudinary to create an application that enables us to serve videos to our users based on how they are feeling at the moment. We also leveraged the Cloudinary Video Player to serve the videos in a playlist format. You can learn more about using the video player and feel free to check out the github repository for the full source code. If you are up for the challenge, clone and submit your own Video Suggestion App below!


Further Reading on Video Manipulation

Recent Blog Posts

Create Lightweight Sites With Low-Code and No-Code Technology

Consumers expect modern websites to be mainly visual. But, the more compelling and complex the related media is, the more data is involved, compounding the site’s weight. In today’s content-craving world, delivering unoptimized media can cost you because it leads to sluggish page loads, resulting in visitors abandoning your site in search of a faster alternative. In fact, a page load that takes more than three seconds can cause as many as 40% of your visitors to bounce. Given this competitive, digital-first environment, you can’t afford to lose page views, for time is of the essence.

Read more
A Blueprint for AWS-Secured Webhook Listeners for Cloudinary

tl;dr: An AWS-secured and optimized Cloudinary webhook listener for extending the Cloudinary service

Code: Github

A webhook is a communication medium for sending notifications from one platform to another about events that occurred. In place are user-defined HTTP callbacks that are triggered by specific events. When a triggered event takes place on the source site, the webhook listens to the event, collects the data, and sends it to the URL you specified in the form of an HTTP request.

Read more
New Accessibility Features for Cloudinary’s Product Gallery Widget

Cloudinary’s Product Gallery widget, which launched in 2019, has enabled many brands to effectively and efficiently showcase their products in a sleek and captivating manner, saving countless hours of development time and accelerating release cycles. By adding Cloudinary’s Product Gallery widget with its customizable UI to their product page, retailers reap numerous benefits, often turning visitors into customers in short order.

Read more
Why Successful Businesses Engage With and Convert Audiences With Visual Media

Most business buyers prefer to research purchase options online, as do many shoppers. No wonder online retail sales in the U.S. rose by 32.4% in 2020—an impressive gain of $105 billion.

For B2B and B2C businesses, text-heavy websites are no longer adequate in attracting shoppers. Instead, engaging visual media—spin images, videos, 3D models, augmented reality—are becoming a must for conveying eye-catching details and differentiators about products or services.

Read more
Making User-Generated Content (UGC) Shoppable With Cloudinary

User-generated content (UGC) is a powerful marketing tool. Not only does video complement marketing efforts for e-commerce by enabling customers to explore products in greater detail, but UGC also adds an element of trust. As a bonus, user-generated video is an exceptional opportunity for e-businesses to attract website traffic without their marketing team having to create promotional videos from scratch. User-generated content drives conversions and brand loyalty as a direct result of authentic interaction.

Read more