Cloudinary Blog
Build a Facial Emotion Recognition Based Video Suggestion App

Developers are always looking for new and creative ways to deliver content that resonates with the way users feel. Often using the latest technical innovations the market has to offer such as Artificial Intelligence (AI) and Machine Learning (ML). What better way to demonstrate innovative uses of these technology in a consumer market than capturing expressions from your users and then serving content based on that expression!

In this article we are going to build an app that suggests videos to users based on their facial expressions (i.e. emotion). To do this, we will use Cloudinary’s Advanced Facial Attributes Detection Add-on and the Cloudinary Video Player.


Sign up for Cloudinary free today!


Set Up Cloudinary

Cloudinary makes adding image and video optimizations to applications a breeze. Head over to the Solutions Page to learn more about the features offered.

First, sign up for Cloudinary. Once you have created an account, you are able to use the “Advanced Facial Attributes Detection” add-on.

You can add the “Advanced Facial Attributes Detection” to your Cloudinary account here and follow the on-screen instructions.

Advanced Facial Attributes Detection Add-on

Make sure to note your API_KEY , API_SECRET and CLOUD_NAME from your developer console. This information is needed when integrating with your application.

Upload Images and Retrieve Facial Emotion Recognition from Images

Using Cloudinary to obtain the emotion from a facial image requires the Advanced Facial Attributes Detection Add-on when uploading the image. We add the adv_face as a tag to let Cloudinary know we are going to use the Advanced Facial Attributes Detection add-on to register the emotions from the detected users face.

When the image is successfully uploaded, emotions detected from the face and their confidence values are returned as part of the result. The emotion with the highest confidence is then selected.

JavaScript Sample:

Copy to clipboard
cloudinary.uploader.upload( req.body.image , function(result) {
  const emotions = result.info.detection.adv_face.data[0].attributes.emotion;
  let arr = JSON.parse(JSON.stringify(emotions));
  let visible_emotion = getMaxKey(arr); // this function gets the emotion with the highest confidence
  return res.json({
    status: true,
    emotion: visible_emotion
  })
},{ detection: "adv_face" });

You can then return the detected emotion to the frontend of your application.

Creating Video Playlists Based on emotion

Once the emotion is obtained on the front end, we want to display a set of videos to the user that correspond to the selected emotion. The Cloudinary Video Player makes creating playlists simple. You can create a video player and then populate it with videos from your media library that are tagged with the highest confidence emotion, such as ‘happiness’.

JavaScript Sample:

Copy to clipboard
let emotion = "happiness";
let cld = cloudinary.Cloudinary.new({ cloud_name: CLOUD_NAME, secure: true});b
// initialize video player
let demoplayer = cld.videoPlayer('elementID');
//  create playlist based on emotion
demoplayer.playlistByTag( emotion, { sourceParams: {angle:0}, autoAdvance: 0, repeat: true, presentUpcoming: 5});

Putting it all together, we have the sample of how it all works below:

Video Suggestion at Work

Conclusion

In this post, we have shown how to use the Advanced Facial Attributes Detection add-on from Cloudinary to create an application that enables us to serve videos to our users based on how they are feeling at the moment. We also leveraged the Cloudinary Video Player to serve the videos in a playlist format. You can learn more about using the video player and feel free to check out the github repository for the full source code. If you are up for the challenge, clone and submit your own Video Suggestion App below!


Further Reading on Video Manipulation

Recent Blog Posts

Top Five Web-Video Formats of 2021

By William Imoh
The Five Most Popular Web-Video Formats and Streaming Protocols

Over the past 15 years, the video industry has undergone a significant change in video formats on the web. In particular, in the early 2010s, the 3GP format, which the 3rd Generation Partnership Project (3GPP) created for 3G-enabled mobile devices, went nearly extinct. The advancement of mobile devices and cellular networks has brought about the need for pioneers to build better formats for a faster user experience.

Read more
Cloudinary Introduces Integration With SAP Commerce Cloud

We’re excited to announce Cloudinary’s integration with SAP Commerce Cloud, through which the latter’s customers can significantly boost the visual media experience on their website or app.

SAP Commerce Cloud powers some of the largest e-commerce sites (B2C, B2B, and B2B2C businesses), complete with building blocks like storefront design and order management. Reinforced with Cloudinary’s laser-sharp focus on optimizing, managing, and delivering images and videos, the new extension will enable SAP Commerce Cloud customers to create unique and engaging visual experiences effortlessly.

Read more
Personalizing Video Email for Marketing Campaigns With Cloudinary

As critical as it is to engage with shoppers in order to succeed in e-commerce, old-style, boring emails are far from being effective. In fact, they tend to be annoying because no one likes to read formulaic, generic messages that are sent en masse. For better results, rethink your email marketing campaigns and try out creative strategies.

Read more
Muted Videos and Subtitles

The bane of our existence is the lack of efficient ways for tackling the plethora of recurring tasks in our lives. One of those tasks is surfing the internet. We consume a lot of web content daily, of which a large percentage are images and videos. We’re constantly quickly scrolling through 30-second videos or checking out pictures of cute items we’d like to buy in our free time.

Read more

Building a Roommate-Matching App With Cloudinary and Jamstack

By Marcelo Ricardo de Oliveira
Building a Roommate-Matching App With Cloudinary and Jamstack

Roommate matching can be a pain—especially during the COVID pandemic when people don't want to meet in person. Matching apps like Flatmates, Roomster, and roommates.com are helpful, and if you're in the roommate-matching space, you know that great video is essential for those seeking roommates. Fortunately, Cloudinary can help.

Read more