Cloudinary Blog
Build a Facial Emotion Recognition Based Video Suggestion App

Developers are always looking for new and creative ways to deliver content that resonates with the way users feel. Often using the latest technical innovations the market has to offer such as Artificial Intelligence (AI) and Machine Learning (ML). What better way to demonstrate innovative uses of these technology in a consumer market than capturing expressions from your users and then serving content based on that expression!

In this article we are going to build an app that suggests videos to users based on their facial expressions (i.e. mood). To do this, we will use Cloudinary’s Advanced Facial Attributes Detection Add-on and the Cloudinary Video Player.

Set Up Cloudinary

Cloudinary makes adding image and video optimizations to applications a breeze. Head over to the Solutions Page to learn more about the features offered.

First, sign up for Cloudinary. Once you have created an account, you are able to use the “Advanced Facial Attributes Detection” add-on.

Note
You can add the “Advanced Facial Attributes Detection” to your Cloudinary account here and follow the on-screen instructions.

Advanced Facial Attributes Detection Add-on

Make sure to note your API_KEY , API_SECRET and CLOUD_NAME from your developer console. This information is needed when integrating with your application.

Upload Images and Retrieve Facial Mood from Images

Using Cloudinary to obtain the emotion from a facial image requires the Advanced Facial Attributes Detection Add-on when uploading the image. We add the adv_face as a tag to let Cloudinary know we are going to use the Advanced Facial Attributes Detection add-on to register the emotions from the detected users face.

When the image is successfully uploaded, emotions detected from the face and their confidence values are returned as part of the result. The emotion with the highest confidence is then selected.

JavaScript Sample:

cloudinary.uploader.upload( req.body.image , function(result) {
  const emotions = result.info.detection.adv_face.data[0].attributes.emotion;
  let arr = JSON.parse(JSON.stringify(emotions));
  let visible_emotion = getMaxKey(arr); // this function gets the emotion with the highest confidence
  return res.json({
    status: true,
    mood: visible_emotion
  })
},{ detection: "adv_face" });

You can then return the detected mood to the frontend of your application.

Creating Video Playlists Based on Mood

Once the mood is obtained on the front end, we want to display a set of videos to the user that correspond to the selected mood. The Cloudinary Video Player makes creating playlists simple. You can create a video player and then populate it with videos from your media library that are tagged with the highest confidence mood, such as ‘happiness’.

JavaScript Sample:

let mood = "happiness";
let cld = cloudinary.Cloudinary.new({ cloud_name: CLOUD_NAME, secure: true});b
// initialize video player
let demoplayer = cld.videoPlayer('elementID');
//  create playlist based on mood
demoplayer.playlistByTag( mood, { sourceParams: {angle:0}, autoAdvance: 0, repeat: true, presentUpcoming: 5});

Putting it all together, we have the sample of how it all works below:

Video Suggestion at Work

Conclusion

In this post, we have shown how to use the Advanced Facial Attributes Detection add-on from Cloudinary to create an application that enables us to serve videos to our users based on how they are feeling at the moment. We also leveraged the Cloudinary Video Player to serve the videos in a playlist format. You can learn more about using the video player and feel free to check out the github repository for the full source code. If you are up for the challenge, clone and submit your own Video Suggestion App below!

Recent Blog Posts

Hipcamp Optimizes Images and Improves Page Load Times With Cloudinary

When creating a website that allows campers to discover great destinations, Hipcamp put a strong emphasis on featuring high-quality images that showcased the list of beautiful locations, regardless of whether users accessed the site on a desktop, tablet, or phone. Since 2015, Hipcamp has relied on Cloudinary’s image management solution to automate cropping and image optimization, enabling instant public delivery of photos, automatic tagging based on content recognition, and faster loading of webpages. In addition, Hipcamp was able to maintain the high standards it holds for the look and feel of its website.

Read more
New Image File Format: FUIF: Why Do We Need a New Image Format

In my last post, I introduced FUIF, a new, free, and universal image format I’ve created. In this post and other follow-up pieces, I will explain the why, what, and how of FUIF.

Even though JPEG is still the most widely-used image file format on the web, it has limitations, especially the subset of the format that has been implemented in browsers and that has, therefore, become the de facto standard. Because JPEG has a relatively verbose header, it cannot be used (at least not as is) for low-quality image placeholders (LQIP), for which you need a budget of a few hundred bytes. JPEG cannot encode alpha channels (transparency); it is restricted to 8 bits per channel; and its entropy coding is no longer state of the art. Also, JPEG is not fully “responsive by design.” There is no easy way to find a file’s truncation offsets and it is limited to a 1:8 downscale (the DC coefficients). If you want to use the same file for an 8K UHD display (7,680 pixels wide) and for a smart watch (320 pixels wide), 1:8 is not enough. And finally, JPEG does not work well with nonphotographic images and cannot do fully lossless compression.

Read more
 New Image File Format: FUIF:Lossy, Lossless, and Free

I've been working to create a new image format, which I'm calling FUIF, or Free Universal Image Format. That’s a rather pretentious name, I know. But I couldn’t call it the Free Lossy Image Format (FLIF) because that acronym is not available any more (see below) and FUIF can do lossless, too, so it wouldn’t be accurate either.

Read more
Optimizing Video Streaming and Delivery: Q&A with Doug Sillars

Doug Sillars, a digital nomad and a freelance mobile-performance expert, answers questions about video streaming and delivery, website optimization, and more.

Doug Sillars, a freelance mobile-performance expert and developer advocate, is a Google Developer Expert and the author of O’Reilly’s High Performance Android Apps. Given his extensive travels across the globe—from the UK to Siberia—with his wife, kids, and 11-year-old dog, Max, he has been referred to as a “digital nomad.” So far in 2018, Doug has spoken at more than 75 meetups and conferences!

Read more