{"id":21742,"date":"2018-06-04T16:52:46","date_gmt":"2018-06-04T16:52:46","guid":{"rendered":"http:\/\/facial_emotion_recognition_based_video_suggestion_app"},"modified":"2024-06-05T16:01:51","modified_gmt":"2024-06-05T23:01:51","slug":"facial_emotion_recognition_based_video_suggestion_app","status":"publish","type":"post","link":"https:\/\/cloudinary.com\/blog\/facial_emotion_recognition_based_video_suggestion_app","title":{"rendered":"Facial Emotion Recognition Based Video Suggestion App"},"content":{"rendered":"

Developers are always looking for new and creative ways to deliver content that resonates with the way users feel. Often using the latest technical innovations the market has to offer such as Artificial Intelligence (AI) and Machine Learning (ML). What better way to demonstrate innovative uses of these technology in a consumer market than capturing expressions from your users and then serving content based on that expression!<\/p>\n

Facial Emotion Recognition<\/h2>\n

In this article we are going to build an app that suggests videos<\/a> to users based on their facial expressions (i.e. emotion). To do this, we will use Cloudinary\u2019s Advanced Facial Attributes Detection Add-on and the Cloudinary Video Player.<\/p>\n

Set Up Cloudinary<\/h2>\n

Cloudinary<\/a> makes adding image and video optimizations to applications a breeze. Head over to the Solutions Page<\/a> to learn more about the features offered.<\/p>\n

First, sign up for Cloudinary. Once you have created an account, you are able to use the \u201cAdvanced Facial Attributes Detection\u201d add-on.<\/p>\n

You can add the \u201cAdvanced Facial Attributes Detection\u201d to your Cloudinary account here<\/a> and follow the on-screen instructions.<\/p>\n

\"Advanced<\/p>\n

Make sure to note your API_KEY<\/code> , API_SECRET<\/code> and CLOUD_NAME<\/code> from your developer console<\/a>. This information is needed when integrating with your application.<\/p>\n

Upload Images and Retrieve Facial Emotion Recognition from Images<\/h2>\n

Using Cloudinary to obtain the emotion from a facial image requires the Advanced Facial Attributes Detection Add-on<\/code> when uploading the image. We add the adv_face<\/code> as a tag to let Cloudinary know we are going to use the Advanced Facial Attributes Detection<\/code> add-on to register the emotions from the detected users face.<\/p>\n

When the image is successfully uploaded, emotions detected from the face and their confidence values are returned as part of the result. The emotion with the highest confidence is then selected.<\/p>\n

JavaScript Sample:<\/p>\n

cloudinary.uploader.upload( req.body.image , function<\/span>(result<\/span>) <\/span>{\n  const<\/span> emotions = result.info.detection.adv_face.data[0<\/span>].attributes.emotion;\n  let<\/span> arr = JSON<\/span>.parse(JSON<\/span>.stringify(emotions));\n  let<\/span> visible_emotion = getMaxKey(arr); \/\/ this function gets the emotion with the highest confidence<\/span>\n  return<\/span> res.json({\n    status<\/span>: true<\/span>,\n    emotion<\/span>: visible_emotion\n  })\n},{ detection<\/span>: \"adv_face\"<\/span> });\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n
You can then return the detected emotion<\/code> to the frontend of your application.<\/p>\n
Creating Video Playlists Based on emotion<\/h2>\n
Once the emotion is obtained on the front end, we want to display a set of videos to the user that correspond to the selected emotion. The Cloudinary Video Player<\/a> makes creating playlists simple. You can create a video player and then populate it with videos from your media library that are tagged with the highest confidence<\/em> emotion<\/code>, such as \u2018happiness\u2019.<\/p>\n
JavaScript Sample:<\/p>\n
let<\/span> emotion = \"happiness\"<\/span>;\nlet<\/span> cld = cloudinary.Cloudinary.new({ cloud_name<\/span>: CLOUD_NAME, secure<\/span>: true<\/span>});b\n\/\/ initialize video player<\/span>\nlet<\/span> demoplayer = cld.videoPlayer('elementID'<\/span>);\n\/\/  create playlist based on emotion<\/span>\ndemoplayer.playlistByTag( emotion, { sourceParams<\/span>: {angle<\/span>:0<\/span>}, autoAdvance<\/span>: 0<\/span>, repeat<\/span>: true<\/span>, presentUpcoming<\/span>: 5<\/span>});\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n
Putting it all together, we have the sample of how it all works below:<\/p>\n
\"Video<\/p>\n
Conclusion<\/h2>\n
In this post, we have shown how to use the facial emotion recognition add-on from Cloudinary to create an application that enables us to serve videos to our users based on how they are feeling at the moment. We also leveraged the Cloudinary Video Player to serve the videos in a playlist format. You can learn more about using the video player<\/a> and feel free to check out the github repository<\/a> for the full source code. If you are up for the challenge, clone and submit your own Video Suggestion App below!<\/p>\n
\n
Further Reading on Video Manipulation<\/h2>\n