Cloudinary Blog

Overlaying Glasses Or Masks on Avatars With Vue.js and Cloudinary

Overlaying Glasses Or Masks on Avatars With Vue.js and Cloudinary

Deep learning, a subset of machine learning, breaks down tasks in ways that make all kinds of machine assists possible. With deep learning, you can identify images by machine, i.e., instead of hand-coding software with specific instructions for a task, you train the machine with large amounts of data and algorithms that enable it to learn how to perform the task.

Cloudinary

Even if you have the technical expertise to train models to identify images and detect next-level facial attributes, you can now leverage cognitive services to do your bidding. It’s amazing!

This article shows you how to, with Cloudinary and Microsoft Cognitive Services, build an app that shows how users look wearing various glasses or masks, helping them decide what to purchase.

Before and after

By way of background, Cloudinary is a media platform on which web and app developers like you can manage the media lifecycle in simple steps: upload rich media, including images and videos, to the cloud for storage; transform and optimize them with dynamic URLs; and deliver them through a multi-CDN system. In essence, do all that without leaving your application code.

Additionally, SDKs are available from Cloudinary for most popular programming languages, including server side, client side, and mobile. As shown in this post, we’re also applying deep learning to enhance our offering.

The Procedure

Follow the steps below.

Step 1: Create a Cloudinary account.

Sign up for a free Cloudinary account.

Sign up for Cloudinary

Afterwards, log in and you will be redirected to the dashboard on which are displayed your credentials: your cloud name, API key, and API secret. Take note of them for use later in this tutorial.

Cloudinary dashboard

Step 2: Set up a Node server.

Initialize a package.json file:

Copy to clipboard
 npm init

Install the modules:

Copy to clipboard
npm install express connect-multiparty cloudinary cors body-parser --save

Here’s what the modules are for:

  • express: For API routes.
  • connect-multiparty: For parsing of HTTP requests with multipart content-type or form-data.
  • cloudinary: For using Cloudinary’s Node.js SDK.
  • cors: For enabling cross-origin resource sharing (CORS).
  • body-parser: For attaching the request body to express’s req object.

Step 3: Activate the add-on for advanced detection of facial attributes.

From your dashboard, go to the Cloudinary Add-Ons section. Click Rekognition Celebrity Detection and select the Free Plan.

Note
You can change to another plan as your usage increases.

Step 4: Identify the facial attributes.

In your root directory, create a server.js file in which to set up require parameters for the modules you just installed, like this:

Copy to clipboard
const express = require('express');
const app = express();
const multipart = require('connect-multiparty');
const cloudinary = require('cloudinary');
const cors = require('cors');
const bodyParser = require('body-parser');

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(cors());

const multipartMiddleware = multipart();

Next, configure Cloudinary:

Copy to clipboard
cloudinary.config({
    cloud_name: 'xxxxxxxx',
    api_key: 'xxxxxxxx',
    api_secret: 'xxxxxxx'
});

Replace the three variables xxxxxx with the values from your dashboard.

Add the route for uploading and call it /upload:

Copy to clipboard
app.post('/upload', multipartMiddleware, function(req, res) {
  // Upload to Cloudinary
  cloudinary.v2.uploader.upload(req.files.image.path,
    function(error, result) {
      res.json(result);
    },
    // Specify transformation and detection of facial attributes.
    {
      transformation: [
        { width: 700, radius: "max", crop: "scale" },
        { flags: "region_relative", gravity: "adv_eyes", overlay: req.body.item, width: "1.7" }
      ]
    });

The overlay parameter takes the value of req.body.item. In this app, that value is glasses or harlequinmask.

Note
I’ve uploaded the two photos below to my Cloudinary account and renamed the top one harlequinmask and the bottom one glasses, which you’ll use as overlays for this app. Upload these two images to your account.

mask

Glasses

Cloudinary’s Advanced Facial Attribute Detection add-on detects specific facial attributes, including the exact position of the eyes of each of the faces in a photo. Based on that information, Cloudinary can position overlays on top of all the detected eye-pairs in an image.

To smartly overlay glasses or harlequinmask on top of the detected eye-pairs in the image uploaded by the user, I’ve set the overlay parameter to the ID of the harlequinmask or glasses image, and the gravity parameter to adv_eyes. I’ve also set region_relative_ flag with a 1.7 width to scale the overlay to 170 percent of the width of the detected eyes, and resized the image to an oval thumbnail with a width of 700 pixels.

Once a user makes a POST request to the /upload route, the route grabs the image file from the HTTP request, uploads the file to Cloudinary, identifies the pair of eyes, and overlays them with the option the user chooses ( glasses or harlequinmask), and returns the correct URL.

Note
Cloudnary’s Advanced Facial Attribute Detection add-on is an integrated face-detection tool that uses Microsoft Cognitive Services to detect the location of high-precision faces with state-of-the-art, cloud-based algorithms that can recognize up to 64 human faces in an image. Afterwards, the add-on returns the detected faces with rectangles (left, top, width, and height) to pinpoint the location of the faces in the image in pixels, the exact position of the eyes, mouth, eyebrows, nose, and lips, as well as other related attributes, such as pose, gender, and age, from each of the faces.

Finally, test the face-detection capability with Postman.

Step 5: Build the front end.

Flesh out the front end with the progressive framework Vue.js by following these steps:

1. Install the Vue command-line interface (CLI):

Copy to clipboard
 npm install -g vue-cli

2. Create a simple Vue project with the Vue CLI:

Copy to clipboard
vue init simple productshowcase

3. Go to the productshowcase directory and create an index.html file with the following code:

Copy to clipboard
<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Vue</title>
  <script src="https://unpkg.com/vue"></script>
  <script src="https://unpkg.com/axios/dist/axios.min.js"></script>
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
</head>
<body>
  <div id="app">

    <div class="container" style="margin-top: 3%; margin-left: 2%;">
      <div class="col-md-6">
        <div>
          <div class="col-md-6">
          <img src="http://res.cloudinary.com/unicodeveloper/image/upload/v1505797868/glasses.png" width="200" height="100" />
          <span> Glasses </span>
          </div>
          <div class="col-md-6">
          <img src="http://res.cloudinary.com/unicodeveloper/image/upload/v1505794374/oie_transparent.png" width="200" height="100" />
          <span> Harlequin Mask </span>
          </div>
        </div>

        <hr />

        <form enctype="multipart/form-data" @submit.prevent="onSubmit">
          <div class="form-group">
            <select class="form-control" name="item" v-model="model.item">
              <option disabled value="">Choose an item</option>
              <option value="glasses"> Glasses </option>
              <option value="harlequinmask"> Harlequin Mask </option>
            </select>
          </div>
          <div class="form-group">
            <label for="">File:</label>
            <input type="file" class="form-control" accept="image/*" name="image" v-on:change="upload($event.target.files)">
          </div>
          <div class="form-group">
            <button class="btn btn-primary" >Upload</button>
            {{ loading }}
          </div>
        </form>
      </div>

      <div class="col-md-4">
        <div class="col-md-6" style="margin-top: 20%;">
          <img id="originalface" class="img-responsive" alt="" width="600" height="600">
        </div>
        <div class="col-md-6" style="margin-top: 20%;">
          <img :src="maskedface" class="img-responsive" alt="" width="400" height="400">
        </div>
      </div>
    </div>
  </div>

  <script>
    new Vue({
      el: '#app',
      data: function() {
        return {
          model: {
           text: '',
           image: null,
           item: ''
          },
          maskedface: null,
          loading: '',
        }
      },
      methods: {
        upload: function(files) {
          this.model.image = files[0]
          this.showPreview(files[0]);
        },
        showPreview: function(file) {
          var reader = new FileReader();
          reader.onload = function (e) {
              document.getElementById("originalface").src = e.target.result;
          };
          // read the image file as a data URL.
          reader.readAsDataURL(file);
        },
        onSubmit: function() {
          // Assemble form data
          const formData = new FormData()
          formData.append('image', this.model.image);
          formData.append('item', this.model.item);
          this.loading = "Processing....Please be patient."

          // Post to server
          axios.post('http://localhost:3333/upload', formData)
          .then(res => {
            // Update UI
            this.maskedface = res.data.url
            this.loading = ''
          })
        }
      }
    })
  </script>
</body>
</html>

4. Run the app.

Let’s step through the code. First, take a look at the form for uploading images.

Copy to clipboard
<form enctype="multipart/form-data" @submit.prevent="onSubmit">
          <div class="form-group">
            <select class="form-control" name="item" v-model="model.item">
              <option disabled value="">Choose an item</option>
              <option value="glasses"> Glasses </option>
              <option value="harlequinmask"> Harlequin Mask </option>
            </select>
          </div>
          <div class="form-group">
            <label for="">File:</label>
            <input type="file" class="form-control" accept="image/*" name="image" v-on:change="upload($event.target.files)">
          </div>
          <div class="form-group">
            <button class="btn btn-primary" >Upload</button>
            {{ loading }}
          </div>
   </form>

Note that you’ve bound the upload form to an upload event-handler and attached a change event to the Choose File button. Once a user has picked a file, the showPreview method called in the Vue instance below is invoked, showing a thumbnail preview of the image about to be uploaded.

Thumbnail preview of the image about to be uploaded
Thumbnail preview of the image about to be uploaded

Check out the methods, model, and data properties on your Vue instance:

Copy to clipboard
new Vue({
      el: '#app',
      data: function() {
        return {
          model: {
           text: '',
           image: null,
           item: ''
          },
          maskedface: null,
          loading: '',
        }
      },
      methods: {
        upload: function(files) {
          this.model.image = files[0]
          this.showPreview(files[0]);
        },
        showPreview: function(file) {
          var reader = new FileReader();
          reader.onload = function (e) {
              document.getElementById("originalface").src = e.target.result;
          };
          // read the image file as a data URL.
          reader.readAsDataURL(file);
        },
        onSubmit: function() {
          // Assemble form data
          const formData = new FormData()
          formData.append('image', this.model.image);
          formData.append('item', this.model.item);
          this.loading = "Processing....Please be patient."

          // Post to server
          axios.post('http://localhost:3333/upload', formData)
          .then(res => {
            // Update UI
            this.maskedface = res.data.url
            this.loading = ''
          })
        }
      }
    })

On submission, the form calls the onSubmit function in the Vue method to make a POST request to the back end and then return the data to the front end.

The data returned is the modified image with the overlay, which is reflected in the UI.

An uploaded image of Rihanna wearing the selected Harlequin mask
An uploaded image of Rihanna wearing the selected Harlequin mask

An uploaded Image of Christian Nwamba, aka codebeast, wearing the selected glasses
An uploaded Image of Christian Nwamba, aka codebeast, wearing the selected glasses

Feel free to check out the source code.

Conclusion

You’ve now learned how to use Cloudinary capabilities to detect facial attributes in an image and transform it with an overlay. The transformation options are virtually limitless. So, let your creative juices flow and enhance your business with products that users will love. As shown in this tutorial, you don’t have to spend time building them: Cloudinary’s got you!

Recent Blog Posts

Create Lightweight Sites With Low-Code and No-Code Technology

Consumers expect modern websites to be mainly visual. But, the more compelling and complex the related media is, the more data is involved, compounding the site’s weight. In today’s content-craving world, delivering unoptimized media can cost you because it leads to sluggish page loads, resulting in visitors abandoning your site in search of a faster alternative. In fact, a page load that takes more than three seconds can cause as many as 40% of your visitors to bounce. Given this competitive, digital-first environment, you can’t afford to lose page views, for time is of the essence.

Read more
A Blueprint for AWS-Secured Webhook Listeners for Cloudinary

tl;dr: An AWS-secured and optimized Cloudinary webhook listener for extending the Cloudinary service

Code: Github

A webhook is a communication medium for sending notifications from one platform to another about events that occurred. In place are user-defined HTTP callbacks that are triggered by specific events. When a triggered event takes place on the source site, the webhook listens to the event, collects the data, and sends it to the URL you specified in the form of an HTTP request.

Read more
New Accessibility Features for Cloudinary’s Product Gallery Widget

Cloudinary’s Product Gallery widget, which launched in 2019, has enabled many brands to effectively and efficiently showcase their products in a sleek and captivating manner, saving countless hours of development time and accelerating release cycles. By adding Cloudinary’s Product Gallery widget with its customizable UI to their product page, retailers reap numerous benefits, often turning visitors into customers in short order.

Read more
Why Successful Businesses Engage With and Convert Audiences With Visual Media

Most business buyers prefer to research purchase options online, as do many shoppers. No wonder online retail sales in the U.S. rose by 32.4% in 2020—an impressive gain of $105 billion.

For B2B and B2C businesses, text-heavy websites are no longer adequate in attracting shoppers. Instead, engaging visual media—spin images, videos, 3D models, augmented reality—are becoming a must for conveying eye-catching details and differentiators about products or services.

Read more
Making User-Generated Content (UGC) Shoppable With Cloudinary

User-generated content (UGC) is a powerful marketing tool. Not only does video complement marketing efforts for e-commerce by enabling customers to explore products in greater detail, but UGC also adds an element of trust. As a bonus, user-generated video is an exceptional opportunity for e-businesses to attract website traffic without their marketing team having to create promotional videos from scratch. User-generated content drives conversions and brand loyalty as a direct result of authentic interaction.

Read more