Cloudinary provides an Advanced Facial Attribute Detection add-on, an integrated face detection solution that utilizes Microsoft Cognitive Services to automatically extract meaningful advanced data about the face(s) in an image. It also extracts other face-related attributes and the exact location of notable facial features. This add-on is fully integrated into Cloudinary’s image management and transformation pipeline, allowing us to extend Cloudinary’s features that involve semantic photo data extraction, image cropping, and the positioning of image overlays.
In this post, we’ll create a simple app that illustrates how to extract advanced facial attributes from an image, crop, and add an image overlay based on the extracted data.
Here is a link to the demo CodeSandbox.
Create a new Next.js application using the following command:
npx create-next-app facial-attributes-detection
Run these commands to navigate into the project directory and install the required dependencies:
cd facial-attributes-detection
npm install cloudinary axios
The Cloudinary Node SDK will provide easy-to-use methods to interact with the Cloudinary APIs, while axios will serve as our HTTP client.
Now we can start our application on http://localhost:3000/ using the following command:
npm run dev
First, sign up for a free Cloudinary account if you don’t have one already. Displayed on your account’s Management Console (aka Dashboard) are important details: your cloud name, API key, etc.
Next, let’s create environment variables to hold the details of our Cloudinary account. Create a new file called .env at the root of your project and add the following to it:
CLOUD_NAME = YOUR CLOUD NAME HERE
API_KEY = YOUR API API KEY
API_SECRET = YOUR API API SECRET
This will be used as a default when the project is set up on another system. To update your local environment, create a copy of the .env file using the following command:
cp .env .env.local
Code language: CSS (css)By default, this local file resides in the .gitignore folder, mitigating the security risk of inadvertently exposing secret credentials to the public. You can update the .env.local file with your Cloudinary credentials.
Cloudinary makes it compulsory that we subscribe to an add-on before we can use it. To register for the Advanced facial attributes detection add-on, follow the steps below:
- Click on the Add-ons link in your Cloudinary console.
- You should see a page consisting of all the available Cloudinary add-ons. Scroll down to locate the Advanced Facial Attributes Detection add-on, click on it and select your preferred plan. We’ll be using the free plan for this project, which gives us 50 free detections monthly.
A detailed object comprising facial attributes of faces detected in an image can be extracted by setting the detection parameter to adv_face when uploading an image to Cloudinary using the upload API. Data detected and extracted by the add-on are stored in a data key nested in an info node of the JSON response.
The value stored in the data key is an array of objects, with each object holding full details about the individual faces detected. The details are divided into attributes, bounding_box, and facial_landmarks.
-
attributes: holds a key-value pair of general information such as the expression expressed by an individual, details about the hair, gender, make-up, and so on.
-
bounding_box: contains details about the bounding box surrounding a detected face, height, width, etc.
-
facial_landmarks: contains the exact position details of specific elements of the mouth, eyebrows, eyes, and nose.
Let’s upload an image to Cloudinary and set the detection parameter to adv_face to see the complete response returned. Create a file named upload.js in the pages/api directory and add the following to it:
const cloudinary = require("cloudinary").v2;
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
secure: true,
});
export default async function handler(req, res) {
try {
const response = await cloudinary.uploader.upload(req.body.image, {
detection: "adv_face",
});
res.status(200).json(response);
} catch (error) {
res.status(500).json(error);
}
}
export const config = {
api: {
bodyParser: {
sizeLimit: "4mb",
},
},
};
Code language: JavaScript (javascript)In the code above, we defined an upload API route to handle file upload to Cloudinary. We import Cloudinary and configure it with an object consisting of our Cloudinary credentials. Next, we define a route handler, which calls the Cloudinary upload method and passes the expected base64 image with an object to set the detection parameter as arguments. The response is sent back to the client; otherwise, an error is sent.
At the bottom of the file, we export the Next.js default config object to set the default payload size limit to 4MB.
Now let’s create a client-side for selecting and forwarding any selected image to the /upload route. Clear the existing content in your pages/index.js file and update it with the following:
import { useState } from "react";
import axios from "axios";
import styles from "../styles/Home.module.css";
export default function Home() {
const [image, setImage] = useState("");
const [uploadStatus, setUploadStatus] = useState();
const [imageId, setImageId] = useState("");
const handleImageChange = (e, setStateFunc) => {
const reader = new FileReader();
if (!e.target.files[0]) return;
reader.readAsDataURL(e.target.files[0]);
reader.onload = function (e) {
setStateFunc(e.target.result);
};
};
const handleUpload = async () => {
setUploadStatus("Uploading...");
try {
const response = await axios.post("/api/upload", { image });
setImageId(response.data.public_id);
setUploadStatus("Upload successful");
console.log(response.data);
} catch (error) {
setUploadStatus("Upload failed..");
}
};
return (
<main className={styles.main}>
<h2>Facial attributes detection</h2>
<div>
<div className={styles.input}>
<div>
<label htmlFor="image">
{image ? (
<img src={image} alt="image" />
) : (
"Click to select image"
)}
</label>
<input
type="file"
id="image"
onChange={(e) => handleImageChange(e, setImage)}
/>
</div>
<button onClick={handleUpload}>Upload</button>
<p>{uploadStatus}</p>
</div>
</div>
</main>
);
}
Code language: JavaScript (javascript)In the code above, we defined the Home component to hold three states for the selected image, the request status, and a Cloudinary-generated ID. Next, we rendered a file input field and worked around opening the custom file picker that triggers the handleImageChange function when a file is selected. The function then converts the selected image to its base64 equivalent.
We also rendered a button that calls the handleUpload function on click. handleUpload makes an Axios call to our API route and sets the required states accordingly. We also logged the complete response to the console.
Now let’s add some styles to give our application a decent look. Copy the styles in this codeSandbox link to your styles/Home.module.css file.
Next, preview the application in your browser and upload an image with faces. Then open the developer console to see the complete JSON response object.
Displayed below is a closer look at the response object.
We can also use Cloudinary’s Admin API to apply automatic face attribute detection to uploaded images based on their public IDs. To achieve this, call the update method of the Admin API and set the detection parameter to adv_face, as shown below.
const response = await cloudinary.v2.api.update("public-id", {
detection: "adv_face",
});
Code language: JavaScript (javascript)As mentioned earlier, the Advanced Facial Attribute Detection add-on is fully integrated into Cloudinary’s image management and transformation pipeline. Therefore, we can crop and apply other transformations to the image based on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on.
To crop a processed image so it focuses on the detected faces in the image, we need to set the gravity parameter to adv_faces or adv_face to focus on the single largest detected face in the image when calling the image method of Cloudinary’s image transformation API.
We also need to specify the width and height parameters and set the crop parameter to either crop, thumb, or fill. Click here to learn more about the various image resizing and cropping options.
To add the cropping functionality to our application, create a crop.js file in the pages/api folder and add the following to it:
const cloudinary = require("cloudinary").v2;
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
secure: true,
});
export default async function handler(req, res) {
try {
const response = await cloudinary.image(`${req.body.imageId}.jpg`, {
gravity: "adv_faces",
height: 240,
width: 240,
crop: "thumb",
sign_url: true,
});
res.status(200).json(response);
} catch (error) {
res.status(500).json(error);
}
}
Code language: JavaScript (javascript)In the code above, in addition to the gravity, width, height, and crop parameters, we set the sign_url parameter to true to reduce the potential costs of users accessing unplanned dynamic URLs with the Advanced Facial Attribute Detection cropping directives.
The expected response sent back to the client-side will be an <img> element with a URL that links to the cropped image.
Let’s update the client-side of the application to reflect the changes. Update your pages/index.js file with the following:
export default function Home() {
//...
// Add this
const [cldData, setCldData] = useState("");
const handleImageChange = (e, setStateFunc) => {
//...
};
const handleUpload = async () => {
//...
};
const handleCrop = async () => {
setUploadStatus("Cropping...");
try {
const response = await axios.post("/api/crop", { imageId });
const imageUrl = /'(.+)'/.exec(response.data)[1].split("' ")[0];
setCldData(imageUrl);
setUploadStatus("done");
} catch (error) {
setUploadStatus("failed..");
}
};
return (
<main className={styles.main}>
<h2>Facial attributes detection</h2>
<div>
<div className={styles.input}>
<div>
<label htmlFor="image">
{image ? (
<img src={image} alt="image" />
) : (
"Click to select image"
)}
</label>
<input
type="file"
id="image"
onChange={(e) => handleImageChange(e, setImage)}
/>
</div>
<button onClick={handleUpload}>Upload</button>
<p>{uploadStatus}</p>
{/* Add this */}
<div className={styles.btns}>
<button disabled={!imageId} onClick={handleCrop}>
Crop
</button>
</div>
</div>
{/* Add this */}
<div className={styles.output}>
{cldData ? <img src={cldData} alt=" " /> : "Output image"}
</div>
</div>
</main>
);
}
Code language: JavaScript (javascript)In the updated code, we defined a state called cldData to hold the expected URL of the cropped image. We also rendered a button and an image with the URL saved in the cldData state.
The button gets disabled until a valid Cloudinary image ID is returned after uploading; it triggers the handleCrop function when clicked. The function initiates an Axios call to the /crop API route to get the <img> element response returned by Cloudinary. It then extracts the URL from the response and sets the states accordingly.
Save the changes and preview the application in your browser. You should be able to upload and crop an image based on the detected faces.
In addition to cropping processed images based on the detected faces, Cloudinary also supports eye detection-based cropping. It automatically crops images based on the position of detected eyes, leveraging the data detected by the add-on. To implement this, change the update your pages/api/crop.js file with the following:
const response = await cloudinary.image(`${req.body.imageId}.jpg`, {
gravity: "adv_eyes", // add this
height: 240,
width: 240,
crop: "thumb",
sign_url: true,
});
Code language: JavaScript (javascript)While considering the pose of the face detected in the extracted facial attribute, Cloudinary can position overlays on top of detected faces and even automatically scale and rotate the overlay according to how the underlying face is positioned.
To properly place an overlay on all detected faces in a processed image, set an overlay parameter to the public ID of your preferred overlay image and the gravity parameter of the added overlay to adv_faces. We also need to set the region_relative flag together with a width and crop value. The width takes a relative value that scales the overlay to 110% of the width of the detected face.
Let’s update our application to include this functionality. Create a file called overlay.js in the pages/api folder and add the following to it:
const cloudinary = require("cloudinary").v2;
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
secure: true,
});
export default async function handler(req, res) {
const { imageId, overlay } = req.body;
try {
await cloudinary.uploader.upload(
overlay,
async function (error, uploadedOverlay) {
const response = await cloudinary.image(`${imageId}.jpg`, {
transformation: [
{ overlay: `${uploadedOverlay.public_id}` },
{ flags: "region_relative", width: "1.1", crop: "scale" },
{ flags: "layer_apply", gravity: "adv_faces" },
],
sign_url: true,
});
res.status(200).json(response);
}
);
} catch (error) {
res.status(500).json(error);
}
}
export const config = {
api: {
bodyParser: {
sizeLimit: "4mb",
},
},
};
Code language: JavaScript (javascript)With the code above, we created a new API route to handle applying overlay to a processed image. It expects the public id of the image that needs to be transformed and an overlay image from the client side.
The approach is similar to the one used in the previous API route files, except that we now configured the route handler to upload the overlay image to Cloudinary first to extract its public id from the response. Next, we called the image transformation method and set the overlay parameter to the extracted public id of the uploaded overlay image.
Let’s update the frontend code. Open your pages/index.js file and update the code as shown below:
export default function Home() {
//...
//add this
const [overlay, setOverlay] = useState("");
const handleImageChange = (e, setStateFunc) => {
//...
};
const handleUpload = async () => {
//...
};
const handleCrop = async () => {
//...
};
// add this
const handleAddOverlay = async () => {
setUploadStatus("Adding overlay...");
try {
const response = await axios.post("/api/overlay", { imageId, overlay });
const imageUrl = /'(.+)'/.exec(response.data)[1];
setCldData(imageUrl);
setUploadStatus("done");
} catch (error) {
setUploadStatus("failed..");
}
};
return (
<main className={styles.main}>
<h2>Facial attributes detection</h2>
<div>
<div className={styles.input}>
<div>
<label htmlFor="image">
{image ? (
<img src={image} alt="image" />
) : (
"Click to select image"
)}
</label>
<input
type="file"
id="image"
onChange={(e) => handleImageChange(e, setImage)}
/>
</div>
<button onClick={handleUpload}>Upload</button>
<p>{uploadStatus}</p>
<div className={styles.btns}>
<button disabled={!imageId} onClick={handleCrop}>
Crop
</button>
{/* add this */}
<button disabled={!imageId || !overlay} onClick={handleAddOverlay}>
Add Overlay
</button>
</div>
{/* add this */}
<div className={styles.overlay}>
<label>Select Overlay</label>
<input
type="file"
onChange={(e) => handleImageChange(e, setOverlay)}
/>
</div>
</div>
<div className={styles.output}>
{cldData ? <img src={cldData} alt=" " /> : "Output image"}
</div>
</div>
</main>
);
}
Code language: JavaScript (javascript)We added a new state called overlay to hold the base64 equivalent of the overlay image selected by the user. Next, we added a <input> tag with a file type to select an overlay image and a button that triggers the handleAddOverlay function when clicked.
The function initiates an Axios call to the /overlay API route and attaches the image ID and the overlay image to the request’s body. Next, it formats the response to extract the output image URL and sets the states accordingly.
Now you can save the changes and preview the application in your browser.
Find the complete project here on GitHub.
The Advanced Facial Attribute Detection add-on powered by Cloudinary’s integration with Microsoft’s Cognitive Services provides a high-precision mechanism that can seamlessly analyze images to extract specific information about facial attributes. Using a simple Next.js application, we’ve seen how to use this add-on to extract advanced face attributes and smartly crop, position, rotate, and add overlay images based on these attributes.
Resources You May Find Helpful
