Cloudinary AI Background Removal

Cloudinary is a cloud-based service that provides solutions for image and video management, including server or client-side upload, on-the-fly image and video manipulations, quick CDN delivery, and a variety of asset management options.

The Cloudinary AI Background Removal add-on combines a variety of deep-learning algorithms to recognize the primary foreground subject of a photo and accurately remove the background in a matter of seconds.

Activating the add-on

You activate the add-on by setting the background_removal parameter to cloudinary_ai when uploading an image (Upload method) or using the Update method of the Admin API for existing images. The background removal process is very quick, but is still handled asynchronously, so you may want to include a notification_url in your method call.

Removing the background while uploading

The code below uploads the photo shown below on the left and activates the Cloudinary AI background removal add-on. The photo that ultimately gets stored in the account is the dog image shown on the right, with a fully transparent background.

Ruby:
Cloudinary::Uploader.upload("dog_couch.jpg",
  :public_id => "dog_couch",
  :background_removal => 'cloudinary_ai',
  :notification_url => "https://mysite.example.com/hooks")
PHP:
\Cloudinary\Uploader::upload("dog_couch.jpg", 
  array(
    "public_id" => "dog_couch",
    "background_removal" => "cloudinary_ai",
    "notification_url" => "https://mysite.example.com/hooks"));
Python:
cloudinary.uploader.upload("dog_couch.jpg",
  public_id = "dog_couch",
  background_removal = "cloudinary_ai",
  notification_url = "https://mysite.example.com/hooks")
Node.js:
cloudinary.v2.uploader.upload("dog_couch.jpg", 
  { public_id: "dog_couch",
    background_removal: "cloudinary_ai",
    notification_url: "https://mysite.example.com/hooks" }),
  function(error, result){console.log(result);});
Java:
cloudinary.uploader().upload("dog_couch.jpg", 
  ObjectUtils.asMap(
    "public_id", "dog_couch",
    "background_removal", "cloudinary_ai",
    "notification_url", "https://mysite.example.com/hooks"));
.Net:
var uploadParams = new ImageUploadParams(){
  File = new FileDescription(@"dog_couch.jpg"),
  PublicId = "dog_couch",
  BackgroundRemoval = "cloudinary_ai",
  NotificationUrl = "https://mysite.example.com/hooks"};
var uploadResult = cloudinary.Upload(uploadParams);
Original image to upload Original image to upload Uploaded image with background removal add-on applied Uploaded image with the add-on applied

Tip
You can use upload presets to centrally define a set of upload options including add-on operations to apply, instead of specifying them in each upload call. You can define multiple upload presets, and apply different presets in different upload scenarios. You can create new upload presets in the Upload page of the Management Console settings or using the upload_presets Admin API method. From the Upload page of the console settings, you can also select default upload presets to use for image, video, and raw API uploads (respectively) as well as default presets for image, video, and raw uploads performed via the Media Library UI.

Removing the background from existing images

You can remove the background from existing images, using the Update Admin API method:

Ruby:
Cloudinary::Api.update("woman", 
  :background_removal => "cloudinary_ai",
  :notification_url => "https://mysite.example.com/hooks")
PHP:
$api = new \Cloudinary\Api();
$api->update("woman", 
  array(
  "background_removal" => "cloudinary_ai",
  "notification_url" => "https://mysite.example.com/hooks"));
Python:
cloudinary.api.update("woman",
  background_removal = "cloudinary_ai",
  notification_url = "https://mysite.example.com/hooks")
Node.js:
cloudinary.v2.api.update("woman", 
  { background_removal: "cloudinary_ai",
    notification_url: "https://mysite.example.com/hooks" }),
  function(error, result){console.log(result);});
Java:
cloudinary.api().update("woman", 
  ObjectUtils.asMap(
    "background_removal", "cloudinary_ai",
    "notification_url", "https://mysite.example.com/hooks" ));
.Net:
var updateParams = new UpdateParams("woman"){
  BackgroundRemoval = "cloudinary_ai",
  NotificationUrl = "https://mysite.example.com/hooks"};
var updateResult = cloudinary.UpdateResource(updateParams);

Asynchronous handling

Although the background removal algorithm usually takes only a few seconds, it is still handled asynchronously after the original file is uploaded or updated. Therefore, the immediate upload response indicates that the background_removal status is pending. For example, here's the upload response from the above dog-on-couch image.

{
  "public_id": "dog_couch",
  "version": 1551099901,
   ...
   ...
  "url": "https://res.cloudinary.com/demo/image/upload/v1551101478/dog_couch.jpg",
  "secure_url": "https://res.cloudinary.com/demo/image/upload/v1551101478/dog_couch.jpg",
  "info": {
    "background_removal": {
      "cloudinary_ai": {
        "status": "pending"
      }
    }
  },
  "original_filename": "dog_couch"
}

Immediately after you perform the upload or update method, the image in your account is your original image. When the background removal process is complete, the original image is overwritten with a PNG containing the foreground image and transparent background.

If you used a notification_url in your upload or update call, the endpoint you specified will receive a JSON POST request when the background removal is complete, including the new url & secure_url with the updated .png file extension and new version number:

{
    "info_kind": "cloudinary_ai",
    "info_status": "complete",
    "public_id": "dog_couch",
    "uploaded_at": "2019-02-25T17:33:45Z",
    "version": 1551104931,
    "url": "http://res.cloudinary.com/demo/image/upload/v1551104931/dog_couch.png",
    "secure_url": "https://res.cloudinary.com/demo/image/upload/v1551104931/dog_couch.png",
    "etag": "6567d798ca4087468dc7d23bcb8a45ec",
    "notification_type": "info"
}

Evaluating the AI results

This AI add-on is based on a combination of neural networks that were trained on a large dataset to create precise segmentation maps of salient objects and refine those maps to accurately separate the edges of the foreground from the background. These neural networks were also optimized to enable returning the result within seconds regardless of image file size.

In the great majority of cases, these deep-learning algorithms return high quality results. However, as with any AI-based algorithm, the results may not always be as expected. Additionally, the neural networks are continually training on new data and thus results may be different over time.

If you are not satisfied with the results of a background removal operation, you can apply one of the following options:

  • Restore from backup: When the newly generated (removed background) image overwrites the old one, the original image is backed up and is accessible from the View backed up versions button, available when you open the Edit Transformation page for that image in the Media Library. This backup is performed for all images where the Cloudinary AI Remove the Background add-on is applied, even if you haven't enabled backups globally for your account.

Originals are stored as backups when you activate the AI Remove the Background add-on

  • Pixelz Remove the background add-on: The Pixelz Remove the Background add-on automatically uploads your original image to the Pixelz team of human experts who manually remove the background of your image within 24 hours. When complete, the new image replaces your original one in your account similar to the behavior of the Cloudinary AI Remove the Background add-on. You can monitor the status using a notification_url.

Delivering and transforming your new image

After the background has been removed from your image, you can take advantage of a variety of image transformations to deliver the new image to your users.

Use case #1 - Add a shadow to a product image

If all the product images in your online store have a shadow, then after you've applied the Cloudinary AI Remove the Background add-on to this router photo, you could use the transformation code below to apply a shadow to the delivered image.

Ruby:
cl_image_tag("docs/rmv_bgd/router.png", :transformation=>[
  {:height=>105, :crop=>"scale"},
  {:effect=>"shadow:50", :x=>10, :y=>10}
  ])
PHP:
cl_image_tag("docs/rmv_bgd/router.png", array("transformation"=>array(
  array("height"=>105, "crop"=>"scale"),
  array("effect"=>"shadow:50", "x"=>10, "y"=>10)
  )))
Python:
CloudinaryImage("docs/rmv_bgd/router.png").image(transformation=[
  {'height': 105, 'crop': "scale"},
  {'effect': "shadow:50", 'x': 10, 'y': 10}
  ])
Node.js:
cloudinary.image("docs/rmv_bgd/router.png", {transformation: [
  {height: 105, crop: "scale"},
  {effect: "shadow:50", x: 10, y: 10}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .height(105).crop("scale").chain()
  .effect("shadow:50").x(10).y(10)).imageTag("docs/rmv_bgd/router.png");
JS:
cloudinary.imageTag('docs/rmv_bgd/router.png', {transformation: [
  {height: 105, crop: "scale"},
  {effect: "shadow:50", x: 10, y: 10}
  ]}).toHtml();
jQuery:
$.cloudinary.image("docs/rmv_bgd/router.png", {transformation: [
  {height: 105, crop: "scale"},
  {effect: "shadow:50", x: 10, y: 10}
  ]})
React:
<Image publicId="docs/rmv_bgd/router.png" >
  <Transformation height="105" crop="scale" />
  <Transformation effect="shadow:50" x="10" y="10" />
</Image>
Angular:
<cl-image public-id="docs/rmv_bgd/router.png" >
  <cl-transformation height="105" crop="scale">
  </cl-transformation>
  <cl-transformation effect="shadow:50" x="10" y="10">
  </cl-transformation>
</cl-image>
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Height(105).Crop("scale").Chain()
  .Effect("shadow:50").X(10).Y(10)).BuildImageTag("docs/rmv_bgd/router.png")
Android:
MediaManager.get().url().transformation(new Transformation()
  .height(105).crop("scale").chain()
  .effect("shadow:50").x(10).y(10)).generate("docs/rmv_bgd/router.png");
iOS:
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation()
  .setHeight(105).setCrop("scale").chain()
  .setEffect("shadow:50").setX(10).setY(10)).generate("docs/rmv_bgd/router.png")!, cloudinary: cloudinary)

Original router image Original router image Router without background No background No background + shadow No background + shadow

Use case #2 - Apply green screen style backdrops

You can take advantage of the Cloudinary AI Remove the Background add-on to provide an app with fun effects. For example, you could allow users to upload images of themselves and then select new background locations by adding the desired scenery as an underlay layer behind your transparent background.

For example, after running the add-on on the original image to remove the background, you could use the following delivery code to relocate this woman from her office to the beach:

Ruby:
cl_image_tag("docs/rmv_bgd/woman.png", :transformation=>[
  {:height=>375, :crop=>"scale"},
  {:underlay=>"docs:bg_beach", :gravity=>"south", :width=>800}
  ])
PHP:
cl_image_tag("docs/rmv_bgd/woman.png", array("transformation"=>array(
  array("height"=>375, "crop"=>"scale"),
  array("underlay"=>"docs:bg_beach", "gravity"=>"south", "width"=>800)
  )))
Python:
CloudinaryImage("docs/rmv_bgd/woman.png").image(transformation=[
  {'height': 375, 'crop': "scale"},
  {'underlay': "docs:bg_beach", 'gravity': "south", 'width': 800}
  ])
Node.js:
cloudinary.image("docs/rmv_bgd/woman.png", {transformation: [
  {height: 375, crop: "scale"},
  {underlay: "docs:bg_beach", gravity: "south", width: 800}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .height(375).crop("scale").chain()
  .underlay(new Layer().publicId("docs:bg_beach")).gravity("south").width(800)).imageTag("docs/rmv_bgd/woman.png");
JS:
cloudinary.imageTag('docs/rmv_bgd/woman.png', {transformation: [
  {height: 375, crop: "scale"},
  {underlay: new cloudinary.Layer().publicId("docs:bg_beach"), gravity: "south", width: 800}
  ]}).toHtml();
jQuery:
$.cloudinary.image("docs/rmv_bgd/woman.png", {transformation: [
  {height: 375, crop: "scale"},
  {underlay: new cloudinary.Layer().publicId("docs:bg_beach"), gravity: "south", width: 800}
  ]})
React:
<Image publicId="docs/rmv_bgd/woman.png" >
  <Transformation height="375" crop="scale" />
  <Transformation underlay="docs:bg_beach" gravity="south" width="800" />
</Image>
Angular:
<cl-image public-id="docs/rmv_bgd/woman.png" >
  <cl-transformation height="375" crop="scale">
  </cl-transformation>
  <cl-transformation underlay="docs:bg_beach" gravity="south" width="800">
  </cl-transformation>
</cl-image>
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Height(375).Crop("scale").Chain()
  .Underlay(new Layer().PublicId("docs:bg_beach")).Gravity("south").Width(800)).BuildImageTag("docs/rmv_bgd/woman.png")
Android:
MediaManager.get().url().transformation(new Transformation()
  .height(375).crop("scale").chain()
  .underlay(new Layer().publicId("docs:bg_beach")).gravity("south").width(800)).generate("docs/rmv_bgd/woman.png");
iOS:
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation()
  .setHeight(375).setCrop("scale").chain()
  .setUnderlay("docs:bg_beach").setGravity("south").setWidth(800)).generate("docs/rmv_bgd/woman.png")!, cloudinary: cloudinary)

Original personal photo Original personal photo No background No background

 

No background + rainbow scene No background + rainbow scene No background + beach scene No background + beach scene