Programmable Media

Cloudinary AI Background Removal

Last updated: Mar-17-2024

Cloudinary is a cloud-based service that provides solutions for image and video management, including server or client-side upload, on-the-fly image and video transformations, quick CDN delivery, and a variety of asset management options.

The Cloudinary AI Background Removal add-on combines a variety of deep-learning algorithms to recognize the primary foreground object(s) in an image and accurately remove the background in a matter of seconds. You can optionally specify one of a set of object names to instruct the add-on to remove everything except that object.

Note
This add-on only supports images, however there are ways to make backgrounds transparent in video overlays to effectively change the background of a video. See Video effects and enhancements for details.

Tip
This page describes how to use the Cloudinary AI Background Removal add-on programmatically in Programmable Media, but that you can also use the add-on for DAM use-cases in Assets. For more information, see Cloudinary AI Background Removal in the Assets user guide.

Getting started

Before you can use the Cloudinary AI Background Removal add-on:

  • You must have a Cloudinary account. If you don't already have one, you can sign up for a free account.

  • Register for the add-on: make sure you're logged in to your account and then go to the Add-ons page. For more information about add-on registrations, see Registering for add-ons.

  • Keep in mind that many of the examples on this page use our SDKs. For SDK installation and configuration details, see the relevant SDK guide.

  • If you are new to Cloudinary, you may want to take a look at How to integrate Cloudinary in your app for a walk through on the basics of creating and setting up your account, working with SDKs, and then uploading, transforming and delivering assets.

Tip
MediaFlows, Cloudinary’s drag-and-drop workflow builder for image and video, offers an out-of-the-box, low-code workflow template to automate the process of removing image backgrounds at scale.

Activating the add-on

The default background removal behavior detects the foreground object(s) of the image and removes the background. You can activate this behavior in one of two ways:

  • Recommended: On the fly, by specifying the background_removal effect transformation parameter when delivering images (e_background_removal in URLs).
  • Legacy (retained for backward compatibility): By setting the background_removal parameter to cloudinary_ai when uploading an image (Upload method), or by using the Update method of the Admin API for existing images.

You can also include additional directives for indicating what the AI algorithm should treat as the foreground object to keep. For details see: Specifying the foreground object below.

Removing the background on the fly

Removing the background on the fly keeps the original image intact and creates a derived version of the image with the background removed.

When background removal is requested on the fly, initially a 423 error response is returned, until the background removal process has completed. This only happens for the first request. After this, the image is cached, so subsequent requests to the same URL will return the background-removed image. You can request background removal as an eager transformation to prepare the derived versions in advance of the first request.

To remove the background of an image on the fly, specify the background_removal effect (e_background_removal in URLs). For example:

Original image Original image Uploaded image with background removal add-on applied Image with the add-on applied

See the full syntax in the transformation reference.

You can combine this transformation with other transformation parameters, but the background removal effect must be the first component in the chain.

For example, to apply the grayscale effect, first specify e_background_removal, then specify e_grayscale:

Dog on couch in grayscale with background removed

To replace the background with a different image, apply the new background as an underlay. For example, replacing the couch with a road, scaling and positioning the dog appropriately:

Dog on road


See background removal being applied to product images on the fly in a React app.

Notes
  • The background-removed version is saved so that when other transformations are applied, the add-on is not called again for that asset.
  • The first time the add-on is called for an asset, a 423 error response is returned until the processing has completed.
  • You can combine the background_removal effect with other transformation parameters, but the background_removal effect must be the first component in the chain.
  • Background removal on the fly cannot currently be used for image overlays. Instead, apply the base image as an underlay.
  • Background removal on the fly is not supported for fetched images.
  • Background removal on the fly is not supported for incoming transformations. If you need this use case, you can remove the background on upload.

Specifying the foreground object

In some cases, the AI algorithm may identify more than one object in the foreground. If you know which type of item you expect in the photos you are applying the add-on to, and that item is among the supported objects listed below, you can request to keep only object(s) of that type and treat everything else as background.

To do this on the fly, set the effect parameter to background_removal:hints_(<object_to_keep>).

Tip
To do this when uploading or updating an image, set the background_removal parameter to cloudinary_ai:<object_to_keep>.

For example, in the original image below, both the cars and the road sign are identified by the add-on algorithm as foreground objects. However, by specifying the object to keep as car using e_background_removal:hints_(car) (or background_removal:"cloudinary_ai:car" on upload/update), the sign is also treated as part of the background, and only the car remains:

Original Original Default background removal Default background removal With car hint Specifying car

Similarly, if you want to deliver images containing only dogs or only cats, you can ensure you only keep the animal you want:

Original Original
Default background removal Default background removal Specifying cat Specifying cat Specifying dog Specifying dog

You can also specify more than one object to keep, for example to keep both cats and dogs, where there may be other objects in the picture:

  • On the fly: e_background_removal:hints_(cat;dog)
  • On upload/update: background_removal:"cloudinary_ai:cat:dog"

Supported foreground objects

You can specify any of the following as the foreground object to keep when you remove the background:

airplane bus dining-table sheep
bicycle car dog sofa
bird cat horse train
boat chair person tv
bottle cow potted-plant motorbike

Defining the edges of the foreground object

If you know that your image has a foreground object with fine detail around its edges, for example fur, you can request to keep this detail by specifying the fine edges option.

Notes
  • You should not use this option if you know that the object has more clear-cut edges.
  • There is no need to use this option on images of human hair, as a special algorithm is applied if hair around a face is detected.

To do this on the fly, set the effect parameter to background_removal:fineedges_y:

Original Original Default background removal Default background removal Specifying fine edges Specifying fine edges

You can see the difference more clearly when you zoom in (click each image to open in a new tab):

Original Original Default background removal Default background removal cloudinary_ai:fine_edges Specifying fine edges

Tip

When uploading or updating an image, set the background_removal parameter to cloudinary_ai:fine_edges:

Combining hints and fine edges

To specify both the foreground object(s) and fine edges, separate the parameters with a semicolon (e_background_removal:hints_(dog);fineedges_y):

Dog hint with fine edges specified

Tip

When uploading or updating an image, separate the parameters with a colon (cloudinary_ai:dog:fine_edges):

Delivering and transforming your new image

After the background has been removed from your image, you can take advantage of a variety of image transformations to deliver the new image to your users.

Use case #1 - Add a shadow to a product image

If all the product images in your online store have a shadow, then after you've applied the Cloudinary AI Remove the Background add-on to this router photo, you could use the transformation code below to apply a shadow to the delivered image.

Original router image Original router image Router without background No background No background + shadow No background + shadow

Learn more: Apply a drop shadow effect

Use case #2 - Apply green screen style backdrops

You can take advantage of the Cloudinary AI Remove the Background add-on to provide an app with fun effects. For example, you could allow users to upload images of themselves and then select new background locations by adding the desired scenery as an underlay behind your transparent background.

For example, you could use the following delivery code to relocate this woman from her office to the beach:

Original personal photo Original personal photo No background No background

 

No background + rainbow scene No background + rainbow scene No background + beach scene No background + beach scene

Guidelines and Best Practices

This add-on is based on a combination of neural networks that were trained on a large dataset to create precise segmentation maps of salient objects and refine those maps to accurately separate the edges of the foreground from the background. These neural networks were also optimized to enable returning the result within seconds regardless of the image size.1

In the great majority of cases, these deep-learning algorithms return high quality results. The following best practices can further increase the likelihood of good results:

  • Make sure there is enough background around the foreground object(s) on all sides.
  • Try to avoid large shadows. This is especially relevant if the background is a single color. For example, if the main subject is photographed against a plain light colored background, the object's shadow might be classified as part of the foreground.
  • In some cases, hair or other very small or blurry parts on the edges of a foreground object might be blended with the background. For example, try to avoid hairs blowing in the wind.
  • In some cases, strong changes in contrast, such as a bright lamp or a bright sun included in the photo, can impact the results.
  • Light-colored reflections on a shiny object (e.g. shine from the camera flash) may be treated as part of the background and removed, especially if the color of the reflection is similar to the main background color.
  • Images containing transparent objects, such as a glass jug, may not give good results.

However, as with any Deep Learning-based algorithm, even if you follow these guidelines, the results may not always be perfect. Additionally, keep in mind that the neural networks are continually training on new data and thus results may be different over time.

If you are not satisfied with the results of a background removal operation, you can try the Pixelz Remove the Background add-on, which automatically uploads your original image to the Pixelz team of human experts who manually remove the background of your image within 24 hours. When complete, the new image replaces your original one in your product environment. You can monitor the status using a notification_url.

1The add-on imposes a limit of 4,194,304 (2048 x 2048) total pixels on its input images. If an image exceeds this limit, the add-on first scales down the image to fit the limit, and then processes it. The scaling does not affect the aspect ratio of the image, but it does alter its output dimensions.

Removing the background on upload/update

Important
This is legacy functionality, kept for backward compatibility reasons. We recommend removing image backgrounds on the fly.

Unlike removing the background on the fly, when you remove a background on upload, only the background-removed image is stored, not the original. If it isn't already a PNG, it is converted to a PNG with a transparent background.

Note
Even if the originally uploaded image is within your file size product environment limits, large resolution images may result in very large file sizes for the returned PNG, and may exceed the max file size limits for your product environment. If the returned file exceeds your product environment limits, then it will not be stored in your product environment and the original will remain.

The code below uploads the photo shown below on the left and activates the Cloudinary AI background removal add-on. The photo that ultimately gets stored in the product environment is the dog image shown on the right, with a fully transparent background.

Original image to upload Original image to upload Uploaded image with background removal add-on applied Uploaded image with the add-on applied

Tip
You can use upload presets to centrally define a set of upload options including add-on operations to apply, instead of specifying them in each upload call. You can define multiple upload presets, and apply different presets in different upload scenarios. You can create new upload presets in the Upload page of the Console Settings or using the upload_presets Admin API method. From the Upload page of the Console Settings, you can also select default upload presets to use for image, video, and raw API uploads (respectively) as well as default presets for image, video, and raw uploads performed via the Media Library UI.

Learn more: Upload presets

Removing the background from existing images

You can remove the background from existing images using the Update Admin API method. The existing image is overwritten with the background-removed image.

Asynchronous handling for upload/update

Although the background removal algorithm usually takes only a few seconds, it is still handled asynchronously after the original file is uploaded or updated. Therefore, the immediate upload response indicates that the background_removal status is pending. For example, here's the upload response from the above dog-on-couch image.

Immediately after you perform the upload or update method, the image in your product environment is your original image. When the background removal process is complete, the original image is overwritten with a PNG containing the foreground image and transparent background.

If you requested a notification in your upload or update call by adding the notification_url parameter, the endpoint you specified will receive a JSON POST request when the background removal is complete, including the new url & secure_url with the updated .png file extension and new version number:

Confidence score

The notification response returns a confidence score, which is a measure of how successful the background removal, and hence the resulting image, is likely to be. For low scores, you could for example, send the image for manual moderation using the explicit method, or perhaps use the Pixelz - Remove the Background add-on instead.

In addition to the notification response, the confidence score is also reported in responses to calls to the Admin API resources endpoint where asset details are returned.

For example:

Sample response:

Restore from backup

When the newly generated (background-removed) image overwrites the old one, both the original image and background-removed image are backed up and are accessible from the View backed up versions button, available when you open the Edit Transformation page for that image in the Media Library. This backup is performed for all images where the Cloudinary AI Remove the Background add-on is applied, even if you haven't enabled backups globally for your product environment.

Originals are stored as backups when you activate the AI Remove the Background add-on

Note
If you don't want the images to be backed up, contact support to apply a setting to your product environment that disables this specific backup. With this setting applied:
  • If your product environment does not have global backup configured and backup was not requested during the upload (either in the request or in the upload preset), then none of the images are backed up.
  • If your product environment does have global backup configured, or backup was requested during the upload (either in the request or in the upload preset), then only the original image is backed up, and not the background-removed one.

✔️ Feedback sent!

Rate this page: