Cloudinary Blog

API for extracting semantic image data - colors, faces, Exif data and more

Extracting Metadata and Exif Data from Images via API
When images are involved, web developers have a large set of relevant tools at their disposal. You can display images in your web sites and mobile applications. You can manipulate and transform such images using image editing and manipulation software or cloud-based solutions like Cloudinary. But there are other types of data embedded in image files that can add unique semantic information to the images and are hardly ever used.
 
Consider what new designs can appear if your graphics designer could assume that only blue themed user uploaded photos will be featured on your homepage. What about featuring only photos that show your users' faces? How about photos taken with new DSLR model cameras rather than older pocket ones? Only photos taken in the GPS vicinity of your website visitor? We believe that such capabilities can offer a new, important tool for web design and development. 
 
Unfortunately, such semantic data is usually locked safely within the images and rarely utilized by developers and designers. We hope that we can change that by introducing a new Cloudinary API that allows you to easily extract rich information regarding your website and mobile application's photos. Using this information you can search, sort and classify your images in amazing new ways.
 

Predominant Colors & Color Histogram

Image search services such as Google Image Search allow you to filter your image search to show only images of a certain color. How is it done? Each image is analyzed and the colors of the images are mapped to one or more leading colors.
 
Cloudinary now supports finding the leading colors of a given image using a standard palette of 12 main colors. Since Cloudinary is a cloud-based service, all image processing is done online and no software installation is required on your side.
 
Finding the predominant colors in an image is also useful for stock-photo sites that wants to allow you to narrow photo searching by colors (see our previous post of how-to quickly build a stock-photo site with Cloudinary) and for e-commerce sites. For example: if you have a fashion site, and you want your users to browse only blue or red shirts.
 
For example, the following image with the public ID 'fashion1' was uploaded to Cloudinary:
 
 
 
Using Cloudinary's Admin API, you can extract the photo's main colors by setting the 'colors' parameter to true (see reference documentation). Here are examples for Ruby, PHP, Node.js and Python:
Cloudinary::Api.resource('fashion1', :colors => true)
$api->resource("fashion1", array("colors" => TRUE));
cloudinary.api.resource('fashion1',  
                        function(result)  { console.log(result) }, { colors: true });
cloudinary.api.resource("fashion1", colors = True)
 
Below is the JSON result of this API call. It seems that the main colors of this image are white (50.7%) and blue (27.8%), with touches of gray and brown. Cool.
{
  "public_id": "fashion1",
  "width": 225,
  "height": 380,
  ...
  "predominant": {
    "google": [
      [ "white", 50.7 ],
      [ "blue",  27.8 ],
      [ "gray", 11.2 ],
      [ "brown", 5.1]
    ]
  }
}
Using this info, you can keep the color mapping in your model and allow clothes to be searched based on colors. Searching for blue clothes should return this product.
 
Another result you get as part of the color information API is a histogram of 32 RGB colors that best represent the image. The following JSON snippet was also included in the result of the API call.
{
  "public_id": "fashion1",
  ...
  "colors": [["#FFFFFF", 50.7], ["#011B43", 5.8], ["#5077A7", 4.9], ["#031235", 4.3], ["#F4CBB4", 3.3], ["#3A6498", 1.9], ["#6284AF", 1.9], ["#2D5E95", 1.9], ["#30578B", 1.8], ["#080918", 1.8], ["#E5B09D", 1.8], ["#36262F", 1.7], ["#264876", 1.6], ["#281A25", 1.5], ["#486A99", 1.4], ["#E3D6CF", 1.4], ["#4D3135", 1.4], ["#07264F", 1.2], ["#664E55", 1.1], ["#6E443C", 1.0]]
}
As you can see, you get RGB format and percentage breakdown of the 32 colors that best represent the image. '#FFFFFF' is white, representing around half of the image, followed by multiple blue shades (e.g., '#011B43' is 5.8%).
 

Face detection info 

Cloudinary supports face detection based cropping and pixelation. Either a single face or multiple faces can be automatically detected. Our API now supports returning additional information regarding the detected faces in an uploaded photo. 
 
Simply set the 'faces' parameter to true in the same method we showed above for 'colors'. Note that you can enable multiple flags in a single call for fetching all information at once. The result includes the exact coordinates of all detected faces, allowing you to easily find out how many faces are available in the photo and their exact positions.
 
The following Ruby command asks for the faces information of the 'fashion1' image:
Cloudinary::Api.resource('fashion1', :faces => true)
Here is the JSON result:
{
  "public_id": "fashion1",
  ...
  "faces": [[99, 21, 64, 87]]
}
 
As you can see, a single face was correctly detected. It is positioned in the 99,21 - 64,87 rectangle of the original image.
 
Same works for images with multiple faces:
 
{
  ...
  "faces":  [ [513, 19, 38, 52], [409, 26, 40, 54], [79, 31, 43, 59], [232, 32, 40, 54], [321, 33, 41, 57], [160, 37, 43, 59], [211, 153, 43, 59], [503, 151, 43, 59], [113, 162, 40, 54], [427, 160, 45, 61], [307, 172, 48, 65] ]
}
Note that face detection does not achieve 100% accuracy. If you need better accuracy, human moderation is recommended.
 

Camera information - Exchangeable image file format (Exif)

**Update March 2017: The exif parameter has been deprecated. The exif data can now be extracted using the image_metadata parameter.
 
Modern digital cameras and smartphones store additional metadata as part of the image files you shoot. Such information includes picture orientation, timestamps, camera model information, photo exposure, GPS location and more.
 
By setting the 'exif' parameter to true, Cloudinary's API can return the image's metadata (see our reference documentation). In the sections above we've shown how to use the Admin API for fetching information of previously uploaded images. You can also request this information while uploading the photos, so it is returned as part of an upload response.
 
For example, the following PHP command uploaded to Cloudinary a photo that was taken by an iPhone 4 in a portrait orientation.
\Cloudinary\Uploader::upload("exif_sample.jpeg", 
   array("public_id" => "exif_sample", "colors" => TRUE, "exif" => TRUE))
 
Here is the JSON of the upload response including the requested Exif and Colors information:
{ 
  "public_id": "exif_sample",
  "width": 2592,
  "height": 1936,
  ...
  "exif": {
    "ApertureValue": "4281/1441",
    "ColorSpace": "1",
    "ComponentsConfiguration": "1, 2, 3, 0",
    "Compression": "6",
    "DateTime": "2010:12:27 11:17:34",
    "DateTimeDigitized": "2010:12:27 11:17:34",
    "DateTimeOriginal": "2010:12:27 11:17:34",
    "ExifImageLength": "1936",
    "ExifImageWidth": "2592",
    "ExifOffset": "204",
    "ExifVersion": "48, 50, 50, 49",
    "ExposureMode": "0",
    "ExposureProgram": "2",
    "ExposureTime": "1/4309",
    "Flash": "24",
    "FlashPixVersion": "48, 49, 48, 48",
    "FNumber": "14/5",
    "FocalLength": "77/20",
    "GPSAltitude": "20723/924",
    "GPSAltitudeRef": "0",
    "GPSImgDirection": "42155/344",
    "GPSImgDirectionRef": "T",
    "GPSInfo": "574",
    "GPSLatitude": "21/1, 768/100, 0/1",
    "GPSLatitudeRef": "N",
    "GPSLongitude": "86/1, 4500/100, 0/1",
    "GPSLongitudeRef": "W",
    "GPSTimeStamp": "17/1, 17/1, 3326/100",
    "ISOSpeedRatings": "80",
    "JPEGInterchangeFormat": "870",
    "JPEGInterchangeFormatLength": "9932",
    "Make": "Apple",
    "MeteringMode": "1",
    "Model": "iPhone 4",
    "Orientation": "6",
    "ResolutionUnit": "2",
    "SceneCaptureType": "0",
    "SensingMethod": "2",
    "Sharpness": "2",
    "ShutterSpeedValue": "4781/396",
    "Software": "4.2.1",
    "SubjectArea": "1295, 967, 699, 696",
    "WhiteBalance": "0",
    "XResolution": "72/1",
    "YCbCrPositioning": "1",
    "YResolution": "72/1"
  },
  "colors":[["#CBC9C5",10.2],["#C4BCB4",9.0],["#1888AB",6.0],["#202618",6.0],["#226391",5.4],["#223A62",4.3],["#B9B4AD",3.8],["#2F88A1",3.5],["#C9C3BA",3.4],["#7492B2",3.4],["#157193",3.1],["#96ABCC",2.9],["#C8B495",2.8],["#4F97AB",2.8],["#484033",2.7],["#669FAD",2.5],["#A0A29E",2.4],["#38A7C8",2.3],["#57A5B7",2.3],["#2D8FAF",2.2],["#ACCADC",2.1],["#073554",2.0],["#60AFC7",2.0],["#1D4A6F",2.0],["#A39477",1.9],["#D1C4A0",1.8],["#296F96",1.7],["#4F6E91",1.5],["#5F5F57",1.4],["#90AECB",1.0]],
  "predominant": {"google":[["teal",41.7],["brown",35.6],["blue",12.1],["green",8.4]]
}
By the way, you can also use Cloudinary's Exif-based automatic rotation by setting the 'angle' parameter ('a' for URLs) to 'exif'. For example:
 
 
 

Summary

With the additional knowledge of image metadata and semantic information, you can enhance your image rich web and mobile applications with little effort, while Cloudinary does all the heavy lifting for you. These additional layers of information adds an important aspect that allows Cloudinary to offer a better than ever cloud-based solution to all your online image management and manipulation needs. 
 
All these new features were requested by Cloudinary's users and we thank all of you for that. We have plenty more ideas for enhancing Cloudinary's capabilities in this area and would love to hear your feedback and suggestions.
 
The ability to fetch Exif, FacesPredominant colors and Color histogram is now available to all of Cloudinary's plans, free and paid. Click here to setup a free Cloudinary account.

Recent Blog Posts

 New Image File Format: FUIF:Lossy, Lossless, and Free

I've been working to create a new image format, which I'm calling FUIF, or Free Universal Image Format. That’s a rather pretentious name, I know. But I couldn’t call it the Free Lossy Image Format (FLIF) because that acronym is not available any more (see below) and FUIF can do lossless, too, so it wouldn’t be accurate either.

Read more
Optimizing Video Streaming and Delivery: Q&A with Doug Sillars

Doug Sillars, a digital nomad and a freelance mobile-performance expert, answers questions about video streaming and delivery, website optimization, and more.

Doug Sillars, a freelance mobile-performance expert and developer advocate, is a Google Developer Expert and the author of O’Reilly’s High Performance Android Apps. Given his extensive travels across the globe—from the UK to Siberia—with his wife, kids, and 11-year-old dog, Max, he has been referred to as a “digital nomad.” So far in 2018, Doug has spoken at more than 75 meetups and conferences!

Read more
Building a Music Discovery Service

In May 2018, Cloudinary sponsored Capitol Music Group’s first hackathon held by its new Capitol360 Innovation Center, which aims at connecting musicians and software technologists to facilitate and stimulate the creation of music. See this interview for details. As a starter project for the hackathon, we built a sample app called Music Discovery Service.

Read more
Once a Hackathon Participant, Now a Judge

Over the past several years, I've had a chance to participate in hackathons, as part of teams that developed a social payment app and helped users decide meals to cook. But it wasn't until last month that I got to experience a hackathon from the other side, as a judge.

Read more
10 Website Videos Mistakes and How to Solve Them

It should come as no surprise that video use on the internet is exploding. You can see the dramatic growth of video on the average site in this SpeedCurve blog post.

With the growth in video comes greater bandwidth use, which is not only costly for your IT budget, but for your visitors as well. Beyond the expense, there is the user experience to consider. The heavier the page, the longer it will take to load, and the greater likelihood visitors will abandon your site. Page load speed is also an important factor in SEO ranking, so clearly video is something we need to take seriously and get right. Video is challenging, presenting terms still unfamiliar to developers - like codecs, bitrate and adaptive bitrate streaming. As a result, mistakes are being made in video implementation.

Read more
Android Data Saver: Optimizing Mobile Data Usage with Cloudinary

Over the life of a mobile device, the cost of a cellular data plan often exceeds that of the device itself. To optimize data usage and purge useless data on their mobile devices, users can enable Data Saver from Android 7.0 (API level 24). To do so, users toggle Data Saver in quick settings under the Notification shade or under Settings > Data usage. With Data Saver enabled, apps that aren't whitelisted cannot use cellular data in the background. They are also directed to consume less data while active.

Read more