Image recognition used to need some serious local resources–servers, GPUs, and machine learning knowledge just for us to have basic image recognition. Thankfully, things have changed, and now we can run classification models right in the browser with some JavaScript code. Now we can get instant results without uploading images to external servers.
In this guide we’ll build a very basic, working image classifier using TensorFlow.js and MobileNet. We’ll also see how Cloudinary’s transformations help preprocess images for better accuracy, and how its AI auto-tagging add-ons can handle recognition on the server side when we need it.
Key Takeaways:
- TensorFlow.js runs machine learning models right in the browser
- MobileNet classifies images into 1000 categories without much code
- Consistent image preprocessing helps with image recognition accuracy
In this article:
- Understanding the Basics of Image Recognition in JavaScript
- Preparing Our Project Environment
- Loading and Processing Images in JavaScript
- Running a Simple Image Recognition Model
- Uploading and Managing Assets with Cloudinary
- Using Cloudinary Transformations to Improve Recognition Accuracy
- Integrating Recognition Results into Web Interfaces
- How Cloudinary Supports Complete Image Recognition Pipelines
Understanding the Basics of Image Recognition in JavaScript
The simplified version is that image recognition models work by analyzing pixel data in an image. We feed them images in a specific image format, and they generate predictions that come out as confidence scores of what the model thinks it has detected in the image.
With JavaScript we can use the browser to give us everything that we need for image recognition. To make it work we need to load images into <img> elements and then draw them to Canvas. From there, pixel data is extracted as typed arrays, and TensorFlow.js does all the hard work with neural networks, using WebGL acceleration behind the scenes.
We’ll be using the MobileNet model for classification in the browser. It’s relatively small at around 17MB, and it’s quite fast. MobileNet recognizes around 1000 different categories from the ImageNet dataset. The tradeoff here is that we get lower accuracy than larger models, and although it is great for classifying dogs and cats, it struggles with more specific objects.
Preparing Our Project Environment
We don’t need to use build tools for basic image recognition; all we need to do is include TensorFlow.js and the MobileNet model via CDN:
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script> <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/mobilenet"></script>
Our project structure is very simple:
project/ ├── index.html ├── styles.css └── script.js
The 17MB model downloads automatically on first use. After that, the browser caches it for faster loads going forward.
Loading and Processing Images in JavaScript
Before we can classify an image we’ll need to load it into a format that the model understands. MobileNet can use <img>, <canvas>, and <video> elements directly, which makes it quite easy to implement.
// Load image and wait for it to be ready
function loadImage(src) {
return new Promise((resolve, reject) => {
const img = new Image();
img.crossOrigin = 'anonymous';
img.onload = () => resolve(img);
img.onerror = reject;
img.src = src;
});
}
The crossOrigin attribute is very important when we load images from external URLs like Cloudinary–we can’t access pixel data without it because of browser security restrictions.
For preprocessing, Canvas lets us resize images to fixed dimensions:
function preprocessImage(img, targetSize = 224) {
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
canvas.width = targetSize;
canvas.height = targetSize;
ctx.drawImage(img, 0, 0, targetSize, targetSize);
return canvas;
}
It’s important to know that MobileNet expects 224×224 pixel inputs from us. The library handles resizing internally, but preprocessing it ourselves gives us more control over how images are scaled and cropped before sending them.
Running a Simple Image Recognition Model
With TensorFlow.js and MobileNet loaded, classification only takes a few lines:
let model = null;
async function initModel() {
model = await mobilenet.load();
console.log('Model loaded');
}
async function classifyImage(imgElement) {
if (!model) await initModel();
const predictions = await model.classify(imgElement);
return predictions;
}
The classify() method returns an array of predictions that are sorted by confidence level:
[
{ className: 'golden retriever', probability: 0.89 },
{ className: 'Labrador retriever', probability: 0.05 },
{ className: 'cocker spaniel', probability: 0.02 }
]
Then we can filter by the confidence threshold to show only strong predictions:
function filterPredictions(predictions, threshold = 0.5) {
return predictions.filter(p => p.probability >= threshold);
}
In this case, we only need to load the model once when the page opens up, and then we can reuse it for all our classifications. Loading the model takes a few seconds on page load, but after that the predictions run in milliseconds.
Uploading and Managing Assets with Cloudinary
For production apps, we should store images in Cloudinary instead of relying on local files. This is because it gives us reliable URLs and automatic optimization, and we can use Cloudinary’s transformation features.
The Upload Widget takes care of file selection and uploads it all in one step:
<script src="https://upload-widget.cloudinary.com/global/all.js"></script>
const widget = cloudinary.createUploadWidget({
cloudName: 'your-cloud-name',
uploadPreset: 'your-preset'
}, (error, result) => {
if (result.event === 'success') {
const imageUrl = result.info.secure_url;
classifyCloudinaryImage(imageUrl);
}
});
Once it has uploaded, the images get permanent URLs that we can use for classification:
async function classifyCloudinaryImage(url) {
const img = await loadImage(url);
const predictions = await classifyImage(img);
displayResults(predictions);
}
Using Cloudinary Transformations to Improve Recognition Accuracy
Recognition models usually do better with consistent inputs, and using Cloudinary’s URL transformations let us standardize images before they reach the model.
function getOptimizedUrl(baseUrl, publicId) {
return `https://res.cloudinary.com/${cloudName}/image/upload/w_224,h_224,c_fill,q_auto,f_auto/${publicId}`;
}
The Main transformations are:
w_224,h_224,c_fill: Resize to model’s expected dimensionsq_auto: Optimize quality without artifactsf_auto: Best format for the browsere_improve: Auto-enhance lighting and contrast
The c_fill crop mode makes sure that images meet the right image dimensions without adding in any distortion. For centered objects that need to be identified, g_auto gives us smart cropping.
These transformations happen on Cloudinary’s servers, so the browser can download images that have already been optimized. This is a win for us because it reduces bandwidth usage and gives every image a chance to hit the model in the same format.
Integrating Recognition Results into Web Interfaces
Now let’s connect everything together with a simple UI that shows predictions as they happen:
async function handleImageUpload(file) {
// Show loading state
showStatus('Analyzing...');
// Create preview
const preview = URL.createObjectURL(file);
displayPreview(preview);
// Load and classify
const img = await loadImage(preview);
const predictions = await classifyImage(img);
// Show results
displayResults(predictions);
URL.revokeObjectURL(preview);
}
function displayResults(predictions) {
const container = document.getElementById('results');
container.innerHTML = predictions
.filter(p => p.probability > 0.1)
.map(p => `
<div class="prediction">
<span class="label">${p.className}</span>
<span class="confidence">${(p.probability * 100).toFixed(1)}%</span>
</div>
`).join('');
}
For better performance and user experience, we can run the classification in a non-blocking way and have it show a loading bar or spinner. The model runs on our local GPU with WebGL, so it won’t freeze up the UI – and users should see feedback right away.
How Cloudinary Supports Complete Image Recognition Pipelines
Cloudinary also has server side recognition with its AI add-ons. These add-ons tag images automatically when they are uploaded, and it works great for our browser approach when we need to use persistent metadata.
Enable auto-tagging in an upload preset:
// Upload with automatic AI tagging
const formData = new FormData();
formData.append('file', file);
formData.append('upload_preset', 'auto-tag-preset');
const response = await fetch(
`https://api.cloudinary.com/v1_1/${cloudName}/image/upload`,
{ method: 'POST', body: formData }
);
const data = await response.json();
console.log('Tags:', data.tags);
The upload preset lets us configure which AI service to use (Google Vision, Amazon Rekognition, or Imagga) and the confidence threshold for tags. This runs server-side, so the results should be consistent no matter what the user’s device is.
The hybrid approach works well because we can use browser-based recognition for instant feedback, and then get Cloudinary’s AI to handle permanent tagging when images are uploaded for storage.
Wrapping Up
JavaScript image recognition has gotten to the point where we can run real neural networks in the browser without much setup. TensorFlow.js and MobileNet handle the complicated machine learning and we can focus on the user experience. The examples in the demo code are quite basic, but it should be enough for us to implement a working solution for ourselves.
Cloudinary fits very well into this workflow because its transformations let us standardize inputs for better accuracy, and using the Upload Widget makes file handling very straightforward. Cloudinary’s AI add-ons give us server side recognition when we need to use persistent tags, which saves us a ton of extra work.
Empower your development team with Cloudinary’s easy-to-use APIs and SDKs. Sign up for free today!
Frequently Asked Questions
How accurate is browser-based image recognition?
MobileNet achieves around 70% top 1 accuracy on ImageNet, meaning it correctly identifies the primary subject about 70% of the time. For top 5 accuracy (It gets the correct answer in the top five predictions), it’s over 89%. Real-world performance depends on image quality and lighting as well as how similar subjects are to the training data.
Can I train custom models for JavaScript image recognition?
Yes. TensorFlow.js supports transfer learning, where we take MobileNet’s learned features and train a new classifier on top for our specific categories. This needs less data than training from scratch and we’ll usually need just a few dozen examples per category to work well for specific use cases.
Does image recognition work on mobile browsers?
It does, but performance will vary from device to device. Newer smartphones handle MobileNet classification in under a second, but for older devices we can reduce the model size using the alpha parameter. This trades accuracy for speed, or we can fall back to server side recognition with Cloudinary’s AI add-ons.
