Teams today juggle massive libraries of product shots, user photos, and marketing visuals. When you need to find the right picture fast, AI-powered image search can feel like magic.
Hi folks,
I want to add visual search to my app so users can type a description or upload an example image and get similar results. In short, how to use an AI-powered image search tool end to end?
What are the key components, how do I build the index, what should I watch out for around accuracy and performance, and how can I integrate this with an image management and delivery workflow? Thanks!
AI image search typically has two query modes: text to image and image to image. Under the hood, both rely on computing vector embeddings for your images, then using approximate nearest neighbor search to retrieve the closest matches. Here is a straightforward path you can run locally, then extend for production.
- Embed: Use a vision-language model such as CLIP to convert each image to a vector.
- Index: Store those vectors in a fast nearest neighbor index like FAISS.
- Query: Convert an input text prompt or image to a vector, then search the index.
- Filter and rank: Apply metadata filters, re-rank results, and return URLs plus scores.
The example below uses sentence-transformers with a CLIP model and FAISS. It handles both text and image queries.
# pip install sentence-transformers faiss-cpu pillow
import os
import numpy as np
import faiss
from PIL import Image
from sentence_transformers import SentenceTransformer
# Load a CLIP model that can embed images and text
model = SentenceTransformer("clip-ViT-B-32")
# 1) Build your image corpus
image_dir = "images"
paths = [os.path.join(image_dir, f) for f in os.listdir(image_dir) if f.lower().endswith((".jpg", ".jpeg", ".png"))]
def embed_image(path):
img = Image.open(path).convert("RGB")
vec = model.encode([img], normalize_embeddings=True)
return vec[0].astype("float32")
embeddings = np.stack([embed_image(p) for p in paths]) # shape: (N, D)
# 2) Create a FAISS index (inner product on normalized vectors is cosine similarity)
index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings)
# 3a) Text-to-image search
def search_by_text(text, k=5):
q = model.encode([text], normalize_embeddings=True).astype("float32")
scores, idxs = index.search(q, k)
return [(paths[i], float(scores[0][j])) for j, i in enumerate(idxs[0])]
# 3b) Image-to-image search
def search_by_image(query_path, k=5):
q = embed_image(query_path).reshape(1, -1)
scores, idxs = index.search(q, k)
return [(paths[i], float(scores[0][j])) for j, i in enumerate(idxs[0])]
# Example:
print(search_by_text("red running shoes", k=3))
# print(search_by_image("query.jpg", k=3))Code language: PHP (php)- Preprocessing: Normalize sizes and color space before embedding. For Python-centric pipelines, this roundup is handy: 6 ways to save images in Python.
- Index choice: For millions of images, use an approximate index like FAISS IVF or HNSW. Start with exact search for simplicity, then switch to ANN when needed.
- Filtering: Keep structured metadata alongside vectors. Filter by product category, brand, or user permissions before final ranking.
- Re-ranking: After the vector search, re-rank on business signals like clicks, sales, or quality scores for better relevance.
- Duplicates: Consider perceptual hashing to detect near-duplicates during ingestion.
- Delivery: Store canonical image URLs and thumbnails for fast rendering.
- Use a labeled test set of query prompts and image sets to track precision at K and recall.
- Check robustness with varied lighting, angles, and backgrounds.
- Strip sensitive EXIF when appropriate during ingestion.
- Gate access to private vectors and images with auth checks before returning URLs.
You can manage your media assets, automate tagging, and deliver optimized results at scale with Cloudinary. In the Media Library, visual search can surface similar images for curators. From code, the Search API lets you fetch images by tags or other attributes that your pipeline assigns, then deliver responsive thumbnails from the same URLs.
// npm i cloudinary
const cloudinary = require("cloudinary").v2;
cloudinary.config({
cloud_name: "YOUR_CLOUD",
api_key: "YOUR_KEY",
api_secret: "YOUR_SECRET"
});
// Find images auto-tagged or labeled as "sneaker"
async function findSneakers() {
const res = await cloudinary.search
.expression('resource_type:image AND tags=sneaker')
.with_field("tags")
.sort_by("created_at", "desc")
.max_results(30)
.execute();
// Build fast thumbnails via URL-based transformations
return res.resources.map(r => ({
id: r.public_id,
thumb: `https://res.cloudinary.com/${cloudinary.config().cloud_name}/image/upload/c_fill,w_300,h_300/${r.public_id}.${r.format}`
}));
}
findSneakers().then(console.log).catch(console.error);Code language: JavaScript (javascript)This complements your vector search: use your ANN index to decide which public_ids to show, then retrieve and transform those assets on the fly for a fast UI. Cloud platforms also help centralize storage, CDN delivery, and governance so your search results render quickly and consistently across devices.
- Compute embeddings for your catalog with a vision-language model, index with FAISS, and query by text or image.
- Add metadata filters and re-ranking to improve relevance and business outcomes.
- Manage ingestion, delivery, and thumbnails with a robust hosting workflow to keep results blazing fast.
- Cloudinary can store, tag, search, and deliver results at scale, alongside your custom vector search stack.
Ready to build smart visual discovery and deliver results fast? Create your free Cloudinary account and start managing, transforming, and searching your images at scale.