AI & ML

Visual Search for Ecommerce: Building Shop-by-Photo in India

iJurug Soft2026-11-145 min read

Visual search for ecommerce lets a shopper upload or snap a photo and instantly see similar products from your catalog. It works by detecting the item in the image, turning it into a numeric embedding with a vision model, and finding the nearest catalog items in a vector index, filtered by stock, size and merchandising rules you control.

If you run a store and want to know whether shop-by-photo suits your catalog, share your category mix and platform with us for a practical assessment.

Why shoppers search with images

Text search assumes the shopper knows the right words. Many do not. Someone who saw a block-printed kurta at a wedding, a brass lamp on Instagram or a sofa in a cafe cannot easily describe the pattern, weave or silhouette in a search box, especially when product names mix English, Hindi and regional terms. A photo carries all of that at once.

Visual search works best in categories where appearance drives the decision:

It adds far less for commodity items where specification, not look, decides the purchase.

How visual search for ecommerce works under the hood

Step one: find the product in the photo

User photos are messy: a person wearing a dupatta and a handbag, a living room with five objects. An object detection model locates candidate items and lets the shopper tap the one they mean, so the search is not confused by background.

Step two: turn images into embeddings

A vision model converts each image into a vector that captures shape, texture, pattern and color. General-purpose image-text models give a strong starting point, but fine-tuning on your own catalog usually improves results noticeably in niche categories such as Indian textiles, where subtle differences between weaves or borders matter to buyers.

Step three: search the vector index

Catalog embeddings are stored in a vector database or search engine with approximate nearest-neighbor indexing, so similar items return in milliseconds even across large catalogs. Our guide to vector database consulting covers choosing between dedicated vector stores and search engines with vector support.

Step four: apply business rules

Raw similarity is not enough. Results should hide out-of-stock items, respect size availability and delivery pincode where relevant, and allow merchandising boosts. Combining image similarity with text filters, a hybrid approach, produces results that are both visually close and actually purchasable.

Your catalog is the foundation

Visual search can only be as good as the product images behind it. Before any model work, review:

Features worth considering beyond photo upload

Shop the look and similar items

The same embeddings power "more like this" carousels on product pages and "complete the look" suggestions, which often reach more shoppers than the camera button itself.

Duplicate and counterfeit detection

Marketplaces can use image similarity to spot duplicate listings or suspicious copies of branded products.

Combined text and image queries

Shoppers can upload a photo and add "in blue" or "cotton only", which combines a visual match with a text constraint. This is close to the semantic search approaches in our article on RAG product search for retail.

Measuring whether it works

Decide success metrics before launch. Offline, measure how often a relevant product appears in the top results for a test set of real shopper photos. Online, run an A/B test and track search-to-product-view, add-to-cart and conversion for visual search users compared with a control group. Watch for zero-result and low-engagement queries; they show which categories or photo types need more training data. Visual search also generates a useful retail analytics signal: what shoppers photograph tells you about demand you may not stock yet.

Building it with iJurug Soft

iJurug Soft builds visual search as part of our AI and machine learning services: detection and embedding models, vector search infrastructure, the APIs your web and mobile apps call, and the pipelines that keep the index fresh. It fits custom storefronts, headless setups and platforms such as Shopify through apps or APIs. Senior engineers lead every engagement, and our fixed-milestone process means we start by testing the approach on a sample of your catalog before a full build. See also our view on AI for retail in India for related use cases.

Frequently asked questions

There is no strict minimum, but it pays off most when the catalog is too large or too visual for shoppers to browse comfortably, typically fashion, decor and marketplace stores.

Can it run inside our mobile app?

Yes. The app captures or uploads the photo, can run light detection on the device, and calls a search API. Results appear as a normal product list.

Do we need to retrain models when new products arrive?

No. New products only need to be embedded and added to the index. Models are fine-tuned periodically when categories change or results show gaps.

How is shopper photo data handled?

Photos should be processed for the search, stored only if your policy allows, and covered in your privacy notice. Designing retention rules into the system from the start keeps this simple.

Every catalog and platform is different, so we do not publish prices. Tell us about your store, categories and tech stack through our contact form or write to info@ijurugsoft.com, and we will come back with a pilot plan and a clear quote.