Ajish Pradeep

WorkRetailCase file 04

Shelf recognition that learns new products without retraining

One model finds products on the shelf and a second identifies them by comparing vectors, so the catalogue can grow without touching either model. Presented at NVIDIA GTC 2025.

Where
President Information Corp, 2023 — 2025
My part
Lead engineer: architecture, training, production deployment
Built with
YOLO, Triplet loss, TensorRT, DeepStream, TAO Toolkit, GCP, Metropolis

A store's product range changes constantly. The architecture had to let the catalogue grow without adding a training cycle for every new product.

7,000+
7-ELEVEN stores running it (Scientific Reports, 2025)
+30%
recognition of products in a customer’s hand
Store #8
Taiwan’s 8th unmanned 7-Eleven, deployed

Where it is, and what it is

An object detector finds products on the shelf. A separate embedding model, fine-tuned with triplet loss, decides what each product is by finding the closest match in a catalogue of reference vectors. Adding a product means adding its vectors. No model is retrained.

QuestionComponentA new product costs
Where is a product?object detectornothing
Which product is it?fine-tuned triplet-loss embedding, nearest-neighbour lookupreference vectors, no retraining
Does the shelf match the plan?comparison against the planogramnothing
Three questions, the part that answers each, and what a new product costs it.

Treating product identity as a lookup, instead of a fixed list of classes, has been the standard approach to open-set product recognition since Tonioni and Di Stefano (2018). It lets the system recognise products it has seen only once or a few times. A fixed classifier would require retraining whenever the catalogue changed, so product identity was implemented as vector lookup.

I fine-tuned the embedding model because general embeddings separated categories but struggled with near-identical flavours and pack sizes.

On top of that, the system compares what is on the shelf with the planned layout (the planogram) and flags the differences. That is the question the business actually asks.

Designing for hidden products

In a cashier-free store, products are recognised while they are in a customer's hand: partly covered, at any angle, and blurred by movement. In that setting this is the normal case, and a test that treats it as rare will overstate how well the system works.

Most of the industry avoids solving this with cameras alone. Amazon's Just Walk Out combines cameras with shelf weight sensors, and the sensors are there for exactly the small, hidden items. The ITRI and 7-Eleven X-STORE in Taiwan went further, arguing that weight sensors and light curtains need less computing than solving it by vision. Doing it with cameras only is harder, and fewer people are working on it.

SystemHandles hidden products with
Amazon Just Walk Outcameras combined with shelf weight sensors
ITRI / 7-Eleven X-STOREweight sensors and light curtains, chosen over vision for cost
This workcameras only: custom detection with vector search, +30%
How different systems handle small, hidden products.

Custom detection plus vector search in these conditions gave a 30% performance gain, and the system went into Taiwan's 8th unmanned 7-Eleven store.

Running it on NVIDIA hardware

Production ran on NVIDIA's stack end to end, built together with NVIDIA: TAO Toolkit for training, DeepStream for video inference, TensorRT for fast execution on the hardware, and Metropolis microservices for cloud and edge deployment.

Related work: an LLM-assisted pipeline on GCP Document AI that pulls product names and details out of product images and cut a lot of manual labelling, and a predictive model over 5M+ data points, with 95% confidence intervals, used for marketing and stock decisions.

What came out of it

  • Accepted as a technical poster at NVIDIA GTC 2025.
  • Deployed in an unmanned store, running on edge hardware with real customers.
  • A catalogue design that took retraining out of day-to-day operations.