# Shelf recognition that learns new products without retraining

- Author: Ajish Pradeep (https://ajishpradeep.com/about/)
- Canonical: https://ajishpradeep.com/work/planogram-vision-ai/

> One model finds products on the shelf and a second identifies them by comparing vectors, so the catalogue can grow without touching either model. Presented at NVIDIA GTC 2025.

- Organisation: President Information Corp
- Period: 2023 to 2025
- Role: Lead engineer: architecture, training, production deployment
- Domain: Computer Vision, Embeddings, Edge Deployment
- Stack: YOLO, Triplet loss, TensorRT, DeepStream, TAO Toolkit, GCP, Metropolis

## Results

- 7,000+: 7-ELEVEN stores running it (Scientific Reports, 2025)
- +30%: recognition of products in a customer’s hand
- Store #8: Taiwan’s 8th unmanned 7-Eleven, deployed

## The problem

A store's product range changes constantly. The architecture had to let the catalogue grow without adding a training cycle for every new product.

## Where it is, and what it is

An object detector finds products on the shelf. A separate embedding model, fine-tuned with triplet loss, decides what each product is by finding the closest match in a catalogue of reference vectors. **Adding a product means adding its vectors. No model is retrained.**

Table: Three questions, the part that answers each, and what a new product costs it.

| Question | Component | A new product costs |
|---|---|---|
| Where is a product? | object detector | nothing |
| Which product is it? | fine-tuned triplet-loss embedding, nearest-neighbour lookup | reference vectors, no retraining |
| Does the shelf match the plan? | comparison against the planogram | nothing |

Treating product identity as a lookup, instead of a fixed list of classes, has been the standard approach to open-set product recognition since Tonioni and Di Stefano (2018). It lets the system recognise products it has seen only once or a few times. A fixed classifier would require retraining whenever the catalogue changed, so product identity was implemented as vector lookup.

Reference: Tonioni & Di Stefano, 2018 (product recognition). Product identity as a lookup instead of a fixed list of classes. The approach this system follows.

I fine-tuned the embedding model because general embeddings separated categories but struggled with near-identical flavours and pack sizes.

On top of that, the system compares what is on the shelf with the planned layout (the planogram) and flags the differences. That is the question the business actually asks.

## Designing for hidden products

In a cashier-free store, products are recognised while they are in a customer's hand: partly covered, at any angle, and blurred by movement. **In that setting this is the normal case**, and a test that treats it as rare will overstate how well the system works.

Most of the industry avoids solving this with cameras alone. Amazon's Just Walk Out combines cameras with shelf weight sensors, and the sensors are there for exactly the small, hidden items. The ITRI and 7-Eleven X-STORE in Taiwan went further, arguing that weight sensors and light curtains need less computing than solving it by vision. Doing it with cameras only is harder, and fewer people are working on it.

Table: How different systems handle small, hidden products.

| System | Handles hidden products with |
|---|---|
| Amazon Just Walk Out | cameras combined with shelf weight sensors |
| ITRI / 7-Eleven X-STORE | weight sensors and light curtains, chosen over vision for cost |
| This work | cameras only: custom detection with vector search, +30% |

Custom detection plus vector search in these conditions gave a **30% performance gain**, and the system went into Taiwan's 8th unmanned 7-Eleven store.

## Running it on NVIDIA hardware

Production ran on NVIDIA's stack end to end, built together with NVIDIA: TAO Toolkit for training, DeepStream for video inference, TensorRT for fast execution on the hardware, and Metropolis microservices for cloud and edge deployment.

Related work: an LLM-assisted pipeline on GCP Document AI that pulls product names and details out of product images and cut a lot of manual labelling, and a predictive model over 5M+ data points, with 95% confidence intervals, used for marketing and stock decisions.

## Outcome

- Accepted as a technical poster at NVIDIA GTC 2025.
- Deployed in an unmanned store, running on edge hardware with real customers.
- A catalogue design that took retraining out of day-to-day operations.