Robot-assisted shopping for people with visual impairments

Type of project: Prototype

Disability concerned: Visual impairment

Topics: Autonomy, Participation in cultural life, sports and leisure activities

Status: In progress

The aim is to develop an autonomous robotic shopping cart to assist visually impaired people with their shopping.

Barrier-free shopping will be possible with the VIRAS shopping cart. The autonomous shopping cart is equipped with a stereo camera and integrated voice recognition and synthesis functions. The cart is motorised and moves by itself. In the future, users will be able to be guided autonomously to the products they want. To ensure safety, the cart must avoid people and obstacles in the supermarket. The stereo camera not only maps the supermarket in 3D, it also identifies the products and the user’s hand and guides him or her to the target product using acoustic instructions.

Product recognition is based on AI object recognition. This means there’s no need to scan the product’s barcode, and products can be reliably detected regardless of their orientation. If a shelf is empty, the user can be warned.

A shopping process for blind and visually impaired people could then look like this:

  1. Locating the cart in the supermarket using sound signals.
  2. Verbal input of the shopping list.
  3. Navigation towards the products, avoiding obstacles or people.

4. The cart stops when the product is reached.
5. Product recognition by AI object recognition.
6. Acoustic indication of product position in shelf.
7. User hand recognition.
8. Acoustic indication of the position of the user’s hand in relation to the product.
9. Audible confirmation of the product name when the user has selected the correct product.
10. Navigation to the next product or to the checkout.

Image of the VIRAS shopping cart prototype.
Skip to content