Augmented reality and AI: when the camera understands what it sees

For years, augmented reality was mainly good at one thing: understanding the geometry of a room. The phone detects the floor, a table, a wall, and you place a virtual object that stays put as you move around. That is already useful for seeing a piece of furniture or a machine at full scale. But the phone did not know what it was looking at. A table was just a horizontal plane, an industrial pump just another volume.

That is what has been shifting over the past two years. So-called multimodal AI models, which handle images, text and voice together, give the camera a kind of understanding. The device no longer sees only surfaces: it identifies objects, reads text and can answer a spoken question about what is in front of it.

From geometry to meaning

Put simply, classic augmented reality answers the question "where?". Where is the floor, where should the object go, how do we keep it stable. AI adds the question "what?". Which machine is in front of me, which part is faulty, which button should be pressed now.

This combination opens up uses that used to require markers stuck everywhere or 3D models prepared in advance for each piece of equipment. A technician can point their phone at an electrical panel and see the right diagram appear. A museum visitor can look at a work of art and get an explanation suited to their age. A salesperson can show a product and let the customer ask questions freely.

Smart glasses are back. For good this time?

The hardware is following. After high-end mixed reality headsets, smart glasses have made a notable comeback, with consumer models that can film, listen and, in the latest ones, display information right in the lens. The major platforms all have a glasses project or an operating system built for this format.

It pays to stay realistic: these glasses still show very little, battery life is short, and using them at work raises privacy questions that cannot be ignored. For a project launched today, smartphones and tablets remain the safest choice, and web-based augmented reality, with no app to install, is still the easiest way to reach a wide audience. Glasses are worth watching closely, with content designed so it can move to them later.

Where the combination already works

• Maintenance and field work: the device recognises the equipment, shows the procedure step by step in the right place and can check that a step has been done.
• On-the-job training: learners handle the real equipment, guided by visual cues, with an assistant that answers their questions without them putting down their tools.
• Visits and cultural mediation: the artwork or site is recognised without a QR code, and the content adapts to the visitor.
• Sales and trade shows: the product is shown at full scale and customers can question it, extending the sales pitch.

The precautions that make the difference

An assistant that gets a maintenance procedure wrong is not a minor issue. For serious use, we do not let a general-purpose model make up its answers: it relies on the company's actual documentation, using a RAG approach that retrieves information from approved sources and shows where it comes from. Critical steps are still written by experts. The AI's job is to recognise the situation and find the right content.

Real-world conditions matter too. A poorly lit workshop, no network at the back of a building, gloves that make the touchscreen useless: these are the constraints that decide success, far more than the demo in the meeting room. We test them on site before signing off on an approach.

In short

Augmented reality changes in nature when AI lets it understand what the camera sees. We move from objects placed in space to help that recognises a situation and responds to it. Glasses will one day make that help feel more natural, but the strongest projects are being built right now on smartphones and tablets, with AI grounded in reliable content and tested in real working conditions.