What we learned putting AI glasses on a robot bench
Most AI-glasses demos end with an answer in your ear. We're interested in the case where the answer has to become a record: a service order, a purchase order, something with an id that somebody closes later.
That combination is what we're trying to learn. Glasses as a sensor, pointed at a physical machine, writing into an operational system. So we built the whole loop on our own robot arm, on a fault we'd already lived through, and paid attention to what was harder than expected.
The fault that started it
In June we brought up a six-axis arm with a gripper: seven motors on a CAN bus, a 48 V supply, a USB-CAN adapter running classic CAN at 1 Mbps. The first scan found one motor.
We spent most of a day on it. We checked the bitrate, checked termination, power-cycled and re-scanned, read the adapter's error counters, and walked the motor IDs one at a time.
It was a cable. The inter-joint link after joint 1 had come loose somewhere in shipping. We reseated the connector and all seven motors came back.
What we built
A demo of the whole loop, as a canvas in the scene that holds the arm. It's embedded below: the real canvas from the scene, not a screenshot of one.
Five steps: inspect, stream, service order, guided repair, fixed. The inspection and repair footage was shot through a pair of AI glasses, a technician walking the bench hands free. It ends with a closed ticket in the scene's log.
A scene, if you haven't met one here before: every workflow becomes a scene — its data, its tools, the agent that runs it, and the history of everything it did.
What runs and what doesn't:
| Step | Status |
|---|---|
| The fault it walks through | Real. It's the cable from June |
| The inspection and repair footage | Real, shot through the glasses. Recorded clips, replayed, not a live stream |
| Telemetry in the stream and verify steps | Real. Live HTTP reads against the arm: armed, bus voltage, joint angles |
| The service order | Real. The job writes it into the scene's ticket log, with the telemetry values it read at that moment |
| Closing the ticket | Real. Written against the same ticket id |
| The anomaly flag | Scripted. It fires seven seconds into the clip, on a timer. No model is looking at the frame |
| The repair instructions | Scripted. The button reveals four steps we wrote by hand |
Everything that moves data between the arm, the job and the scene works. Noticing the fault and knowing the fix are the two we staged.
The machine's telemetry and the glasses see different things
The arm serves its state over HTTP: armed, bus voltage, each joint's angle and limits. Nothing was wrong with that endpoint during the fault. It reported armed: false and listed the joints it could still reach. Neither of those says anything about a connector.
A connector that is most of the way seated has no sensor on it. From downstream, a loose cable and a dead motor look identical. Someone standing at the bench can tell them apart, because the connector is visibly hanging off the joint.
The demo ended up using both for different jobs. The glasses are what notices. The telemetry is what proves the fix, because seven motors back online is a measurement, and that's what closes the ticket.
Hands-busy is the constraint that shapes the interface
During the repair both hands are on the connector. That rules out most of what you'd normally build.
What survived is very small: one step in view at a time, one line of text, advancing on voice. No 3D tracking, nothing projected onto the cable, no menu. We started out thinking of it as an AR overlay and ended up with something closer to a teleprompter.
Keeping the list to four steps mattered more than we expected. A checklist of twenty on a head-mounted display would be worse than the same checklist on paper.
The glasses only see what the wearer looked at
The footage is head-mounted POV: wide, moving, pointed wherever the technician's attention was. That's a different input than a fixed inspection camera.
The wearer walks around the machine, so occlusion mostly solves itself. A fault that's only visible from one angle stays invisible until somebody happens to face it.
We expect that to be the dominant failure mode and we haven't measured it yet. It's the first thing the detector test below has to answer.
The latency budget is much looser than we assumed
We already run detectors on the same node: face and person detection over the camera feed, on downscaled frames, at about 11 fps and 5-6 fps. Fast, and completely useless here, because neither has any concept of a seated connector.
The thing that could see the cable is a vision model, and a model call is orders of magnitude slower. Then we walked the actual task and it stopped mattering. An inspection pass takes a minute of somebody walking around a bench. A frame every few seconds is plenty.
That's specific to inspection. Anything in a control loop has the opposite budget, and glasses would be the wrong instrument for it.
The output has to be a record somebody closes
The easy version of this sends a notification. We think that's the version that gets ignored, so the flag opens a ticket instead.
The entry carries the fault, the severity, and the telemetry the job read at that moment: armed=false, vbus=47.9V. Those are the values the arm returned when the job asked for them. The log is a document in the scene, so the next job can read it, a technician can open it on a phone, and it's still there the next time this connector works loose.
The parts side works the same way. The other half of the demo points the glasses at a kit before assembly and counts it against the bill of materials. It found six of the eight corner brackets the station needed, and the shortage opened a procurement order with that count logged against it.
The repair knowledge was already ours
The four steps we hardcoded were copied out of our own bring-up note, in the same scene, written the day we found the cable. It reads: the arm read 1/7 online until a loose inter-joint cable after joint 1 was reconnected.
One of the four is the step that matters. Before touching the arm, confirm the joints are holding on gear friction. Disabled joints keep their pose through the gearing, so nothing drops when you cut power. We learned that on this arm, and no manual we had would have told us.
So the "generate the repair steps" half isn't a research problem. The document to generate them from already existed, in the same place the ticket lands, because we wrote our debugging down. Whatever we build there starts as retrieval.
What's next is the detector
A frame every few seconds to a vision model, with one question: is anything wrong with this assembly. Everything else in the loop already runs.
Whether that works is unknown, and rerunning the fault we already found wouldn't answer it. Back the connector out, add a handful of faults nobody has written down, run the passes blind, report the hit rate and the misses.
We'll post those numbers when we have them.
Why this is where we're spending time
A camera doesn't need the equipment to cooperate. Most of what sits on a shop floor will never expose an interface to read, so the input has to work without one.
Where the record lands is the easier half. Here it's a ticket in the scene next to the arm; elsewhere it's whatever system that team already runs.