Sky-C2
An AR heads-up display that labels the planes overhead with live flight data. It runs on a phone in a $10 Google Cardboard, and you can ask it about any plane out loud.
Which dot is which plane?
In a control tower, controllers look out the window, often with binoculars, then down at the radar, then at paper flight strips and callsign tables to match them up. Every second of attention counts in that job.
Put the flight info where you’re looking.
We kept watching the traffic over Georgia Tech on its way into Atlanta, the world’s busiest airport. With Sky-C2 you look at a plane and its callsign, route and status show up right next to it.
What the headset shows.
- Compass headband
Every aircraft sits at its real direction on a compass strip across the top, with plane and helicopter icons.
- Flight card
Center a plane and a card shows its callsign, airline, type, route, altitude, climb rate, speed and distance.
- Radar in view
A heading-up mini-map shows the airspace around you and turns as you turn your head.
- Hands-free questions
Say “Hey Grok” and ask about the plane you’re looking at. The answer comes back out loud.
A live view also streams what the wearer sees, overlay included, to another screen, so a teammate or supervisor sees the same picture.
Built on phone sensors and live flight data.
There’s no computer vision in it. The headset knows where it is from GPS and which way it’s facing from the compass and gyroscope, and live ADS-B data says where every aircraft is. That flattens the problem to 2D.
For each plane we compute its bearing from you, then the signed angle δ between that bearing and where you’re looking. A pinhole-camera projection turns δ into a spot on screen, so the icons line up with the camera image.
The voice assistant.
I designed and built the voice assistant, so you can ask about the plane you’re looking at without taking the headset off. Here’s what happens between “Hey Grok” and the answer.
- 01Wake word
Always-on speech recognition in the browser, with fuzzy matching on “Hey Grok” so near-misses still wake it up.
- 02Your question
It waits for 1.3 seconds of silence before sending, so a short pause doesn’t cut you off.
- 03Live context
My FastAPI service builds the prompt from the focused plane’s live ADS-B data and where you’re standing.
- 04Grok answers
grok-4.3 with reasoning off, about 0.9 seconds per answer.
- 05Spoken reply
Grok’s text-to-speech voice reads it out. Start to finish, it’s under three seconds.
Every step has a fallback, so the demo never went silent in front of the judges.
What we ran into.
A phone’s compass reports the direction of its top edge, but in the headset the phone sits sideways and we care about the rear camera. We compute the camera’s direction from the full device rotation, then correct for Atlanta’s roughly 5° magnetic declination.
A phone sitting slightly crooked gave us double vision, and cropping the camera stream made everything look zoomed in. We added in-headset calibration for tilt, lens spacing, shift, zoom and field of view, locked by default so a brushing cheek can’t change it.
Chrome’s speech recognition decides you’re done at the first short pause. I switched to waiting for 1.3 seconds of silence before sending the question.
At one update per second, adsb.lol started rejecting requests, and GPS jitter kept slipping past our cache. We snap positions to a roughly 1 km grid, back off when limited and fall back to the last good data, so the sky never goes empty.
Helicopters without a type code were described as slow-moving planes, and a medical-priority status looked like an emergency. The server now works out each aircraft’s kind and status and passes them to both the HUD and the AI.
Sensors and geometry covered a lot of what I assumed would need computer vision. We also built an indoor demo mode, a fallback for every API and a live view so the judges could see inside the headset. Clear contracts between the three of us meant almost no merge conflicts.
Phone compasses drift, especially indoors, and public ADS-B feeds aren’t certified for real operations. Next would be an AR headset with precise head tracking, a surveyed tower position, the airport’s own surveillance feeds, and labels placed at each plane’s true height in the sky.