Edge AI Architecture for Wearable Smart Glasses with Real-Time Multimodal Processing

meta_/_ray_ban · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Wearable edge AI for smart glasses

Meta Reality Labs needed Ray-Ban Meta glasses to provide always-available AI assistance close to the user's perception while fitting severe wearable constraints around battery, thermals, connectivity, latency, reliability, and privacy.

Constraints:

The glasses must be light enough to wear and distribute weight across the frame.
Battery size and chemistry constrain peak power, total power, and simultaneous workloads.
Some commands such as photo capture must run quickly and reliably before the moment passes.
Advanced AI requests may require media, phone connectivity, and server round trips.
The system handles sensitive contacts, messages, photos, audio, and visual context.

Architecture

Meta uses a four-part architecture: case and glasses hardware, the phone, the Meta AI app, and Meta servers. The glasses include cameras, microphones, buttons, a microcontroller, SOC, and accelerators. The microcontroller keeps wake-word detection running in a power-efficient way, while the SOC runs product logic and models directly on the frames for quick offline commands. The phone connects over Bluetooth/Wi-Fi and relays to Meta servers over LTE or Wi-Fi for heavier AI responses.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  glasses-hardware[Glasses hardware] --> wake-word-microcontroller[Wake word microcontroller]
  glasses-hardware --> on-frame-soc[On-frame SOC]
  on-frame-soc --> phone-and-meta-ai-app[Phone and Meta AI app]
  phone-and-meta-ai-app --> meta-servers[Meta servers]
  charging-case[Charging case] --> glasses-hardware

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Glasses hardware`	Cameras, microphones, buttons, speakers, battery, microcontroller, SOC, and accelerators in the frames.	`Phone and Meta AI app`, `Wake word microcontroller`, `On-frame SOC`
`Wake word microcontroller`	Always-on, power-efficient wake word and short-command trigger.	`On-frame SOC`
`On-frame SOC`	Runs product logic and local models for offline or low-latency commands.	`Phone and Meta AI app`
`Phone and Meta AI app`	Connectivity bridge and app layer connected to the glasses over Bluetooth and Wi-Fi.	`Glasses hardware`, `Meta servers`
`Meta servers`	Cloud AI services for advanced responses requiring media, network, and model hops.	`Phone and Meta AI app`
`Charging case`	Provides additional charge capacity for the glasses away from wall power.	`Glasses hardware`

Data flow

A user speaks or interacts with the glasses. The microcontroller detects the wake word, local hardware captures audio or visual context, and the SOC handles fast on-frame commands when possible. For richer AI, the glasses send context through the phone and Meta AI app to Meta servers, then return a response while preserving privacy and maintaining latency/reliability targets.

“our microcontroller allows us to respond to a wake word in a power efficient manner.”transcript, stated

“And then we have our Meta servers which are connected to our phone via LT or Wi-Fi.”transcript, stated

Why this architecture

The architecture splits work by latency, power, privacy, and compute needs. Wake words and short commands stay on the glasses for speed and availability; advanced multimodal tasks use the phone and cloud. This preserves the wearable form factor while letting the assistant see, hear, and respond in real-world context.

“We use a fourpart architecture. We have our case and glasses. We have the phone which is connected via Bluetooth and Wi-Fi to our glasses”transcript, stated

“because we want to process some commands offline and really quickly on device, we have compute for our product logic.”transcript, stated

Models used: Meta AI · wake word model · on-frame models · cloud AI models

Scale

The glasses have a 4-hour battery in the frames.
“We have a 4hour battery on our frames which allows us to be able to take photos or walk around and listen to music or take phone calls.”transcript, stated
The case provides an extra 36 hours of charge.
“And then we also have an extra 36 hours available of charge in the case.”transcript, stated
Advanced AI responses average under 3 seconds despite phone and server hops.
“we can still respond on average in under 3 seconds.”transcript, stated
Certain commands need reliability close to 100%.
“you don't want the reliability of certain commands to not be close to 100%. Because then the user can't depend on these actions being taken.”transcript, stated

Alternatives considered (roads not taken)

Handle all AI work in the cloud.

Rejected because: Short commands like photo capture need to respond very quickly and reliably without waiting for multiple hops.

“take a photo and other short commands really really quickly and super super reliably.”transcript, stated

Run every workload simultaneously on the glasses.

Rejected because: Battery chemistry and peak power limit how many things can run at the same time.

“your peak power is something interesting because it's how many things you can do at the same time.”transcript, stated

Patterns

`latency-optimization` (operations)

Wake-word detection and short commands run on-device for low-latency responses, while cloud paths handle advanced requests.

“So we can respond very quickly to users commands. For example, really quickly to take a photo or to take a video or to call someone.”transcript, stated

`model-routing` (model-operations)

The system routes between microcontroller, SOC, accelerators, phone, and cloud based on workload needs.

“we run our models and we run our logic directly on the frames itself. We also have accelerators for certain use cases”transcript, stated

`privacy-preserving-processing` (governance)

The design explicitly treats contacts, messages, photos, and user sensory context as privacy-sensitive.

“we have to make sure that we preserve user privacy wherever we can in order to make the the the device useful in many scenarios.”transcript, stated

Tradeoffs

On-device versus cloud execution

Chose Hybrid edge-phone-cloud architecture. over Pure cloud AI or pure on-device AI..

Rationale: The hybrid architecture balances quick short commands against heavier advanced AI responses.

“advanced AI responses which require multiple hops through the phone and through our servers, sending media and dealing all sorts of network connectivity”transcript, stated

Always-on wake word

Chose Power-efficient microcontroller wake-word path. over Waking heavier compute for all listening..

Rationale: The microcontroller keeps the assistant available while preserving wearable battery.

“we can have the wake word running the entire time that the user is wearing it.”transcript, stated

Gotchas (what bit them)

Wearable peak power limits concurrent AI actions.

Root cause: Battery size and chemistry constrain both total and peak power in a lightweight frame.

Resolution: Meta splits workloads across microcontroller, SOC, accelerators, phone, and cloud.

“Depending on the battery chemistry and the outside temperatures and other things, we might be able to pull more or less power”transcript, stated

Latency is user-visible in wearable capture moments.

Root cause: A slow photo command can miss the moment the user wanted to capture.

Resolution: Short commands are kept quick and reliable on the device path.

“you don't want take a photo to take too long cuz the moment might be over.”transcript, stated

Expanding markets and live translation increase workload contention.

Root cause: More markets and offline translation add competing model workloads and model-miss risk.

Resolution: Meta flags agents, live translation, and market expansion as future reliability and power challenges.

“We have more competing workloads and we have more potential for model misses as we move to different markets”transcript, stated

Evaluation

The talk evaluates the system through latency, reliability, battery/charge capacity, and qualitative wearable constraints rather than a formal offline benchmark.

Metrics: 4-hour frame battery · 36-hour charging case capacity · Advanced AI responses under 3 seconds on average · Reliability target close to 100% for certain commands

“we can still respond on average in under 3 seconds.”transcript, stated

“We're also able to provide these kind of commands with really high reliability, which makes them useful in all scenarios”transcript, stated

Outcomes

Delivered Meta AI capabilities on Ray-Ban Meta smart glasses.
Enabled quick wake-word, photo, video, call, and advanced AI interactions.
Supported advanced AI responses averaging under 3 seconds.
Established a four-part wearable AI architecture balancing edge, phone, and cloud.

The case reports 4 hours of frame battery, 36 hours in the case, under-3-second average advanced AI responses, and expansion to markets including India and Mexico.

“enabling features like visual question answering, photo capture, and voice commands with sub-second response times for on-device operations and under 3-second response times”summary, stated