Building Custom Tracing Tools and Development Infrastructure for AI-Powered Meeting Notes

granola · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-powered meeting notes and development tooling

Granola needed to ship LLM-based meeting notes and chat features without treating the model as a black box. Real users exposed web-search cost spikes, provider regressions, role-specific output expectations, and desktop-app testing friction. The team needed internal observability for tool calls, reasoning, and cost, plus a faster way to preview and test AI feature variants despite shipping an Electron app.

Constraints:

Architecture

Granola built custom tracing around its LLM features and separately turned the Electron renderer into a web-deployable shell. The tracing layer wraps the AI SDK, records tool calls, search and reasoning steps, and costs to a database, and exposes a purpose-built frontend so internal users can inspect an agent loop from start to finish without CloudWatch queries. The development-infrastructure layer abstracts Electron IPC/system APIs to fall back to web standards, adapts React routing/session/query layers for web execution, and deploys the renderer online. CI creates a preview link for each PR, and Cursor tests the preview and uploads screenshots to the pull request, allowing multiple variants to be experienced in practice before shipping.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  granola-desktop-app[Granola desktop app] --> audio-capture[Audio capture]
  granola-desktop-app --> manual-notes[Manual notes]
  granola-desktop-app --> llm-meeting-summary[LLM meeting summary]
  granola-desktop-app --> chat-feature[Chat feature]
  granola-desktop-app --> electron-main-process[Electron main process]
  granola-desktop-app --> electron-render-process[Electron render process]
  audio-capture --> real-time-transcription[Real-time transcription]
  real-time-transcription --> llm-meeting-summary
  real-time-transcription --> chat-feature
  manual-notes --> llm-meeting-summary
  llm-meeting-summary --> tracing-wrapper[Tracing wrapper]
  chat-feature --> web-search-tool[Web search tool]
  chat-feature --> tracing-wrapper
  tracing-wrapper --> trace-database[Trace database]
  trace-database --> trace-ui[Trace UI]
  trace-ui --> internal-reviewers[Internal reviewers]
  electron-main-process --> ipc-abstraction[IPC abstraction]
  electron-render-process --> ipc-abstraction
  electron-render-process --> web-shell[Web shell]
  ipc-abstraction --> web-standards-adapters[Web standards adapters]
  web-shell --> ci-preview-deployment[CI preview deployment]
  web-shell --> cursor-test-agent[Cursor test agent]
  cursor-test-agent --> pull-request[Pull request]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Granola desktop app Electron meeting-notes application installed on the user's dock. Audio capture, Manual notes, LLM meeting summary, Chat feature, Electron main process, Electron render process
Audio capture System and microphone audio capture feeding real-time transcription. Real-time transcription
Real-time transcription Meeting transcript generated from audio and combined with user notes. LLM meeting summary, Chat feature
Manual notes User-written notes that steer the final generated summary toward what the user would naturally write. LLM meeting summary
LLM meeting summary LLM feature that generates meeting notes from transcript and manual annotations. Tracing wrapper
Chat feature LLM chat feature for asking questions about current, past, or shared meeting context. Web search tool, Tracing wrapper
Web search tool External search capability whose cost, context impact, and provider behavior need monitoring. —
Tracing wrapper Internal instrumentation layer around AI SDK that captures tool calls, reasoning, search, and cost. Trace database
Trace database Storage for structured trace events used by internal debugging UI. Trace UI
Trace UI Purpose-built frontend for employees to follow agent loops and diagnose failures. Internal reviewers
Internal reviewers Engineers, product managers, data analysts, CX, and founders who use traces for debugging. —
Electron main process Electron process that works with system APIs. IPC abstraction
Electron render process Frontend process that Granola made agnostic to Electron so it can run as a web app. IPC abstraction, Web shell
IPC abstraction Platform abstraction that falls back to web standards outside Electron. Web standards adapters
Web standards adapters Adapted routing, sessions, and query-layer behavior for browser execution. —
Web shell Deployed web version of the Electron frontend used for previews and testing. CI preview deployment, Cursor test agent
CI preview deployment Continuous integration path that creates preview links for pull requests. —
Cursor test agent AI-assisted testing step that opens PR previews, tests changes, and uploads screenshots. Pull request
Pull request Code review artifact containing preview link and AI-generated test screenshots. —

Data flow

During a meeting, Granola captures system audio and microphone audio, produces real-time transcription, and lets users add manual notes. At summary or chat time, the LLM feature may call tools such as web search and reason over meeting context. The tracing wrapper around the AI SDK stores each tool call, reason, search action, and cost in a trace database. Internal users open the trace UI to follow the agent loop from beginning to end and identify the exact failure point. For development, a pull request triggers CI to deploy the Electron renderer as a web shell; reviewers and an AI testing agent open the preview, exercise the change, and attach screenshots back to the PR.

4 supporting quotes
“it has access to your system transcription system audio as well as your microphone audio, which means we have real-time transcription.”transcript, stated
“full visibility over the individual tool calls, why it's making those tool calls, the search tools, the reasoning tools, the cost”transcript, stated
“obviously you can use OpenTelemetry or like other providers. But we essentially just like save things to a DB, wrap around like AI SDK, and then the front-end is like kind of like the most important part cuz that's what people are going to use to figure out”transcript, stated
“now our CI, whenever we open a PR, we get a preview link and we can go and test those things. And this generally sped up our development time so much more. And like the cooler part of this is that because LLMs can now self-verify their work, these guys are”transcript, stated

Why this architecture

Granola chose custom tracing because generic logs and SaaS observability did not match how product, data, CX, founders, and engineers needed to inspect LLM behavior. Owning the trace schema and UI let them move from vague output complaints to exact failure points in agent loops. The web-shell architecture solved a different bottleneck: Electron made variant testing and coworker review slow, while deploying the renderer as a web app gave Granola web-style preview links and made AI-assisted self-verification practical on every PR.

4 supporting quotes
“we want to kind of go very deep into the details and figure out exactly what's going on. So, what we did recently at Cronulla is we started building our own tracing tools. And obviously, thanks to LLMs, you can actually one-shot these things. And this is”transcript, stated
“structured the data exactly how we want it. And the UI is built to like serve our our employees internally, not just like engineers, but also product, data, and like CX, and everyone. So, you don't have to like, you know, go into CloudWatch and do like very”transcript, stated
“the problem for us specifically at Cronulla was that we are a desktop app, which means you can only run one instance of the app at a time. And there was a lot of friction when it came to like testing new features, different variants, and actually testing those in parallel. >> [clears throat] >> So,”transcript, stated
“we took our Electron app and we turned the front-end of the Electron app into a web shell. And this was deployed online. So, now our CI, whenever we open a PR, we get a preview link and we can go and test those things. And this generally sped up our”transcript, stated

Models used: LLMs · AI SDK · Cursor

Scale

Alternatives considered (roads not taken)

Use generic SaaS tracing or CloudWatch queries.

Rejected because: The team wanted a trace schema and UI that matched its own employees' debugging workflows rather than requiring complex log queries.

“you don't have to like, you know, go into CloudWatch and do like very complex queries to figure out why something failed.”transcript, stated
“previously, obviously, building this kind of tools would be like up to using a SaaS provider. And it simply wouldn't you simply wouldn't have the time. But now you actually can spend time building this tracing tool that actually serves what you need. And”transcript, stated

Treat web search as a one-line LLM tool.

Rejected because: Web search introduced token blowups, costs, provider behavior changes, and loss of control that affected user experience.

“examples is is web search. So, web search for most LLM providers looks like a line of code. You simply add the web search tool and you expect it to just work. That's what the labs want you to believe, but once you get into it, there's lots of other complications. So, for example,”transcript, stated
“expect it to just work. That's what the labs want you to believe, but once you get into it, there's lots of other complications. So, for example, the token usage and token cost can bubble up quite a lot, especially for complex queries. It's going to”transcript, stated

Keep Electron-only local testing.

Rejected because: Local Electron setup required dependencies and blocked quick coworker testing, parallel variants, and web-style preview links.

“before you'd have to like run the Electron app locally, install the dependencies, and test things. If you wanted a coworker to to test those changes, you'd have to get them to do those things as well. We We didn't have the same luxuries as like web apps do.”transcript, stated
“wanted a coworker to to test those changes, you'd have to get them to do those things as well. We We didn't have the same luxuries as like web apps do. So, essentially, what we did is we took our Electron app and we turned the front-end of the Electron app into a web shell.”transcript, stated

Re-platform from Electron to Tauri.

Rejected because: Granola had tried Tauri and discussed it, but did not see the performance gains they cared about.

“questions? >> [applause] >> What do you think about re-platforming from Electron to Tauri? We've thought about moving to Tauri a couple times. Um I think the way Electron serves us right now has been super nice. Like the APIs changing quite a quite a lot. We've tried”transcript, stated
“quite a lot. We've tried Tauri before as well and we didn't really see massive performance gains, which we which is what we care about the most. So, yeah, it's been discussed before. We've played around with it, but haven't shipped it. Cool. Thank you, guys. >> [applause] >> Yay! >> [music]”transcript, stated

Patterns

observability-tracing (operations)

Granola records the full LLM interaction lifecycle, including tool calls, reasons, search use, and costs, then exposes it through an internal trace UI.

“complete visibility on the tool calls straight from the beginning to the end.”transcript, stated
“following the agent loop completely front to back to figure out exactly what went wrong.”transcript, stated

tool-use-orchestration (architecture)

The chat feature uses tools such as web search, and tracing makes tool-call behavior inspectable so the team can debug provider and workflow failures.

“You simply add the web search tool and you expect it to just work.”transcript, stated
“individual tool calls, why it's making those tool calls, the search tools, the reasoning tools”transcript, stated

cost-aware-routing (operations)

Granola treats web-search token growth and per-chat cost as first-class trace data because provider tools can make chat interactions economically infeasible at scale.

“So, for example, the token usage and token cost can bubble up quite a lot, especially for complex queries. It's going to blow up your context.”transcript, stated
“It's going to blow up your context. And each chat could be costing you like 10 pence. Obviously, at scale when you have millions of users, this is not really feasible.”transcript, stated

eval-driven-development (evaluation)

The web-shell preview flow lets humans and Cursor test AI-feature changes in practice on every PR before release.

“And like the cooler part of this is that because LLMs can now self-verify their work, these guys are now like once we open a PR, Cursor goes and tests it, uploads a screenshot into our PRs, which speeds up the testing so much more.”transcript, stated
“we actually felt those products in in like in practice rather than just like seeing it in Figma.”transcript, stated

feedback-loop-iteration (operations)

Granola emphasizes tracing and preview infrastructure as feedback loops for iterating with LLMs rather than relying on one-shot prompts.

“So, essentially, this this is basically a long talk to tell you that the answer isn't to one-shot better. It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM.”transcript, stated
“figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM.”transcript, stated

Tradeoffs

Build custom tracing tools.

Chose A purpose-built trace schema and UI wrapped around AI SDK over Generic SaaS tracing or CloudWatch-based debugging.

Rationale: Custom tooling required some engineering work, but LLM-assisted development made it feasible and gave Granola exactly the trace structure and UI its internal teams needed.

“building this kind of tools would be like up to using a SaaS provider.”transcript, stated
“now you actually can spend time building this tracing tool that actually serves what you need.”transcript, stated

Wrap AI SDK and save traces to a database.

Chose A basic internal implementation with custom frontend emphasis over Waiting for a sophisticated observability platform to cover the exact use case.

Rationale: The most important part was the frontend people used for diagnosis; the backend trace capture could remain simple.

“we essentially just like save things to a DB, wrap around like AI SDK”transcript, stated
“the front-end is like kind of like the most important part cuz that's what people are going to use”transcript, stated

Turn the Electron renderer into a web shell.

Chose Abstract IPC and React platform concerns to web standards over Keeping all testing inside the installed desktop app.

Rationale: This allowed browser previews, parallel variant testing, and AI-assisted PR screenshots without abandoning the Electron product.

“we abstracted our IPC APIs, which is the system APIs, to fall back to web standards”transcript, stated
“this just made the render agnostic of of Electron. And we can just simply run it as a web app.”transcript, stated

Stay with Electron rather than ship Tauri.

Chose Electron plus renderer/web-shell abstraction over Re-platforming to Tauri.

Rationale: The team had tried Tauri and discussed it, but did not see the performance gains it cared about.

“Um I think the way Electron serves us right now has been super nice. Like the APIs changing quite a quite a lot.”transcript, stated
“We've tried Tauri before as well and we didn't really see massive performance gains, which we which is what we care about the most.”transcript, stated

Gotchas (what bit them)

A one-shot chat implementation breaks down with real user requests.

Root cause: Users ask for web search, personalized emails, coaching, cross-meeting context, and role-specific outputs that a generic chatbot was not designed to handle.

Resolution: Granola built tracing to understand exact failures and iterated toward use-case-specific behavior.

3 supporting quotes
“So, let's say I built, you know, like a one-shot this chat system. It's very easy to do. And I put it into production in my fake Cronulla app.”transcript, stated
“very common problems that you're going to run into when you make a generic chatbot.”transcript, stated
“molding the LLM to work to your specific use case can be super hard.”transcript, stated

Provider-managed web search can change underneath the product.

Root cause: LLM provider updates can degrade search behavior without exposing enough detail for the application team to diagnose or control.

Resolution: Granola built internal tracing and kept provider switching/control visible rather than treating web search as a black box.

“overnight, they shipped an update and for some reason web search degraded and it was completely out of our control.”transcript, stated
“we generally had no idea like what was going on apart from just like switching providers.”transcript, stated

Generic summaries do not satisfy every role.

Root cause: Sales, engineering, and HR users look for different information structures and downstream actions.

Resolution: Granola uses tracing and iteration to customize behavior instead of assuming one prompt can cover every role.

“someone in sales might expect more of a deal focus. Someone in engineering might expect like action items, blockers, or like linear tickets.”transcript, stated
“And the thing is that one prompt can't generally serve everyone. And you know, LLMs are stubborn and we need to figure out how to get inside them and make them work how we want it to work.”transcript, stated

CloudWatch and generic logs were too high-friction for broad debugging.

Root cause: Non-engineering internal users need to inspect LLM failures, but complex log queries block product, data, CX, and founder workflows.

Resolution: Granola built a custom trace UI structured for internal users.

“not just like engineers, but also product, data, and like CX, and everyone.”transcript, stated
“go into CloudWatch and do like very complex queries to figure out why something failed.”transcript, stated

Electron app testing made AI-feature variants slow to validate.

Root cause: The desktop app could only run one instance, and local testing required dependencies and setup for each reviewer.

Resolution: Granola made the renderer Electron-agnostic, deployed it as a web shell, and generated PR preview links.

3 supporting quotes
“But the problem for us specifically at Cronulla was that we are a desktop app, which means you can only run one instance of the app at a time. And there was a lot of friction when it came to like testing new features, different variants, and actually testing those in parallel.”transcript, stated
“before you'd have to like run the Electron app locally, install the dependencies, and test things.”transcript, stated
“So, essentially, what we did is we took our Electron app and we turned the front-end of the Electron app into a web shell. And this was deployed online.”transcript, stated

Evaluation

Granola evaluates AI features through internal trace inspection, role-specific output debugging, PR preview deployments, AI-assisted screenshot testing, and hands-on variant testing in the web shell. The aim is to identify exact LLM/tool failure points and validate user experience before release.

Metrics: Tool-call sequence visibility · Reasoning/tool-call explanations · Search tool usage · Cost per interaction · Context blowup from web search · PR preview availability · AI-generated screenshot feedback · Variant testing across feature designs · Internal user ability to pinpoint exact failure

3 supporting quotes
“you can actually figure out like, you know, this output feels off to like exactly what failed.”transcript, stated
“Cursor goes and tests it, uploads a screenshot into our PRs, which speeds up the testing so much more.”transcript, stated
“So, this has helped us on top of the LLM improvements was like we were able to just like change and like test like one feature in like multiple different variants. So, like whatever the end product is actually feels super good cuz we know that we've tried so many different variants.”transcript, stated

Outcomes

The case mentions each complex web-search chat costing about 10 pence, millions-of-users infeasibility, one feature appearing differently for four users, one-instance desktop app constraints, preview links on every PR, and Electron's main/render two-process architecture.

4 supporting quotes
“complete visibility on the tool calls straight from the beginning to the end.”transcript, stated
“our founder literally goes into like the details like following the agent loop completely front to back”transcript, stated
“we get a preview link and we can go and test those things. And this generally sped up our development time so much more.”transcript, stated
“So, like whatever the end product is actually feels super good cuz we know that we've tried so many different variants. And we actually felt those products in in like in practice rather than just like seeing it in Figma.”transcript, stated