Building Enterprise AI-Powered Software Engineering Tools with Multi-Modal Agent Architecture

windsurf · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise AI software engineering platforms

Windsurf was building an enterprise-focused AI software engineering platform that could move beyond code generation into the broader work of software engineering. The problem was to deliver useful human-AI collaboration now, while expanding the product surface, data context, retrieval, memory, and custom model stack needed for agents to operate across IDE work, browser research, PR review, documentation, CI/CD, and enterprise-governed data.

Constraints:

Architecture

Windsurf built a layered software-engineering agent platform. At the application layer, a VS Code fork controls the IDE surface, browser previews pass screenshots/logs/UI context into the agent, a Wave 10 browser captures web/Jira/GitHub research context, and PR review brings the agent into code review. At the data layer, Windsurf progressed from open-file context to Smart RAG, Riptide reranking, MCP, direct enterprise integrations, and memory. At the model layer, it built custom autocomplete, Riptide, and SWE-1-style agent models, then evaluated them offline on GitHub PR tasks and online with human code-acceptance signals. The whole strategy is a flywheel: better product surfaces yield user data, user data improves models, and improved models expand what the agent can do.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-developer[Enterprise developer] --> windsurf-ide[Windsurf IDE]
  enterprise-developer --> windsurf-browser[Windsurf browser]
  enterprise-developer --> pr-review-tool[PR review tool]
  windsurf-ide --> browser-previews[Browser previews]
  windsurf-ide --> cascade-agent[Cascade agent]
  windsurf-ide --> code-context[Code context]
  browser-previews --> cascade-agent
  windsurf-browser --> cascade-agent
  windsurf-browser --> enterprise-workflow-context[Enterprise workflow context]
  pr-review-tool --> cascade-agent
  pr-review-tool --> memory-system[Memory system]
  cascade-agent --> retrieval-layer[Retrieval layer]
  cascade-agent --> mcp-connections[MCP connections]
  cascade-agent --> direct-integrations[Direct integrations]
  cascade-agent --> memory-system
  cascade-agent --> custom-model-layer[Custom model layer]
  code-context --> retrieval-layer
  enterprise-workflow-context --> mcp-connections
  enterprise-workflow-context --> direct-integrations
  retrieval-layer --> riptide-reranker[Riptide reranker]
  retrieval-layer --> custom-model-layer
  riptide-reranker --> custom-model-layer
  mcp-connections --> cascade-agent
  direct-integrations --> cascade-agent
  memory-system --> cascade-agent
  custom-model-layer --> offline-pr-benchmark[Offline PR benchmark]
  custom-model-layer --> online-accept-rate-evaluation[Online accept-rate evaluation]
  offline-pr-benchmark --> custom-model-layer
  online-accept-rate-evaluation --> data-flywheel[Data flywheel]
  data-flywheel --> custom-model-layer
  data-flywheel --> windsurf-ide
  data-flywheel --> windsurf-browser
  data-flywheel --> pr-review-tool
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Enterprise developer Human software engineer collaborating with AI across IDE, browser, research, code review, and CI contexts. Windsurf IDE, Windsurf browser, PR review tool
Windsurf IDE VS Code fork that controls the product surface, APIs, visuals, and under-the-hood agent experience. Browser previews, Cascade agent, Code context
Browser previews In-IDE frontend preview surface that captures clicks, screenshots, logs, and UI context for the agent. Cascade agent
Windsurf browser Browser surface for research, Jira, GitHub, and other web-based software engineering context. Cascade agent, Enterprise workflow context
PR review tool Enterprise beta review surface for catching bugs and improving code-review culture as generated code volume grows. Cascade agent, Memory system
Cascade agent Human-in-the-loop coding agent that uses product-surface context, retrieval, tools, memory, and custom models. Retrieval layer, MCP connections, Direct integrations, Memory system, Custom model layer
Code context Open files, codebase context, and parsed language structures used by retrieval and autocomplete systems. Retrieval layer
Enterprise workflow context Research, Jira, GitHub, documentation, and other non-code context needed to complete software engineering work. MCP connections, Direct integrations
Retrieval layer Successive retrieval generations including vector search, Smart RAG, parsers, reranking heuristics, app mentions, context pinning, and Riptide. Riptide reranker, Custom model layer
Riptide reranker Custom code-retrieval/reranking model running with GPU burst capacity for more accurate context selection. Custom model layer
MCP connections Flexible mechanism for bringing arbitrary context sources and actions outside the IDE into the agent. Cascade agent
Direct integrations Enterprise-governed integrations with admin controls, preprocessing, indexing, and lower barrier to entry than raw MCP. Cascade agent
Memory system Persistent engineering decisions, lessons, and best practices that extend from IDE work into PR review and CI/CD. Cascade agent
Custom model layer Windsurf-built autocomplete, Riptide, and SWE-1/SUI-1-style agent models for code completion, retrieval, and complex software tasks. Offline PR benchmark, Online accept-rate evaluation
Offline PR benchmark GitHub PR reconstruction benchmark that measures whether models can complete tasks and pass unit tests. Custom model layer
Online accept-rate evaluation Production evaluation based on users accepting or rejecting generated code. Data flywheel
Data flywheel Loop where better agents attract users, free-user acceptance data informs the frontier, and model/product improvements expand the work horizon. Custom model layer, Windsurf IDE, Windsurf browser, PR review tool

Data flow

Developers work in Windsurf IDE, browser previews, the Windsurf browser, and PR review surfaces. Those surfaces capture code, screenshots, logs, research context, Jira/GitHub activity, documents, and engineering memories. Context flows through retrieval systems such as Smart RAG and Riptide or through direct integrations with enterprise controls before it reaches Cascade and custom models. Offline PR benchmarks and online acceptance/rejection data evaluate the models, and accepted/rejected usage from free users feeds the data flywheel for future model and product improvements.

5 supporting quotes
“software engineering is not the same as coding. Codegen is just a very small sliver of what software is engineer software engineers do”transcript, stated
“click various elements, you can screenshot, you can take logs and easily pass it into our AI agent to then go iterate on.”transcript, stated
“This is giving us the ability to be in the location that developers are doing research that they are working with Jira that they are working with GitHub”transcript, stated
“if you ship the best agent you can bring a number of users onto the platform that will love your product. From there you can understand what the frontier is naturally um we have a lot of data around what users accept or reject. We do this of course only on”transcript, stated
“we have a lot of data around what users accept or reject. We do this of course only on our free users um and do not train on on paying users.”transcript, stated

Why this architecture

Windsurf chose a broad product-and-model architecture because software engineering work is spread across IDEs, browsers, PR review, docs, Jira, GitHub, and CI/CD rather than just codegen. Controlling the product surfaces let Windsurf capture richer context and user feedback, retrieval and direct integrations expanded the agent's context beyond open files, memory preserved enterprise practices, and custom models let Windsurf optimize for software-engineering workflows rather than depend only on generic frontier models.

4 supporting quotes
“controlling the product surface. It allows you to control the underlying APIs. It allows you to control both the visuals as well as what's happening under the hood”transcript, stated
“when we're talking about how do we get broader? How do we do all of software engineering when software engineers are in their IDE hopefully 20 to 30% of the day? So, ironically, the first one I'm going to put up of what we released is in the IDE, but this was the the the first evidence”transcript, stated
“They're only as good as the data that we can feed into the agent or feed into the models. And this is where we started. We started with just code. We started with open files. That was our V1. We saw about an 11% higher acceptance rate of the autocompletes”transcript, stated
“space might debate the counterfactual of that but uh controlling the models is important. So what did we do? We came out with SUI 1 and this was our first um display of us releasing a model that is able to reason over a complex task and work like um reason over complex task and power an agent.”transcript, stated

Models used: Autocomplete model · Riptide reranking model · SUI 1 · Cascade agent · Claude Sonnet 3.5 · Claude Sonnet 3.7 · Vector databases · Smart RAG

Scale

Alternatives considered (roads not taken)

Promise full software-engineering automation immediately.

Rejected because: The team judged current AI as not yet capable of fully automating software engineering, so it built human-AI flows that deliver present value while expanding the future frontier.

“AI is not at a place where it's able to fully automate software engineering or really most tasks that exist. Um, and in order to work around that, I think it's important that you have a balance of what is realistic. Now, the a lot of the founding team of of”transcript, stated
“windsurf is that we wanted to constantly figure out where the frontier was, deliver that now while keeping our focus in the future. So, where we are now is very much still human in the loop. People know this concept and as work goes, you know, you make some edits to some files or you're just”transcript, stated

Stay only inside the IDE and focus on code generation.

Rejected because: The team saw software engineering as broader than coding and the IDE as only a minority of the developer's day.

“Codegen is just a very small sliver of what software is engineer software engineers do and for us to provide a great experience we're going to need to expand beyond just codegen and figure out how to solve all of it. So this is our thesis on how how to do it”transcript, stated
“being just an IDE is a surface area problem when we're talking about how do we get broader? How do we do all of software engineering when software engineers are in their IDE hopefully 20 to 30% of the day? So, ironically, the first one I'm going to put up of”transcript, stated

Use plain vector databases as the main code retrieval approach.

Rejected because: The first semantic vector-search attempt performed worse, so Windsurf moved to parsers, reranking heuristics, app mentions, context pinning, and eventually Riptide.

“we actually saw it got worse to be honest when we released just vector database um as a way of finding semantically similar code”transcript, stated
“just vector database um as a way of finding semantically similar code to um do inference on. We then released what we called smart rag. It's probably just a flashy marketing term, but inside of that we released a parsers for various languages. We came up with”transcript, stated

Rely only on MCP for enterprise context integrations.

Rejected because: MCP was flexible but ephemeral and scary for large enterprises, so Windsurf added direct integrations with admin controls, preprocessing, and indexing.

“MCP is quite scary. So direct integration really lowers the barrier to entry for them. They don't need to worry about building their own integration. It allows us to put admin controls on top of it as well as it allows us to do a lot of the pre-processing”transcript, stated
“It allows us to put admin controls on top of it as well as it allows us to do a lot of the pre-processing and indexing”transcript, stated

Optimize only autocomplete because it was an existing near-term product area.

Rejected because: Windsurf intentionally invested in browser and other surfaces because the team believed future agent capability would require broader product context.

“There's all kinds of investments we could have made to just say make autocomplete the best in the market because we know where the world is going”transcript, stated
“we've laid the foundation with things that seem random like browser but they're really not because that is the surface that we need once these models are ready.”transcript, stated

Patterns

human-in-the-loop-gate (agent workflow)

Windsurf frames current software engineering automation as shared human-AI flows rather than full autonomy, with more work moving to AI only as systems improve.

“So, where we are now is very much still human in the loop. People know this concept and as work goes, you know, you make some edits to some files or you're just doing some amount of work as a software engineer, we want there to be some kind of back and forth between the human and the AI.”transcript, stated
“the work between what a human is doing and what the AI is doing to accomplish some task should be shared between the two”transcript, stated

tool-use-orchestration (product architecture)

The platform coordinates IDE actions, browser screenshots/logs, Jira/GitHub research, PR review, MCP, direct integrations, and CI/CD context through the agent.

“you can screenshot, you can take logs and easily pass it into our AI agent to then go iterate on.”transcript, stated
“allow um engineers to bring in any any forms of context that they thought would be important for the agent as well as doing some amount of actions um outside of the IDE”transcript, stated

retrieval-ranking (context engineering)

Windsurf evolved from open-file context and weak vector search to Smart RAG and Riptide, a custom reranking model trained to identify task-relevant code.

“we released a parsers for various languages. We came up with a number of different reranking heristics that worked really well.”transcript, stated
“Riptide. This was our departure from embedding based search. I guess this is more of a reanker.”transcript, stated

rag-retrieval (context engineering)

The agent depends on retrieval over code, documentation, web/Jira/GitHub activity, and enterprise integrations because model performance is constrained by supplied context.

“They're only as good as the data that we can feed into the agent or feed into the models.”transcript, stated
“MCP was about extending in the first step around how do we find context that isn't just code.”transcript, stated

agent-memory (context persistence)

Windsurf encodes engineering decisions and lessons as memories so the agent can carry enterprise practices across IDE work, PR review, and CI/CD interactions.

“encoding lessons that happen as you're as you're building software.”transcript, stated
“memories will naturally extend beyond just the IDE. They will be in how we do PR reviews. They will be in if something breaks in your CI/CD pipeline.”transcript, stated

eval-driven-development (model evaluation)

Windsurf evaluates custom SWE models offline on GitHub PR reconstruction and conversation-continuation tasks, then checks online performance through user code acceptance.

“The first thing we did was test end toend workflows. So what we would do is we would take some PR from from GitHub.”transcript, stated
“the metrics that we use here are very much centered around how often are users um accepting the code that is generated.”transcript, stated

data-quality-loop (model improvement)

The product creates a flywheel where useful agents bring users, acceptance/rejection data identifies the frontier, and that data improves the model and product surface.

“what is the data flywheel that we can produce here and tying it all the way back to the application layer”transcript, stated
“You've built better models, you have all of the data and you have a better application surface that touches more of what a user is doing”transcript, stated

Tradeoffs

Keep software engineering agents human-in-the-loop.

Chose Flows that share work between humans and AI. over Claiming full autonomous software engineering..

Rationale: The team believed current AI was not ready for full automation, and shared flows could deliver value while capability improved.

“AI is not at a place where it's able to fully automate software engineering or really most tasks that exist.”transcript, stated
“the work between what a human is doing and what the AI is doing to accomplish some task should be shared between the two”transcript, stated

Fork VS Code to control the application surface.

Chose A custom Windsurf IDE. over Remaining only an extension or external assistant with less control over APIs and UI..

Rationale: Owning the IDE surface let Windsurf control visuals, APIs, under-the-hood behavior, and the data it needed for a better agent experience.

“It was a fork of VS Code which became a bit of a meme at the time as everyone was doing it at the time.”transcript, stated
“It allows you to control the underlying APIs. It allows you to control both the visuals as well as what's happening under the hood”transcript, stated

Build direct enterprise integrations alongside MCP.

Chose Direct integrations with admin controls, preprocessing, and indexing. over Depending only on MCP for arbitrary context..

Rationale: Direct integrations reduced enterprise security concerns and allowed better governance and preprocessing than ephemeral MCP access.

“MCP is quite scary. So direct integration really lowers the barrier to entry for them.”transcript, stated
“MCP in nature is very ephemeral in in how it interacts with that data.”transcript, stated

Invest in custom models.

Chose Windsurf-trained autocomplete, Riptide, and SUI 1 models. over Relying exclusively on third-party frontier models..

Rationale: The team believed controlling models was important for the best software-engineering experience and could be done at lower cost with a small team.

“we also believe that controlling models themselves is important for providing a great experience.”transcript, stated
“If we can do this at a fraction of the cost with a small team compared to the frontier labs, there's no reason we won't continue to make that bet.”transcript, stated

Broaden product surfaces instead of only maximizing autocomplete.

Chose Browser, PR review, docs integrations, memories, and context expansion. over Focusing all investment on autocomplete quality..

Rationale: The team believed future agents needed broader context and surfaces once models became ready, even if those investments seemed random in the present.

“There's all kinds of investments we could have made to just say make autocomplete the best in the market”transcript, stated
“things that seem random like browser but they're really not because that is the surface that we need once these models are ready.”transcript, stated

Gotchas (what bit them)

Over-promising autonomy would repeat the self-driving-car failure pattern.

Root cause: The founding team believed current AI could not fully automate software engineering, and prior autonomous-vehicle experience showed the danger of promising full autonomy next year forever.

Resolution: Windsurf focused on human-AI flows that deliver present value while pushing the frontier incrementally.

“self-driving would come next year and then you'd say the same thing again next year, right?”transcript, stated
“we wanted to constantly figure out where the frontier was, deliver that now while keeping our focus in the future.”transcript, stated

An IDE-only agent misses most of the software engineering workflow.

Root cause: Software engineers spend much of their day in browsers, Jira, GitHub, PR review, documentation, and other contexts outside the IDE.

Resolution: Windsurf expanded into browser previews, a dedicated browser, PR review, MCP, direct integrations, and memory.

“being just an IDE is a surface area problem when we're talking about how do we get broader?”transcript, stated
“developers are doing research that they are working with Jira that they are working with GitHub”transcript, stated

Naive vector retrieval worsened performance.

Root cause: Embedding-based semantic similarity alone did not reliably identify the code relevant to a task.

Resolution: The team added language parsers, reranking heuristics, context pinning, and eventually a custom Riptide reranking model.

“we actually saw it got worse to be honest when we released just vector database”transcript, stated
“we actually trained a large language model to do the task of figuring out what code is important for some task.”transcript, stated

MCP was too dynamic for some enterprise security expectations.

Root cause: Large enterprises care where data goes, and MCP's ephemeral interactions made admin controls, preprocessing, and indexing harder.

Resolution: Windsurf added direct integrations with admin controls and preprocessing for enterprise data sources.

“And why these are important is that enterprises care a lot about where their data is going. So on top of this, we can do all kinds of arback and controlling of what data is going out because having talked to a number of these very large enterprises, MCP is quite scary.”transcript, stated
“MCP is quite scary. So direct integration really lowers the barrier to entry for them.”transcript, stated

Code-only context creates a ceiling for enterprise software engineering agents.

Root cause: Enterprise software work depends on knowledge outside the codebase, including docs, research, requirements, PRs, and organizational practices.

Resolution: Windsurf invested in browser surfaces, documentation integrations, memories, and broader application surfaces to expand context.

“doing all of these retrieval tasks just on code is not going to get you that far.”transcript, stated
“there is so much more about uh the software that was being written in enterprises than is just encoded inside of the code.”transcript, stated

Evaluation

Windsurf evaluated retrieval and models with benchmark uplifts, GitHub PR reconstruction tasks, conversation-continuation tests, unit-test pass outcomes, and online user acceptance rates. The team treated humans accepting generated code as the production judge, while limiting training data collection to free users rather than paying users.

Metrics: Open-file context produced about 11% higher autocomplete acceptance than a base model. · Smart RAG produced about 38% benchmark uplift. · Riptide produced about 4x higher retrieval accuracy. · SUI 1 outperformed Sonnet 3.5 on end-to-end GitHub PR workflow benchmarks. · Online accept rates outperformed Sonnet 3.7.

5 supporting quotes
“We saw about an 11% higher acceptance rate of the autocompletes versus just a base model.”transcript, stated
“We saw about 38% uplift um on our benchmarks. And the last piece that we came out with was Riptide. This was our departure from embedding based search.”transcript, stated
“We saw about four 4x higher accuracy in terms of uh of that retrieval process.”transcript, stated
“take some PR from from GitHub. You would scrap the diff, start it from scratch, and you would use the commit message or comments to show intent”transcript, stated
“the metrics that we use here are very much centered around how often are users um accepting the code that is generated.”transcript, stated

Outcomes

The platform had moved beyond autocomplete into IDE, browser, PR review, MCP, direct documentation integrations, memory, custom models, offline benchmarks, and online human-acceptance feedback. The source is a vendor talk and reports performance directionally rather than publishing benchmark datasets.

5 supporting quotes
“very large deployments with the JP Morgans of the world, the Dells of the world.”transcript, stated
“We have had 10 major releases since then. The text is probably pretty small, but it's not important that you see what we released in that time, but it's more around product velocity.”transcript, stated
“We saw about four 4x higher accuracy in terms of uh of that retrieval process.”transcript, stated
“We were actually outperforming sonnet 35. Um, and we saw pretty frontier performance in the model that we built.”transcript, stated
“the accept rates that we find is actually even outperforming um sonnet 37.”transcript, stated