AI Agent for Automated Feature Flag Removal

duolingo · Education · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Software maintenance automation

Duolingo wanted to remove obsolete feature flags without repeatedly spending engineer time on routine code cleanup.

Constraints:

repo-local code changes
PR creation
non-deterministic agent failures
rapid internal deployment

Architecture

Duolingo built the feature-flag remover as a Temporal-orchestrated coding-agent workflow. A Platform Self-Service UI starts a gateway workflow, which dispatches a feature-flag-removal worker. The worker resolves the requesting user’s GitHub account, clones the target repository into a temporary local directory, invokes Codex CLI in full-auto/quiet mode inside a sandboxed ECS/Temporal setup, and creates an assigned pull request when code changes are found. Temporal supplies local testing, retries, and workflow composition for future coding agents.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  platform-self-service-ui[Platform Self-Service UI] --> gateway-workflow[Gateway workflow]
  gateway-workflow --> feature-flag-removal-worker[Feature flag removal worker]
  feature-flag-removal-worker --> github-account-lookup[GitHub account lookup]
  feature-flag-removal-worker --> repository-clone-activity[Repository clone activity]
  repository-clone-activity --> codex-cli-activity[Codex CLI activity]
  temporal-retry-layer[Temporal retry layer] --> feature-flag-removal-worker
  temporal-retry-layer --> codex-cli-activity
  codex-cli-activity --> pull-request-creator[Pull request creator]
  github-account-lookup --> pull-request-creator

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Platform Self-Service UI`	Internal page where engineers start automated feature-flag cleanup.	`Gateway workflow`
`Gateway workflow`	Temporal dispatcher namespace workflow that kicks off the feature-flag-removal worker.	`Platform Self-Service UI`, `Feature flag removal worker`
`Feature flag removal worker`	Main Temporal workflow coordinating user lookup, repository clone, Codex execution, and PR creation.	`Gateway workflow`, `GitHub account lookup`, `Repository clone activity`, `Codex CLI activity`, `Pull request creator`
`GitHub account lookup`	Activity that resolves the requester so the resulting PR can be assigned correctly.	`Feature flag removal worker`, `Pull request creator`
`Repository clone activity`	Clones the repo to a temporary local directory for local code operation.	`Feature flag removal worker`, `Codex CLI activity`
`Codex CLI activity`	Runs Codex CLI as a full-auto quiet subprocess to edit the cloned code.	`Repository clone activity`, `Pull request creator`
`Temporal retry layer`	Handles non-deterministic agent failures such as crashes, no changes, hangs, or incomplete runs.	`Feature flag removal worker`, `Codex CLI activity`
`Pull request creator`	Creates a GitHub PR and assigns it to the requesting engineer when changes are found.	`Codex CLI activity`, `GitHub account lookup`

Data flow

An engineer launches cleanup from an internal self-service page. The gateway Temporal namespace starts the feature-flag-removal workflow, which first gets the user’s GitHub account, then runs the main work activity. That activity clones the repository locally, asks Codex CLI to remove the feature flag, and if changes exist creates a pull request assigned to the requester.

3 supporting quotes

“Through the Platform Self-Service UI, a workflow is kicked off in gateway, Temporal’s dispatcher namespace. The gateway workflow kicks off another workflow, in this case our feature flag removal worker.”article, stated

“We first run an activity (Temporal-speak for “task”) to get the user’s Github account name, then kick off a new activity to do the main work.”article, stated

“We clone the repo to a temporary directory, then ask Codex CLI to do its work. Once the agent completes, assuming it has found changes to make, it creates a PR, assigning it to the user’s account name we pulled earlier.”article, stated

Why this architecture

Duolingo chose this shape because coding agents operate well on local code, while normal GitHub access can happen outside the AI call. Temporal made rapid prompt iteration and retries practical for non-deterministic agents. Keeping clone, agent work, and PR creation together avoided worker-locality problems where a later activity might run on a different machine without the cloned code.

3 supporting quotes

“Most AI tools efficiently operate on code locally (Codex CLI for instance), and a Github MCP is simply not necessary for tasks that are not operating on open PRs or code history.”article, stated

“Most importantly, it has trivially easy local testing (key for rapid prompt engineering) and strong retry logic.”article, stated

“But in Temporal, each activity may run on a separate worker, so cloning in a separate activity is not a viable option—we could find ourselves cloning the code and then trying to operate on it on a different instance that doesn’t have the code at all.”article, stated

Models used: Codex CLI model

Scale

The team moved from chosen stack to prototype in about one day and production in about one week.
“Once we settled on our tech stack, we had a prototype running in ~1 day, and a productionized version in ~1 week.”article, stated
The initial implementation expanded from a Python-only prototype to also cover Kotlin.
“Much of the week was spent on prompt engineering and feature development (such as extending the Python only prototype to also cover Kotlin), with the rest figuring out reusable patterns.”article, stated
The pattern is intended as a reusable foundation for future Duolingo agents operating on code.
“we wanted to start figuring out patterns for building additional agents that operate on our code.”article, stated

Alternatives considered (roads not taken)

Separate clone, agent operation, and PR creation into reusable Temporal activities

Rejected because: Separate activities may run on different workers, so the agent could lack the cloned code.

“A reasonable design pattern might seem to split this into more reusable activities—one to clone, one to operate agentically, and one to create the PR for instance. But in Temporal, each activity may run on a separate worker”article, stated

Continue LangChain/fast-agent implementations

Rejected because: Codex CLI accepted their merged prompt and worked quickly after one to two weeks of alternative toolchain development.

“These versions were in development for 1-2 weeks, and we were able to get the langchain version working well using a set of 3 prompts run in loops, and the fast-agent was along the way doing the same. Then Codex was released.”article, stated

Wait for a formal Codex SDK

Rejected because: Running Codex CLI as a Python subprocess was good enough to ship without waiting.

“Ideally, we would much rather be using a proper API through some form of Codex SDK. But running the CLI as a Python subprocess works, and means we don’t have to wait.”article, stated

Patterns

`durable-execution` (orchestration)

Temporal provides local testing and retry behavior for nondeterministic agentic cleanup workflows.

“Temporal is a workflow orchestration platform, which can fill a similar niche as Jenkins.”article, stated

`tool-use-orchestration` (software-engineering)

The agent operates on local code in a cloned repository, then standard tooling creates the pull request.

“We clone the repo to a temporary directory, then ask Codex CLI to do its work.”article, stated

Tradeoffs

Which agent integration to use.

Chose Codex CLI as a subprocess over waiting for a formal SDK or continuing bespoke LangChain/fast-agent builds.

Rationale: The CLI worked quickly enough to ship despite being less ideal than an SDK.

“But running the CLI as a Python subprocess works, and means we don’t have to wait.”article, implied

Temporal activity boundaries.

Chose keep clone, agent operation, and PR creation in the main work path over splitting clone and agent work into separate reusable activities.

Rationale: Temporal worker locality means the cloned repository might not exist where a later activity runs.

“we could find ourselves cloning the code and then trying to operate on it on a different instance that doesn’t have the code at all.”article, stated

Agent integration maturity.

Chose Codex CLI subprocess now, SDK later over waiting for a formal SDK before shipping.

Rationale: The CLI was less deterministic and less ideal than an SDK, but it worked quickly enough to productionize the tool.

“But running the CLI as a Python subprocess works, and means we don’t have to wait. We expect to replace this with the Codex SDK if and when it exists.”article, stated

Gotchas (what bit them)

Splitting clone/agent/PR work into separate Temporal activities can break local workspace assumptions.

Root cause: Temporal may schedule each activity on a different worker, so the cloned repo may not exist where the agent runs.

Resolution: Keep clone, agent operation, and PR creation in one activity where local filesystem state is preserved.

Evaluation

The post reports implementation speed and production readiness rather than formal accuracy metrics.

Metrics: prototype in one day · production in about one week

“The team reports moving from initial experimentation to production deployment in approximately one week after settling on their technology stack, with a working prototype operational within a single day.”summary, stated

Outcomes

Duolingo shipped a reusable agent pattern for future code-infrastructure agents.

This is a narrow internal automation, not broad autonomous development.

“We built this agent on the workflow orchestration service Temporal using the Codex CLI, a pattern we intend to extend to future agents.”article, stated