Migrating LLM Fine-tuning Workflows from Slurm to Kubernetes Using Metaflow and Argo

adept.ai · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM fine-tuning and evaluation infrastructure

Adept needed to make LLM training, fine-tuning, evaluation, CI/CD, and infrastructure-monitoring workflows self-serve while gradually moving from a Slurm-centered stack toward Kubernetes without slowing data scientists.

Constraints:

• Existing training, fine-tuning, and evaluation workflows had different config files, entry points, and execution paths.
• The fine-tuning code still ran on Slurm and supported active model work, so the migration could not force a big-bang Kubernetes cutover.
• Legacy code and a roughly 1 GB repository made containerization slow and awkward.
• Data scientists needed links, logs, Slack notifications, and experiment visibility even when Metaflow steps were only orchestrating Slurm jobs.
• Some desired workflow behaviors, such as queueing when resources were unavailable and step-level exception handling, were still missing.

Architecture

Adept put Metaflow plus Argo Workflows on Kubernetes in front of its Slurm training environment. Metaflow packages the workflow DAG and provides Python-flow ergonomics, Argo executes the workflow layer on Kubernetes, and each step temporarily SSHes into a Slurm bastion to run the real srun/sbatch workload. The team also built internal CLI affordances, Slack notifications, links to Slurm job logs, experiment-platform links, CircleCI-driven container builds, and Metaflow/Argo workflow recreation on merge to main. The result is a hybrid orchestration layer that lets users launch fine-tune/eval and nightly workflows now while the infrastructure team gradually moves workloads to fully containerized Kubernetes execution.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  CLI["Internal Adept CLI"] --> Metaflow["Metaflow flows"]
  Metaflow --> Argo["Argo Workflows"]
  Argo --> K8s["Kubernetes cluster"]
  Metaflow --> Bridge["Slurm bastion bridge"]
  Bridge --> Slurm["Slurm training cluster"]
  Circle["CircleCI deployment"] --> Images["Container images"]
  Circle --> Metaflow
  Images --> K8s
  Metaflow --> Obs["Observability notifications"]

Components

Data flow

A data scientist launches a Metaflow flow, often through an internal Adept CLI. Metaflow packages the flow DAG and Argo runs the workflow on Kubernetes. During the hybrid phase, each workflow step SSHes into a bastion on the Slurm cluster and runs the underlying srun or sbatch command. Long-running fine-tune steps freeze a model and fan out to evaluation tasks, while Slack messages, logs, experiment links, and Slurm job-log links make the hidden execution observable.

Why this architecture

Metaflow plus Argo gave Adept a Kubernetes-native workflow layer without requiring immediate migration of Slurm execution. That let the infra team improve self-service, observability, nightlies, and CI/CD while preserving training velocity for data scientists. The hybrid bridge is explicitly a stopgap toward full Kubernetes containerized workloads.

Models used: Adept computer-use agent model · fine-tuned LLMs · evaluation datasets for browser action types

Scale

• The training environment could involve 200 to 300 nodes with 8 GPUs each.
  “when you want, let's say, 200, 300 nodes, each having eight GPUs.”transcript, stated
• One example workflow launches a 16-node job with 8 GPUs per node.
  “launch it on 16 nodes, each node having eight GPUs." So that's 16 * 8 kind of number of uh uh GPUs kind of thing.”transcript, stated
• The legacy repository was roughly 1 GB because model/data artifacts had accumulated in Git LFS.
  “the code repo was large. And that that came in from some you know, initial attempts at using Git LFS for some storing some versions of the models or some data inside the code base and that bloated to like about almost about a GB.”transcript, stated
• Fine-tuning jobs could run for 1 to 3 hours.
  “this is like that S batch command that runs it and then waits for that fine-tuning, which might take an hour or 3 hours to to complete.”transcript, stated
• Nightly training could spin up a 32-node, roughly 240-GPU job.
  “nightly training, which, you know, just spins up a 32-node job and trains uh, trains stuff. So, 32 nodes again, for those who do a quick math, is 240 GPUs.”transcript, stated

Alternatives considered (roads not taken)

Continue with fragmented Slurm entry points

Rejected because: Self-serve fine-tuning became too complex as config loading, workflow entry points, and execution conventions diverged.

Force an immediate full Kubernetes migration

Rejected because: The team needed to keep existing Slurm-backed fine-tuning velocity while code cleanup and containerization were still in progress.

Use a workflow alternative with custom orchestration

Rejected because: Adept was more comfortable with Metaflow because it used Argo, a robust Kubernetes-native orchestration system.

Patterns

durable-execution (workflow)

Metaflow and Argo provide a DAG-based workflow layer for fine-tune/eval, nightlies, monitoring, and housekeeping jobs.

fine-tuning-adaptation (model_optimization)

The platform exists to let users run fine-tuning jobs and nightly fine-tune/eval workflows self-serve.

eval-driven-development (evaluation)

Fine-tuned models are frozen and evaluated across action-type datasets, with nightly and programmatic evaluation workflows.

observability-tracing (operations)

The workflow layer sends Slack updates, exposes dashboards/logs, links to experiment management, and monitors Slurm jobs and zombie workflows.

batch-offline-processing (workflow)

The system runs long fine-tuning jobs, nightly jobs, training runs, monitoring workflows, and housekeeping tasks asynchronously.

canary-rollout (deployment)

Workflow definitions are recreated automatically on merge to main through CI/CD rather than manually updated.

Tradeoffs

Use a hybrid Slurm/Kubernetes bridge

Chose Metaflow and Argo orchestrating steps that SSH into Slurm over Blocking users until all training workloads ran natively on Kubernetes.

Rationale: The hybrid bridge preserved data-science velocity while the team untangled legacy code and moved toward full Kubernetes execution.

Choose Metaflow because it used Argo

Chose Metaflow with Argo Workflows on Kubernetes over An alternative workflow tool with custom orchestration.

Rationale: Adept trusted Argo's Kubernetes-native, battle-tested workflow execution more than a custom orchestration layer.

Own container builds instead of relying only on Metaflow packaging

Chose Custom PyTorch/NVIDIA-based images built for multiple infrastructure uses over Using Metaflow packaging alone for the legacy codebase.

Rationale: The legacy dependencies were not easily packageable, and the team wanted one containerization approach for workflows, dev boxes, CI/CD, and future gang-scheduled training.

Gotchas (what bit them)

Workflow modernization stalls if legacy code and config paths are tangled.

Root cause: Training, fine-tuning, and nightly jobs had multiple configs, entry points, and loading patterns from different contributors.

Resolution: The team spent sprints refactoring logical workflow steps and normalizing config loading for Metaflow containerization.

Hybrid orchestration can hide the real execution environment from users.

Root cause: Metaflow and Argo logs describe the container layer, but the actual training still runs through SSH into Slurm.

Resolution: The internal CLI prints Slurm job IDs and log links, while workflow steps expose notifications and experiment-management links.

Large ML repositories make per-commit container execution expensive.

Root cause: Historical Git LFS usage stored model versions or data in the repo, growing it to about 1 GB.

Resolution: Adept reduced the repository size, removed data and dead code, and built PyTorch/NVIDIA-based containers per commit.

Resource shortages still crashed large workflows instead of queueing them.

Root cause: The system lacked queueing for flows when requested Slurm resources were unavailable.

Resolution: Queue management was identified as a desired feature for future work.

Evaluation

Adept evaluated the platform operationally through self-serve launchability, fine-tune/eval DAGs, action-type evaluation datasets, Slack/log/experiment observability, nightly training, CI/CD workflow recreation, and infrastructure monitoring workflows.

Metrics: self-serve launch of fine-tune jobs · 1-3 hour fine-tuning steps completing inside flows · evaluation datasets for browser action types · nightly 32-node training runs · long-running and zombie Slurm job detection · workflow recreation on merge to main

Outcomes

• Users can launch fine-tuning jobs self-serve.
• Metaflow plus Argo supports fine-tune/eval, text eval, nightlies, training, monitoring, and housekeeping workflows.
• CI/CD can recreate nightly workflows automatically on merge to main.
• The system provides observability through Slack, dashboards, logs, experiment-platform links, and Slurm-job links.
• Full fine-tuning and training directly on Kubernetes remained work in progress.

The transcript cites 16-node jobs with 8 GPUs each, possible 200-300 node clusters with 8 GPUs per node, 32-node nightly training equivalent to about 240 GPUs, and fine-tuning steps lasting 1-3 hours.