AgentKernelArena is a controlled experimentation platform for developing AI agents on real GPU kernel optimization tasks. It enables reproducible A/B testing across models, prompts, tools, and agent policies, while providing objective compilation, correctness, and performance signals that can serve as rewards for agent reinforcement learning.
AgentKernelArena makes changes to an agent measurable. Run the same task set with a baseline and a treatment—such as a different model, prompt, MCP server, skill, tool, memory strategy, or policy—and compare the outcomes under the same execution and scoring pipeline.
The platform provides:
- Controlled A/B experiments: Label and compare repeated runs while holding tasks, hardware, environment, and evaluation rules constant.
- RL-ready feedback: Produce per-task compilation, correctness, runtime, speedup, and score signals that can be consumed as rewards by an external reinforcement-learning system.
- Multiple agent integrations: Run Cursor Agent, Claude Code, Codex, DeepSeek Harness, GEAK-based agents, or custom agents through a shared interface.
- Real GPU task environments: Work with HIP, Triton, FlyDSL, PyTorch-to-kernel conversion, instruction-to-kernel generation, and image-backed kernel optimization tasks.
- Isolated and reproducible execution: Give every task its own timestamped workspace and preserve logs, modified sources, and structured results.
- Centralized evaluation: Measure compilation, correctness, and GPU performance independently of the optimizing agent.
- Multi-GPU scheduling: Start one isolated Docker worker per GPU and dynamically claim tasks from a shared queue.
- Slurm/Spur login-node workflow: Allocate one or eight MI355X GPUs, then launch the same Docker runtime on the assigned compute node.
- Resumable experiments: Resume a run without repeating tasks whose framework completion and source evidence remain valid.
- Held-out evaluation: Test optimized kernels on unseen shapes and measure the generalization gap.
- Task validation and visualization: Validate task quality with a dedicated agent and compare local run reports in a dashboard.
AgentKernelArena supplies an environment and objective reward signals; it does not currently include an RL trainer, replay buffer, or policy-update loop. Its per-task workspaces provide reproducibility and concurrent-run separation, not a security sandbox: agent processes run permissively inside a privileged container and can access mounted repository and authentication state.
Run the same configuration twice with a distinct suffix. Change only the capability under test between the two runs.
# Choose the run configuration used for both sides of the experiment.
CONFIG_PATH=example_configs/quickstart_claude_mi300.yaml
# Baseline: capability disabled
make docker-run CONFIG="$CONFIG_PATH" RUN_ARGS="--run-suffix baseline"
# Treatment: capability enabled
make docker-run CONFIG="$CONFIG_PATH" RUN_ARGS="--run-suffix treatment"Compare the generated reports directly:
python3 src/tools/compare_runs.py \
workspace_MI300_claude_code/run_<timestamp>_baseline \
workspace_MI300_claude_code/run_<timestamp>_treatmentFor visual comparison, build the local dashboard as described in the visualization module README. For stochastic agents, repeat matched baseline/treatment pairs and interpret the observed deltas together with run-to-run variance.
AgentKernelArena/
├── main.py # Run orchestration, resume, and parallel queue
├── example_configs/ # Quickstart and curated benchmark run configs
├── src/
│ ├── module_registration.py # Agent registration and handler selection
│ ├── preprocessing.py # Workspace and source setup
│ ├── prompt_builder.py # Task prompt construction
│ ├── evaluator.py # Compilation and correctness evaluation
│ ├── performance.py # Baseline and optimized timing
│ ├── score.py # Reward/score calculation
│ ├── postprocessing.py # Aggregate run reports
│ ├── held_out/ # Optional generalization evaluation module
│ ├── visualization/ # Dashboard module and static frontend
│ └── tools/
│ ├── compare_runs.py # Compare two completed experiment runs
│ └── perf/ # Canonical performance helpers
├── agents/
│ ├── cursor/ # Cursor Agent CLI
│ ├── claude_code/ # Claude Code CLI
│ ├── codex/ # Codex CLI
│ ├── deepseek_harness/ # DeepSeek Harness headless CLI
│ ├── forge/ # KernelForge through the shared task contract
│ ├── geak/ # Native GEAK Workflow integration
│ └── task_validator/ # Task quality validator
├── tasks/
│ ├── hip2hip/
│ ├── triton2triton/
│ ├── instruction2triton/
│ ├── torch2hip/
│ ├── torch2flydsl/
│ ├── triton2flydsl/
│ ├── flydsl2flydsl/
│ ├── Aiter-task/ # Production GEMM and MoE to FlyDSL
│ └── image_kernel/ # Kernels from declared in-image source trees
└── docs/ # Full documentation
- Load the run configuration and selected agent.
- Discover task
config.yamlfiles matching the configured selectors. - Create an isolated task workspace and materialize its declared sources.
- Validate the task, establish its independent baseline, and check an existing initial candidate when present. Generation targets may start unimplemented.
- Build the task prompt and run the selected agent inside the workspace.
- Independently compile, check, and time the agent's modified implementation.
- Write a structured
task_result.yamlcontaining reward signals and a score, then run any task-declared exports for an accepted candidate. - Aggregate all task results into run-level CSV, JSON, and text reports.
task_validator follows a validation-specific path and writes validation_report.yaml instead of optimizing and scoring a kernel.
For multi-GPU runs, the host-side Docker runner creates a shared .parallel/ queue under the run directory. Each long-lived worker sees one logical GPU, atomically claims one task at a time, and returns for more work until the queue is empty. Final aggregation runs once after all workers finish.
Each run selects one agent.template. Repeated runs can compare different agents or different configurations of the same agent.
| Template | Purpose |
|---|---|
cursor |
Cursor Agent CLI integration |
claude_code |
Claude Code CLI integration |
codex |
Codex CLI integration |
deepseek_harness |
DeepSeek Harness headless CLI (setup) |
forge |
KernelForge search through the shared task interface; initialize, translate, or optimize as required |
geak |
GEAK multi-agent Workflow engine through the shared v2 task interface |
task_validator |
Task quality validation; does not optimize kernels |
The registry maps the compatibility names geak_v4 to geak and
forge_operator2flydsl to forge. Each uses the canonical launcher, defaults
and post-processing; there are no separate legacy agent directories. See
src/module_registration.py for selectable names
and the GEAK and Forge guides
for their shared v2 interfaces and runtime requirements.
Agent-specific defaults live under agents/<agent_name>/agent_config.yaml or in the selected agent CLI. Supported run-level overrides take precedence over agent-local defaults. Specialized agents may require additional setup; inspect their directories and agent-specific README files where present.
See CLI agent defaults and verification for tested CLI/model versions, run-level overrides, and the scope of live checks.
| Task type | Objective |
|---|---|
hip2hip |
Optimize an existing HIP implementation |
triton2triton |
Optimize an existing Triton implementation |
instruction2triton |
Implement a Triton kernel from a specification |
torch2hip |
Replace a PyTorch reference with a HIP implementation |
torch2flydsl |
Replace a PyTorch reference with a FlyDSL implementation |
triton2flydsl |
Translate a Triton implementation to FlyDSL |
flydsl2flydsl |
Optimize an existing FlyDSL implementation |
image_kernel |
Optimize a kernel from a declared source tree in the runtime image |
Aiter-task |
Reimplement production BF16 GEMM and MXFP4 MoE operators in FlyDSL |
These names organize task selection; the candidate and evaluation declarations
determine behavior. Every retained suite uses the unified task contract in
Task definition, schema, and authoring.
The prompt system also recognizes cuda2hip; the current bundled task tree does not include a cuda2hip/ suite.
- A Linux host with a supported AMDGPU kernel driver;
/dev/kfdand/dev/drimust be present - Docker Engine; the current user must be able to access the Docker daemon
without
sudo - Git
- Node.js 22+ and npm when using the alternative npm installation of Claude Code (or another npm-installed agent CLI); DeepSeek Harness uses a dedicated Node.js 24 prefix as described in its setup guide
- For MI300/MI355X, use the GPU-specific SGLang image:
gfx942useslmsysorg/sglang:v0.5.12-rocm720-mi30x;gfx950useslmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260705 - For RDNA4
gfx1201, the runner automatically builds the default pinned RDNA4 runtime on first use if it is missing. - A supported agent CLI installed and logged in on the host, or the dependencies required by a specialized agent
git clone https://github.com/AMD-AGI/AgentKernelArena.git
cd AgentKernelArena
# Verify the container runtime, GPU, and Python environment.
make docker-smoke
# Install the primary quickstart agent with Claude Code's native installer.
curl -fsSL https://claude.ai/install.sh | bash
claude --version
# Start Claude Code and complete login on the host, then exit.
claude
claude auth status
# Select the quickstart config for the physical GPU.
CONFIG_PATH=example_configs/quickstart_claude_mi300.yaml
# For MI355X instead:
# CONFIG_PATH=example_configs/quickstart_claude_mi355x.yaml
# Verify the agent selected by the config, then run one task.
make docker-check-agents CONFIG="$CONFIG_PATH"
make docker-run CONFIG="$CONFIG_PATH"
# Other options:
# make install-cursor-agent
# Install Codex using its official CLI instructions and ensure `codex` is in PATH.Installing an agent CLI is not enough: authenticate it on the host before
running docker-check-agents or an experiment. The Docker runner supports both
Claude Code's recommended native installation and its alternative npm
installation. The npm path requires Node.js 22+ and npm. See the
official Claude Code setup guide
for the current alternatives.
The following configurations provide starting points for supported runtimes:
| Configuration | Purpose |
|---|---|
example_configs/quickstart_claude_mi300.yaml |
One Claude Code GELU task on MI300/MI300X (gfx942); use this for a first run on MI300-series hardware. |
example_configs/quickstart_claude_mi355x.yaml |
One Claude Code GELU task on MI355X (gfx950); use this for a first run on MI355X. |
example_configs/quickstart_claude_rdna4.yaml |
One Claude Code GELU task on RDNA4 (gfx1201); builds the default runtime on first use if missing. |
example_configs/quickstart_deepseek_harness_mi300.yaml |
One DeepSeek Harness GELU task on MI300/MI300X; requires the pinned CLI and API key. |
example_configs/quickstart_deepseek_harness_mi355x.yaml |
One DeepSeek Harness GELU task on MI355X; see setup. |
example_configs/benchmark_cursor_mi355x.yaml |
Curated 60-task Cursor Agent benchmark on MI355X; use this for a longer benchmark only after installing and authenticating Cursor Agent. |
Running make docker-run without CONFIG uses the MI300/MI300X Claude
quickstart. On another GPU, pass the matching configuration explicitly; for
example, use CONFIG=example_configs/quickstart_claude_mi355x.yaml on MI355X.
FlyDSL tasks require FlyDSL in the container. The pinned image may already provide it; otherwise run:
make docker-setup-flydslPerformance timing helpers are maintained in src/tools/perf/ and materialized into run workspaces. See src/tools/perf/README.md before changing task timing code.
For detailed installation and compatibility information, see docs/install/install.md and docs/reference/compatibility-matrix.md.
Start from the example that matches the physical GPU and the agent you installed. Experiment preflight intentionally fails when the configured and detected GPU architectures differ. To create a custom experiment, copy an example and edit the copy:
cp example_configs/quickstart_claude_mi300.yaml my_experiment.yamlFor Codex and Claude Code, the run config's agent mapping can override
model, effort, max_iterations, and timeout_seconds from the selected
agent's agent_config.yaml. Cursor accepts the same overrides except a
standalone effort. See run-level overrides
for exact semantics; other integrations define their own supported settings.
For a Cursor, Claude Code, Codex, or task-validator config, verify only the selected first-class host CLI (the validator resolves to its configured backend):
CONFIG_PATH=my_experiment.yaml
make docker-check-agents CONFIG="$CONFIG_PATH"Use AGENTS=claude_code,codex to check an explicit subset or AGENTS=all to
check Cursor, Claude Code, and Codex together. Specialized integrations such as
Forge and legacy mini-swe have their own dependency checks. GEAK and its v2 aliases
resolve to Claude Code for this CLI check; a normal run additionally checks the
pinned GEAK engine and SDK.
DeepSeek Harness also supports config-selected docker-check-agents, or an
explicit AGENTS=deepseek_harness. It uses an API key and isolated per-task
state; see its setup guide. AGENTS=all
continues to check the original three CLIs.
make docker-run CONFIG="$CONFIG_PATH"
make docker-run CONFIG="$CONFIG_PATH" RUN_ARGS="--run-suffix experiment_name"# Explicit host GPU IDs
make docker-parallel-run CONFIG="$CONFIG_PATH" GPU_IDS=0,1,2,3
# Discover GPUs with rocm-smi --showid
make docker-parallel-run CONFIG="$CONFIG_PATH"The Docker parallel path is verified for cursor, claude_code, codex, and
task_validator. Specialized GEAK integrations need their own
dependencies and GPU-ID configuration before they are used with isolated
workers.
When the login node has no GPU or Docker daemon, allocate the compute node before entering the existing Docker workflow:
# One-GPU development smoke/run
make slurm-smoke
make slurm-run CONFIG=config_codex_mi355x_spur.yaml
# One node, eight GPUs, one isolated Docker worker per GPU
make slurm-parallel-smoke
make slurm-parallel-submit CONFIG=example_configs/benchmark_cursor_mi355x.yamlSee Run on Slurm/Spur GPU nodes for synchronous, batch, interactive, resource-override, authentication, and logging details.
make docker-run CONFIG="$CONFIG_PATH" RUN_ARGS="--resume-latest"
make docker-run CONFIG="$CONFIG_PATH" RUN_ARGS="--resume-run run_20260702_041903_experiment"
make docker-parallel-run CONFIG="$CONFIG_PATH" GPU_IDS=0,1,2,3 RUN_ARGS="--resume-latest"Task selectors are paths relative to tasks/. A selector can name one task or any parent directory.
The Aiter-task suite was previously named SIKL-task. Update custom selectors
to Aiter-task/.... This rename changes task IDs and report grouping; start a
new run with the new selectors, since existing runs retain their original IDs.
tasks:
- hip2hip/gpumode
- triton2triton/vllm
- triton2triton/rocmbench
- instruction2triton/rocmbench
- torch2hip
- torch2flydsl
- triton2flydsl
- flydsl2flydslEvery normal optimization task produces task_result.yaml with compilation and correctness status, baseline and optimized times, speedup, timing-method metadata, and the final score.
The default score is:
| Component | Points | Condition |
|---|---|---|
| Compilation | 20 | The optimized implementation compiles |
| Correctness | 100 | Correctness passes |
| Performance | speedup_ratio × 100 |
Added only when compilation and correctness pass |
Therefore:
- Compilation fails →
0 - Compilation passes but correctness fails →
20 - Both pass →
120 + speedup_ratio × 100
For multi-case tasks, the evaluator prefers the explicit per-case average speedup_ratio rather than deriving the score from only the two aggregate timing values. The scoring policy can be replaced in src/score.py when an experiment needs a different reward function.
Each task has one config.yaml plus task-local implementation and evaluation
files. Read Task definition, schema, and authoring
for the canonical schema, examples, command/result contracts, baseline and
candidate lifecycle, optional sanitizers, and migration instructions.
All 438 retained tasks declare schema v2 and task-owned evaluation actions. GPU qualification and agent campaign results are tracked separately from that migration. Task-family directory names remain useful selectors; no family has a separate configuration schema.
Create agents/your_agent/launch_agent.py with the interface used by main.py:
from agents import register_agent
@register_agent("your_agent")
def launch_agent(
eval_config: dict,
task_config_dir: str,
workspace: str,
) -> str:
# Build the prompt, invoke the agent in `workspace`, and return its output.
return resultThen:
- Add the name to
AgentTypeinsrc/module_registration.py. - Add its import branch in
load_agent_launcher. - Select the standard or custom prompt-building path in the launcher.
- Add it to
load_post_processing_handlerif it should use normal run aggregation. - Add an
agent_config.yamland focused documentation for agent-specific settings.
Read Task definition, schema, and authoring before changing task code, config, references, or harnesses. Follow its authoring workflow. When converting an external legacy task, also read the compatibility and migration guidance.
New tasks and material task-contract/harness changes require a fresh
framework-finalized validation_report.yaml on compatible GPU hardware before
PR submission. A clean pass has overall_status: PASS; WARN needs an explicit
maintainer-approved justification. See the
validator guide for the Docker command and
report interpretation.
- Held-out evaluation: Generate unseen shapes and evaluate generalization.
- Visualization dashboard: Compare local experiment reports.
- Full documentation: Installation, configuration, task authoring, parallel execution, and methodology.
- Improve interactive A/B comparison and experiment tracking.
- Export richer episode traces and reward data for external agent-RL pipelines.
- Expand standardized private held-out coverage.
- Support heterogeneous agent configurations within one multi-GPU experiment.
- Continue expanding task coverage across the supported kernel environments.