Skip to content

feat(evals): add adversarial agent tool-use scenarios #8513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sudoKrishna wants to merge 9 commits into from
+2,015 −0
Closed

feat(evals): add adversarial agent tool-use scenarios #8513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
86c79d7
feat(evals): add agent tool-use evaluation harness
sudoKrishna Sep 29, 2026
af2e6ca
feat(evals): add live DeepSeek model runs to the agent eval suite
sudoKrishna Sep 29, 2026
433fd0f
test(evals): assert grounded behavior instead of exact phrasing in li…
sudoKrishna Sep 29, 2026
85525a4
feat(evals): run agent scenarios through the DAGExecutor
sudoKrishna Sep 29, 2026
fb7f54c
test(evals): assert executor block retry on agent failure
sudoKrishna Sep 29, 2026
c93426b
test(evals): assert executor model fallback on primary failure
sudoKrishna Sep 29, 2026
9d2ec19
feat(evals): add adversarial agent tool-use scenarios
sudoKrishna Oct 1, 2026
cdebc4d
test(evals): fix adversarial scenario design after the live run
sudoKrishna Oct 1, 2026
fc549db
test(evals): fix chain and near-duplicate scenarios for live models
sudoKrishna Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions apps/sim/evals/README.md
View file
Open in desktop
81 changes: 81 additions & 0 deletions apps/sim/evals/agent-tool-use/agent-tool-use.eval.test.ts
View file
Open in desktop
83 changes: 83 additions & 0 deletions apps/sim/evals/agent-tool-use/agent-tool-use.live.test.ts
View file
Open in desktop
351 changes: 351 additions & 0 deletions apps/sim/evals/agent-tool-use/executor-harness.ts
View file
Open in desktop
418 changes: 418 additions & 0 deletions apps/sim/evals/agent-tool-use/harness.ts
View file
Open in desktop
61 changes: 61 additions & 0 deletions apps/sim/evals/agent-tool-use/live.ts
View file
Open in desktop
129 changes: 129 additions & 0 deletions apps/sim/evals/agent-tool-use/report.ts
View file
Open in desktop
600 changes: 600 additions & 0 deletions apps/sim/evals/agent-tool-use/scenarios.ts
View file
Open in desktop
147 changes: 147 additions & 0 deletions apps/sim/evals/agent-tool-use/types.ts
View file
Open in desktop
2 changes: 2 additions & 0 deletions apps/sim/package.json
View file
Open in desktop