Skip to content

refactor: move resource parsing and org connection helpers into config-free modules - #59

Merged
scott-lowe-vapi merged 1 commit into
mainfrom
refactor/config-free-engine-modules
Oct 3, 2026
Merged

scott-lowe-vapi merged 1 commit into
mainfrom
refactor/config-free-engine-modules

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: project — PR 3 of 10 for inline simulation PR checks (TEST-141); this PR is a behaviour-preserving refactor.

  • Problem: the PR check (PRs 4–7) has to read an org's resource files and resolve org API keys. All the code that does that sits behind config.ts, which parses argv, binds one org and can process.exit at import time. Copying it into the check would let push and the check drift apart, so the check could test a different payload from the one push sends.
  • Who it affects: gitops users, whose PR checks must test exactly what push would deploy; and maintainers, who get one copy of the loader instead of two (three for the .md system-prompt injection).
  • What changes: code moves into two config-free modules. resources.ts, config.ts and promote-cmd.ts delegate to them and keep their exports, logs and error messages.
    • src/resource-parse.ts:
      • from resources.ts: FOLDER_MAP, VALID_EXTENSIONS, frontmatter/file parsing, the directory scan and the loader;
      • from config.ts: .vapi-ignore reading and matching;
      • new: orgResourcesRead(rootDir, org), which returns every resource of an org keyed type:id;
      • the three copies of the .md body → system message injection become markdownResourceParse.
    • src/org-connection.ts: envValue, tokensParse(envName), connectionLoad and childRun, moved from promote-cmd.ts and parameterised on the root dir, the token variable and the base URL.
    • api.ts: takes VapiApiError, parseApiMessage, shouldRetry and the backoff constants from vapi-client.ts, and re-exports VapiApiError. Both clients now share one error class (so instanceof checks see one class) and one retry rule.
    • promotion.ts:
      • imports FOLDER_MAP / VALID_EXTENSIONS instead of keeping its own copies;
      • exports SLUG_RE, promotionBindingsParse, promotionBindingsResolve, promotionBindingsApply and PromotionBindingsResolved, for the check config (PR 4) and the payload builder (PR 5).
    • Also dropped: the unused FOLDER_TO_TYPE map in resources.ts.

Evidence of value

The output is byte-identical before and after. The same fixture org was run at both refs against a local stub API. Nothing reached a real org.

The fixture:

  • an .md assistant whose frontmatter has a system message the body must replace;
  • a nested assistant;
  • an ignored assistant;
  • a backup copy and an unsupported file;
  • a .ts tool and a YAML tool;
  • a squad, a structured output and all four simulation types;
  • a two-org promotion.yml.
Command fix/sim-false-green (06a1492) This branch (0518af1)
push ev-dev --dry-run (new-file gate refusal) identical identical
push ev-dev --dry-run --allow-new-files (full plan, every would-POST) identical identical
loadResources JSON for every type (full parsed data, incl. the .md system message) identical identical
validate ev-dev identical identical
promote --pipeline release --from ev-dev --to ev-prod (plan) identical identical
Combined stdout + stderr + stub request log sha256 3f3b609f…1d3b, 14,927 bytes sha256 3f3b609f…1d3b, 14,927 bytes
  • Dry-run placeholder IDs (dry-run-post-<Date.now()>) and the stub's port are normalised before hashing.
  • --allow-new-files is passed only because every fixture file is new by construction.

Tests: npm test goes from 377 to 398 passing (21 new), and npm run build (src + tests) is clean.

Testing plan

  • tests/resource-parse.test.ts (temp-dir fixtures) covers:
    • .md body replacing a frontmatter system message, and an empty body;
    • missing frontmatter;
    • .md parsing being the same for parseResourceDataFromFile and the loader;
    • sorted order across .yml/.yaml/.ts and nested dirs;
    • hidden and .bkp files skipped;
    • duplicate IDs refused, with the exact YAML-not-an-object message;
    • missing directory;
    • .vapi-ignore comments, blanks and !;
    • * vs ** vs ? matching;
    • orgResourcesRead reading one org and applying its ignore file, and the caller's override.
  • tests/org-connection.test.ts covers:
    • plain and quoted .env values;
    • token-map parsing, with the variable named in every error;
    • token precedence (map, then .env.<org>) and base URL precedence (configured, then .env.<org>);
    • the missing-token error;
    • childRun passing the org and key to a real child, and dropping an inherited VAPI_BASE_URL;
    • a failing child.
  • The existing push dry-run, .vapi-ignore push, promotion, audit and cleanup-safety suites pass unchanged.
  • Not tested:
    • a push against a real org (the comparison uses a stub API that returns empty lists, so update/PATCH paths for existing resources aren't exercised; the code they call is unchanged);
    • promote --apply (the moved connectionLoad/childRun are covered by unit tests, but no live child pull/apply was run);
    • Node 20 locally (left to CI).

Stacked on #58. Review with git diff --color-moved=dimmed-zebra fix/sim-false-green... to see that most of the diff is moved lines.

Refs TEST-141

🤖 Generated with Claude Code

scott-lowe-vapi commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Merge activity

  • Oct 3, 5:59 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Oct 3, 6:03 AM UTC: Graphite rebased this pull request as part of a merge.
  • Oct 3, 6:03 AM UTC: @scott-lowe-vapi merged this pull request with Graphite.

@scott-lowe-vapi
scott-lowe-vapi changed the base branch from fix/sim-false-green to graphite-base/59 October 3, 2026 06:00
@scott-lowe-vapi
scott-lowe-vapi changed the base branch from graphite-base/59 to main October 3, 2026 06:01
…g-free modules

The PR check has to read an org's resource files and resolve org keys
without importing config.ts, which parses argv and exits at import time.
Move the code rather than copy it, so push and the check read files the
same way:

- src/resource-parse.ts: FOLDER_MAP, VALID_EXTENSIONS, frontmatter and
  file parsing, the directory scan and loader, .vapi-ignore reading and
  matching, and a new orgResourcesRead(rootDir, org). The two copies of
  the .md body -> system message injection become one function.
- src/org-connection.ts: envValue, tokensParse(envName), connectionLoad
  and childRun from promote-cmd.ts, parameterised on the root dir, token
  variable and base URL.
- api.ts takes VapiApiError, parseApiMessage, shouldRetry and the
  backoff constants from vapi-client.ts and re-exports VapiApiError.
- promotion.ts imports FOLDER_MAP/VALID_EXTENSIONS and exports SLUG_RE
  and the bindings parse/resolve/apply helpers.

resources.ts, config.ts and promote-cmd.ts delegate, and keep their
exports, logs and error messages unchanged.

Refs TEST-141

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scott-lowe-vapi
scott-lowe-vapi force-pushed the refactor/config-free-engine-modules branch from 0518af1 to 8de0fed Compare October 3, 2026 06:02
@scott-lowe-vapi
scott-lowe-vapi merged commit 93deaf3 into main Oct 3, 2026
2 checks passed
scott-lowe-vapi added a commit that referenced this pull request Oct 3, 2026
## Value

**V.A.L.U.E. tier:** project — PR 4 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this PR adds a config parser with no caller yet.

- **Problem:** a customer needs one place to say which agents to test, with which suites, in which org, and how strictly. `promotion.yml` can't hold it: `promotionConfigParse` requires a pipeline of two or more orgs, and most customers have one.
- **Who it affects:** gitops users setting up PR checks, single-org and CI-org alike. The PR check (PRs 5–8) and the promotion gate (PR 10) both read this file, so the gate runs the same tests as the PR check.
- **What changes:**
  - `src/check-config.ts` parses a root `vapi-checks.yml` into typed check definitions:
    - `org`, plus an optional `runOrg` and `baseUrl`;
    - `targets` (`assistants/<id>` or `squads/<id>`, nested IDs allowed);
    - `suites` and/or `simulations`;
    - `bindings` (promotion's shape, via `promotionBindingsParse`);
    - extra `paths`;
    - run settings with repo-wide `defaults`: transport `chat`, 1 iteration, 20 min, `toolMocks: strict`, `stripWebhooks: true`.
  - `vapi-checks.example.yml` is the commented starting point: a single-org check, plus a commented CI-org check.
  - Every key is validated, and unknown keys fail, so a typo can't silently run fewer tests. A `mode:` key fails with a message explaining that checks always build from the branch, since a "run what's deployed" mode was deliberately left out.

## Evidence of value

| Check | Result |
|---|---|
| Valid configs | Minimal check (defaults applied, `runOrg` = `org`, credentials `bind` / phones `omit`); default and per-check overrides; a CI-org check with EU `baseUrl` and bindings |
| Rejections | 31 invalid shapes, each asserted against its exact message naming the field (table in `tests/check-config.test.ts`) |
| Shipped example | Parses as-is, and the commented CI-org check parses once uncommented, so the docs can't drift from the parser |
| `npm test` | 398 → 404 passing; `npm run build` clean |

## Testing plan

- `tests/check-config.test.ts` covers:
  - parse results for valid configs;
  - one table row per rejection: version, empty or unknown keys at each level, `mode`, slugs, target shape, extension and `..` in IDs, duplicates, missing suites/simulations, each setting's range, `baseUrl`, `paths`, bindings;
  - the example file;
  - `checksConfigLoad`, which returns `null` when no `vapi-checks.yml` exists (checks are opt-in).
- **Not tested:** nothing uses the config yet. Resolving targets and suites against real files is PR 5, and the `baseUrl` fallback order (`.env.<runOrg>`, `$VAPI_BASE_URL`, `api.vapi.ai`) is applied at run time in PR 7.

Stacked on #59.

Refs TEST-141

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants