feat(validate): catch broken references and show findings on the PR

AGENTS.md

@@ -133,7 +133,8 @@ precisely.
 3. **Validate:** `npm run validate -- <org>` (offline). CI's **Validate
    resources** check runs it for every org on every PR; if that check fails,
    run it locally for the org it names and fix the errors. Don't weaken the
-   check or the workflow to get past it.
+   check or the workflow to get past it, and never edit the state file to
+   make a reference resolve: fix the name, or pull.
 4. **Build PR checks offline** if `vapi-checks.yml` exists:
    `npm run check -- --all --dry-run`. Fix anything it reports.
 5. **Deploy only with a yes** (safety rule 1): `npm run apply -- <org>`, or

docs/guides/commands.md

@@ -10,7 +10,7 @@ The other commands are direct only.
 | Command | Usage | What it does |
 | --- | --- | --- |
 | `npm run setup` | `npm run setup [-- <org>]` | Connect an org: creates `.env.<org>` and `resources/<org>/`. |
-| `npm run validate` | `npm run validate -- <org>` | Check resource files offline. Run it before every `apply`. |
+| `npm run validate` | `npm run validate -- <org>` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. |
 | `npm run apply` | `npm run apply -- <org> [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). |
 | `npm run pull` | `npm run pull -- <org> [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. |
 | `npm run push` | `npm run push -- <org> [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. |

docs/guides/troubleshooting.md

@@ -2,7 +2,8 @@
 
 ## "Reference not found" warnings
 
-The referenced resource doesn't exist. Check:
+The referenced resource doesn't exist. `npm run validate` reports these as
+`dangling-reference` errors before you deploy. Check:
 
 1. File exists in correct folder
 2. Filename matches exactly (case-sensitive)
@@ -30,7 +31,9 @@ bypassed).
 
 ## "Credential with ID not found" errors
 
-The credential UUID doesn't exist in the target org. Fix:
+The credential UUID doesn't exist in the target org. `npm run validate`
+warns about a credential name that isn't in the state file
+(`unresolved-credential`). Fix:
 
 1. Run `npm run pull -- <org>` to fetch credentials into the state file
 2. If the credential doesn't exist, create it in the Vapi dashboard with the same name
@@ -92,11 +95,23 @@ findings:
 npm run validate -- <org>
 ```
 
-Each error names the file, field and rule. Plain `push` only warns about
-these errors, so a repository that has been deploying with `push` can carry
-some from before the check existed; they show up on the next pull request,
-whatever it changes. Fix them in that PR or a separate one first. `apply`
-refuses to deploy until they're fixed anyway.
+Each finding names the file and the rule, and on GitHub it's also shown on
+the file in the pull request. Plain `push` only warns about these errors (unless `--strict`), so
+a repository that has been deploying with `push` can carry some from before
+the check existed; they show up on the next pull request, whatever it
+changes. Fix them in that PR or a separate one first. `apply` refuses to
+deploy until they're fixed anyway.
+
+| Rule | Severity | What to do |
+| --- | --- | --- |
+| `dangling-reference` | error | A reference names no local file and no state entry. Fix the name (it's the file name without extension, including any folder), or run `npm run pull -- <org>` if the resource was created in the dashboard. Don't add a state entry by hand. |
+| `override-tool-by-name` | error | References inside `assistantOverrides`, `membersOverrides` and `targetOverrides` aren't resolved. Put the tool inline in the override's `model.tools`. |
+| `reference-to-ignored` | error | The referenced resource matches `.vapi-ignore`, so it's never deployed. Stop ignoring it, or remove the reference. |
+| `name-length` | error | Shorten the name to 40 characters or fewer. |
+| `voice-provider-schema` | error | Move the setting to where that voice provider expects it; the message says where. |
+| `unresolved-credential` | warning | The credential name isn't in the state file. Run `npm run pull -- <org> --bootstrap` and commit the state file, or create the credential in the dashboard first. |
+| `reference-by-uuid` | warning | A UUID only exists in one org and breaks promotion. Reference the file by name. Vapi's stock personalities are exempt. |
+| `so-assistant-lockstep`, `prompt-duplicate-*`, `max-tokens-floor` | warning | Follow the message; see [structured outputs](../learnings/structured-outputs.md) and [writing prompts](writing-prompts.md). |
 
 A folder under `resources/` that isn't a valid org name (lowercase letters,
 digits and hyphens) fails too. Rename it, or move it out of `resources/`.

improvements.md

@@ -82,7 +82,7 @@ you which stack PR closes the row.**
 | 28  | Handoff tools 400 on first push into an empty org          | Push aborts before the assistant-linking pass runs  | None       | RESOLVED 2026-08-01                             |
 | 29  | SO linking sent filtered `assistantIds` arrays              | Silent unlink of live-but-untracked assistants      | None       | RESOLVED 2026-08-03 (#51)                       |
 | 30  | Tool-linking pass could PATCH a raw assistant slug          | Mid-push 400 naming the wrong resource              | None       | RESOLVED 2026-08-03 (#51)                       |
-| 31  | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Open                                            |
+| 31  | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | RESOLVED 2026-10-03 (validation)                |
 | 32  | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56)                      |
 | 33  | `npm run sim` reported every run as passed                | A failing suite exited 0 — false green             | None       | RESOLVED 2026-10-01                             |
 | 34  | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green           | #33        | RESOLVED 2026-10-01                             |
@@ -1568,6 +1568,8 @@ the repo and a subsequent push runs.
 
 ## 31. Unresolved references are handled three different ways depending on the field, and `validate.ts` has no dangling-reference check
 
+**[RESOLVED 2026-10-03]** by validation; the three runtime behaviours remain.
+
 **Discovered:** while fixing #29 and #30 — those two entries close the
 loudest and quietest failure modes for their specific fields, but the
 underlying question ("what happens when a reference resolves to nothing")
@@ -1646,9 +1648,24 @@ conditions; it doesn't require picking one runtime behavior (filter vs.
 defer vs. 400) for every field, since it stops the push before any of those
 three behaviors gets a chance to run.
 
+### Possible fix (landed)
+
+`src/validate-refs.ts` reuses the `extractReferencedIds` walk, plus scenario
+judges' `evaluations[].structuredOutputId`, and reports an error for any
+name that matches no local file and no state entry (`dangling-reference`).
+Names matched by `.vapi-ignore` are left to `reference-to-ignored`, which
+`npm run validate` now runs too. Alongside it: `override-tool-by-name`
+(an error: push never resolves `toolIds` inside overrides),
+`unresolved-credential` and `reference-by-uuid` (warnings). `validate`
+reads the committed state file, so the check runs offline and in CI, and
+`apply` stops on it before its pull. `push` runs the same checks with its
+other validators: warnings by default, blocking under `--strict`.
+
 ### Status
 
-**Open.**
+**RESOLVED 2026-10-03** by validation. The runtime still filters, defers or
+sends an unresolved reference depending on the field, but `validate` and
+`apply` stop before any of that runs.
 
 ---
 

src/push.ts

@@ -39,6 +39,7 @@ import {
   validateNoIgnoredReferences,
   validateResources,
 } from "./validate.ts";
+import { validateReferences } from "./validate-refs.ts";
 
 // Map a resource label to its state-file key. Used for snapshotting —
 // snapshot directories are keyed by the same names the state file uses.
@@ -1714,6 +1715,12 @@ async function main(): Promise<void> {
       // a config that references an ignored resource is a contradiction the
       // operator should see.
       ...validateNoIgnoredReferences(loadedResources, loadIgnorePatterns()),
+      ...validateReferences({
+        loaded: loadedResources,
+        org: VAPI_ENV,
+        state,
+        ignorePatterns: loadIgnorePatterns(),
+      }),
     ];
     if (findings.length > 0) {
       console.log(summarizeFindings(findings));

src/state.ts

@@ -48,7 +48,7 @@ function migrateSection(
 // State Management
 // ─────────────────────────────────────────────────────────────────────────────
 
-function createEmptyState(): StateFile {
+export function createEmptyState(): StateFile {
   return {
     credentials: {},
     assistants: {},

src/validate-cmd.ts

@@ -5,19 +5,32 @@
 // and prints findings. Exit code 0 if no errors, 1 if any error-severity
 // finding is present.
 
-import { resolve } from "path";
+import { existsSync } from "fs";
+import { relative, resolve } from "path";
 import { fileURLToPath } from "url";
-import { VAPI_BASE_URL, VAPI_ENV } from "./config.ts";
+import {
+  BASE_DIR,
+  loadIgnorePatterns,
+  STATE_FILE_PATH,
+  VAPI_ENV,
+} from "./config.ts";
 import { loadResources } from "./resources.ts";
+import { createEmptyState, loadState } from "./state.ts";
 import type { LoadedResources } from "./types.ts";
-import { summarizeFindings, validateResources } from "./validate.ts";
+import {
+  findingAnnotation,
+  summarizeFindings,
+  validateNoIgnoredReferences,
+  validateResources,
+} from "./validate.ts";
+import { validateReferences } from "./validate-refs.ts";
 
 async function main(): Promise<void> {
   console.log(
     "═══════════════════════════════════════════════════════════════",
   );
   console.log(`🔎 Vapi GitOps Validate - Environment: ${VAPI_ENV}`);
-  console.log(`   API: ${VAPI_BASE_URL}`);
+  console.log("   Offline: no API calls");
   console.log(
     "═══════════════════════════════════════════════════════════════\n",
   );
@@ -35,9 +48,33 @@ async function main(): Promise<void> {
     evals: await loadResources("evals"),
   };
 
-  const findings = validateResources(resources);
+  // References resolve through the committed state file, as they do on push.
+  const stateExists = existsSync(STATE_FILE_PATH);
+  if (!stateExists)
+    console.log("📄 No state file yet: references must name local files.");
+  const state = stateExists ? loadState() : createEmptyState();
+  const ignorePatterns = loadIgnorePatterns();
+  const findings = [
+    ...validateResources(resources),
+    ...validateNoIgnoredReferences(resources, ignorePatterns),
+    ...validateReferences({
+      loaded: resources,
+      org: VAPI_ENV,
+      state,
+      ignorePatterns,
+    }),
+  ];
   console.log(`\n${summarizeFindings(findings)}\n`);
 
+  if (process.env.GITHUB_ACTIONS === "true") {
+    for (const finding of findings) {
+      const file = resources[finding.type].find(
+        (r) => r.resourceId === finding.resourceId,
+      )?.filePath;
+      console.log(findingAnnotation(finding, file && relative(BASE_DIR, file)));
+    }
+  }
+
   const errorCount = findings.filter((f) => f.severity === "error").length;
   if (errorCount > 0) {
     console.error(