Add api-contract-audit skill for type hint checking

doc/changes/unreleased.md

@@ -2,7 +2,7 @@
 
 ## Features
 
-- #940: Added shared validation for packaged agent skills and the `skills:check` Nox session.
-- #938: Added the `skills:install` Nox session for installing the packaged PTB agent skill.
-
-## Summary
+* #940: Added shared validation for packaged agent skills and the `skills:check` Nox session.
+* #938: Added the `skills:install` Nox session for installing the packaged PTB agent skill.
+* #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime
+  behavior

exasol/toolbox/skills/api-contract-audit/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: api-contract-audit
+description: Audit a Python library's public API for inconsistencies between type annotations, docstrings, user-facing documentation/examples, and actual runtime behavior. Use when reviewing API changes, checking whether public methods accept undocumented parameter shapes, or validating that docs and type hints match enforcement in code.
+---
+
+# API Contract Audit
+
+Use this skill when the task is to review a Python package's public API contract rather than implement features.
+
+Focus on externally visible behavior:
+- public functions and methods
+- exported classes
+- user-facing docs and examples
+- runtime validation and coercion
+
+Do not assume the annotation is the source of truth. The goal is to find drift between multiple sources of truth.
+
+## Inputs To Compare
+
+For each relevant public API entrypoint, compare:
+- signature and type annotations
+- docstring parameter and return descriptions
+- examples in docs, README, and example scripts
+- runtime behavior in the implementation path
+
+Treat these as separate claims. Report when they disagree.
+
+## What To Look For
+
+Prioritize these mismatch patterns:
+- annotation says `str`, but implementation accepts or requires tuple-like schema-qualified identifiers
+- annotation says one scalar type, but runtime hard-checks another with `isinstance(...)`
+- docstring says a parameter or return type that does not match the signature
+- docs/examples call the API with arguments that disagree with the annotation or actual signature
+- implementation silently accepts more forms than the public docs mention
+- wrappers expose narrower types than the lower-level public method they forward to
+- runtime coercion like `int(val)` or `str(val)` that makes the public contract broader than the annotation suggests
+
+Typical search signals:
+- `isinstance(`
+- `type(`
+- `raise ValueError`
+- identifier-formatting helpers
+- tuple-specific branches
+- wrapper methods that pass through parameters unchanged
+
+## Workflow
+
+1. Enumerate the public API surface relevant to the request.
+2. Read the implementation of each public method and the immediate downstream code it calls.
+3. Trace parameter handling until the real runtime constraint is clear.
+4. Cross-check docstrings and user-facing docs/examples.
+5. Report only concrete inconsistencies or clearly label residual uncertainty.
+
+Prefer `rg` for discovery. Good starter patterns:
+
+```bash
+rg -n "^class |^    def " package_dir
+rg -n "isinstance\\(|type\\(|raise ValueError|raise TypeError" package_dir
+rg -n "function_name\\(" README.md doc examples test
+```
+
+## Output Format
+
+Present findings first, ordered by severity.
+
+For each finding include:
+- severity: High, Medium, or Low
+- affected API
+- what the annotation/doc claims
+- what the implementation really does
+- file references for both sides of the mismatch
+
+After findings, optionally include:
+- open questions where intended behavior is unclear
+- a short summary of recurring patterns
+
+If no findings are discovered, say that explicitly and mention any coverage limits.
+
+## Severity Guidance
+
+- High: likely to mislead callers, break type-checked usage, or document the wrong accepted input shape
+- Medium: accepted behavior is real but under-documented, or docs/examples contradict each other
+- Low: naming, docstring argument labels, stale prose, or smaller clarity issues
+
+## Boundaries
+
+- Do not rewrite the API contract on your own. If code, docs, and examples disagree, report the disagreement.
+- Do not stop at the first example. Check for the same pattern across sibling APIs.
+- Do not treat private helper inconsistencies as findings unless they affect public behavior.

test/resources/skills/api-contract-audit/eval_cases.yml

@@ -0,0 +1,117 @@
+version: 1
+skill: "api-contract-audit"
+cases:
+  - id: "audit-public-api-contract"
+    category: "audit"
+    prompt: >
+      Audit the target Python library's public API for concrete inconsistencies between its
+      type annotations, docstrings, user-facing documentation or examples,
+      and actual runtime behavior. Report findings first, ordered by severity.
+      For each finding, identify the affected API, the claimed contract, the
+      observed behavior, and relevant source file references. Do not modify
+      the code or rewrite the API contract.
+    expected:
+      must_include:
+        - "severity"
+        - "affected API"
+        - "claimed contract"
+        - "observed behavior"
+        - "source file"
+      must_not_include:
+        - "change the implementation"
+        - "propose a code fix"
+
+  - id: "interpret-explicit-type-check"
+    category: "runtime-validation"
+    prompt: >
+      A public API implementation contains an isinstance() check. Audit
+      whether that check is consistent with the public annotation and
+      documentation. Explain what the check actually accepts or rejects and
+      whether the available evidence establishes a contract mismatch. Do not
+      treat the presence of isinstance() alone as proof of a defect.
+    expected:
+      must_include:
+        - "isinstance"
+        - "accepted"
+        - "rejected"
+        - "annotation"
+        - "documentation"
+        - "evidence"
+      must_not_include:
+        - "isinstance() alone proves a mismatch"
+        - "propose a code fix"
+
+  - id: "interpret-coercion-and-errors"
+    category: "runtime-validation"
+    prompt: >
+      Audit a public API whose implementation coerces or rejects values at
+      runtime, for example through int(), str(), ValueError, or TypeError.
+      Compare the accepted and rejected inputs with the public annotations and
+      documentation. Report any concrete contract mismatch, its severity, and
+      the relevant source file reference.
+    expected:
+      must_include:
+        - "coercion"
+        - "accepted inputs"
+        - "rejected inputs"
+        - "annotation"
+        - "documentation"
+        - "severity"
+        - "source file"
+      must_not_include:
+        - "change the implementation"
+        - "propose a code fix"
+
+  - id: "compare-docstrings-and-signatures"
+    category: "documentation"
+    prompt: >
+      Inspect two concrete public functions from the target Python library,
+      including a module-level function and a public method. Name the functions
+      in the report. Check whether their runtime signatures, type annotations,
+      and docstrings describe the same contract.
+      Use inspect.signature(), inspect.get_annotations(),
+      typing.get_type_hints(), and inspect.getdoc() where applicable to make
+      the comparison evidence-based instead of treating any single source as
+      authoritative. Report concrete mismatches with the affected function,
+      severity, and source file references. If no finding can be established,
+      state the coverage limitation instead of guessing.
+    expected:
+      must_include:
+        - "signature"
+        - "docstring"
+        - "annotation"
+        - "function"
+        - "inspect.signature"
+        - "inspect.get_annotations"
+        - "typing.get_type_hints"
+        - "inspect.getdoc"
+        - "mismatch"
+        - "severity"
+        - "source file"
+      must_not_include:
+        # The skill reports discrepancies; it must not infer a contract or
+        # prescribe implementation changes.
+        - "treat annotations as authoritative"
+        - "invent a finding"
+
+  - id: "compare-user-facing-examples"
+    category: "documentation"
+    prompt: >
+      Check whether the target Python library's user-facing examples in documentation, including
+      RST files, README content, and example scripts, agree with the public
+      API signature, annotations, and runtime behavior. Report concrete
+      inconsistencies with the affected API, severity, and source file
+      references. Do not assume an example is authoritative without checking
+      the implementation.
+    expected:
+      must_include:
+        - "example"
+        - "documentation"
+        - "signature"
+        - "implementation"
+        - "runtime"
+        - "severity"
+        - "source file"
+      must_not_include:
+        - "treat the example as authoritative"
+        - "propose a code fix"