[Perf]: pygraph.execute host overhead ~30 us above cudnnBackendExecute per call

[YangXu1990uiuc]

Summary

pygraph.execute() costs ~45-50 µs of host time per call on the python frontend for an SDPA graph with 13 tensors (4 ragged offsets + 2 cu_seq_lens + Q/K/V/O/Stats + Stats ragged offset), of which only ~14 µs is cudnnBackendExecute (measured by interposing the symbol the frontend resolves with dlsym). The remaining ~30 µs is the binding: python dict → std::unordered_map<int64_t, void*>, the override_uids/override_shapes/override_strides lists when execute-time override is used, workspace handling, and the handle lookup.

For comparison FlashInfer's FA2 path launches its kernel through TVM-FFI in ~14 µs total, so on small serving steps (decode-context-parallel verify, chunk remainders) the frontend glue is the difference between the two backends.

Environment

cudnn-frontend 1.30 (release pybind), cuDNN 9.27, torch 2.13, SM100 box; medians over 2000 calls with the GPU idle.

Suggestion

A pre-bound execution object: build the variant pack once from {uid: tensor} (or accept raw pointers / a preallocated int64_t*+void** pair), and let execute take only the pointers that changed (plus override shapes as a flat buffer). Same spirit as the stream caching already done in cudnn.Handle (set_stream is now 0.09 µs when unchanged vs 1.5 µs raw).

_{note to self: claude::61d24ed2-7c90-4a97-9cbb-b91ae18136fd — "flashinfer frost prefill/GEMM enablement" · cwd /home/scratch.yanxu_libs/flashinfer}

[YangXu1990uiuc]

Prototype of the bottom-up route (pointer/int kernel ABI, then plan-time prepared launch), FROST SM100 THD f16: pygraph.execute 149 → 86 → 29 µs, FlashInfer suites green under FROST: #1099. The remaining 29 is the CuTe JIT executor call (~13), the graph layer (~7) and make_ptr (~6); a bind API / exported C entry is the optional next step.

[YangXu1990uiuc]

Status after #1119 (merged d469317de), measured on the FlashInfer-shaped THD graph (b=64 × q4, kv 1024, GQA 8/2, d128 bf16, Stats, ragged declaration), public graph.execute(), SM100 dev box, uninstrumented medians:

tree	host µs	where the time went
develop before #1119	132–137	per-execute variant-pack dict + binding re-resolution (~16), adapter THD pack rewrite: memset launch, stream contexts, workspace carver, as_strided views, compile(**kwargs) lru re-key (~65), DLPack tensor marshalling ×10 (~15)
#1119, f16/bf16 SM100 / SM107 THD	≈30	_uid_to_data 1.1 · _normalize 4.5 · bind_thd (Python) 10 · positional tvm_ffi call 6.6 · plan / graph glue ≈6
cuDNN backend plan, same graph	24	reference

Kernel device time and outputs are unchanged (SM100 and SM107 A/B tables in the PR). What is left to reach the 8–12 µs end game, in order: (1) dense f16 through the same positional entry (today the dense arm still goes through the DSL keyword wrapper + make_ptr, ≈35 µs); (2) a native batched override (override_many(indices, shapes, strides), one crossing) — the bounded-override case costs +45 µs today, all of it the graph layer's Python override loop, and the backend plan pays the same loop (24 → 35 with override); (3) a native binder replacing the 10 µs Python bind_thd; (4) the other kernel families (FP8 / MXFP8, decode tiles, SM120, SM80) joining the explicit entry one PR each, deleting their tensor arms. #1099 stays closed as the reference for (4).

_{note to self: claude::61d24ed2-7c90-4a97-9cbb-b91ae18136fd — "flashinfer frost prefill/GEMM enablement" · cwd /home/scratch.yanxu_libs/flashinfer}

[YangXu1990uiuc]

Status after #1119 and #1132 (merged today as 5567e2733), local SM100 box, public graph.execute(), medians of 500:

	develop before #1119	now
FROST THD f16 (FlashInfer-shaped b64 × q4, kv 1024, d128 bf16, Stats), fixed geometry	97.5 µs	31
same, bounded batch override (32 of 64, 10 operands)	98 (override ignored)	36
FROST dense bf16 b2 h8 s1024 d128, fixed / override	28 / 28 (ignored)	31 / 34 (honored)
backend plan, THD fixed / override	12.4 / 15.1	11.6–12.3 / 14.6–15.2 (unchanged)

Through FlashInfer's ragged prefill wrapper with FROST opted in: 124 → 88 µs per call; backend 67. Remaining FROST-vs-backend gap at fixed geometry ≈ 19 µs = the Python binders (bind_thd / bind_dense, ~13 µs) + the positional call; next: native binder, and the decode tiles onto the explicit ABI (FlashInfer decode is still on the tensor arm at ~95 µs).

[0z5a]

Hi @YangXu1990uiuc, I'd like to take the pre-bound execution path proposed in this issue.

Proposed scope:

• Prototype a fixed-plan execution binding that reuses UID/slot metadata and host-side argument storage, with a lightweight path for updating tensor pointers and execute-time overrides.
• Keep device workspace caller-owned and preserve the existing execute API. Define plan invalidation, device/stream handling, tensor lifetime, and concurrent-execution behavior explicitly.
• Benchmark host submission overhead separately from backend execution and GPU time on RTX 5090, starting with small SDPA workloads and then checking per-rank submission latency in 2/4/8-GPU runs.

This would focus on the Python/C++ execution-binding layer, separately from the dtype/stride memoization in #589 and the workspace changes in #1181.

Could you assign this scope to me if it is available? Please also flag any internal execution-binding work or preferred API direction so the implementation can align with it.