Conversation
Every $ld:ai:* event that execute_and_track sends carries the LaunchDarkly environment id, because AI Config Monitoring needs it to match a trace to the config that produced it. The two payloads built outside that path did not: make_track_data, used by every native graph adapter, and the graph-level graph_track_data in graph.py. Both now read the id through the same _try_get_environment_id helper, and omit the key when there is no id to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A native graph adapter has no track data of its own: make_track_data describes one node, and the run as a whole is the graph flag. Add make_graph_track_data, which names the graph flag as both the config key and the graph key, the same choice the SDK's own graph runner makes. Adapters pass it to set_ld_span_attributes in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
to_openai_agents, to_claude_agents and to_lang_graph each open a launchdarkly.graph span that carried only the graph key and token counts. The AI Config Monitoring tab finds a trace by a feature_flag span event and the config key, so none of these runs appeared against their AI Config. Tag the span through set_ld_span_attributes, the same helper every handler already uses, so the graph span now carries launchdarkly.config.key, launchdarkly.variation.key, launchdarkly.run.id, the context keys, and the feature_flag event with feature_flag.set.id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
bugbot run |
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Reviewed with the 1.0 freeze in mind. The fix is right: native graph spans now carry the config identity Monitoring needs, and the environmentId handling matches execute_and_track. Tests pass locally on 6c19b28 (1407 passed, 11 skipped).
Two things I'd like settled before GA, because they are hard to change once 1.0 ships:
- New public name.
make_graph_track_datais exported from the package root. The 1.0 API-surface plan makesmake_track_datainternal and renames itmake_node_track_data, so this one should be internal too. Same formakeGraphTrackDatain launchdarkly/js-ai-sdk#101. Details inline. - Graph span and graph events disagree on the config. The span now says
config.key = <graph key>. The$ld:ai:graph:invocation_success/duration:total/total_tokensevents in all three adapters still usemake_track_data(root, ...), so theirconfigKeyis the root node's key. The SDK's owngraph()uses the graph key for those events (graph.py_build_graph). That's not new in this PR, but event payloads are a data contract with Monitoring, and changing them after GA would quietly shift customers' dashboards. Could we pick one, either here or in a follow-up before GA? The monorepoTESTING.mddoesn't say whichconfigKeygraph events carry, so it should be written down there too.
Smaller notes:
- The SDK's own
graph()span (graph.py:919invoke,:1135stream) still doesn't getset_ld_span_attributes, and neither does JSgraph.ts:722. After this PR, native-adapter graph spans can be found by config key butgraph()spans can't. Is that intended, or does Monitoring reachgraph()traces through the node spans? launchdarkly.graph.keyis now set twice on the native span: once directly, then again byset_ld_span_attributes. It's harmless; the direct call could go.- Not from this PR: in openai-agents the span starts before the
trythat ends it, so aValueErrorduring agent setup (the "not built" checks) leaves the span un-ended and never exported. JS #101 adds an "ends the span once" test. Python may want the same. - I didn't find a matching ai-sdks-monorepo spec PR. The span attributes and
feature_flagevent onlaunchdarkly.graphare worth adding toTESTING.mdso both SDKs stay in step.
| "init_evaluations", | ||
| # utils | ||
| "create_handler", | ||
| "make_graph_track_data", |
There was a problem hiding this comment.
This adds a new name to the public surface that freezes at 1.0. Only the three adapter packages call it, so it's internal plumbing. The API-surface plan moves make_track_data to internal (as make_node_track_data) for the same reason. Could the adapters import it from launchdarkly_ai_server.utils instead, so it stays out of __all__? JS #101 has the same export in index.ts.
| track_data: dict[str, Any] = { | ||
| "runId": run_id, | ||
| "configKey": graph_key, | ||
| "variationKey": "", |
There was a problem hiding this comment.
variationKey is always "" here (and version always 1), while graph() reports the real values from the graph flag's meta. Native-graph spans will always have an empty launchdarkly.variation.key, so per-variation Monitoring views won't work for them. If GraphDefinition exposes meta later, this signature (graph_key, run_id) has to change. That's one more reason to keep the helper private until then.
| set_ld_span_attributes( | ||
| span, | ||
| { | ||
| "__ld": make_graph_track_data(def_obj.key, run_id), |
There was a problem hiding this comment.
The span's config key is now the graph key, but invocation_success / duration:total / total_tokens below (lines ~279 and ~316) still use make_track_data(root, ...), so those events carry the root node's key. Same in claude-agents and langchain-agents. The SDK graph() runner keys those events to the graph. Should the native adapters do the same, so a graph run means the same thing whichever runner you use?
| **model_stamps_from_meta(meta), | ||
| "graphKey": key, | ||
| } | ||
| _environment_id = _try_get_environment_id() |
There was a problem hiding this comment.
Good to see environmentId here. The launchdarkly.graph span this runner opens (lines ~919 and ~1135) still doesn't call set_ld_span_attributes, though, so graph() spans don't carry launchdarkly.config.key or the feature_flag event that native-adapter spans now have. Is that intended?
The launchdarkly.graph span that graph() opens, on invoke and on stream, carried only launchdarkly.graph.key. It now goes through set_ld_span_attributes with the same track data its graph events use, so it has the config key, variation, run id, context keys and the feature_flag event, like the native adapters' graph span. TELEMETRY-CONTRACT.md section 10 now says which config key the graph span and the graph-level events carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ternal The $ld:ai:graph:invocation_success, invocation_failure, duration:total and total_tokens events from the three native adapters carried the root node's config key, while their graph span and graph() use the graph key. They now use make_graph_track_data, so a graph run reports the same config key whichever runner produced it. make_graph_track_data is no longer exported from the package root. Only the adapters use it, and its signature will change if GraphDefinition gains the graph's variation metadata, so it stays out of the 1.0 surface. The adapters no longer set launchdarkly.graph.key by hand, since set_ld_span_attributes already does. to_openai_agents and to_lang_graph now open the graph span only after the agents and graph are built. A setup error such as a child agent that was not built used to leave a started span that was never ended or exported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit d693d68. Configure here.
2 participants
Summary
Spans from the native graph adapters (
to_openai_agents,to_claude_agents,to_lang_graph) had nothing tying them to the graph's AI Config, so Monitoring couldn't find them by config key.launchdarkly.graphspan now getsset_ld_span_attributes, like every other handler: config key, run id, context keys and thefeature_flagevent. New helper:make_graph_track_data.LD_ENVIRONMENT_IDis set.$ld:ai:graph:invocation_successand friends) now use the graph key asconfigKey, matching the span andgraph(). They used the root node's key.graph()runner's ownlaunchdarkly.graphspan gets the same identity.to_openai_agentsandto_lang_graphopen the span only after setup, so a setup error no longer leaves it un-ended.make_graph_track_datastays internal (imported fromlaunchdarkly_ai_server.utils, not in__all__).TELEMETRY-CONTRACT.mdsection 10 says which config key graph spans and events carry.JS twin: launchdarkly/js-ai-sdk#101.
Testing
🤖 Generated with Claude Code
Note
Overview
Aligns
launchdarkly.graphtelemetry with AI Config Monitoring so traces and metrics can be found by graph/config key, matching the TypeScript SDK (js-ai-sdk#101).All
launchdarkly.graphspans—fromgraph()and the OpenAI/LangChain/Claude native adapters—now go throughset_ld_span_attributes(config key, run id, context keys,feature_flagevent) instead of only settinglaunchdarkly.graph.key. A new internal helpermake_graph_track_databuilds graph-level track payloads with graph key asconfigKey(replacing the root node key on graph events likeinvocation_success,duration:total, andtotal_tokens). Node-level events still use per-node keys.environmentIdis added to track payloads whenLD_ENVIRONMENT_IDis set (make_track_data,make_graph_track_data, and SDKgraph()build path). Native adapters use emptyvariationKeyand version1on graph spans/events;graph()keeps real variation metadata.OpenAI and LangChain adapters defer opening the graph span until after setup so setup failures do not leave spans un-ended.
TELEMETRY-CONTRACT.md§10 documents the contract.Reviewed by Cursor Bugbot for commit d693d68. Bugbot is set up for automated code reviews on this repo. Configure here.