Repository navigation
Missing parent trace activities (spans) when switching context between SDK and CLI #2803
Description
Activity
Investigated
dotnet/src/Telemetry.cs(RestoreTraceContext) and its call site indotnet/src/Session.cs(~line 701, tool handler invocation).Finding: this is a bug in the .NET SDK.
RestoreTraceContextcreatesnew Activity("copilot.tool_handler")outside anyActivitySource, parents it to the CLI'straceparent, and callsStart(). It becomesActivity.Currentwith a freshly generated span id. Any child activity created by a tool handler therefore gets that new span id as its parent id. Because the activity is never sampled or exported, the span is never collected, so child spans point at a parent that doesn't exist in the backend. This matches the tree in the report (external_tool→ invisiblecopilot.tool_handler→GET). The code comment says the activity is meant to "carry the remote parent context", but the extra span id defeats that goal.Likely fix: make the restored context's span id be the CLI's span id, so children reference
external_tooldirectly. Options are (a) creating the activity so itsIdequals the incomingtraceparent, or (b) starting a real activity from anActivitySourceso it is exported and forms a visible hop. Your suggestion of emitting it from a properActivitySourceis option (b).The CLI-side concern can't be verified from this repo, since the CLI isn't open source here, so it would need to be checked separately.
Labeling as
bug.Generated by Bug Handler for #2803 · copilot · auto · 15.9 AIC · ⌖ 5.42 AIC · ⊞ 7.4K · ◷
I don't think option (a) is, well, an option. 😕 I don't think
Activityallows to specify its ID, it always tries to generate one when you callStart(): https://github.com/dotnet/dotnet/blob/main/src/runtime/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Activity.cs#L834-L838, so the only option is likely just creating an activity from a proper source, so it can be sampled.
We are using Copilot SDK for .NET and it seems like the thinking behind
RestoreTraceContext(this) is not entirely correct. It creates a newActivitywithout theActivitySource, which, indeed, prevents it from being sampled, but it still carries its own id (span id), which is then used as a parent span id for all children activities. So in the trace tree (In Grafana, for example) we see unparented spans from the tool implementations, which I think should be parented underexternal_toolspans from the Copilot CLI:the
GETspan (idd2c10abb001cb97a) has parent span id set tocabcf362cbf15a7a(which is nowhere to be found), where asexternal_toolspan has an id of2cd84acd8fb4e994. So, I think, the tree is:Why don't we just emit this span properly from
ActivitySource(maybe matching SourceName/defaultgithub.copilot)?I also think a similar problem exists in the CLI itself, because transitioning from SDK to CLI also leaves all "top-level" CLI spans unparented. CLI code doesn't seem to be open source, so I can't confirm, but my gut feeling is that a similar approach was used there in CLI, where it creates one span and doesn't attach it to a source that is being sampled/exported, so it never gets collected, leaving all other CLI spans unparented. 😞