TTFT and total time events for agent host sessions #320194
Draft
amunger wants to merge 2 commits into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-turn performance telemetry for Agent Host sessions by tracking time-to-first-progress (TTFT) and total turn duration, enabling comparisons with existing local agent-loop telemetry.
Changes:
- Introduces
agentHost.turnCompletedtelemetry event with TTFT/total time measurements. - Adds
AgentHostTurnTrackerto maintain per-turn timing state and emit telemetry on completion. - Hooks turn start/progress/completion tracking into
AgentSideEffectsfor both direct and queued message flows.
Show a summary per file
| File | Description |
|---|---|
| src/vs/platform/agentHost/node/agentSideEffects.ts | Starts per-turn timers when sending messages; marks first progress; completes turns on success/cancel/error signals. |
| src/vs/platform/agentHost/node/agentHostTurnTracker.ts | New helper to track per-turn stopwatches and emit completion reports. |
| src/vs/platform/agentHost/node/agentHostTelemetryReporter.ts | Adds the agentHost.turnCompleted telemetry definition and reporter method. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 6
Comment on lines
+73
to
+75
| private _key(session: string, turnId: string): string { | ||
| return `${session}:${turnId}`; | ||
| } |
Comment on lines
425
to
+432
| if (action.type === ActionType.SessionTurnComplete) { | ||
| this._turnTracker.turnCompleted(sessionKey, turnId, 'success'); | ||
| this._runTurnCompleteSideEffects(sessionKey, turnId); | ||
| } | ||
|
|
||
| if (action.type === ActionType.SessionTurnCancelled) { | ||
| this._turnTracker.turnCompleted(sessionKey, turnId, 'cancelled'); | ||
| } |
Comment on lines
+761
to
765
| const { model, permissionLevel } = this._getTurnTelemetryContext(state); | ||
| this._turnTracker.turnStarted(agent.id, channel, action.turnId, model, permissionLevel); | ||
| agent.sendMessage(URI.parse(channel), action.message.text, attachments, action.turnId).catch(err => { | ||
| const errCode = (err as { code?: number })?.code; | ||
| this._logService.error(`[AgentSideEffects] sendMessage failed for session=${channel}: code=${errCode}, message=${err instanceof Error ? err.message : String(err)}, type=${err?.constructor?.name}`, err); |
Comment on lines
+1051
to
1056
| const queuedState = this._stateManager.getSessionState(session); | ||
| this._telemetryReporter.userMessageSent(agent.id, session, queuedState, 'queued', attachments); | ||
| const { model, permissionLevel } = this._getTurnTelemetryContext(queuedState); | ||
| this._turnTracker.turnStarted(agent.id, session, turnId, model, permissionLevel); | ||
| agent.sendMessage(URI.parse(session), msg.message.text, attachments, turnId).catch(err => { | ||
| this._logService.error('[AgentSideEffects] sendMessage failed (queued)', err); |
Comment on lines
+95
to
+104
| turnCompleted(report: IAgentHostTurnCompletedReport): void { | ||
| this._telemetryService.publicLog2<IAgentHostTurnCompletedEvent, IAgentHostTurnCompletedClassification>('agentHost.turnCompleted', { | ||
| provider: report.provider, | ||
| agentSessionId: AgentSession.id(report.session), | ||
| timeToFirstProgress: report.timeToFirstProgress, | ||
| totalTime: report.totalTime, | ||
| result: report.result, | ||
| model: report.model, | ||
| permissionLevel: report.permissionLevel, | ||
| }); |
Comment on lines
+95
to
+104
| turnCompleted(report: IAgentHostTurnCompletedReport): void { | ||
| this._telemetryService.publicLog2<IAgentHostTurnCompletedEvent, IAgentHostTurnCompletedClassification>('agentHost.turnCompleted', { | ||
| provider: report.provider, | ||
| agentSessionId: AgentSession.id(report.session), | ||
| timeToFirstProgress: report.timeToFirstProgress, | ||
| totalTime: report.totalTime, | ||
| result: report.result, | ||
| model: report.model, | ||
| permissionLevel: report.permissionLevel, | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
top 2 priorities from https://github.com/microsoft/vscode-internalbacklog/issues/7876#issuecomment-4627207234 on a new event
agentHost.turnCompletedcomparison between the new event for AH
agentHost.turnCompletedand the existing corresponding event for the local agent loopinteractiveSessionProviderInvoked