Skip to content

feat(providers): [copilot.ts]Add OTel cache token parsing including tests and regression checks.#477

Open
steelp02 wants to merge 1 commit into
getagentseal:mainfrom
steelp02:copilot-otel
Open

feat(providers): [copilot.ts]Add OTel cache token parsing including tests and regression checks.#477
steelp02 wants to merge 1 commit into
getagentseal:mainfrom
steelp02:copilot-otel

Conversation

@steelp02

Copy link
Copy Markdown

Signed-off-by: steelp02 pieter.steel@pfizer.com

Summary

Testing

  • I have tested this locally against real data (not just unit tests). The costs codeburn displays align with github copilot costs online
  • npm test passes
  • npm run build succeeds

For new providers only:

  • I installed the tool and generated real sessions by using it
  • npm run dev -- today shows correct costs and session counts for this provider
  • npm run dev -- models --provider <name> shows correct model names and pricing
  • Screenshot or terminal output attached below proving it works with real data

[ Today ] 7 Days 30 Days This Month 6 Months
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ CodeBurn Today │
│ $55.58 cost 806 calls 131 sessions 46.0% cache hit │
│ 16.1M in 256.6K out 14.8M cached 1.3M written │
│ Custom (all): $59.30 API-equivalent vs $1000.00 plan ▓░░░░░░░░░ 5.9% │
│ Well within plan. Projected month: $76.13 (reset in 21 days). │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────╮╭────────────────────────────────────────────────────────────────╮
│ Daily Activity ││ By Project │
│ cost calls ││ cost avg/s sess │
│ 06-10 ██████████ $55.58 806 ││ ██████████ copilot/chat $55.58 $0.431 129 │
│ ││ ░░░░░░░░░░ codeburn $0.00 $0.00 1 │
│ ││ ░░░░░░░░░░ xxxxx $0.00 $0.00 1 │
╰────────────────────────────────────────────────────────────────╯╰────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────╮╭────────────────────────────────────────────────────────────────╮
│ By Activity ││ By Model │
│ cost turns 1-shot ││ cost cache calls 1-shot │
│ ██████████ Exploration $52.13 303 - ││ ██████████ Sonnet 4.6 $52.93 46.0% 147 - │
│ █░░░░░░░░░ Conversation $3.46 496 - ││ ░░░░░░░░░░ Haiku 4.5 $2.60 46.1% 37 - │
│ ░░░░░░░░░░ Debugging $0.00 4 - ││ ░░░░░░░░░░ GPT-4o Mini $0.047 45.7% 115 - │
│ ░░░░░░░░░░ Feature Dev $0.00 1 - ││ ░░░░░░░░░░ copilot-sugges $0.00 0.0% 4 - │
│ ░░░░░░░░░░ Brainstorming $0.00 2 - ││ ░░░░░░░░░░ copilot-nes-oc $0.00 0.0% 4 - │
│ ││ ░░░░░░░░░░ Copilot (Anthr $0.00 - 499 - │
╰────────────────────────────────────────────────────────────────╯╰────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────╮╭────────────────────────────────────────────────────────────────╮
│ Core Tools ││ Shell Commands │
│ calls ││ No shell commands │
│ ██████████ Shell 5358 ││ │
│ █████░░░░░ Read 2883 ││ │
│ ███░░░░░░░ replace_string_in_file 1585 ││ │
│ ██░░░░░░░░ manage_todo_list 1126 ││ │
│ █░░░░░░░░░ grep_search 609 ││ │
│ █░░░░░░░░░ file_search 269 ││ │
│ ░░░░░░░░░░ create_file 191 ││ │
│ ░░░░░░░░░░ list_dir 57 ││ │
│ ░░░░░░░░░░ kill_terminal 38 ││ │
│ ░░░░░░░░░░ get_terminal_output 16 ││ │
╰────────────────────────────────────────────────────────────────╯╰────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────╮╭────────────────────────────────────────────────────────────────╮
│ Skills & Agents ││ MCP Servers │
│ No skill/agent usage ││ No MCP usage │
╰────────────────────────────────────────────────────────────────╯╰────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ <> switch q quit 1 today 2 week 3 30 days 4 month 5 6 months d yesterday o optimize c compare │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

…ests and regression checks.

Signed-off-by: steelp02 <pieter.steel@pfizer.com>
@steelp02

Copy link
Copy Markdown
Author

Otel information: https://code.visualstudio.com/docs/agents/guides/monitoring-agents
Additional instructions to get it working: enable otel by adding these two lines to settings.json in vscode:
"github.copilot.chat.otel.enabled": true, "github.copilot.chat.otel.dbSpanExporter.enabled": true

@dat-intellis

Copy link
Copy Markdown

Otel information: https://code.visualstudio.com/docs/agents/guides/monitoring-agents Additional instructions to get it working: enable otel by adding these two lines to settings.json in vscode: "github.copilot.chat.otel.enabled": true, "github.copilot.chat.otel.dbSpanExporter.enabled": true

is there a corresponding copilot-cli setting to enable otel?

@steelp02

Copy link
Copy Markdown
Author

Otel information: https://code.visualstudio.com/docs/agents/guides/monitoring-agents Additional instructions to get it working: enable otel by adding these two lines to settings.json in vscode: "github.copilot.chat.otel.enabled": true, "github.copilot.chat.otel.dbSpanExporter.enabled": true

is there a corresponding copilot-cli setting to enable otel?

export COPILOT_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants