Skip to content

Log Anthropic cache tokens and a round's cost - #127

Open
Mattbusel wants to merge 1 commit into
webstonehq:mainfrom
Mattbusel:use-llm-wasm
Open

Mattbusel wants to merge 1 commit into
webstonehq:mainfrom
Mattbusel:use-llm-wasm

Conversation

@Mattbusel

Copy link
Copy Markdown

On message_start, crates/seaquel-ai/src/wire/anthropic.rs reads only usage.input_tokens, so Anthropic's cache_read_input_tokens and cache_creation_input_tokens are dropped and the "Token usage" log line undercounts a round's input once prompt caching is in play. This PR adds both counts to wire::Usage, logs them in turn.rs, and adds Usage::cost_usd(model), which prices the round from the llm-wasm crate's model price table (pure data, no I/O, builds for wasm32; None for a model the table doesn't list). OpenAI-compatible decoding and every other behaviour are unchanged; the seaquel-ai tests (with a new cache-token test), Core's ai_turn and ai_replay, clippy, the wasm32 check and check-crate-deps all pass. I maintain llm-wasm (MIT) and will keep its price table current.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant