Repository navigation
Token Allowance: per-user token and cost budgets for ChatClient, as a Boot starter #7287
Ultron03
started this conversation in
Show and tell
Replies: 1 comment 2 replies
|
This is a useful place to put the guardrail: I would keep the admission check at the advisor layer, but make the accounting model explicit enough for concurrency and retries. A few things I would want before enforcing it in a multi-instance service:
Advisor ordering seems like the right hook for request admission because it can short-circuit before memory/tool activity. The per-model-call meter should remain close to the actual model invocation, as you describe, so a tool loop is still accounted for. I would document the failure policy explicitly too: if the ledger is unavailable, does a request fail closed, fall back to a local limit, or run in observe-only mode? That decision matters as much as the quota itself. |
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I've been building a Spring Boot starter that puts token and dollar budgets on Spring AI
ChatClientcalls, and 0.1.1 is now on Maven Central. I'd like feedback from people running Spring AI in production.The problem: one user, one bug or one tool loop that never ends can run up an LLM bill, and the provider dashboard shows it the next day.
Setup: add the dependency next to your model starter, set prices and limits in
application.yml, and keep using theChatClient.BuilderSpring AI gives you.How it works:
ToolCallingAdvisor, and denies a request whose budget is used up. In a web app that is a429withProblemDetailandRetry-After.observemode only logs, so you can compare its numbers with your provider's dashboard before enforcing.Spend is kept in memory in 0.1; a Redis ledger for multiple instances is next.
I'd especially like to hear:
ChatModellevel?Usagedoesn't match their bill?Related issue: #6683
All reactions