Repository navigation
Conversation
…example The "Locally Hosted" Ollama example sets llm and summary_llm to the ollama/ litellm provider, which routes to Ollama's /api/generate endpoint, not documented by litellm as supporting tool/function calling (https://docs.litellm.ai/docs/providers/ollama). Every PaperQA2 agent ends a turn with a forced tool call, so a model reached through ollama/ returns an empty message and the agent answers "I cannot answer this question due to having no papers." - the exact symptom in Future-House#1128, diagnosed there as an LLM limitation. ollama_chat/ routes to /api/chat, litellm's documented tool-calling endpoint. Switching the example's chat-model entries to that prefix is enough for the same question to get a real, cited answer. Embeddings stay on ollama/, since litellm's embedding call uses that provider regardless of chat support. Generated-by: Claude Code:claude-sonnet-5
There was a problem hiding this comment.
🟡 Changes recommended
Configure the agent model and router settings to match the local Ollama setup.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the Locally Hosted Ollama example to use LiteLLM’s chat provider for tool-calling models.
Changes:
- Switches chat models to
ollama_chat/llama3.2. - Explains the provider distinction.
- Keeps embeddings on
ollama/.
File summaries
| File | Summary |
|---|---|
README.md |
Updates Ollama guidance; agent settings still need to be configured for a fully local setup. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| llm="ollama_chat/llama3.2", | ||
| llm_config=local_llm_config, | ||
| summary_llm="ollama/llama3.2", | ||
| summary_llm="ollama_chat/llama3.2", | ||
| summary_llm_config=local_llm_config, |
There was a problem hiding this comment.
Good catch, fixed in e2260ab: added AgentSettings(agent_llm=..., agent_llm_config=...) so the example actually sets it.
The llama.cpp example right above this one (README.md, the Locally Hosted section before this one) has the same gap: its Settings(...) call also never sets agent.agent_llm, so it would hit the same default-gpt-4o problem. Leaving that one alone since it's outside this PR's diff, but flagging it here in case it's worth a follow-up.
Copilot correctly flagged that the example told the reader to route agent_llm through ollama_chat/ but never set it: Settings() left agent.agent_llm at its default gpt-4o-2024-11-20, so the example still called OpenAI for the agent step despite the instruction above it. Add AgentSettings(agent_llm=..., agent_llm_config=...) to close that gap. Generated-by: Claude Code:claude-sonnet-5
What
The "Locally Hosted" Ollama example sets
llmandsummary_llmto theollama/litellm provider, which routes to Ollama's/api/generateendpoint - not documented by litellm as supporting tool/function calling (https://docs.litellm.ai/docs/providers/ollama). Every PaperQA2 agent ends a turn with a forced tool call, so a model reached throughollama/returns an empty message and the agent answers "I cannot answer this question due to having no papers." - the exact symptom in #1128, diagnosed there as an LLM limitation ("This is an LLM failure beyond the control of PaperQA").ollama_chat/routes to/api/chat, litellm's documented tool-calling endpoint. Switching the example's chat-model entries to that prefix is enough for the same question to get a real, cited answer.Fix
Change
ollama/llama3.2toollama_chat/llama3.2in the three chat-model spots (model_name,litellm_params.model, and the top-levelllm/summary_llm), and add one sentence explaining why. The embedding line stays onollama/, since litellm's embedding call uses that provider regardless of chat support.Verified locally against a running Ollama server with
qwen3.5:9b:ollama/-> empty message, agent finishes with no answer;ollama_chat/-> tool calls succeed, agent produces a cited answer.Not addressing the separate "
agent_llmandembedding_configmust also be set for local models" gap tracked in #1321 - out of scope here.