Skip to content

fix: propagate configured temperature through reflect calls - #3828

Open
koriyoshi2041 wants to merge 1 commit into
vectorize-io:mainfrom
koriyoshi2041:fix/reflect-temperature
Open

fix: propagate configured temperature through reflect calls#3828
koriyoshi2041 wants to merge 1 commit into
vectorize-io:mainfrom
koriyoshi2041:fix/reflect-temperature

Conversation

@koriyoshi2041

Copy link
Copy Markdown
Contributor

Problem

HINDSIGHT_API_LLM_TEMPERATURE_REFLECT is resolved by configuration but reflect requests omit it, so providers apply their defaults. This affects tool turns, final synthesis, structured extraction, and over-budget rewrites.

Closes #3825.

Fix

Forward llm_temperature_reflect through every reflect LLM call path. Add deterministic coverage for the tool loop, rewrite, and structured-output paths.

Test

  • uv run pytest tests/test_reflect_agent.py::TestReflectStructuredOutput::test_structured_output_forwards_reflect_temperature tests/test_reflect_agent.py::TestReflectAgentMocked::test_done_tool_answer_respects_max_tokens -q (2 passed)
  • uv run ruff check hindsight_api/engine/reflect/agent.py tests/test_reflect_agent.py
  • uv run ruff format --check hindsight_api/engine/reflect/agent.py tests/test_reflect_agent.py
  • git diff --check

The full reflect-agent file also passed 45 tests; four environment-dependent tests could not start because the local checkout lacks pg0-embedded and provider credentials. The repository-wide lint wrapper reached unrelated TypeScript linting and stopped because workspace npm dependencies were not installed.

Risk

Low. The provider interfaces already accept optional temperature values, and the default remains None, preserving existing provider-default behavior when the setting is unset.

@strix-security

strix-security Bot commented Aug 27, 2026

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Updated for 11128e2.


Reviewed by Strix
Re-run review · Configure security review settings

@Sanderhoff-alt Sanderhoff-alt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threading itself is complete (all four reflect call sites plus the pre-existing think path), but the tool-loop call reaches a provider path that never had to handle a temperature before, and the resolved default is not None.

tools=tools,
scope="reflect_tool_call",
tool_choice=iter_tool_choice,
temperature=get_config().llm_temperature_reflect,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent.py:874 is the only in-repo caller of call_with_tools, so this is the first time a temperature reaches that path — and OpenAICompatibleLLM.call_with_tools does not apply the reasoning-model suppression its own call() does (providers/openai_compatible_llm.py:930: if temperature is not None and not is_reasoning_model, versus :1363 where the guard is absent). OpenAIResponsesLLM has it on both paths (:456, :574).

Concrete break: provider=openai, model=gpt-5 (or o1/o3) routes to OpenAICompatibleLLM; with no config set the reflect tool loop now sends temperature=0.9, which chat/completions rejects with a 400 ('temperature' does not support 0.9 with this model), so every reflect fails after retries. Before this change no temperature was sent and it worked.

Suggested fix in the provider, to keep the two methods symmetric:

if temperature is not None and not self._supports_reasoning_model():
    ...
    call_params["temperature"] = temperature

response_format=DynamicModel,
scope="reflect_structured",
strict_schema=get_config().llm_strict_schema_reflect,
temperature=get_config().llm_temperature_reflect,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llm_temperature_reflect resolves to DEFAULT_LLM_TEMPERATURE_REFLECT = 0.9 when unset (config.py:222, _resolve_operation_temperature), not None — so contrary to the risk note in the description, every deployment that never set the env var changes behaviour here, and this particular call is the JSON-schema extraction of the final answer, which previously ran at the provider default. Please either confirm 0.9 is intended for structured extraction or use a deterministic value for this call, and correct the description so reviewers of the release notes are not misled about the blast radius.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reflect ignores HINDSIGHT_API_LLM_TEMPERATURE_REFLECT: no temperature sent on reflect or mental-model refresh LLM calls

2 participants