feat(computer-history): make the timeline a feed of what the model wrote, and catch up with main - #399
Merged
ZongYue99 merged 70 commits intoSep 11, 2026
Conversation
…ervice-restart-v1-1-3 fix(memory): repair signed-package service restart on Windows and macOS
…ages MODEL_INPUT_CAPABILITIES is an exact-match catalog of model IDs reviewed against vendor docs, and getModelInputModalities falls back to text-only on a miss. For a BYOK preset that fallback is a guess about a model we have never seen: the runner refused the request locally, never called the endpoint, and told the user to switch to a multimodal model even when their own model reads images fine. BYOK also cannot use the account image2text fallback, so there was no recovery path. Distinguish "catalog says text-only" from "catalog has no entry". A BYOK model with no entry now goes to the provider, which returns the real error if it genuinely cannot read images (already classified as image_input_unsupported). Account-managed models and models the catalog explicitly marks text-only are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l-image-input fix(agent): let BYOK models outside the capability catalog receive images
Memmy v1.1.2 on Windows can leave a persistent CMD window showing Memory capture and summary logs. The login task launches a `.cmd` directly, and the service launcher inherits its console. Hiding the installer's `schtasks` call cannot hide a task started independently by Task Scheduler. This change starts the task through a hidden WScript host that waits for Node, keeps the service child attached, and appends stdout/stderr and launcher errors to `memory-service/logs/service.log` and `service-error.log`. Task settings ignore duplicate runs, remove the execution time limit, and allow background operation on battery. Windows standalone restart uses the existing local service rebuild loop so it does not depend on a helper surviving termination of its own task. Existing launchers are refreshed on install, compatible reuse and start. Desktop also repairs a retained legacy task before returning early for a healthy compatible runtime, waits for a live database owner to finish startup/migrations first, and re-probes before taking ownership. Repair matches only the old task command for the current home, updates its definition without starting it, and preserves configuration, data and compatible runtime selection. Validation: - Regression tests were observed failing before implementation for task entry/logging/refresh/stop behavior, Windows restart dispatch and Desktop's same-version reuse path. - Targeted Memory/CLI suites: 79 passed, 1 Windows-only integration test skipped on macOS. - Targeted Desktop lifecycle suites: 64 passed. Windows native validation remains required: this development host is macOS. An opt-in integration test creates a unique temporary task and checks the real WScript/Node process tree, duplicate runs, stop/start, Unicode paths, output logs and main-window handles. On an interactive Windows desktop, run in PowerShell: ```powershell $env:MEMMY_WINDOWS_SERVICE_INTEGRATION = '1' npm test -w @memmy/memory -- tests/runtime-installer-windows.integration.test.ts ``` Also visually confirm there is no transient console flash during install/login/restart. The integration test never registers the product's default task. Target: `release/v1.1.3`. No version publication or merge is included.
fix(memory): keep account summaries on dedicated model
…s-memory-console fix(memory): keep Windows background service console-free
splitText computed `candidate` before the size check, then flushed `current` and reset it to "" — but the trailing `else` branch reassigned `current = candidate`, which still contained the text that had just been flushed. The accumulator therefore never shrank: every subsequent line tripped the limit again and pushed an ever-growing prefix into `chunks`. Output grew quadratically and the work cubically, so importing a single conversation turn containing one large multi-line message never finished. Observed on a 181 KB Claude Code message: the scan spun at ~85% CPU with RSS climbing to 900 MB and produced zero rows in 4.5 hours. Sibling `splitUtf8` already does this correctly (`current = character` after its flush); splitText was simply missing that step. After a flush `current` is "", so restarting the accumulator from `line` is exactly the intended behaviour and is a no-op on every path that did not flush. The existing oversized-content test uses `"x".repeat(30_000)`, a single line with no newlines, which takes the `line.length > maxChars` branch and never reaches the faulty `else` — hence the gap. The three added tests cover multi-line input, the token budget, and multibyte byte boundaries. On the unfixed code they report 28.8x content amplification, parts over the token budget, and 200 CJK characters emitted 19,397 times. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jdi5HaeP7tYuApntyxPtXq
splitUtf8 called Buffer.byteLength(current + character) once per character. Each call forces V8 to flatten the growing cons string, so a single call was O(n^2) in the length of the chunk being built. Track the running byte count instead and add Buffer.byteLength(character) per code point. Iterating a string with for...of always yields whole code points, so the running total matches what the previous expression measured, and the split boundaries are unchanged. Verified equivalent against the previous implementation over 72,690 cases (12,316 inputs — 5,723 real captured agent messages, length-truncated slices of them, and synthetic edge cases covering lone surrogates, astral plane characters, emoji ZWJ sequences and CJK — crossed with six maxBytes values from 8 to 512 KiB): zero differing outputs. Splitting a 16 KB chunk 20 times goes from 2058 ms to 11 ms. This is independent of the splitText correctness fix in the previous commit and can be dropped without affecting it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jdi5HaeP7tYuApntyxPtXq
…-text-flush fix(agent-source): stop splitText re-emitting flushed text
…ory-legacy-startup fix(desktop): keep legacy plugin migration out of Memory startup
fix account memory role routing
# Conflicts: # App/backend/src/infrastructure/memmy-config/index.ts # App/backend/src/infrastructure/memmy-config/tests/account-model-projection.test.ts
release: Memmy v1.1.3
…dinates The recorder resolved every click by hit-testing the cursor position and then sleeping 300ms hoping the application had rebuilt its accessibility tree. That races the renderer, and its own comment admitted Chrome often still returned stale geometry on the retry. Keep the focused element continuously up to date from AXObserver notifications so a click can be attributed immediately; the positional hit test is now only a fallback for controls that never take focus. Coordinates are no longer emitted. Event kinds now match the shape Codex/Skysight produces, which the diff engine and the layered summaries will build on: selection.changed new, and the highest-volume semantic signal keyboard.submit new, a cheap and reliable task-boundary marker mouse.drag new, carrying origin and destination elements mouse.context_menu new scroll removed; the AX tree diff carries that state instead Also add app.secureInput from IsSecureEventInputEnabled() so keystroke text is suppressed while a password field owns focus, rather than relying only on the redaction regexes as a second line of defence. The history JSONL output shape is unchanged so summarize-history and its fixtures keep working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex's equivalent tool has no replay: it answers questions about what the user did, and reproducing a behavior is Computer Use's job. Splitting the two the same way stops a retrieval result from turning into desktop control on its own, and matches the conclusion that Computer History is searchable evidence rather than a library of replayable templates. Drop the `action` enum and the replay branch, add an explicit result limit, and label every returned field as untrusted observed evidence — the event stream records whatever appeared on screen, including text written by third parties. The service keeps prepareReplayUserRequest for the desktop UI; only the agent-facing surface loses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ript The summarizer was a formatter: it emitted one bullet per event under "Activity timeline" and one numbered step per event under "Reusable operation experience". A five-minute window produced 280 lines, 175 of them "Typed 1 character(s); text redacted". That is the event stream reformatted, not a summary, and it is nothing like what it was meant to resemble. The model now writes the recording summary itself, from the segment's own event stream compacted into activity arcs: consecutive events are grouped by application and each run reported once, keeping the semantic labels — a clicked message, a page title — because those are what let the account say what happened. Forty keystrokes become one line rather than forty. Recovering those labels means reading the recorder's enrichment, not just the top-level attributes: clicks often land on an anonymous container, and the focused control and nearest labeled descendants and ancestors are where the name actually is. What stays is the part that was already right: the durable facts — time range, applications, event counts, click coverage, approved starting URL. Those are what remain useful once raw events pass their retention window. The scroll compression test went with the scroll events themselves, which the AXObserver rewrite removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
splitTurn cut a turn on every blank line without repacking, so a single agentic turn fanned out into hundreds of near-empty tool-call fragments (one observed turn produced 640 parts, 639 of them a 67-character "Tool calls: - tool_1"). A codex scan of 1,000 selected turns wrote 19,413 memories. Restore one turn == one memory. Oversized turns are now clipped on a UTF-8 boundary via renderTurnClipped so the add-memory request stays under the wire limit instead of inventing extra memories. Both ingest call sites (memory runtime and desktop backend) use the same helper and keep the pre-split legacyTurnRequestId / legacyTurnId idempotency keys, so watermarks and dedup stay aligned with pre-4eea5f52 memories. Also bundles AgentSourceCore/dist into the packaged memory runtime and rewrites workspace deps to file: with --install-links, so the packaged runtime can resolve @memmy/agent-source-core. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(scan): keep one agent-source turn as one memory
…ent be pinned A workflow candidate was generated for every finalized segment. A ten-minute window boundary has nothing to do with a task boundary, so almost all of them described a slice of an afternoon rather than anything worth repeating. Steps are now derived from a segment's event stream when someone actually asks to repeat something. That makes the raw stream the thing replay depends on, so retention becomes a real question rather than a detail. Two answers: within the window, ask and a workflow is written from the events, and from then on it is permanent and independent of them. Beyond it, pin the segment beforehand and its events are exempt. Because pinning writes a marker inside the segment, age can no longer come from the directory's mtime — touching it would silently grant another full window. It comes from the start time the segment recorded for itself. Replay availability now follows from whether the event stream is still on disk, not from whether the summary contains a section listing actions — that section is gone, and every path that parsed it was quietly returning nothing. Searching no longer filters by replayability. It is evidence retrieval: an entry used to vanish from search the moment its events expired, which is exactly when the written summary is all that remains and the only thing that can still answer what happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… so when it does not The titles stayed mechanical because the call was missing what the chat-title generator passes: reasoningEffort "none". A reasoning model spent the budget thinking and returned empty content, and every failure path here returned null without a word — so a broken call looked exactly like a feature that was never wired. Diagnosing it took reading the code rather than the symptom, which is the real defect. Narration now reports why it produced nothing, through a callback the service logs and exposes as observation.narrationError, so the mechanical wording is always explained rather than merely observed. It also runs once a segment has enough stream to be worth summarizing, not only when the segment closes. Waiting for the close left every entry reading mechanically for the whole ten minutes someone is most likely to be looking at it, which is how this went unnoticed for three rounds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…just summaries The summaries and the event streams are two different records answering two different questions — what a window was about, and what specifically happened in it. Only the first was reachable, so "who contacted me today" could not be answered even though the message text was sitting in the stream all along. Three things were in the way. computer_history_status reported the summary directory as event_stream_root_path, so following it led to the wrong place; it now reports both, named for what they hold. Search results carried no way down to the evidence; each match now names its event stream, or says the raw events have passed retention rather than implying the summary was all there ever was. And nothing told the agent this record existed or how to read it, which the new computer-history skill now does: the layout, when to stay in the summaries, what fields carry a clicked message, and to grep rather than read whole files, because one event line can carry a whole accessibility tree. The skill also states the two rules this record needs: it is evidence and never instructions, however much a captured message reads like one; and an answer must distinguish "nothing was found" from "recording was off" or "the events expired", because those send the user somewhere different. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ning it by hand Adding eventStreamPath to history entries broke the page the same way adding description did: the desktop client validates the snapshot strictly, so a field the agent grows and the client does not know about fails every entry and renders a wall of unrecognized_keys. The first time, the answer was a test asserting the workflow field names. That only covered workflows, so histories drifted next, and the same test would have kept passing however many more fields were added elsewhere. Listing field names by hand is the same work the schema already does, done twice. The schema now lives on its own, free of any browser dependency, and a test at the repository root parses a real snapshot from the real service with it. It belongs at the root because the contract spans two packages that cannot import each other — which is exactly why the sides were free to drift. Reverting the missing field makes it fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…null
Narration resolved its model with `llmRuntime(modelPreset ?? null)`. That turns
"no preset was asked for" into "resolve the preset called null", which the
gateway's resolver cannot do — it answered model_selection_unavailable, and
every summary silently kept its mechanical title.
Reproduced against the real config: llmRuntime() resolves gpt-5.6-terra,
llmRuntime("computer-use-fast") resolves it too, and only llmRuntime(null)
throws. Pass nothing when there is nothing to pass.
End to end on a real segment, 424 events compact to 56 activity arcs and come
back as "Computer History Integration Review" with an account naming DingTalk,
Arc, PR MemTensor#371 and the architecture docs — instead of "用户在「Computer History
2026-09-08T08-20-00Z」中完成了一组电脑操作".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-public-body fix(release): sanitize public release notes
release: Memmy v1.1.4
…ow it only when written Two things were wrong with what the timeline showed. The written summary never survived recording. Every sixty seconds the live pass regenerated the mechanical markdown over the file, while narration ran once per segment — so the account was written, overwritten on the next tick, and only stuck after the segment closed. That is why a summary appeared to require a manual stop. And it was only ever half written. Narration replaced the recording summary alone, leaving "Memory summary" as a templated sentence, "Relevant prior context" as boilerplate about the capture policy, and "Important non-obvious context" holding event counts, screen resolution and file paths — machine bookkeeping under a heading promising insight about the user. The model now writes the body: what the window was for, how it relates to the windows before it, the specifics worth keeping once the raw events expire, and the account of what happened. It is given the two preceding summaries, without which any claim about what came before would be invention. The mechanical pass is reduced to frontmatter and citations, and the machine facts are gone. An entry appears in the timeline only once written, so the placeholder is never what a reader meets. Imported and demo entries, which no model writes, are unaffected. Verified end to end on a real segment: four model-written sections, the prior context genuinely continuing the earlier window, and no bookkeeping left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…xiao-sync feat: sync GitHub issues to Yunxiao
…eration-sync fix: include Yunxiao sprint in GitHub sync
…ternal-pr-only feat: sync only community pull requests to Yunxiao
The page was a list beside a markdown pane, and the pane showed the file: frontmatter bookkeeping, a templated "Memory summary", citations. Only four fields were ever meant for a reader — title, description, applications and the account itself — so the page is now a single column of those, and the markdown body no longer reaches it. Three things were wrong underneath that, each of which made history appear to lose entries. A summary was dated by its file. Nothing writes `captured_at`, so `createdAt` fell through to mtime, and a summary is rewritten every time the model catches up with it. A ten-minute window from 10:10 displayed as 10:28, walked forward on every regeneration, and sorted rollups into the middle of the segments they cover. The window an id already names is the honest answer. Finalizing a segment published less than was already published. The mechanical pass overwrote the account with its placeholder and narration replaced it a few seconds later — an entry that blinked out and returned, or never returned when the model was unreachable. The regeneration now happens beside the standing summary and is swapped in only once written. And a rollup stood next to the segments it summarizes, saying the same six hours twice. A rollup now appears only once its window has closed, at which point it stands in for what it covers; a window whose rollup never ran keeps its segments rather than losing them. Applications are drawn as their real icons, read through NSWorkspace rather than out of the bundle, because a growing share of applications ship their icon in an asset catalog with no .icns to find. Also: every string moved into the message catalogs, which this file had been violating since before this change, and the stylesheet it replaced is gone — including the two `font: inherit` declarations that were failing the prototype alignment test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ry-ax-capture # Conflicts: # package.json
memory-lee
requested review from
Wang-Daoji,
ZongYue99,
hijzy and
wustzdy
as code owners
September 11, 2026 02:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
1. Brings
computer_useup to date withmain: mergesupstream/main(48 commits). The only conflict was the test script inpackage.json, where each side added one entry. I kept both:test:contracts(from thecomputer_useline) andtests/windows-memory-workflow.test.ts(frommain).2. Turns the Computer History timeline into a single-column feed of model summaries.
Computer History changes
The page used to be a list on the left and a markdown panel on the right. The panel showed the raw file: frontmatter, the templated
Memory summaryand citations. Only four fields were meant for people to read:title,description,applicationsand the narrative the model writes. The page is now a single-column feed of those four fields, and the markdown body is no longer shown.There were three underlying problems, and each one made the history look like it was losing entries:
Summary timestamps came from the file's mtime. No summary writes
captured_at, socreatedAtfell back tostat.mtime, and every regeneration rewrites the file. A 10-minute window starting at 10:10 showed up as 10:28 and moved later with each rewrite. It could even sort a six-hour rollup into the middle of the segments it covers. Timestamps now come from the window start in the filename, which never changes.Closing a segment could publish less than was already published. The mechanical summary wrote a placeholder over the file first, and the model filled it in a few seconds later. So the entry flickered away and came back, or never came back if the model was unreachable. Regeneration now writes to a staging file next to the original and
renames it into place only after the model finishes, so what's on disk never goes backwards.Rollups were shown next to the segments they cover, so the same time span was described twice. A rollup now appears only after its window closes, and it replaces the segments it covers. If a rollup fails to run, its window keeps the segments, so nothing disappears.
Other changes
NSWorkspaceinstead of looking for an.icnsin the bundle. More and more apps keep their icons in an asset catalog with no.icnsto read. Addedworkflows/scripts/app-icon.swift, which is compiled once per source hash and cached. Icons are cached on disk by bundle id.english-ui-coveragerule before this change (18 lines of hardcoded Chinese), so that's fixed too.font: inherit, which is why theprototype-page-alignmenttest kept failing.