Posts carrying a followed tag arrive from the servers it names

[wintermeyer]

Nothing fetches anything yet. A worker should read the tag timeline of each server a follow names, keep the newest handful as plain text plus a link to the original, and skip what is sensitive or behind a content warning. The rows belong in a small table of their own with a cap per tag and a hard ceiling, not in fediverse_posts: a cached remote post drags image ingestion, the AI gate, the screenshot queue and two counting sweepers behind it, and a REST status is not an ActivityPub object to begin with.

How often is the tag's own business: fetch so that roughly five new posts accumulate, at least ten minutes apart and at most three hours. Measured, that puts a tag in the middle of a news event at ten minutes and a quiet local one at three hours, and both drift back on their own. A budget per server keeps twenty busy tags from spending its rate limit, and the clock is stamped on every outcome, skips included, or a pair that can never be fetched holds the front of the queue forever.

Where: a sweeper beside Vutuv.Fediverse.CountsRefresher, HTTP through Vutuv.SocialFeed.Http.

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Handed over from #2125 (merged as 6b788a9ba), two things its review found and deliberately left for the fetcher:

• The SSRF guard in the changeset is only half of it, by design. TagFollowSource does the literal check; it cannot do DNS. metadata.google.internal is accepted and stored (so is it by OrganizationDomain). The sweeper must call Vutuv.Ssrf.vetted_address/1 and pin the connection, not merely resolves_to_internal?/1, or DNS rebinding walks past both layers. Vutuv.SocialFeed.Http — the module this issue names — already calls resolves_to_internal?/1.
• A source is not checked against the instance blocklist. A member can add a host the operator blocked at /admin/fediverse. The check belongs at fetch time, so the blocklist stays authoritative when the operator changes it after the fact.

Also from that review: wanted_tag_sources/0 excludes the local source and answers in one grouped query; a follow with zero source rows still counts as vutuv, so the fetcher must never read "no rows" as "nothing wanted".

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Built in PR #2138 (branch claude/external-tag-fetcher-2126), for whoever picks up #2127–#2129.

The hypotheses in the body. Vutuv.Fediverse.CountsRefresher is where the body says and is the right sibling to copy — a Process.send_after loop, the next tick scheduled after the run finishes, gated by a config flag so it never starts in tests. Vutuv.SocialFeed.Http exists too, but the handover comment's claim that it "already calls resolves_to_internal?/1" is wrong: only fetch_avatar/2 does. get/3 vets nothing, which is why three of its callers (Vutuv.Mastodon, CodeStats.Forgejo, the avatar fetch) each check the host themselves and then hand Req the hostname to resolve a second time.

What shipped. external_tag_posts (text, link, author, declared language, published_at) and external_tag_fetches (the per-pair schedule), both plain additions, N-1 safe. Vutuv.Tags.ExternalTagClient reads GET /api/v1/timelines/tag/:hashtag; Vutuv.Tags.ExternalPosts owns the due query, the cadence and the caps; Vutuv.Tags.ExternalPostFetcher is the two-minute loop. Skipped on arrival: sensitive, content-warned, boosts, replies, non-public, no link, no text, and anything dated more than five minutes in the future (a future-dated post would pin itself to the top of a tag forever).

The two security requirements. Both went in at fetch time, both calibrated. Vutuv.SocialFeed.Http.get_pinned/4 resolves through Ssrf.vetted_address/1, dials the vetted IP and carries the hostname in connect_options[:hostname] (Mint uses it for SNI, certificate verification and the Host header) — with the pin removed, 127.0.0.1 and an unresolvable host are both fetched and the request goes to mastodon.example instead of 93.184.216.34. The blocklist is asked per fetch, so an operator blocking a host afterwards stops the pull; with the check removed the blocked host is fetched. It also drops a status whose author lives on a blocked host, which the requirement did not ask for but "drop everything that host sends" does not stop being true because a third server relayed it.

The clock. checked_at and next_fetch_at are stamped on every outcome. Calibrated: with stamp_skipped made a no-op, the blocked pair is still first in due_sources/10 after a full pass — the #1316 deadlock exactly. A skip takes no strike and waits the ceiling; a failure takes a strike and backs off by doubling. :unresolvable counts as a failure (DNS is the remote side), :blocked / :internal / :gone as skips.

Cadence, measured (next_interval/2, target 5, floor 600 s, ceiling 10,800 s). A tag that keeps answering empty walks 10 → 20 → 40 → 80 → 160 → 180 minutes and reaches the ceiling after eight hours of quiet; one storing 20 a fetch walks 180 → 90 → 45 → 22 → 11 → 10 back down. At the target the interval does not move. That is 144 requests/day per (tag, server) pair at the floor and 8 at the ceiling. At most one halving or doubling per pass, so a single odd fetch cannot fling the pace across the range.

Deliberately not wanted_tag_sources/0. The #2125 review asked for a bound on it, and a LIMIT on that list would cut the tail off permanently. So Vutuv.Tags.wanted_tag_sources_query/0 is now the composable definition — wanted_tag_sources/0 orders and runs it, due_sources/1 adds the schedule join, the due filter and a SQL LIMIT on top. One consequence: the merged-alias exclusion moved into the shared query, so wanted_tag_sources/0 no longer answers about a tag that is an alternative name for another. Tag.hashtag_name/1 was lifted out of VutuvWeb.Fediverse.Docs for the same reason: the hashtag we ask for and the Hashtag object we send must be one spelling.

Dead ends and things worth knowing.

• Http.get/3 merged extra with Keyword.merge/2, which replaces :headers — so the first draft asked strangers' servers for their timelines as req/0.5.18 instead of vutuv/<version> (+url). CodeStats.GitHub had been working around it by hand. get/3 now merges :headers and :connect_options per key, and the test asserts the User-Agent (calibrated: it reads req/0.5.18 without the fix).
• The per-tag cap test failed first with 10 of 20 rows because its fixtures were dated 2026-09-01…25 and today is the 10th — the future-date drop ate half of them. Date test fixtures in the past.
• prune/0 is not called from fetch_due/0. It reacts to somebody dropping a source or unfollowing a tag, which no fetch can bring about, and both its NOT EXISTS deletes scan a table to find nothing on an ordinary day; the fetcher runs it every 30th tick (hourly). The ceiling sweep runs only when a pass actually stored something.
• since_id was considered and left out: the id format is the remote server's and a wrong one could silently return nothing forever. The unique index on (tag_id, source, remote_id) dedupes instead, and the on_conflict: :nothing insert count is what the cadence reads.
• Still open, and not this issue's: wanted_tag_sources_query/0 does not exclude follows of suspended or frozen members; the per-server cap is a third copy of a reduce Vutuv.Fediverse holds twice (both private, both wired to their own config), which wants one shared home; and nothing reads external_tag_posts yet — What the other servers found stands in the feed like any other post #2127 will want an ordered per-tag reader.

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Blind review of PR #2138 — CI green, eleven of twelve claims measured and held (pin dialled at 93.184.216.34 with redirect: false; blocklist refuses after storage and drops a blocked author host; no path reaches the image gate, ingestion or screenshots; flag off makes no request at all; migration pure addition, run twice and rolled back on a copy of production; cadence re-derived, clamped at 600 s and 10,800 s). Claim 1, the headline requirement, fails. Seven items go back to the builder:

1. Blocking. ExternalPost.changeset/2 validates remote_id, author_acct, author_name with validate_length(max: 255), which counts graphemes, against varchar(255), which counts codepoints. 100 ZWJ family emoji = 100 graphemes / 700 codepoints: changeset valid? true, then Postgrex 22001 inside Repo.insert_all in store/2 — outside ExternalTagClient.fetch/2's rescue. Handle.display_name/1 does not normalise emoji, so an ordinary fediverse display name reaches it.
2. The amplifier. The raise aborts the whole Enum.map in fetch_due/0, so no pair is stamped, not even the healthy one. Measured: both pairs keep interval_seconds: nil, sit first under asc_nulls_first, and every later run raises again. That is Die Like-Zahlen fremder Posts bleiben nicht mehr bei null stehen (v7.228.1) #1316 in its exact shape and worse — not one wedged pair but the whole fetcher, permanently, installation-wide.
3. author_host/2 returns nil on a missing acct, and MapSet.member?(blocked, nil) is false, so an unparseable author passes the blocklist. A degraded path must fail closed.
4. 25 distinct pinned hostnames produced 25 Finch instances and +200 processes, never reaped. Cardinality is member-controlled.
5. Vutuv.Tags.Merge @movable says a new table pointing at tags belongs there; both new tables are missing, so preview/2 under-reports and revert/1 cannot restore them.
6. runtime.exs hard-matches [a, b] = String.split(…) on three new env vars — a typo is a MatchError at boot, against the project's own rule.
7. The per-host budget comment claims limit * 4 keeps a busy host from shrinking the run; measured, a batch of 4 served 2.

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Repair round on PR #2138, rebased onto main (#2133 and #2137 merged since). The gate reads 20856 mods/funs, found no issues and 11147 passed.

1. The grapheme/codepoint gap. Correct, and measured on the dev database before choosing: 100 ZWJ family emoji are 100 graphemes, 700 codepoints, 2500 bytes; varchar(255) refuses that value outright while text takes it. The sibling for a remote display identity is fediverse_followers.name / .handle, both text — so author_name and author_acct are text now, and the client clamps rather than refusing, because dropping a stranger's post over the length of their name is the wrong answer. The clamp cuts to the byte budget on a grapheme boundary, or the family emoji this exists for would end on a dangling joiner. Everything still bounded (source, remote_id, language, both URLs) validates with count: :bytes, which in UTF-8 can never be fewer than the codepoints the column counts — Ecto has no :codepoints option, so the conservative unit is the available one. An over-long id drops the status (it names nothing that can be filed); an over-long language keeps the post without one. scrub_nul/1 came along too: a NUL in a display name is the other way that insert raises, and no length check sees it.

2. The amplifier. Correct and the worse half. fetch_source/1 now takes the outcome first and stamps the clock after, with the stamp's own failure caught rather than raised, so no pair's trouble can leave the ones behind it unstamped. The three places a pair's work can throw go through one guard/2 rather than four log strings. Calibrated: with the guards out, ** (exit) :boom propagates out of fetch_due/0 and the healthy pair is never stamped either. I dropped the second, outer guard the first attempt added — a reviewer of this repair showed the test exercised the inner one and that the outer never stamped, so by this project's own calibration rule it was unproven weight.

3. Fail closed. Correct. to_post/5 requires a parseable author host as its first clause; a status naming none is dropped. Calibrated: without it, both unparseable statuses come back.

4. Unbounded Finch instances. Correct, and my first fix was worse than the bug. Starting one instance per request and stopping it bounded the processes and leaked 9 atoms per request instead, which at this sweeper's budget is atom-table exhaustion in about eight days — atoms are never reclaimed. The names now come from Http.pinned_slot_names/0, a compile-time list of 16 borrowed and handed back. Measured over 500 distinct hostnames: 0 atoms created, 0 processes left. A seventeenth concurrent pinned request gets {:error, :busy}, which is a strike; the sweeper is sequential, so it cannot happen there.

4b. And a raise I introduced with it, caught by a review of this repair. pin/3 deleted :connect_options from its own list, but get/3 merges that on top of base_options/3, which supplies it — so every real request carried both :finch and :connect_options and Req raised cannot set both :finch and :connect_options (reproduced directly). ExternalTagClient.fetch/2's rescue would have turned that into a strike on every pair, silently, to the three-hour ceiling. No stubbed test could ever see it: Req.Steps.put_plug/1 swaps the adapter out before the Finch step validates anything. The drop moved to the merged list, and request_options/3 is public so the property can be asserted at all.

5. @movable. Correct. Both tables are in it. external_tag_fetches keys on source (its unique index is exactly (tag_id, source)); external_tag_posts keys on remote_id, the narrower half of (tag_id, source, remote_id), since the mechanism takes one column — conservative in the safe direction, because anything that could have collided is dropped rather than moved, and drop_leftovers/6 captures it for the revert. Calibrated: without the entries preview/2 reports nil.

6. The env vars. Correct. All three read through one lenient parser: an unreadable value keeps the shipped default and warns on stderr, where a MatchError at boot is a site that does not come up. The colon is not new here — FEDIVERSE_COUNTS_LADDER already spells age:interval, with the comma separating whole entries — so a comma would have read as "another one of these". The test evaluates the parser out of runtime.exs itself rather than copying it; calibrated by putting the hard match back, which raises MatchError. Two older blocks in that file (FEDIVERSE_INBOUND_CAPS line 271, FEDIVERSE_COUNTS_LADDER line 373) still hard-match and are outside this issue — the test's moduledoc says so, so nobody reads it as covering the file.

7. The over-fetch comment. The comment was wrong, the behaviour is right, so the comment changed. What limit * 4 buys is that a server holding the first limit due pairs cannot fill the candidate list and hide another server's pair sitting just past the cut. It does not make a run bigger than what is due: one server, per_host: 2, batch 4 serves 2, and that is the budget doing its job. Both halves now have a test.

Also from that review, fixed: a dead is_list guard on parse/2, two spellings of "is this IPv6" four lines apart, max_id/max_language read at runtime where max_display was read at compile time, and two comments that had grown past their point.

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

Shipped in #2138 (merged 3d811041d, 10 September 2026). A followed tag now pulls from the servers it names: a background job reads each one's public tag timeline and keeps the newest posts as plain text plus a link, never a picture, so nothing on this path reaches the AI image gate. They live in their own two tables rather than fediverse_posts, because a row there drags image ingestion, the screenshot queue and two counting sweepers behind it, and a REST status read off a public timeline is none of those.

Deliberately left out: anything sensitive or behind a content warning, boosts and replies, and since_id (the id format belongs to the remote server, and a wrong one could silently return nothing forever, so the unique index dedupes instead). Nothing reads these rows yet; that is #2127–#2129.

An AI agent wrote this text in my name. I know that is problematic.

[wintermeyer]

One thing the merge left standing, recorded so it is not lost: a crash inside store/2 is stamped as last_outcome: "empty" with strikes: 0, so a tag that is failing to store reads to an operator as a quiet tag. The clock advances, so it is not a deadlock — the moduledoc's ":crashed is a strike like any other failure" is true of the client path only. Worth a distinct outcome when the admin surface for this grows.

An AI agent wrote this text in my name. I know that is problematic.