The composer takes PDF, text and Markdown files, within a budget per member

.gitignore

@@ -62,6 +62,8 @@ erl_crash.dump
 # pictures fetched from other networks (Vutuv.RemoteMedia): a followed
 # account's avatar and its posts' attachments
 /remote_media
+# the files a post or a message carries (Vutuv.AttachmentStore)
+/attachments
 # the AI-moderation limbo twin of the served trees (Vutuv.Uploads.quarantine_dir/1)
 /quarantine
 

config/config.exs

@@ -765,6 +765,29 @@ config :vutuv, :organization_images, max_filesize: 4_000_000
 # Runtime overrides: MEDIA_KIT_MAX_MB, MEDIA_KIT_MAX_PHOTOS, MEDIA_KIT_MAX_LOGOS.
 config :vutuv, :press_kit, max_filesize: 30_000_000, max_photos: 10, max_logos: 5
 
+# Files on posts and messages (issue #2104, Vutuv.Attachments). `enabled` is
+# the product switch an installation that wants no files turns off
+# (ATTACHMENT_UPLOADS); `uploaders` is :admins while the milestone is being
+# built and :members once a post can actually carry a file, exactly as video
+# was introduced.
+#
+# 20 MB is a scanned twenty-page contract or a deck with pictures, and small
+# enough that five of them still fit one composer session over a phone
+# connection. The two budgets are per member and counted as **accepted
+# uploads** over a rolling window, so an upload-and-delete loop cannot reset
+# them; admins have none. `pdfinfo` comes from poppler-utils, the package
+# `pdftoppm` (qualification proofs) already asks for — without it PDFs are not
+# offered at all, and text and Markdown carry on.
+config :vutuv, :attachments,
+  enabled: true,
+  uploaders: :admins,
+  max_filesize: 20_000_000,
+  max_per_post: 5,
+  daily_budget: 100_000_000,
+  monthly_budget: 500_000_000,
+  pdfinfo: "pdfinfo",
+  pdfdetach: "pdfdetach"
+
 # Job postings (Vutuv.Jobs, milestone 11).
 #   * default_runtime_days — how long a published posting stays live before it
 #     auto-expires. Flat, no renewals: a still-open role gets a fresh posting.

config/runtime.exs

@@ -747,29 +747,68 @@ if config_env() == :prod do
   # how much of the machine it may take. Unset keeps the config.exs defaults.
   video_defaults = Application.get_env(:vutuv, :post_videos, [])
 
-  video_env_int = fn name ->
+  # Read an integer knob, or nil when it is unset. Shared with the attachment
+  # block below rather than written twice — the version that got its own
+  # closure was the one whose megabyte multiply then drifted.
+  env_int = fn name ->
     case System.get_env(name) do
       nil -> nil
       value -> String.to_integer(String.trim(value))
     end
   end
 
+  # …and the same as a megabyte count, since every size knob here is decimal
+  # (1 MB = 1,000,000 bytes), the unit the forms and the messages name.
+  env_mb = fn name ->
+    case env_int.(name) do
+      nil -> nil
+      megabytes -> megabytes * 1_000_000
+    end
+  end
+
   config :vutuv,
          :post_videos,
          Keyword.merge(video_defaults,
            enabled: System.get_env("VIDEO_UPLOADS") != "false",
            uploaders:
              (System.get_env("VIDEO_UPLOADERS") == "members" && :members) ||
                video_defaults[:uploaders],
-           max_filesize:
-             (video_env_int.("VIDEO_MAX_MB") && video_env_int.("VIDEO_MAX_MB") * 1_000_000) ||
-               video_defaults[:max_filesize],
+           max_filesize: env_mb.("VIDEO_MAX_MB") || video_defaults[:max_filesize],
            max_duration_seconds:
-             video_env_int.("VIDEO_MAX_SECONDS") || video_defaults[:max_duration_seconds],
+             env_int.("VIDEO_MAX_SECONDS") || video_defaults[:max_duration_seconds],
            ffmpeg: System.get_env("FFMPEG_PATH") || video_defaults[:ffmpeg],
            ffprobe: System.get_env("FFPROBE_PATH") || video_defaults[:ffprobe],
-           threads: video_env_int.("VIDEO_THREADS") || video_defaults[:threads],
-           concurrency: video_env_int.("VIDEO_CONCURRENCY") || video_defaults[:concurrency]
+           threads: env_int.("VIDEO_THREADS") || video_defaults[:threads],
+           concurrency: env_int.("VIDEO_CONCURRENCY") || video_defaults[:concurrency]
+         )
+
+  # Files on posts and messages (issue #2104): the switch, who may upload, the
+  # per-file cap, how many go on one post, and the two per-member budgets.
+  # Unset keeps the config.exs defaults. The megabyte knobs are decimal (1 MB =
+  # 1,000,000 bytes), the same unit the composer's message names.
+  attachment_defaults = Application.get_env(:vutuv, :attachments, [])
+
+  config :vutuv,
+         :attachments,
+         Keyword.merge(attachment_defaults,
+           # Unset means "whatever config.exs says", not "on": an operator who
+           # turned files off in a source config must not get them back by
+           # leaving the env var alone.
+           enabled:
+             case System.get_env("ATTACHMENT_UPLOADS") do
+               nil -> attachment_defaults[:enabled]
+               value -> value != "false"
+             end,
+           uploaders:
+             (System.get_env("ATTACHMENT_UPLOADERS") == "members" && :members) ||
+               attachment_defaults[:uploaders],
+           max_filesize: env_mb.("ATTACHMENT_MAX_MB") || attachment_defaults[:max_filesize],
+           max_per_post: env_int.("ATTACHMENTS_PER_POST") || attachment_defaults[:max_per_post],
+           daily_budget: env_mb.("ATTACHMENT_DAILY_MB") || attachment_defaults[:daily_budget],
+           monthly_budget:
+             env_mb.("ATTACHMENT_MONTHLY_MB") || attachment_defaults[:monthly_budget],
+           pdfinfo: System.get_env("PDFINFO_PATH") || attachment_defaults[:pdfinfo],
+           pdfdetach: System.get_env("PDFDETACH_PATH") || attachment_defaults[:pdfdetach]
          )
 
   # Post images are auth-proxied: the app checks the post's audience, then

docs/ADMINS.md

@@ -37,7 +37,12 @@ Related documents: [README](../README.md) (overview) ·
   first page of PDF proof documents that members can attach to their
   certificates & licenses. Without `pdftoppm` on `$PATH`, PDF uploads are
   refused with a clear message ("please upload an image instead"); image
-  proofs keep working.
+  proofs keep working. The same package carries `pdfinfo` and `pdfdetach`,
+  which are what decide whether a PDF attached to a post is safe to publish
+  (encrypted, carrying a program, carrying another file). Without them **PDF
+  attachments are not offered at all** — the composer's picker does not accept
+  `.pdf` and the server refuses one — while text and Markdown files carry on.
+  A check that cannot run is never treated as a check that passed.
 - **ffmpeg** (optional, `apt-get install ffmpeg`) — video on posts: converts
   a member's clip into the files browsers play and pulls the stills the AI
   check looks at. Debian's build carries `libx264` and `libsvtav1`; without
@@ -122,6 +127,13 @@ Everything else has a default (the vutuv.de production value):
 | `VIDEO_THREADS` | `4` | The thread cap every ffmpeg run gets. Raise it on a big idle machine, lower it where the web server shares few cores |
 | `VIDEO_CONCURRENCY` | `2` | How many clips are converted at once. Everything past that queues; the author sees "waiting in line" |
 | `FFMPEG_PATH` / `FFPROBE_PATH` | `ffmpeg` / `ffprobe` | The two binaries, if not on `$PATH` under those names |
+| `ATTACHMENT_UPLOADS` | `true` | `false` turns files on posts off entirely: no picker in the composer, and the server refuses one. For an installation that wants text and pictures and nothing that can carry code |
+| `ATTACHMENT_UPLOADERS` | `admins` | Who may attach a file. `admins` while the feature is being built — a post cannot show or hand out its files yet — `members` opens it to everyone once it can |
+| `ATTACHMENT_MAX_MB` | `20` | The largest file a member may attach, in megabytes. 20 MB is a scanned twenty-page contract or a deck with pictures in it |
+| `ATTACHMENTS_PER_POST` | `5` | How many files one post may carry |
+| `ATTACHMENT_DAILY_MB` | `100` | How much a member may upload in **any 24 hours**, in megabytes. The window rolls rather than resetting at midnight, so there is no hour at which twice the allowance fits. Counted as *accepted uploads*: deleting a file does not give the megabytes back, which is what stops an upload-and-delete loop from filling your disk. Admins have no allowance |
+| `ATTACHMENT_MONTHLY_MB` | `500` | The same over any 30 days |
+| `PDFINFO_PATH` / `PDFDETACH_PATH` | `pdfinfo` / `pdfdetach` | The two poppler binaries the PDF check runs, if not on `$PATH` under those names. Missing either one means PDFs are not offered (see the dependency list above) |
 | `SCREENSHOT_BLOCKLIST` | – | Extra pages never to take a link-preview screenshot of, on top of the shipped `reddit.com` and `heise.de`. Comma-separated domains and/or URLs, copied into the blocklist table the first time you migrate; afterwards the live list is edited in the admin area (see "Screenshot blocklist" below) and this variable is inert. `SCREENSHOT_BLOCKED_HOSTS` is the older name and still works |
 | `SCREENSHOT_PAGE_CHECK` | `true` | Whether each link-preview capture is judged by the Ollama vision model on whether it shows the page or a consent / ad / login wall or a bot check, and the site blocklisted when it does not (see "The list mostly writes itself" below). `false` leaves the blocklist entirely hand-written — the setting for an installation without Ollama. Independent of `IMAGE_MODERATION_ENABLED`: that one is the safety gate, this one is a quality filter |
 | `SCREENSHOT_CHECK_VOTES` | `3` | How many opinions must agree before a site is blocklisted automatically. The first is deterministic, the rest are independent draws; a single dissent leaves the site alone. `1` acts on one opinion |
@@ -609,6 +621,12 @@ vutuv runs fine without internet access:
   AI check over the stills is the same local Ollama the photos use. Leave
   `ffmpeg` uninstalled (or set `VIDEO_UPLOADS=false`) if the installation
   should not take clips at all.
+- Files on posts need nothing from the internet either: the format is read
+  from the bytes and the PDF check is poppler, running locally. There is no
+  virus scanner behind it — the short format whitelist (PDF, plain text,
+  Markdown) and that check are the defence — so an installation that would
+  rather take no files at all sets `ATTACHMENT_UPLOADS=false`, and one that
+  wants text files but not PDFs simply leaves poppler-utils uninstalled.
 - Set `FETCH_BOOK_METADATA=false`: the cover fetch and the
   page-count/publisher lookup behind book-review posts call Open Library, and
   an audiobook's running time is read from a library catalogue (`DNB_SRU_URL`).

docs/architecture/README.md

@@ -31,6 +31,7 @@ installing and operating vutuv in [Running your own vutuv](../ADMINS.md).
 | [email.md](email.md) | the Emailer chokepoint, multipart bodies, opt-outs, bounces & deliverability |
 | [images.md](images.md) | the AVIF pipeline, kept originals, fingerprinted filenames, the shared `images` table and its backfill, URL screenshots, AI image moderation (Ollama) |
 | [video.md](video.md) | video on posts: the ffmpeg pipeline and its resumable job, the AI check over the stills, the post that waits for its clip, the player and the range-answering proxy, what federates and what the Mastodon API says |
+| [attachments.md](attachments.md) | files on posts and messages: the upload chokepoint, the format read from the bytes, the PDF gate and why a raw-byte scan is not enough, the two on-disk copies, the per-member budget |
 | [admin.md](admin.md) | the admin panel: live dashboard, member browser, account deletion, newsletter & audiences, daily report |
 | [ads.md](ads.md) | the daily text ad: booking, review, serving |
 | [company-pages.md](company-pages.md) | the site footer, the English `/system/investors` and `/system/media-kit` pages, brand assets, and the daily head-count history behind the growth curve |

docs/architecture/attachments.md

@@ -0,0 +1,111 @@
+# Files on posts and messages
+
+A post can carry files as well as photos and a clip: PDF, plain text and
+Markdown to begin with (milestone #2102). This document covers the part that
+exists today — how a file gets in — and grows as the rest of the milestone
+lands: the preview pages (#2105), the post that waits for them (#2106), what
+a post shows and hands out (#2108), reports and copyright (#2109), messages
+(#2110).
+
+## One chokepoint
+
+`Vutuv.Attachments.create_pending/3` is the only way a file enters an
+installation. It is modelled on `Vutuv.Videos.create_pending_video/3`: the
+composer uploads eagerly, the row exists with **no parent** while the author
+is still writing, and a row nobody ever claimed is swept after a day
+(`sweep_pending/1`, run by `Vutuv.Posts.PendingImageSweeper` beside the
+pending photos and clips).
+
+Its refusals come cheapest-first and nothing is written to disk until every
+one has passed; the module's own doc lists them in order. Each is an atom the
+composer turns into a sentence (`attachment_error_message/1`), because "that
+file could not be processed" tells a member with a password-protected PDF
+nothing they can act on.
+
+## The format comes from the bytes
+
+`Vutuv.Attachments.Format` decides what a file is, and the extension only
+decides which names the installation offers at all and whether a text file is
+labelled Markdown or plain. The two have to **agree**, so a ZIP called
+`invoice.pdf` and a PDF called `notes.txt` are both refused: a lying name can
+never route a file past its own gate. The module's own doc has the rest.
+
+## The PDF gate
+
+`Vutuv.Uploads.PdfGate` blocks four effects — the file cannot be read
+(encrypted), it runs code when opened, it does something to the reader when
+opened, it carries another file inside it — and **fails closed**: poppler
+missing, poppler failing, or a scan that could not be finished is a refusal.
+Three of the four are poppler's answers (`pdfinfo` for encryption and
+JavaScript, `pdfdetach -list` for embedded files); `/OpenAction` no poppler
+tool reports, so it is read from the bytes.
+
+The one thing worth repeating outside that module: **a raw-byte scan alone is
+not enough, and this was measured.** One `qpdf --object-streams=generate` run
+moves every dictionary into a Flate-compressed object stream, after which
+`grep -ac` finds zero occurrences of `/JavaScript`, `/OpenAction` and
+`/EmbeddedFile` in a file that still does all three. So the scan also inflates
+every stream it can, bounded against a decompression bomb. `attachments_test.exs`
+tries every hostile PDF twice, as written and hidden that way; calibrated by
+removing the inflation pass, the `/OpenAction` file is then **accepted** while
+poppler still catches the other two.
+
+It lives under `Vutuv.Uploads` rather than beside the context that added it
+because two other doors already take a member's PDF and hand it back verbatim
+(`Vutuv.QualificationDocument`, `Vutuv.JobReferenceDocument`), whose only check
+is that page 1 renders. Neither calls it yet.
+
+## Two copies on disk
+
+`Vutuv.AttachmentStore` keeps the upload verbatim in the private
+`originals/attachments/<token>/` tree and a served copy under
+`attachments/<token>/`, both keyed by the row's URL token, never its id.
+They are byte-identical today; they are still two files, because the served
+copy is what #2107 rewrites when an author asks for the metadata to be
+removed, and the private tree's promise — this is exactly what the member sent
+— has to survive that. Neither tree gets a `Plug.Static` mount: every served
+byte will go through an authorizing proxy (#2108). Both are gitignored, and
+`test/vutuv/uploads_gitignore_test.exs` fails the build if that slips.
+
+## The budget
+
+Two rolling windows per member, 24 hours and 30 days, counted from
+`Vutuv.Attachments.Upload` — one ledger row per **accepted** upload, holding a
+member, a byte count and a moment, and nothing about the file.
+
+Two decisions sit in that sentence. **Accepted, not stored**: deleting a file,
+or letting the sweep take it, gives no megabytes back, so an
+upload-and-delete loop cannot run the disk down. **Rolling, not calendar**:
+there is no midnight at which twice the day's allowance fits.
+
+Admins have no budget. The composer shows what is left before the next file
+flows, as a formatted byte figure and a percentage.
+
+## Configuration
+
+Everything is per installation, read in `config/runtime.exs` with the
+`config/config.exs` values as defaults, and documented in the env-var table in
+[ADMINS.md](../ADMINS.md): `ATTACHMENT_UPLOADS`, `ATTACHMENT_UPLOADERS`,
+`ATTACHMENT_MAX_MB`, `ATTACHMENTS_PER_POST`, `ATTACHMENT_DAILY_MB`,
+`ATTACHMENT_MONTHLY_MB`, `PDFINFO_PATH`, `PDFDETACH_PATH`.
+
+`ATTACHMENT_UPLOADERS` is `admins` while the milestone is being built, the way
+video was introduced: a post cannot show or hand out its files until #2106 and
+#2108 land, so nobody else is offered a picker yet.
+
+## The nullable pair
+
+`attachments.post_id` and `attachments.message_id` are the shape CLAUDE.md
+warns about: at most one is set, and both are `nil` while the composer holds
+the file. `Vutuv.Attachments.pending?/1` is the one place that asks, so a
+later reader cannot write its own `is_nil/1` pair and get one of them wrong —
+an inner join to `posts` would silently drop every message's file, and a
+`NOT IN` over these ids without an `is_nil/1` branch is false for every row.
+
+## The media job
+
+The intake writes one `Vutuv.MediaJobs` row of kind `attachment_intake`, so
+`/admin/media` shows it beside the photo scans and video conversions. A
+refusal is a **finished** job with the reason in `detail` — the pipeline did
+its work and the answer was no; only a step that could not be run at all is
+`failed`.