feat: shared static_files across cases (never templated, symlinked not duplicated) - #79
Merged
Conversation
…t duplicated) Models can declare "static_files": files identical across every case (a shared weather CSV, a large reference dataset) that are never templated/substituted, never re-hashed per case, and (for relative paths) not duplicated on disk per case. - Absolute path entries are assumed already present at that same path on the calculator side too; fz never copies/symlinks/transfers them, only hashes them once per fzr()/fzd() call so cache:// still reacts if the shared file's content changes. - Relative path entries are resolved against cwd at call time, identified by basename (not the full declared path, which may contain ".." to reach outside input_path and would otherwise escape the case directory as a symlink name), symlinked into every case's result/temp directory (falling back to a real copy if the platform disallows symlinks), and explicitly transferred to ssh://, slurm:// (remote), and funz:// calculators via their real source path, since they live outside input_path and the generic per-case file transfer never finds them. - fzi() excludes them from variable discovery; .fz_hash always includes them so cache matching stays correct. Also fixes tests/test_funz_udp_fallback.py's mock to accept the new static_entries kwarg threaded through run_single_case_calculation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- test_static_files.py::test_static_files_command_can_read_both_kinds embedded a Windows backslash path directly into a bash script; bash reads backslashes as escapes, corrupting the path. Use .as_posix(). - test_static_files_ssh.py needs a real SSH server + key-based auth, which ci.yml only sets up on Linux (like the existing test_ssh_*.py files); add it to the Windows/macOS ignore lists. It still runs on Linux (both in ci.yml's matrix and the dedicated ssh-localhost.yml). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…_static argument Static file declarations no longer live in the model dict - the model doesn't need this data, since it's about how a run/compile/design is invoked, not about the model definition itself. - fzr(), fzc(), fzi(), fzd() all gain an input_static parameter (list of paths); fzd() passes it through unchanged to each iteration's internal fzr() call. - CLI: new --input_static flag (repeatable, or an inline JSON list) on fzi, fzc, fzr, fzd and the unified fz input/compile/run/design subcommands. - helpers.resolve_static_files()/resolve_static_file_paths() now take the raw path list directly instead of reading model["static_files"]; model validation for the old field is removed in favor of a new _validate_input_static() used by the four public functions. - Docs moved from doc/model-definition.md (no longer a model concern) to doc/core-functions.md, with README/skills/reference.md/NEWS.md updated to match. Semantics (absolute vs relative resolution, symlinking, explicit remote transfer, hashing, fzi exclusion) are unchanged. - Tests and the slurm-localhost.yml CI step updated to pass input_static as an fzr() argument instead of a model key. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…_static fzr() now logs a one-time warning (per file, not per case) when a file under input_path contains no variables and is at least FZ_STATIC_CANDIDATE_MIN_SIZE bytes (default 1 MiB, new config option), suggesting it be passed via input_static instead - such a file is otherwise re-read/re-copied and re-hashed on every case for no benefit. Set FZ_STATIC_CANDIDATE_MIN_SIZE=0 to disable. Detection: for text files, compares content before/after variable substitution and formula evaluation (unchanged => no variables); binary files are inherently variable-free. Deduplicated by resolved source path so the warning fires once per fzr() call, not once per case. New tests/test_static_files_warning.py (4 tests). Docs updated: README, doc/core-functions.md, skills/fz/reference.md, NEWS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Models can now declare
"static_files": files identical across every case (e.g. a shared weather CSV or a large reference dataset) that are never templated/substituted, never re-hashed per case, and (for relative paths) not duplicated on disk per case.fzr()/fzd()call, socache://still reacts if the shared file's content changes.fzr()/fzd()was called from, identified by basename (not the full declared path, which may contain..to reach outsideinput_pathand would otherwise escape the case directory as a symlink name), symlinked into every case's result/temp directory (falling back to a real copy if the platform disallows symlinks, e.g. Windows without developer mode/admin), and explicitly transferred tossh://,slurm://(remote), andfunz://calculators via their real source path, since they live outsideinput_pathand the generic per-case file transfer never finds them.fzi()excludes them from variable discovery;.fz_hashalways includes them (hashed once, memoized) so cache matching stays correct.Also fixes
tests/test_funz_udp_fallback.py's mock to accept the newstatic_entrieskwarg threaded throughrun_single_case_calculation.Docs updated: README.md,
doc/model-definition.md(full write-up),doc/INDEX.md,skills/fz/reference.md,NEWS.md.Test plan
tests/test_static_files.py(8 tests,sh://): relative symlinking + basename hashing, absolute entries hashed-but-not-transferred, both readable by a calculator script,fziexclusion, cache invalidation on shared-file content change, missing-entry warning (non-fatal), invalid-type validation, name-collision handling.tests/test_static_files_ssh.py: real SFTP transfer of a relativestatic_filesentry overssh://to localhost (exercises the actual remote-transfer code path, not just same-filesystem symlink resolution); wired into.github/workflows/ssh-localhost.yml. Also runs automatically in the mainci.ymlLinux matrix (not excluded, markedrequires_ssh/requires_paramikolike existing SSH tests)..github/workflows/slurm-localhost.ymlforstatic_fileswith local SLURM..fz_hashcorrectly records both; cache correctly invalidates when the shared file's content changes;fzidoesn't pick up spurious variables from static files; no-symlink-privilege fallback path.🤖 Generated with Claude Code