Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,12 +218,12 @@ jobs:
- name: Run tests (Windows)
if: runner.os == 'Windows'
run: |
pytest tests/ -v --tb=short -x --ignore=tests/test_examples_advanced.py --ignore=tests/test_examples_modelica.py --ignore=tests/test_examples_perfectgaz.py --ignore=tests/test_examples_runner.py --ignore=tests/test_examples_telemac.py --ignore=tests/test_interrupt_handling.py --ignore=tests/test_ssh_availability_check.py --ignore=tests/test_ssh_directory_uniqueness.py --ignore=tests/test_ssh_localhost.py --ignore=tests/test_ssh_many_cases.py --ignore=tests/test_ssh_perfectgaz.py --ignore=tests/test_funz_integration.py --ignore=tests/test_funz_protocol.py
pytest tests/ -v --tb=short -x --ignore=tests/test_examples_advanced.py --ignore=tests/test_examples_modelica.py --ignore=tests/test_examples_perfectgaz.py --ignore=tests/test_examples_runner.py --ignore=tests/test_examples_telemac.py --ignore=tests/test_interrupt_handling.py --ignore=tests/test_ssh_availability_check.py --ignore=tests/test_ssh_directory_uniqueness.py --ignore=tests/test_ssh_localhost.py --ignore=tests/test_ssh_many_cases.py --ignore=tests/test_ssh_perfectgaz.py --ignore=tests/test_static_files_ssh.py --ignore=tests/test_funz_integration.py --ignore=tests/test_funz_protocol.py

- name: Run tests (macOS)
if: runner.os == 'macOS'
run: |
pytest tests/ -v --tb=short -x --ignore=tests/test_examples_advanced.py --ignore=tests/test_examples_modelica.py --ignore=tests/test_examples_perfectgaz.py --ignore=tests/test_examples_runner.py --ignore=tests/test_examples_telemac.py --ignore=tests/test_ssh_availability_check.py --ignore=tests/test_ssh_directory_uniqueness.py --ignore=tests/test_ssh_localhost.py --ignore=tests/test_ssh_many_cases.py --ignore=tests/test_ssh_perfectgaz.py --ignore=tests/test_funz_integration.py --ignore=tests/test_funz_protocol.py
pytest tests/ -v --tb=short -x --ignore=tests/test_examples_advanced.py --ignore=tests/test_examples_modelica.py --ignore=tests/test_examples_perfectgaz.py --ignore=tests/test_examples_runner.py --ignore=tests/test_examples_telemac.py --ignore=tests/test_ssh_availability_check.py --ignore=tests/test_ssh_directory_uniqueness.py --ignore=tests/test_ssh_localhost.py --ignore=tests/test_ssh_many_cases.py --ignore=tests/test_ssh_perfectgaz.py --ignore=tests/test_static_files_ssh.py --ignore=tests/test_funz_integration.py --ignore=tests/test_funz_protocol.py

- name: Run tests (Linux)
if: runner.os == 'Linux'
Expand Down
51 changes: 51 additions & 0 deletions .github/workflows/slurm-localhost.yml
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,57 @@ jobs:
print("\\n✓ Multiple partition test passed!")
PYTHON

- name: Test SLURM calculator - static_files
run: |
echo "Testing FZ static_files with local SLURM..."
SCRIPT_PATH="$HOME/fz_test/slurm_calc.sh"

python3 << PYTHON
import tempfile
from pathlib import Path
from fz import fzr

print("=" * 60)
print("Test 4: static_files with local SLURM")
print("=" * 60)

script_path = "$SCRIPT_PATH"

with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
assets_dir = root / "assets"
assets_dir.mkdir()
weather = assets_dir / "weather.csv"
weather.write_text("weather-slurm-local\\n")

study_dir = root / "study"
study_dir.mkdir()
input_file = study_dir / "input.txt"
input_file.write_text("x = \${x}\\n")

model = {
"delim": "{}",
"output": {"weather": "cat weather.csv"},
}

import os
os.chdir(study_dir)
calculator_uri = f"slurm://:debug/bash {script_path}"
results = fzr(
str(input_file), {"x": 3}, model,
calculators=calculator_uri, results_dir="results",
input_static=["../assets/weather.csv"],
)

if hasattr(results, "to_dict"):
row = results.to_dict("records")[0]
else:
row = {k: (v[0] if isinstance(v, list) else v) for k, v in results.items()}

assert row["weather"].strip() == "weather-slurm-local", row
print("\\n✓ static_files with local SLURM test passed!")
PYTHON

- name: Run SLURM error reporting integration tests
run: |
python -m pytest tests/test_error_reporting.py::TestSlurmIntegrationErrorReporting -v -s --tb=long
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/ssh-localhost.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ jobs:
pip install paramiko
python -m pytest tests/test_error_reporting.py::TestSSHIntegrationErrorReporting -v -s --tb=long

- name: Run static_files over SSH test
run: |
pip install paramiko
python -m pytest tests/test_static_files_ssh.py -v -s --tb=long

- name: Test summary
if: always()
run: |
Expand Down
35 changes: 35 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,41 @@

## Unreleased

### Shared static files across cases (`input_static`)

- `fzr()`/`fzc()`/`fzi()`/`fzd()` gain an `input_static` parameter (CLI
`--input_static`, repeatable or an inline JSON list): files identical
across every case (e.g. a shared weather CSV or a large reference
dataset) that are never templated/substituted, never re-hashed per case,
and (for relative paths) not duplicated on disk per case. This is a
function argument, not a model field — the model itself doesn't need to
know about it.
- **Absolute path** entries are assumed already present at that same path
on the calculator side too (shared/mounted storage); fz never copies,
symlinks, or transfers them - only hashes them (once per `fzr()`/`fzd()`
call), so `cache://` still reacts if the shared file's content changes.
- **Relative path** entries are resolved against the cwd `fzr()`/`fzd()`
was called from, identified by basename, and symlinked into every case's
result/temp directory (falling back to a real copy if the platform
doesn't allow symlinks, e.g. Windows without developer mode/admin).
Explicitly transferred to `ssh://`, `slurm://` (remote), and `funz://`
calculators, since they live outside `input_path` and wouldn't otherwise
be found by the normal per-case file transfer.
- `fzi()` never scans them for `$variables`; `.fz_hash` always includes
them (once, memoized) so cache matching stays correct.
- `fzd()` passes `input_static` through unchanged to each iteration's
internal `fzr()` call.
- See `doc/core-functions.md` ("fzr" → `input_static`) for the full write-up.
- `fzr()` now logs a one-time warning (per file, not per case) when an
`input_path` file has no variables and is at least
`FZ_STATIC_CANDIDATE_MIN_SIZE` bytes (default 1 MiB), suggesting it be
passed via `input_static` instead; set `FZ_STATIC_CANDIDATE_MIN_SIZE=0`
to disable.
- New `tests/test_static_files.py` (8 tests, `sh://`),
`tests/test_static_files_ssh.py` (real SFTP transfer over `ssh://` to
localhost, wired into `ssh-localhost.yml`), and
`tests/test_static_files_warning.py` (4 tests for the new warning).

### Configurable case directory naming (`case_naming`), thread-safe signal handling

- `fzr()`/CLI `fzr`/`fz run` gain a `case_naming` parameter (`--case_naming`,
Expand Down
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1083,6 +1083,13 @@ print(results)
`info.txt` if the manifest is missing or incomplete). Defaults to the
`FZ_CASE_NAMING` env var, or `"path"`.

- `input_static`: Files identical across every case (a shared weather CSV, a large
reference dataset) that are never templated and never duplicated per case — see
`doc/core-functions.md` ("fzr" → `input_static`) for the full write-up. If a large
variable-free file is left in `input_path` instead, `fzr()` logs a one-time warning
suggesting `input_static` (threshold: `FZ_STATIC_CANDIDATE_MIN_SIZE`, default 1 MiB,
`0` disables it).

**Returns**: pandas DataFrame with all results

### fzd - Run Design of Experiments
Expand Down Expand Up @@ -2467,6 +2474,11 @@ export FZ_RUN_TIMEOUT=3600
# (short content hash, avoids filesystem filename length limits with many
# variables), or "index" (case_<i>)
export FZ_CASE_NAMING=path

# Minimum size (bytes) for a variable-free input_path file to trigger a
# one-time warning suggesting input_static instead (default: 1048576 = 1 MiB;
# 0 disables the warning)
export FZ_STATIC_CANDIDATE_MIN_SIZE=1048576
```

### Shell Path Configuration (FZ_SHELL_PATH)
Expand Down
3 changes: 3 additions & 0 deletions doc/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,12 +218,14 @@ Quick reference index for finding specific topics in the FZ context documentatio
| Use caching | calculators.md → "Cache Calculator" |
| Debug my calculation | quick-examples.md → "Troubleshooting Examples" |
| Avoid filename length limits with many variables | core-functions.md → "fzr" → `case_naming` |
| Share a large/static file across all cases without duplicating it | core-functions.md → "fzr" → `input_static` |

## Configuration & Advanced Topics

| Topic | File | Section |
|-------|------|---------|
| Case directory naming (`case_naming`, `FZ_CASE_NAMING`) | core-functions.md | "fzr" |
| Shared static files across cases (`input_static`) | core-functions.md | "fzr" |
| FZ_SHELL_PATH overview | shell-path.md | "Overview" |
| Shell path setup | shell-path.md | "Usage" |
| Windows path configuration | shell-path.md | "Common Configurations" → "Windows with MSYS2" |
Expand Down Expand Up @@ -271,3 +273,4 @@ Quick keyword search:
- **Performance**: parallel-and-caching.md → "Performance Optimization"
- **case_naming / FZ_CASE_NAMING**: core-functions.md → "fzr"
- **cases.csv manifest**: core-functions.md → "fzo" → "Automatic Variable Extraction"
- **input_static**: core-functions.md → "fzr" → `input_static`
37 changes: 34 additions & 3 deletions doc/core-functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,14 @@ fzl --format json > config.json
```python
import fz

variables = fz.fzi(input_path, model)
variables = fz.fzi(input_path, model, input_static=None)
```

**Parameters**:
- `input_path` (str): Path to input file or directory
- `model` (dict or str): Model definition or alias
- `input_static` (list of str, optional): Files identical across every case (see `fzr`'s
`input_static` below); never scanned for variables, since they're never templated

**Returns**: Dictionary with variable names as keys (values are None)

Expand Down Expand Up @@ -173,14 +175,16 @@ print(variables)
```python
import fz

fz.fzc(input_path, input_variables, model, output_dir)
fz.fzc(input_path, input_variables, model, output_dir, input_static=None)
```

**Parameters**:
- `input_path` (str): Path to input file or directory
- `input_variables` (dict): Variable values (scalar or list)
- `model` (dict or str): Model definition or alias
- `output_dir` (str): Output directory path
- `input_static` (list of str, optional): Files identical across every case (see `fzr`'s
`input_static`); symlinked into `output_dir` rather than templated/duplicated

**Returns**: None (writes files to output_dir)

Expand Down Expand Up @@ -386,6 +390,29 @@ results_df = fz.fzr(
single `cases.csv` manifest is written at the results root mapping each case
directory to its variables (each case's own `info.txt` also has them, as a
fallback). Defaults to the `FZ_CASE_NAMING` env var, or `"path"`.
- `input_static` (list of str, optional): Files identical across every case (e.g. a
shared weather CSV or a large reference dataset) that are never templated/
substituted, never re-hashed per case, and (for relative paths) not duplicated on
disk per case:
- **Absolute path** entries are assumed already present at that same path on the
calculator side too (shared/mounted storage); fz never copies, symlinks, or
transfers them - only hashes them once per `fzr()`/`fzd()` call, so `cache://`
still reacts if the shared file's content changes. The calculator command/script
must reference the absolute path directly.
- **Relative path** entries are resolved against the cwd `fzr()`/`fzd()` was called
from, identified by their **basename** (not the full declared path, which may
contain `..` to reach outside `input_path`), symlinked into every case's
directory (falling back to a real copy if the platform disallows symlinks, e.g.
Windows without developer mode/admin), and explicitly transferred to `ssh://`,
`slurm://` (remote), and `funz://` calculators, since they live outside
`input_path` and the generic per-case file transfer never finds them.
- Either way, `fzi()` never scans them for `$variables`, and `.fz_hash` always
includes them so `cache://` matching stays correct.
- **Detection helper**: if a file under `input_path` has no variables and is at
least `FZ_STATIC_CANDIDATE_MIN_SIZE` bytes (default 1 MiB), `fzr()` logs a
one-time warning suggesting it be passed via `input_static` instead - it's
otherwise re-read/re-copied and re-hashed on every case. Set
`FZ_STATIC_CANDIDATE_MIN_SIZE=0` to disable.

**Returns**: pandas DataFrame with all results and metadata

Expand Down Expand Up @@ -537,7 +564,8 @@ result = fz.fzd(
algorithm,
calculators=None,
algorithm_options=None,
analysis_dir="analysis"
analysis_dir="analysis",
input_static=None
)
```

Expand All @@ -550,6 +578,9 @@ result = fz.fzd(
- `calculators` (str, list, or int): Calculator URI(s) (default: `["sh://"]`); when `model` is a callable, must be an `int` (default: `1`), accepted for API compatibility — calls are always run sequentially, never in parallel (see below)
- `algorithm_options` (dict, str, or None): Algorithm-specific options (dict, JSON string, or JSON file path)
- `analysis_dir` (str): Analysis results directory (default: `"analysis"`)
- `input_static` (list of str, optional): Files identical across every case (see `fzr`'s
`input_static`); passed through unchanged to each iteration's internal `fzr()` call
for file-based models

**Returns**: Dictionary with keys:
- `XY`: pandas DataFrame with all input and output values
Expand Down
Loading
Loading