Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ storage. See [`../CLAUDE.md`](../CLAUDE.md) for how the three layers fit togethe
- `expand(spec) -> Vector{DataKey}` — Cartesian product of list-valued params × `total_samples`.
- `format_path(key, path_keys) -> String` — compact on-disk directory segment.
- `canonical(key) -> String` — stable, order-/version-independent key identity.
- `param(key, name[, T])` — one parameter, resolved dotted-or-leaf; with `T` it refuses a
conversion that would change the number.

## Two facts that trip up callers

Expand All @@ -32,7 +34,7 @@ storage. See [`../CLAUDE.md`](../CLAUDE.md) for how the three layers fit togethe

## Source layout

`src/core/` = public API (`types`, `load`, `expand`, `format`, `canonical`).
`src/core/` = public API (`types`, `load`, `expand`, `format`, `canonical`, `param`).
`src/util/` = internal (`grid`, `flatten`, `path_keys`) — renamable without notice.

## Invariant when changing this package
Expand Down
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name = "ParamIO"
uuid = "938a3ac2-d340-473c-bcf1-88af577e4ccf"
version = "0.4.7"
version = "0.4.8"
authors = ["sota shimozono <shimozono-sota631@g.ecc.u-tokyo.ac.jp>"]

[deps]
Expand Down
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,12 @@ So the two leaf-table forms are duals: a **grid** (`{ start, stop, … }`) is a
`path_keys`:

```julia
key.params["system.N"] # ✅
key.params["N"] # ❌ KeyError
param(key, "system.N") # ✅ dotted
param(key, "N") # ✅ the leaf, when only one group has it
param(key, "N", Float64) # ✅ …and refuses the conversion if it would change the number

key.params["system.N"] # ✅ the raw Dict is still there
key.params["N"] # ❌ KeyError — no resolution, and no report of what does exist
```

A fixed scalar (`J` above) is **not** swept but **is** carried in every `DataKey`
Expand All @@ -124,6 +128,7 @@ name); `canonical` uses *all* params plus the sample index.
| `expand(spec; sweep_order=nothing) -> Vector{DataKey}` | Cartesian product × samples |
| `format_path(key, path_keys) -> String` | compact directory segment |
| `canonical(key) -> String` | stable, Julia-version-independent key identity |
| `param(key, name[, T]) -> value` | one parameter, resolved dotted-or-leaf and optionally typed |
| `resolve_path_keys(blocks) -> Vector{String}` | auto-detect `path_keys` if omitted |

## Source layout
Expand Down
3 changes: 3 additions & 0 deletions src/ParamIO.jl
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ export ConfigSpec, StudySpec, DataKey, AmbiguousPathKeyError
export load, expand, format_path, resolve_path_keys
export build_axis_formats, AxisFloatFmt
export canonical
export param
export diagnose, DiagnosticReport

# ── Core ──────────────────────────────────────────────────────────────────────
Expand All @@ -26,12 +27,14 @@ include("core/types.jl")
# ── Util (core が依存する) ────────────────────────────────────────────────────
include("util/grid.jl")
include("util/flatten.jl")
include("util/resolve.jl")
include("util/path_keys.jl")

# ── Core API ──────────────────────────────────────────────────────────────────
include("core/load.jl")
include("core/expand.jl")
include("core/format.jl")
include("core/param.jl")
include("core/canonical.jl")
include("core/diagnose.jl")

Expand Down
15 changes: 8 additions & 7 deletions src/core/format.jl
Original file line number Diff line number Diff line change
Expand Up @@ -165,13 +165,14 @@ end
# exact-then-leaf lookup. Returns `nothing` when absent or ambiguous (build then
# simply skips it — a non-float / unresolvable axis gets no auto entry).
function _resolve_axis_value(key::DataKey, pk::String)
val = get(key.params, pk, nothing)
val !== nothing && return val
_, leaf = _split_dotted(pk)
leaf == pk || return nothing
matches = Any[v for (k, v) in key.params if _split_dotted(k)[2] == pk]
length(matches) == 1 || return nothing
return matches[1]
# Absent and ambiguous are both "no auto entry" here, so this asks for the outcome rather than
# catching a raise: a `try` would have taken an InterruptException, or a bug in the resolver,
# with the same hand as the two conditions it means to allow — and `build_axis_formats` would
# then report "this axis has no auto format", which is how two distinct values come to share a
# directory name (see the header of this file).
found = _lookup_name(pk, Base.keys(key.params))
found isa String || return nothing
return key.params[found]
end

"""
Expand Down
54 changes: 54 additions & 0 deletions src/core/param.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# core/param.jl — reading one parameter out of a DataKey.

"""
param(key, name) -> Any
param(key, name, T) -> T

The value `name` selects in `key`, resolved by the same rule as a `path_key`: an exact match on the
dotted name, else a unique match on the leaf. See `_resolve_name`.

`work_fn` is otherwise written as `Float64(key.params["system.kbT"])`, and both halves of that are
a hazard.

A misspelled name is a bare `KeyError` naming only the string that was wrong. It is raised inside
the work function, which is running on a worker, dispatched by `run!` — so the config that has the
right spelling and the function that has the wrong one are in different files, and nothing compares
them until a point is computed. `param` reports what the key does carry instead.

The type is the other half. TOML gives `2` as `Int64` and `2.0` as `Float64`, so a config edited
from `[2.0, 2.5]` to `[2, 3]` changes what reaches the kernel without changing anything a reader
would look at.

`param(key, name, T)` converts and then, **for numbers**, checks the value survived it, so any `T`
that would have altered it is refused rather than returned. `convert` alone is not enough: it raises
for `Int` from `2.5`, but silently rounds for `Float64` from `2^53 + 1` and for `Float32` from `2.1`
— the same silent change of number, one type down from the one this exists to prevent.

The check stops at the numeric tower on purpose. A type with no `==` falls back to identity, so
applying it further would refuse a correct `convert(::Type{Celsius}, ::Real)` for returning a
different object rather than a different value. Outside numbers, `param` gives you whatever
`convert` gives you.

```julia
a = param(key, "system.a", Float64)
n = param(key, "numerics.nsteps", Int)
kT = param(key, "kbT", Float64) # leaf form, when it is unambiguous
```
"""
function param(key::DataKey, name::AbstractString)
return key.params[_resolve_name(name, Base.keys(key.params))]
end

function param(key::DataKey, name::AbstractString, ::Type{T}) where {T}
val = param(key, name)
converted = convert(T, val)
# Numbers only. Between an Int and a Float `==` is exact in Julia, so this catches the roundings
# `convert` performs silently. Outside the numeric tower it would not be measuring the same
# thing: a type with no `==` falls back to identity, so a perfectly good
# `convert(::Type{Celsius}, ::Real)` would be refused for being a different object rather than a
# different value.
if val isa Number && converted isa Number
isequal(converted == val, true) || throw(InexactError(:param, T, val))
end
return converted
end
55 changes: 55 additions & 0 deletions src/util/resolve.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# util/resolve.jl — the one place a name is resolved against a set of parameter keys.
#
# The rule is: an exact match on the dotted name, else a unique match on the leaf.
#
# It is also written out inline in `format_path` (both forms) and in `_validate_path_keys`, and
# those copies do not agree on what an ambiguous leaf is: `_validate_path_keys` throws
# `AmbiguousPathKeyError`, both `format_path` copies throw a bare `error`. Anything reusing the rule
# should come here rather than add a fourth reading of it.

"""
_lookup_name(name, available) -> String | Vector{String} | Nothing

The outcome of resolving `name`, without deciding what to do about it:

- a `String` — the key it selects;
- a `Vector{String}` — the groups a plain leaf appears in, when there is more than one;
- `nothing` — nothing matched.

Split out from [`_resolve_name`](@ref) so that the caller which treats "absent" and "ambiguous" as
ordinary outcomes does not have to reach them through a `catch`. A `try` around a raise is a filter
that cannot say what it caught: it takes `InterruptException` and a genuine bug with the same hand
as the two conditions it means to allow.
"""
function _lookup_name(name::AbstractString, available)
name ∈ available && return String(name)

_, leaf = _split_dotted(String(name))
leaf == name || return nothing

matches = sort!(
String[String(k) for k in available if _split_dotted(String(k))[2] == name]
)
length(matches) == 1 && return matches[1]
isempty(matches) && return nothing
return sort!(unique!(String[_split_dotted(k)[1] for k in matches]))
end

"""
_resolve_name(name, available) -> String

The key in `available` that `name` selects. Throws [`AmbiguousPathKeyError`](@ref) when a plain leaf
sits in more than one group, and an error naming what is available when nothing matches.

Returns the KEY, not the value, so callers that need the group prefix (or the value, or neither) all
get what they need from one rule.
"""
function _resolve_name(name::AbstractString, available)
found = _lookup_name(name, available)
found isa String && return found
found isa Vector && throw(AmbiguousPathKeyError(String(name), found))
return error(
"\"$name\" not found. Available: " *
join(sort!(String[String(k) for k in available]), ", "),
)
end
35 changes: 35 additions & 0 deletions test/paramio/test_format_auto.jl
Original file line number Diff line number Diff line change
Expand Up @@ -337,3 +337,38 @@ end
@test spec.float_format == "fixed2"
@test diagnose(spec).ok
end

@testset "build_axis_formats: a leaf path_key, resolved and refused" begin
# `_resolve_axis_value` routes through the shared name resolver and treats BOTH "absent" and
# "ambiguous" as "no auto entry". Every other call in this file uses fully dotted path_keys,
# which take the exact-match short-circuit — so without this the leaf branch, and the catch
# that turns an ambiguous leaf into a skip, are never executed.
study = ParamIO.StudySpec("t", 1, "out")
block = Dict{String,Any}(
"system.a" => [1, 2],
"model.a" => [3],
"system.g" => [0.5, 0.6],
"numerics.h" => [0.25],
)

spec = ParamIO.ConfigSpec(study, ["a", "system.g", "h"], [block], String[], "auto")
af = build_axis_formats(spec)
@test !haskey(af, "a") # leaf in two groups -> skipped, not thrown
@test haskey(af, "system.g") # exact dotted -> resolved
@test haskey(af, "h") # unique leaf -> resolved, which is the branch under test

# …and a name that matches nothing is skipped the same way rather than raising here.
spec2 = ParamIO.ConfigSpec(study, ["system.g", "nope"], [block], String[], "auto")
af2 = build_axis_formats(spec2)
@test !haskey(af2, "nope")
@test haskey(af2, "system.g")
end

@testset "format_path (auto): a leaf path_key drops the group prefix, like the 2-arg form" begin
# The 2-arg form has this pinned; the 3-arg one did not. It matters because the leaf branch
# formats WITHOUT the group prefix on purpose — "user chose plain name intentionally" — so a
# future conversion of this copy to the shared resolver, which returns the resolved KEY, would
# start emitting "modg0.50" and silently move where auto-mode results live on disk.
key = DataKey(Dict{String,Any}("model.g" => 0.5), 1)
@test format_path(key, ["g"], Dict{String,AxisFloatFmt}()) == "g0.50"
end
146 changes: 146 additions & 0 deletions test/paramio/test_param.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# param(key, name[, T]) — reading one parameter out of a DataKey.
#
# The two hazards this closes are the reason it exists, so both are asserted directly rather than
# through the happy path.

using ParamIO
using Test

const _K = ParamIO.DataKey(
Dict{String,Any}(
"system.a" => 2, "system.kbT" => 2.5, "numerics.nsteps" => 400, "model.a" => 9
),
1,
)

@testset "param: dotted, and leaf when it is unambiguous" begin
@test param(_K, "system.kbT") == 2.5 # exact dotted match
@test param(_K, "kbT") == 2.5 # unique leaf
@test param(_K, "nsteps") == 400
end

@testset "param: an ambiguous leaf throws AmbiguousPathKeyError, naming the groups" begin
# NOT the exception `format_path` raises for this condition — that one throws a bare
# `error`. The two disagree, and a `catch` written against one does not catch the other.
# `a` is in both `system` and `model`. Resolving it silently to either would make the value a
# study reads depend on Dict iteration order.
err = try
param(_K, "a")
nothing
catch e
e
end
@test err isa ParamIO.AmbiguousPathKeyError
@test err.groups == ["model", "system"]
@test occursin("model", sprint(showerror, err))
end

@testset "param: a missing name reports what the key does carry" begin
# The failure this replaces is a bare KeyError raised on a worker, naming only the wrong string.
err = try
param(_K, "system.kbt") # lower-case t
nothing
catch e
e
end
@test err isa ErrorException
msg = err.msg
@test occursin("system.kbt", msg) # what was asked for
@test occursin("system.kbT", msg) # …and what exists, which is the point
@test occursin("numerics.nsteps", msg)
end

@testset "param: the type is converted, and a lossy conversion is refused" begin
# TOML gives `2` as Int64 and `2.0` as Float64, so a config edited from [2.0, 2.5] to [2, 3]
# changes what reaches the kernel without changing anything a reader would look at.
@test param(_K, "system.a") isa Int # as stored
@test param(_K, "system.a", Float64) === 2.0 # …and as asked for
@test param(_K, "numerics.nsteps", Int) === 400

# Asking for an Int when the value is not one must fail at the boundary, not two layers down.
@test_throws InexactError param(_K, "system.kbT", Int)

# …and so must the conversions `convert` performs SILENTLY. These are the ones it allows:
# a Float64 cannot hold 2^53+1, and a Float32 cannot hold 2.1. Without the round-trip check
# both come back as a different number with no signal — which is the hazard one type down from
# the one this function exists to close.
# The values matter: 2.5 is exact in Float32, so it cannot show this and would pass for the
# wrong reason. 2.1 is not, and 2^53+1 is the smallest integer Float64 cannot hold.
lossy = ParamIO.DataKey(
Dict{String,Any}("system.seed" => 9007199254740993, "system.x" => 2.1), 1
)
@test_throws InexactError param(lossy, "system.seed", Float64)
@test_throws InexactError param(lossy, "system.x", Float32)
@test param(_K, "system.kbT", Float32) === 2.5f0 # …and an exact narrowing still goes through
end

@testset "param: an exact match wins before ambiguity is considered" begin
# A top-level name and a leaf of the same spelling can coexist. The rule resolves the exact
# match first, so `N` is the top-level one and no ambiguity is reported — even though the leaf
# `N` genuinely sits in two groups. Worth pinning: read the other way round it looks like a bug.
k = ParamIO.DataKey(Dict{String,Any}("N" => 99, "system.N" => 24, "model.N" => 8), 1)
@test param(k, "N") == 99
@test param(k, "system.N") == 24
end

# ── the conversion, across the type space ────────────────────────────────────
#
# `param(key, name, T)` is generic in `T`, so its behaviour is whatever `convert` does plus one
# guard. Testing it on `Int` and `Float64` alone measures two cells of a table with several
# interesting corners, and the guard was in fact wrong in one of them until this file grew.

struct Celsius
v::Float64
end
Base.convert(::Type{Celsius}, x::Real) = Celsius(float(x))

const _T = ParamIO.DataKey(
Dict{String,Any}(
"n" => 3, # TOML integer
"x" => 2.1, # TOML float, not exact in Float32
"half" => 2.5, # …but exactly representable, so it must NOT be refused
"big" => 9007199254740993, # 2^53 + 1: the smallest integer Float64 cannot hold
"s" => "abc", # TOML string
"b" => true, # TOML boolean
"v" => [1, 2, 3], # TOML array carried as a fixed value
),
1,
)

@testset "param(_, _, T): conversions that must go through" begin
@test param(_T, "n", Float64) === 3.0
@test param(_T, "n", Int) === 3
@test param(_T, "half", Float32) === 2.5f0 # exact narrowing is not a loss
@test param(_T, "n", Rational{Int}) == 3//1
@test param(_T, "n", BigInt) == big(3)
@test param(_T, "half", BigFloat) == big"2.5"
@test param(_T, "s", String) == "abc"
@test param(_T, "v", Vector{Float64}) == [1.0, 2.0, 3.0]
@test param(_T, "n", Any) === 3 # identity, not an error
end

@testset "param(_, _, T): conversions that must be refused" begin
@test_throws InexactError param(_T, "x", Float32) # 2.1 does not fit
@test_throws InexactError param(_T, "big", Float64) # 2^53+1 does not fit
@test_throws InexactError param(_T, "x", Int) # convert's own refusal
@test_throws InexactError param(_T, "n", Bool) # 3 is not a Bool
@test_throws MethodError param(_T, "s", Int) # no conversion exists
@test_throws MethodError param(_T, "n", Symbol)
end

@testset "param(_, _, T): the guard stops at the numeric tower" begin
# A user type with a `convert` method and no `==` falls back to identity comparison, so a
# round-trip check applied here would refuse a conversion that is entirely correct — it would be
# measuring "a different object", not "a different value". This is the case that made the first
# version of the guard wrong.
@test param(_T, "n", Celsius) == Celsius(3.0)
@test Celsius(3.0) != 3 # …and the reason it would have been refused
end

@testset "param(_, _, T): Bool is a number in Julia, and that shows" begin
# Not a defect to fix here, but worth pinning so nobody is surprised later: `Bool <: Integer`,
# so a boolean converts to and from the numeric types without complaint.
@test param(_T, "b", Int) === 1
@test param(_T, "b", Float64) === 1.0
@test param(_T, "b", Bool) === true
end
Loading