Repository navigation
Commit 263624c
authored
Speed up decoding of annotation files in rdann (#627)
`proc_ann_bytes` and its helpers decode an annotation file by indexing
the Nx2 NumPy byte array one element at a time (`filebytes[bpi, 1]`,
`np.int64(...)`, `.astype("i1")`), and build each aux note with
`"".join(chr(c) for c in ...)`. Each of those makes a NumPy scalar,
which is slow in a Python loop.
This converts the byte pairs to a list once with `.tolist()` and does
the rest with plain ints. Signed `subtype`/`num` bytes are converted
with a small helper, and aux notes are decoded with
`bytes(...).decode("latin-1")`, which gives the same characters as
`chr()` for 0-255. Since no NumPy scalars are involved any more, the
decoding no longer depends on NumPy's scalar type promotion rules, which
caused the `OverflowError` in #493 and #522 (already worked around on
main with `np.int64`).
The output is unchanged: I compared every field (`sample`, `symbol`,
`subtype`, `chan`, `num`, `aux_note`, `fs`, `label_store`, custom
labels, and dtypes) for all 15 annotation files in `sample-data`, with
and without `sampto`, against main.
Benchmark (median, Apple Silicon, alternating runs of main and this
branch; the machine was busy, so the absolute numbers are noisy):
| file | main | this PR | speedup |
| --- | --- | --- | --- |
| `rdann` 100.atr | 4.10 ms | 2.01 ms | 2.0x |
| `rdann` 1003.atr | 6.24 ms | 3.31 ms | 1.9x |
| `rdann` 12726.wabp (many aux notes) | 16.5 ms | 6.4 ms | 2.6x |
| `rdann` 03700181.gqrsl | 2.99 ms | 1.64 ms | 1.8x |
| `rdann`, 100,000 beats with some aux notes (written with `wrann`) |
176 ms | 76 ms | 2.3x |
| `proc_ann_bytes` only, 12726.wabp | 22.4 ms | 7.6 ms | 2.9x |
Tests: added a round-trip test for `subtype`, `chan` and `num` values
(including negative subtypes). `uv run pytest -m "not network"` passes,
`tests/test_annotation.py` passes with network tests, `ruff
format`/`ruff check` are clean, and the docs build without warnings.
Added a changelog entry.
Open PRs #491 and #498 also touch `annotation.py`, but only how the file
is opened and read into bytes, not the decoding, so they shouldn't
conflict beyond nearby lines.3 files changed
Lines changed: 58 additions & 28 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2209 | 2209 | | |
2210 | 2210 | | |
2211 | 2211 | | |
| 2212 | + | |
| 2213 | + | |
2212 | 2214 | | |
2213 | 2215 | | |
2214 | 2216 | | |
| |||
2218 | 2220 | | |
2219 | 2221 | | |
2220 | 2222 | | |
2221 | | - | |
| 2223 | + | |
2222 | 2224 | | |
2223 | | - | |
| 2225 | + | |
2224 | 2226 | | |
2225 | 2227 | | |
2226 | 2228 | | |
| |||
2231 | 2233 | | |
2232 | 2234 | | |
2233 | 2235 | | |
2234 | | - | |
| 2236 | + | |
2235 | 2237 | | |
2236 | 2238 | | |
2237 | 2239 | | |
2238 | 2240 | | |
2239 | | - | |
| 2241 | + | |
2240 | 2242 | | |
2241 | 2243 | | |
2242 | 2244 | | |
| |||
2245 | 2247 | | |
2246 | 2248 | | |
2247 | 2249 | | |
2248 | | - | |
| 2250 | + | |
2249 | 2251 | | |
2250 | 2252 | | |
2251 | 2253 | | |
| |||
2261 | 2263 | | |
2262 | 2264 | | |
2263 | 2265 | | |
| 2266 | + | |
| 2267 | + | |
| 2268 | + | |
| 2269 | + | |
| 2270 | + | |
2264 | 2271 | | |
2265 | 2272 | | |
2266 | 2273 | | |
2267 | 2274 | | |
2268 | 2275 | | |
2269 | | - | |
2270 | | - | |
| 2276 | + | |
| 2277 | + | |
2271 | 2278 | | |
2272 | 2279 | | |
2273 | 2280 | | |
| |||
2287 | 2294 | | |
2288 | 2295 | | |
2289 | 2296 | | |
2290 | | - | |
| 2297 | + | |
2291 | 2298 | | |
2292 | 2299 | | |
2293 | | - | |
2294 | | - | |
2295 | | - | |
2296 | | - | |
| 2300 | + | |
| 2301 | + | |
| 2302 | + | |
| 2303 | + | |
2297 | 2304 | | |
2298 | 2305 | | |
2299 | 2306 | | |
| |||
2305 | 2312 | | |
2306 | 2313 | | |
2307 | 2314 | | |
2308 | | - | |
2309 | | - | |
| 2315 | + | |
| 2316 | + | |
| 2317 | + | |
2310 | 2318 | | |
2311 | 2319 | | |
2312 | 2320 | | |
| |||
2323 | 2331 | | |
2324 | 2332 | | |
2325 | 2333 | | |
2326 | | - | |
2327 | | - | |
| 2334 | + | |
| 2335 | + | |
2328 | 2336 | | |
2329 | 2337 | | |
2330 | 2338 | | |
| |||
2366 | 2374 | | |
2367 | 2375 | | |
2368 | 2376 | | |
2369 | | - | |
| 2377 | + | |
2370 | 2378 | | |
2371 | 2379 | | |
2372 | 2380 | | |
2373 | 2381 | | |
2374 | 2382 | | |
2375 | | - | |
| 2383 | + | |
2376 | 2384 | | |
2377 | 2385 | | |
2378 | 2386 | | |
2379 | 2387 | | |
2380 | 2388 | | |
2381 | | - | |
| 2389 | + | |
2382 | 2390 | | |
2383 | 2391 | | |
2384 | 2392 | | |
2385 | 2393 | | |
2386 | 2394 | | |
2387 | 2395 | | |
2388 | | - | |
2389 | | - | |
2390 | | - | |
2391 | | - | |
2392 | | - | |
2393 | | - | |
2394 | | - | |
2395 | | - | |
| 2396 | + | |
| 2397 | + | |
| 2398 | + | |
| 2399 | + | |
| 2400 | + | |
| 2401 | + | |
| 2402 | + | |
2396 | 2403 | | |
2397 | | - | |
| 2404 | + | |
2398 | 2405 | | |
2399 | 2406 | | |
2400 | 2407 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
423 | 423 | | |
424 | 424 | | |
425 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
0 commit comments