New fast channel for reproducible random - #12
Open
jpn-- wants to merge 42 commits into
Open
Conversation
* bug fix * add utility based choice option for transit virtual pathbuilder * custom chooser * mem tracing in interaction_sample * bug fix * interaction_sample memory saving implementation * memory work * spelling * delete after last ref, not before * set index * to numpy once * remove comment * interaction sample dev * remove scale and location params from ev1 quartile fct * use numpy's gumbel to draw from ev1 * clean up * gumbel by hand by inverse cdf * memory logging * merge bug * more merge bugs * more merge fixes, example runs through * log eet setting on start up * some variation checks, dev notebooks, to delete * seeding corr test * corr * sign on logsum * removes notebook folder * compute_setting overrides for eet, currently only for interaction_sample to test loc sampling with MC * Adds subcomponent-specific eet setting * default compute settings set before, no need to check if none * comment clean up * identifies eet todos * test SOA MC for disagg access * lint * comments * more comments * jtp custom chooser for EET * lint * move interaction sample without sampling out of MC loop to enable for both MC and eet * update comments * fix no sampling in interaction_sample * fix _interaction_sample return doc * fix bool conversion of None for eet compute settings in sub-components * compute settings in disaggreagte accessibility w/o hack * lint * fix nest_spec is None bug * remove comment * clean up comments * comment clean up * avoid choosing alternatives corresponding to padded utilities in interaction_sample_simulate * lint * patch other instance of padded_utility for eet * adds validate utility method to align with unavailable choices in MC simulation * consistently treat utilities that would lead to zero choices in MC simulation * lint * fix allow_zero_probs for prob calculation in EET interaction_sample * fix unbound variable in estimation mode * blacken * update larch pin to 6.0.46 * np array instead of for loop * memory reduction * no duplicate arrays * bug fix: order of chooser_idx in interaction_simulate * add tests, docstrings for logit * Add basic with/without EET test for interaction simulate * Fix, complete tests for interaction sample, simulate * Linting * Normalise number of choosers, alternatives and minimum tolerance for EET comparison tests * reshape, do not flatten for potential performance * undo stray comment * unify mc and eet reporting during choice making * series not array * reinstate test, up number of draws for comparison * numpy not loop * interaction_sample test to catch index order bug * make test clearer * reset rng offset on iterate_location_choice (shadow pricing) * Add tests for logit NL, ordering, and all models using EET * Add basic docs for EET * Add tests checking that choices made using eet and from probabilities are the same for eval_mnl and eval_nl * Linting, minor changes to test_simulate.py * roll back changes to core tests to minimize noise * Implement Jan's suggestion of how to calculate household ids for testing cdap eet parity * Add test for compute_nested_utilities. Computes nested utilities on a toy example and compares the result * Linting * Move nest_spec to a fixture in simulate.py tests * Finish removing nest_spec definition in simulate.py tests * lint * clean up * reset rng only for eet for now. discuss in engineering meeting * Close out remaining TODO-EETs * Shuffle check to logit.py * Linting * Fix test * Roll back validate_utils * eet doco start * doco part two * add scale comment * Revert linting changes * . * Fix failing tests * doc fix * docs * doco clean up * doco re memory * doco clean up * memory usage doco * clean up doco * undo changes in unrelated code to reduce noise * testing for semcog model, update to shadow pricing reset * lint * disable test unitl further investigation * mp rng reset, disabling one new test for no * semcog test with different seed for eet mp to work around non-determinism in trip_scheduling (probabilistic, no EET branch) * no need to restrict to specific choosers for reset, each process has its own state * Add eet multiple zone test for zone=2 * Change formatting from using ruff to black * add back accidentally removed import * re-add 3-zone tests * base seed same as non-eet, update regress eet data * progressive 2 zone test * Copy changes from m-richards:matt/poisson_eet_2026_04 * Tidy, complete implementation of alts_context, add tests, rebuild regress for multi zone tests * Commenting/fix alts context constructions, update semcog test * Deterministic randoms when adding/removing tours for trip scheduling choice * PR noise * Roll back final_trips.csv for production_semcog * PR noise, make sure alts_context built on un-modified alternatives in trip_dest * linting * updates outdated comment * re-add arc test, remove parking choice column from regress trips * re-instate arc tests * Revert "Deterministic randoms when adding/removing tours for trip scheduling choice" This reverts commit 2d181b2. * no alts_context for trip_scheduling * updates arc regress files * removes outdated comment * trip scheduling comment for EET * disable arc sharrow test until regress trips are updated * doco update * Matt's Poisson sampling changes from PR 1065 * bug fix * add second return arg for zero probs as well * eet for nested logit * towards a unified choice interface * test and doco updates * jtp fix Co-authored-by: Copilot <copilot@github.com> * fused location sampling Co-authored-by: Copilot <copilot@github.com> * c order to map the first n_alt rands to all alternatives for the first sample, etc Co-authored-by: Copilot <copilot@github.com> * fused mnl ev1 argmax with alt_info Co-authored-by: Copilot <copilot@github.com> * stable sampling Co-authored-by: Copilot <copilot@github.com> * some doco updates * doc clean up * poisson sample tests and runtime improvements * tests for random sampling as fallback, some doc strings and removal of superfluous probs * avoid unneccessary dataframe construction, better doco * update interaction sample tests for poisson * integration of sampling methods with clear separation from eet as simulation method * poisson sampling optimization * poisson sampling with stable rands across scenarios where attractions flip from 0 to non-zero, other clean ups * do not throw on poisson fallback sampling with sample_size > alternatives * debug logging * clean up * sampling doco * clean up * doco update * linting * razzmatazz. I mean maz for taz. * test fixes and stable sorting for tour/trip loc choice * no stable alts for MC * lint * stable two-zone maz_from_taz for Poisson * more doco * no op if no alts_context * do not throw on interaction_sample_simulate w/o alts_context * eet special casing * tour_od with maz_taz stable, clean up * arc eet golden trips * switch base seed to avoid trip_scheduling (probabilistic) to come up with edge cases for small sample test. note this also happens for non-eet, e.g. with base seed 1 * forgot the corresponding golden trips * test multiple_zone golden * removes requirement of interaction_sample_simulate to have alts_context * fix test by using eet as intended, remove stable_indexing in tour_dest for mc * clean up * conditional stable sample indexes * stable alts only for eet with poisson sampling for two-zone * trip maz-for-taz stable alts for eet with poisson sampling * clean up * stable two-zone alts for tour_od * tour_od stable alt cond * resolve_smapling_method * decouple sample and simulation methods * separate RNG for shadow pricing to enable loc choiec rng reset for simulation method * lint * logging and comments * debug logging for sample method, add warning for disagg acc and poisson * doco * info, nto warning * compute settings, not model settings * compute settings, not model settings * variable name * log log on subset of rands only * optionally bias poisson disagg acc * bias also for non-mand * avoid copy in favour of log log for gumbel choice * switch to bias all location choice logsums for poisson sampling * add setting * adds missing import in trip dest * adds missing import in trip dest * ling * addresses claude code review * consistent spelling * mal-formed html comment * doc ref * addresses copilot code review * remove try block * clean up * logit clean up * clean up * lint * estimation safe guard only for poisson * deletes arc test as discussed in engineering meeting in June * adding skip_failed_choices into test, removing dead function parameter * fix eet trip_dest presample test * removes duplicate call * addresses pr review comments * changes poisson sampling to be deterministic when no alternative sampled * do not allow zero probs for nl leafs * handle non-available alternatives in make_choices_utility_based like MC in make_choices * harmonize sample choice maker naming --------- Co-authored-by: Jan Zill <jan.zill@veitchlister.com.au> Co-authored-by: Jan Zill <jzill@anl.gov> Co-authored-by: Jeff Newman <jeff@driftless.xyz> Co-authored-by: Tom Stephen <tom.stephen@icloud.com> Co-authored-by: Tyler Pearn <tyler@outerloop.io> Co-authored-by: Copilot <copilot@github.com>
* Support explicit_chunk for mode-choice components Location, destination, and scheduling components already honor the `explicit_chunk` setting (a fixed chunk size, used when chunk_training_mode: explicit), but mode-choice components did not: the setting was never plumbed from the component settings into simple_simulate's chunk loop. This wires it through: add `explicit_chunk` to TemplatedLogitComponentSettings (the base for mode-choice settings), and pass it via mode_choice_simulate -> simple_simulate -> adaptive_chunked_choosers. Defaults to 0 (unchanged behavior). This lets tour_mode_choice and trip_mode_choice run with a fixed, bounded per-chunk memory footprint, which is what makes large samples fit on memory-constrained machines under chunk_training_mode: explicit. * docs: sync mode_choice_simulate docstring with explicit_chunk_size Remove the stale `chunk_size` entry (not a parameter), mark compute_settings as optional, and document the new explicit_chunk_size argument. Addresses the automated review note on PR ActivitySim#1088. --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* fallback poisson without zero prob alts, mathjax rendering for doco * removes todo * fix trip_schedulin_choice comment * use standard terminology * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* location choice logsum not overwritten * test: cover location choice logsum survey overrides * Set sampling options in location choice test --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* ENH: add trip scheduling choice explicit chunking # Conflicts: # activitysim/abm/test/test_misc/test_trip_scheduling_choice.py * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * TST: strengthen trip scheduling explicit chunking coverage --------- Co-authored-by: Matt Richards <matt.richards@veitchlister.com.au> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Introduce a configurable RNG channel type and exercise both implementations in tests. Add Settings.rng_channel_type to choose between the new FastChannel (PCG64 vectorised) and legacy SimpleChannel for reproducibility. Random now accepts a channel_type on init and add_channel accepts fast=None to default to the global channel_type; existing code will pick up settings.rng_channel_type via State initialization and rng access. Implement FastChannel.extend_domain to allow adding new domain rows (initialising per-row PCG64 state when a step is active) and tighten index handling. Update many pipeline tests to parametrize over channel types, isolate per-channel output dirs, and include per-channel expected regression values and checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new
activitysim.core.fast_randompackage. It has one public class,FastChannel, which exactly reproduces (I believe) the entire functionality ofactivitysim.core.random.SimpleChannel, except forinit_row_states_for_step, which is only ever used internally bySimpleChannel.We also add plumbing to
activitysim.core.random.Randomandactivitysim.core.configuration.topto use the new fast channel.The
FastChannelimplementation usesnumpy'sPCG64bit generator, and stores the actual 256 bit random state for each index value in the channel. Since we store this state, we do not need to reseed inside a step, nor do we do any state forwarding.We do still need to reseed for each index value once per ActivitySim step to maintain reproducibility, so the runtime benefits for this change will mostly accrue to models that need to make multiple random draws inside a single ActivitySim step.
Also, all existing tests that are contingent on having a stable random result will fail if you change the random generator. Prior code used the (old and slow) Mersenne Twister via the old
numpy.random.RandomState, and the new code uses a different mathematical algorithm, so it cannot be set to give the same stream of random values. Therefore, the default setting is to not use this new feature unless it is explicitly activated.This PR includes unit tests for
FastChannelwhich have been tested and pass on Linux, Mac and Windows.