Skip to content

Enable benchmarking a real GLASS workflow #1086

New issue
New issue
Closed
5 of 7 issues completed
Closed
Enable benchmarking a real GLASS workflow#1086
5 of 7 issues completed
Assignees
connoraird
Labels
array-apiWork is related to the Array APIbenchmarksBenchmarking workenhancementNew feature or request

Description

@connoraird
connoraird
opened on May 20, 2026

Is Your Feature Request Related to a Problem? Please Describe

We would like to provide a mechanism for benchmarking a realistic GLASS workflows on various machines (HPC clusters, local laptops, etc). This will be different from the regression tests we have now which only test individual functions within GLASS and are intended to catch regressions in performance purely coming from changes to GLASS code. The intention of these new benchmarks will be to understand the ideal hardware configurations and GLASS workflows combined.

Activity

  1. connoraird
    assigned
    connoraird
    and
    paddyroddy
    on May 20, 2026
  2. connoraird
    added
    enhancementNew feature or request
    array-apiWork is related to the Array API
    benchmarksBenchmarking work
    on May 20, 2026
  3. connoraird
    changed the title [-]nable benchmarking a real GLASS workflow[/-] [+]Enable benchmarking a real GLASS workflow[/+] on May 20, 2026
  4. connoraird
    added sub-issues on May 20, 2026
  5. paddyroddy
    added a sub-issue on May 20, 2026
  6. connoraird
    unassigned
    paddyroddy
    on Jun 8, 2026

  7. connoraird commented on Jun 8, 2026

    @connoraird
    connoraird
    on Jun 8, 2026
    ContributorAuthor

    To allow benchmarking on Archer2, we need to prevent HTTP calls happening mid execution. Therefore, we should add a global option to define a path to the healpy pixel weight data files.

  8. paddyroddy commented on Jun 8, 2026

    @paddyroddy
    paddyroddy
    on Jun 8, 2026
    Member

    To allow benchmarking on Archer2, we need to prevent HTTP calls happening mid execution. Therefore, we should add a global option to define a path to the healpy pixel weight data files.

    Raised in #1121

  9. connoraird
    added a sub-issue on Jun 8, 2026

  10. connoraird commented on Jul 15, 2026

    @connoraird
    connoraird
    on Jul 15, 2026
    ContributorAuthor

    Storing some benchmark results for the lensing.py benchmark on Archer2 GPU node

    Running benchmark with nside/lmax = 128
Running benchmarks for backends:  jax.numpy
Took 189.210 seconds with jax.numpy
Running benchmark with nside/lmax = 256
Running benchmarks for backends:  jax.numpy
Took 371.657 seconds with jax.numpy
Running benchmark with nside/lmax = 512
Running benchmarks for backends:  jax.numpy
Took 2225.858 seconds with jax.numpy
Running benchmark with nside/lmax = 1024
srun: Job 14369784 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Step created for job 14369784
Running benchmarks for backends:  jax.numpy
Took 4756.359 seconds with jax.numpy

  11. connoraird commented on Jul 15, 2026

    @connoraird
    connoraird
    on Jul 15, 2026
    ContributorAuthor

    A good resource for profiling on Archer2 with gpu

  12. connoraird commented on Jul 16, 2026

    @connoraird
    connoraird
    on Jul 16, 2026
    ContributorAuthor

    Equivalent results for the same benchmarks using CPU

    Running benchmark with nside/lmax = 128
Running benchmarks for backends:  numpy, array_api_strict, jax.numpy
Took 2.639 seconds with numpy
Took 2.226 seconds with array_api_strict
Took 50.071 seconds with jax.numpy
Running benchmark with nside/lmax = 256
Running benchmarks for backends:  numpy, array_api_strict, jax.numpy
Took 10.723 seconds with numpy
Took 11.445 seconds with array_api_strict
Took 103.338 seconds with jax.numpy
Running benchmark with nside/lmax = 512
Running benchmarks for backends:  numpy, array_api_strict, jax.numpy
Took 71.864 seconds with numpy
Took 73.217 seconds with array_api_strict
Took 2910.300 seconds with jax.numpy
Running benchmark with nside/lmax = 1024
Running benchmarks for backends:  numpy, array_api_strict, jax.numpy
Took 513.699 seconds with numpy
Took 516.891 seconds with array_api_strict
Took 6370.869 seconds with jax.numpy

  13. paddyroddy commented on Jul 16, 2026

    @paddyroddy
    paddyroddy
    on Jul 16, 2026
    Member

    Where L = 2^n

    n numpy array-api-strict jax.numpy CPU jax.numpy GPU
    7 2.639 2.226 50.071 189.210
    8 10.723 11.445 103.338 371.657
    9 71.864 73.217 2910.300 2225.858
    10 513.699 516.891 6370.869 4756.359

    Not looking great...

  14. paddyroddy commented on Jul 16, 2026

    @paddyroddy
    paddyroddy
    on Jul 16, 2026
    Member

    Suspect we need to go much higher n to see a meaninful benefit

  15. connoraird commented on Jul 16, 2026

    @connoraird
    connoraird
    on Jul 16, 2026
    ContributorAuthor

    Suspect we need to go much higher n to see a meaninful benefit

    Yes I agree. I'm pretty sure the issue is memory transfers to and from the GPU which dominate over the compute time.

  16. connoraird
    added sub-issues on Sep 14, 2026
  17. connoraird
    added a parent issue on Sep 14, 2026
  18. paddyroddy
    added a sub-issue on Sep 21, 2026
  19. connoraird
    added a commit that references this issue on Oct 15, 2026

    gh-1086: Document how to profile GLASS (#1229)

    5662ae9

  20. paddyroddy commented on Oct 20, 2026

    @paddyroddy
    paddyroddy
    on Oct 20, 2026
    Member

    Closing this as it is essentially done. There are more useful profiling results that we'll leave documented in #1187.

  21. paddyroddy
    closed this as completedon Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

array-apiWork is related to the Array APIbenchmarksBenchmarking workenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions