Repository navigation
Sparsity review and changes for BinnedData #586
Description
A recent issue (now resolved) raised by PR #570 brought to light issues with how we create BinnedData objects. These objects are created from unbinned data sets, which usually come with a time axis. When binning on the time axis, most time bins are likely to be empty, and so the resulting binned Histogram is likely to be sparse. Accidentally creating it as a dense Histogram can therefore cause resource usage to blow up.
That said, the original motivation for dense binned data is that most BinnedData users in cosipy immediately project away the time dimension and then convert the resulting reduced Histogram from sparse to dense. This is a real performance concern -- the sparse projection operation is extremely expensive! A better solution to this issue would be to allow BinnedData to create dense histograms from the unbinned data without the time axis in the first place.
Activity
jdbuhler commentedon Apr 20, 2026
To put specific numbers on this: the notebook diffuse_511_spectral_fit.ipynb loads a sparse binned data file "combined_binned_data.hdf5" that was created from two unbinned data files, each using a time binning of 0.5s. The shape of this binned data is
Time x Em x Phi x PsiChi = (15959910, 1, 60, 3072)
A dense version of this data in float64 would be nearly 22 GB, but it contains fewer than 5 million nonzero entries.
The first thing the notebook does with the binned data after loading is to project away its time axis, after which it has nearly 90% nonzero entries and so no longer needs to be sparse. (The notebook does not convert this data to dense, and I don't think the analysis it calls does either.) If we could create this data densely in the first place without the time axis, it would occupy only 1.4 MB and, as a dense matrix, would likely be considerably faster to process.
Note that the notebook avoids the csots associated with binning the data in the first place by downloading a pre-binned version (though oddly enough, it also downloads the unbinned data). But this should not be needed if we have a reasonably efficient time-independent binning implementation.
jdbuhler commentedon Apr 20, 2026
Some of the other notebooks work around this issue by defining the time axis bins large enough that all the data collapses to one time bin. But rather than resort to such hacks, we should be able to create the binned data densely without the time axis if that's how it is going to be used. This would hopefully remove the need to create and download pre-cooked versions of the binned data entirely.
jdbuhler commentedon Apr 20, 2026
Proposed fix (probably post-DC4): the BinnedData class should be simplified so that its primary method, get_binned_data(), is passed an UnbinnedData object and returns a Histogram with the binned data. All the remaining methods (most of which are just plotting code) could be rewritten to pass in the binned Histogram. The class would not need to retain any state except the binning axis info from the YAML file.
get_binned_data() could take a parameter indicating whether to save the time axis; alternatively, we could have to different functions to produce binnings with or without time. The sparsity could be chosen automatically based on whether the time axis is present, or the user could be required to specify sparsity (and so would be responsible for not blowing out of memory if the time axis is actually wanted!)
jdbuhler commentedon Apr 22, 2026
Additional note: the only good cases I am aware of for using sparse Histograms in cosipy right now are (1) time-dependent binned data, and (2) the coordinate conversion matrix used in Hiroki's image deconvolution code. Do we know of other examples where it would be detrimental to switch to dense?
israelmcmc commentedon May 15, 2026
@jdbuhler I'm reverting to
sparse=Truein #609. However, I don't want to automatically close this issue since you raised many good point that go beyond just changing the default. I'll leave it up to you if you want to move your comments to new issue(s), or just editing this one to reflect other comments concerning an overall improved behavior.[-]Make sure BinnedData remains sparse if needed for memory reasons[/-][+]Sparsity review and changes for BinnedData[/+]jdbuhler commentedon May 15, 2026
Thanks @israelmcmc . I will keep this issue number and have changed the text to reflect the current state of the code base after #609.
Revert binning default to sparse