First-class R-universe support instead of treating it as a CRAN-like repository

[eitsupi]

Currently, rig treats R-universe repositories as generic CRAN-like repositories.
While this works for installing currently available packages, I think R-universe deserves a distinct source model.

R-universe exposes CRAN-compatible package indexes, but its underlying model is different from CRAN:

• R-universe builds packages from upstream Git repositories and exposes provenance metadata such as RemoteUrl, RemoteSha, and RemoteSubdir.
• Unlike CRAN, R-universe does not archive previous package versions. An artifact referenced by a lockfile may disappear when a newer build is published.
• A package's version alone is not sufficient to identify its source. Different Git commits may produce builds with the same R package version.

For example, suppose a project depends on polars from R-multiverse Community.
https://community.r-multiverse.org/polars

After a newer version is published, its previous artifact may no longer be available.
The lockfile should ideally retain enough upstream Git provenance to restore the selected revision rather than relying solely on the original R-universe download URL.

I would suggest considering:

1. A first-class R-universe repository type, distinct from CRAN and generic CRAN-like repositories, with access to R-universe's API and provenance metadata.
2. Separate source identity from distribution artifacts. The lockfile could preserve the upstream Git URL, commit SHA, and subdirectory alongside available R-universe source/binary artifacts.
3. Restoration from pinned Git provenance when the original artifact is no longer available, without silently selecting a different version or commit. A reconstructed source build would not necessarily be byte-identical to the original artifact, so this distinction should remain explicit.

We have adopted a similar modeling approach in the experimental https://github.com/eitsupi/rsolve, which distinguishes RUniverse from CranLike and models R-universe package identities using Git provenance.
This is a reference for the data model, not a claim that all of the restoration behavior described above is implemented there.

I am not suggesting that rig needs to adopt rsolve's implementation or schema. However, I think treating R-universe as a first-class source would provide a better foundation than adding special-case fallback behavior to a generic CRAN-like repository.

Would this approach make sense for rig's package management architecture?

[gaborcsardi]

Yes, this would indeed make a lot of sense. OTOH I am hoping that R-universe could be improved to serve better metadata and have stable URLs to older R universe builds. Otherwise using R universe packages in lock files is not very useful because the locked builds will be unavailable as soon as there is a new commit in the package's repository.

@jeroen does R universe delete the older binary (and source) builds, or these still exist but they are not in the PACKAGES files?

[gaborcsardi]

What exactly do you mean? All data we have is exposed via the APIs?

Data about older package versions and builds. Which you don't have any more, so I guess the question in mute.

Currently things get deleted for each update. If it helps we could instead expire things after n days or something. But I think it would be a waste of money to retain all binaries that were ever built forever...

Sure, i think that's perfectly fine. But it also means that it does not make too much sense to put R-universe repos into lock files, because the URLs are unstable, the same URL is going to serve a different build in the future, and/or the URL will be broken in the future.

[jeroen]

Yeah that is true. I am dreaming of a system where we just use the sha256 hashes from the binaries to quickly lookup all the binaries that are still available from whatever content-addressable storage or caches, without worrying about stable URLs...

[gaborcsardi]

You can put them on ghcr.io if you want, that's just content addressable storage and it is also free. IDK if they'd be bothered if you put too much stuff there. I have been putting a lot of stuff there, but probably not nearly as much as R-universe would if you saved every build. Do you know your build volume? E.g. the amount of data you put on R2 (say) in a month?

[jeroen]

Yes we have about 1TB active storage, and about 4TB/mo upload. But I also don't want to (over)promise features that I need to backpaddle on them when ghcr decides this is considered abusive or starts charging.

And I think a technical hurdle is that to download a file from ghcr.io you need to pass some auth headers, which install.packages does not support I think?

[gaborcsardi]

For https://github.com/r-hub/containers I upload about 60GB daily, i.e. about 1.8TB per month and never deleted anything for many years now.

[gaborcsardi]

I looked at the information that R-universe provides.

The PACKAGES files do not contain the git hashes. The JSON file at api/packages does, but that API does not seem very practical for larger repositories, e.g. for cran.r-universe.dev takes ~25 seconds for the 40MB of the first 3000 packages and I don't see a way to paginate. So we'd need to get the metadata package by package with api/packages/{package}. I guess this is possible and with some caching the speed might be acceptable as well.

Because both source and binary packages are deleted as soon as there is a new build, the user experience will not be great, at least not with lock files. But even simple rig pkg install will occasionally fail or fall back to building from the git tree if the files are updated between the dependency resolution and the package downloads.

@jeroen If you could keep the package file around for a bit that would improve the experience a lot. E.g. keep the last (say) 3 builds plus the builds that are newer than a day or week. That would improve rig pkg install at least, but using R-universe with lock files still seems impractical.

Maybe keeping all binary builds could be an option for a universe, e.g. cran.r-universe.dev only has builds for proper versions. Especially for platforms that PPM does not have, like aarch64 Windows, that'd be quite valuable to keep.