Website to browse modules by tag

[willgearty]

Right now we have catered sets of modules in our workshop website, but it would be cool to be able to browse all of the available modules (both for developing a new workshop and for random users) on some website, by title/theme/etc.

[willgearty]

https://quarto.org/docs/websites/website-listings.html

[etiennebacher]

I think it would make sense to have this on the main website, e.g. like this:

(this is just a local test, it's not deployed) I have added this to the new website to get feedback but no problem to revert

The idea would be that anytime there is a push that changes the modules folder here, it triggers a workflow in the website repo that copies the modules folder to the website repo and updates the website. That way we can continue to develop modules in this repo without having to understand the entire website structure, and the website is always up to date with the contents here. However, there are two constraints:

• this means that only "finished" content should be pushed to main, otherwise incomplete content would appear on the website
• the .qmd and the rendered files must be pushed to main, not just .qmd. This is because we don't want to render all modules every time in the workflow of the website repo (because of changes in package versions for instance)

What do you think?

[etiennebacher]

Basically the structure I have in mind for this repo is:

modules/    # this repo

    ggplot/
      index.qmd
      images/
      _freeze

    pkg_dev/
      index.qmd
      _freeze

This way each module can be developed independently here, and a CI workflow can copy each module to the appropriate location in the website repo. Having one _freeze per module means that the workflow that updates the website will not re-compute R code chunks, which saves time and avoids reproducibility issues.

Edit: just realized this is basically what Lewis suggested in Zulip

[willgearty]

I'm not necessarily opposed to the interface living on the main website (although see my comments in the feedback doc about the current implementation). However, at the moment, I'm not convinced having them on the main website is required/better. I like the separation of the workshop materials to their own subdomain and keeping the two/three websites looking the same would be fairly easy with some github actions pulling from the resources repo. Maybe more work on the navigation of the modules would change my opinion (I understand the current implementation is just hosting what I had already put together).

On Zulip @LewisAJones mentioned avoiding archiving website infrastructure. Does that include _freeze folders? For our workshop repositories, we only keep the _freeze content on the main workshop repository and then keep only the source files in the individual workshop repositories (e.g., https://github.com/palaeoverse/2025-GSA-workshop). That way when we archive the individual workshops, only the source files are archived. Another option is to build a custom GitHub Actions workflow that pushes to Zenodo (instead of using the automatic release webhook), and we could exclude whichever files we want from the Zenodo upload.

Also, I'm not sure I understand the value of separate _freeze folders for each module vs. one _freeze folder. With freeze: auto, I'm pretty sure only changed modules would be updated anyways, right?

Another thought: are we looking to archive the individual modules separately or all as one archive? (I imagine the former would be possible with a GitHub Action)

Regardless of the infrastructure strategy, it might be nice to have a way to preview what work-in-progress modules would look like, especially if a new module can't be pushed to main until it is finished. Maybe we could use https://github.com/rstudio/education-workflows/blob/main/examples/pkgdown.yaml?

Sounds like there's a lot to discuss here; I'm happy to jump on a call if it's easier.

[LewisAJones]

Good points for consideration here @willgearty. There are a few things to discuss related to this, and some other WP1 items.

@etiennebacher is going to set up a meeting between the three of us.

[willgearty]

Some decisions from our meeting:

• We want to archive each module separately (ideally) on Zenodo, with just the source files archived (no _freeze, no shared files). The citation information should be pulled from the module itself (perhaps with a shared CITATION file if needed). The modules would all be stored in a Zenodo collection and each module would only be updated when source files for that module are changed. @LewisAJones suggests we could also have a README or CITATION for each module.
• The _freeze folder should only be generated in the rendering repository (e.g., the main palaeoverse website), not in this repository. This _freeze folder should be flushed at some time interval (two weeks? monthly?).
• Website repo will pull in any changes to this repo, then re-render.
• All module submissions should be made as PRs. Ideally these PRs would have some sort of render preview feature. Each module requires [n] reviews before approval.
• Each module should be its own folder in the root of the repository.
• Will will continue investigating rendering webpage/slides from a single qmd file.
• Include a Zenodo DOI badge on the live module webpages?

[etiennebacher]

I have implemented the infrastructure as discussed, except for the parts related to citation and Zenodo.

In palaeoverse/modules (this repo)

• one folder per module at the root of the repo
• when a PR is opened, a workflow runs to generate a preview of the website and put a link in a PR comment (see [WIP] Module on "Writing functions" #3 for instance)
• all modules must follow a few rules in the YAML:
  • they must have a difficulty field that can take Beginner, Intermediate, or Advanced (this is used on the website to have a dropdown to filter by difficulty)
  • they must have freeze: auto (so that they are rendered in the website repo only when they change)
  • a CI workflow will error if those rules are not respected
• any change in the main branch sends a notification to the website repo

In the website repo

• when the repo receives the notification from the modules repo, it copies the new or modified modules and adds them to the website.
• it then triggers the workflow to build the website. Modules that haven't changed are not re-rendered (thanks to the _freeze folder stored in the repo). New or modified modules are rendered and their _freeze output is pushed to the repo so that they don't get re-rendered if they don't change in the future.
• every week, a workflow runs to delete the _freeze folder and render the website again

[willgearty]

Preview functionality works amazingly, thanks @etiennebacher (and Claude?)!

[etiennebacher]

Yes Claude was quite involved in this module-website interaction

[willgearty]

I've added branch rules to require PRs with two reviews and to ensure that the yaml check passes.