Create a way to clean up or prevent unattached packages (e.g. packages with no PRC)

[daviddavis]

Currently, it's possible to have packages that belong to a repo but are effectively orphaned due to some missing structure (Release, ReleaseComponent, or PackageReleaseComponent). These are packages that are effectively orphaned within the repository. When you publish a structured publication for such a repo, these packages can't be installed using clients and don't show up in the pool/ folder.

One common use case that causes this issue is when a release or component is removed from a repository. All of its packages remain and appear in the repo when listing the packages via the Pulp API but they are unreachable by clients.

Ideally, I think pulp_deb would provide some feature to prevent this from happening. One option would be to prevent unattached packages entirely by automatically removing them. Perhaps this could be a method called when finalizing the repo version. I think this is an issue though because for simple publications, unattached packages are a supported use case?

Another possibility I think would be to introduce some sort of option to enable this cleanup behavior. The possible options I can think of are a global setting, a repo setting, or a parameter on the repository modify endpoint.

[quba42]

First, I want to state for the record that your diagnosis is entirely correct and also my apologies for the huge post:

Second, I want to introduce some clearly defined terms to facilitate precise communication:

• ReleaseComponent (RC), PackageReleaseComponent (PRC), and ReleaseArchitecture (RA) are "structure content" types. They are artifact less content types that are placed in a repository version to define how the APT repo published from this repository version should be structured.
• The Release content type is not actually structure content, it just provides a way to store certain metadata values that will be placed within the corresponding published Release file, but will not impact the repo structure. It is "metadata content".
• ReleaseFile, PackageIndex, InstallerFileIndex, and SourceIndex are content types containing upstream repo metadata only used for verbatim publications, and can be ignored for now.
• For berevity I will use "publish" to mean "structured publish", if I want to talk about "simple" or "verbatim" publish (or publication), I will do so explicitly.
• I now want to define what makes a pulp_deb repository version "structurally consistent":
  • The empty repository version is consistent by definition. (Starting with pulp_deb 3.9 we publish some minimal empty default metadata for empty repo versions).
  • Adding a RC to a consistent repo version will result in a new repo version that is still consistent. (APT clients will consume repos containing empty package indices without error or complaint).
  • A Release within a given repo version is structurally consistent if and only if there is also an RC in the repo version with the same value for it's "distribution" field. (otherwise it is a dangling Release content).
  • A RA within a given repo version is structurally consistent if and only if there is also an RC in the repo version with the same value for it's "distribution" field. (otherwise it is a dangling RA content).
  • A PRC within a given repo version is structurally consistent if and only if, both the package and the RC referenced by the PRC are also in the same repo version. (otherwise it is a dangling PRC).
  • A Package within a given repo version is structurally consistent if and only if it is referenced by a PRC that is also in the same repo version and is also structurally consistent within that repo version. (otherwise it is "dangling" within the repo version).
• It follows that any repo version containing any "dangling" content is structurally inconsistent.

Next I want to describe my longterm design goal regarding structure:

1. No pulp_deb workflow should ever result in a structurally inconsistent repo version.
2. Users should not need to understand what makes a pulp_deb repo version consistent.
3. Users should not need to manually manage structure content. They should describe what components, releases/distributions, and architectures they want, and pulp_deb workflows should "make it so".

Continuing, I want to describe some of the history of how we got here, since I think this will inform how to move forward, as well as what to watch out for:

• When we first built pulp3 pulp_deb, the only easy way to get consistent structure content was sync.
• The simple publish was a kind of bandaid, providing an easy way to publish all packages in a structurally inconsistent repo version using a simple default structure. Note that a publication can use both simple=True and structured=True simultaneously.
• If you wanted to create structured repos for uploaded packages, or if you wanted to modify a synced repo, it was up to you to figure out how to, and then manually create things consistently.
• Over time some workflows were added to make it easier to create structurally consistent repo versions:
  • The pulp_deb advanced copy will drag any structure content for any packages being copied from the source repo along for the ride into the target repo.
  • Package upload directly to a repository now automatically creates structure content for the package being uploaded.
  • Most recently you added the option to request structure be respected to the repo modify API.
• Originally pulp_deb users had to explicitly choose which of simple=True and/or structured=True a new publication should use. Since pulp_deb 3.0.0, we have made it simple=False, structured=True by default.
• Katello heavily relied on simple publish (both for uploaded content), but also for Katello specific reasons independent from pulp_deb repo version consistency. (For a long time, Katello had no way to tell consuming clients what repo structure they need to use.) Katello did not ensure pulp_deb repo versions being created are always structurally consistent.
• Katello 4.15 added the option to enable "structured APT" for Katello, and Katello 4.19 made "structured APT" mode mandatory, automatically transitioning all installations during upgrade.
• For backwards compatibility, Katello with structured APT enabled still contains a fallback mechanism (to use simple publish) for old structurally inconsistent repository versions, that were created before the transition.

Finally, what makes sense as a next step, and what about your proposals?

Validating for structural consistency when finalizing repository versions:

• Performance is a concern here. If this ends up adding many minutes to version creation for large repository versions, then that is a very heavy price to pay! I do not know how efficiently we could check for complete structural consistency, this is an empirical question.
• Backwards compatibility is a huge concern. We can't make it so users can no longer build new versions on top of their existing repo versions, just because they contain some structural inconsistency users have been happily working around.
• Providing a way to repair existing inconsistent repo versions is not impossible, but likely very difficult if the solution is anything beyond simply dropping all dangling content from them, which is not a solution for people happily using inconsistent repo versions along with simple publish.
• Providing some way to check a existing repo version for consistency is a completely uncontroversial way to introduce this feature, albeit one with limited impact.
• Providing integrity checking but disabling it by default for all repositories created in the past would be one step up, that still respects backwards compatibility.

Alternative (especially if the performance concern proves to be prohibitive):

• Deprecate and ultimately drop more and more workflows that can result in inconsistent repo versions. Replace them with workflows that either force users to ask for a particular structure behaviour, or default to something sane when users do not provide it.
• So regarding your example "removing an RC from a repo version" might throw an error if there are still things attached to it, or else remove everything attached to it going forward. Checking would happen during this action not on repository version finalize.
• The downside is that we would not catch inconsistent repo versions resulting from bugs in our workflows.
• It might also result in many small breaking changes (rather than one big one).

[daviddavis]

Thank you for the detailed information. This is really helpful and provides a lot of useful context around the problem.

I think this is a feature we'd ultimately like to have as we've had issues with inconsistent structure in the past, but it also looks fairly complex to implement a solution. In the short term, I think we could maybe implement a feature to manually remove content when a release or component is removed. It's not ideal but I think it solves our current need to remove packages when removing releases or components.

I have a PR that would allow to remove packages for releases and components, so my thought is that it could serve as a stop-gap solution for now:

#1524