WITHDRAWN PACKAGES
The problem
A target's gap measurement generates its build order as (desktop closure) minus (what upstream ships), so the moment upstream adopts a package this factory correctly stops rebuilding it. That is only half of reacting to an upstream publish: nothing withdrew the copy already published. The build order shrinks; the served index does not, and the stale rebuild stays indexed at whatever version it was cut at, indefinitely.
That would be harmless if version comparison decided which copy a consumer
installs. It does not. The desktop manifests that enable a target's rebuild
repository do so at priority: 5, and dnf's priority is absolute, not
a tie-break: a package from a higher-priority repository wins even when a
lower-priority one (the upstream the rebuild exists to track) offers a
newer build. So for any name both repositories carry, ours is what
installs -- and when ours is the older build, upstream's fix never reaches
the image. See tunaos-packages#525.
scripts/check-upstream-shadowing.py, wired into upstream-drift.yml,
detects this: it compares a target's served index against its declared
upstream index and reports every name where ours is older.
The remedy is exclusion, not deletion
The obvious remedy -- delete the stale RPM from the live bucket -- is
exactly the shape of mistake #124 / INCIDENT-repo-wipe-gnome
exists to prevent, and scripts/publish-rpm-wave.sh's NEVER SHRINK guard
would refuse a publish whose local tree came out smaller than the one it
synced down -- correctly, since crossing that guard on purpose needs an
audited path the guard cannot tell apart from the bug it exists to catch.
It does not have to cross it. createrepo_c -x PATTERN excludes a file
from the repodata it generates without touching the file:
- the RPM stays in the synced-down tree exactly where it was;
count_rpms(what the NEVER SHRINK guard checks) still counts it, so the guard never observes a withdrawal happening and never has to be told to allow one;- and the served index -- what
check-upstream-shadowing.pymeasures, and what dnf resolves against -- simply stops naming the withdrawn build, so upstream's copy (served from its own repository, carrying no priority) is what a consumer installs.
Removing the manifest entry below un-withdraws the package on the very
next publish: the file never left, so the next createrepo_c run indexes
it back in. No bucket deletion, no new destructive capability, no new
credential, and nothing a contributor runs against production --
publication stays exactly where
CONTRIBUTING.md puts it, in trusted
GitHub environments only.
How to withdraw a package
-
Run
python3 scripts/check-upstream-shadowing.py --target <target>(or read theupstream-shadowing-<target>artifactupstream-drift.ymlalready uploads) to get the exactpackageandserved_evrof what is shadowed. -
Add one entry per
(name, evr)under that target inmanifests/withdrawn-packages.yaml:targets:hummingbird:- name: sudoevr: "1.9.17-1.p2.fc43"reason: "public-hummingbird ships 1.9.17-16.p2.3.hum1 (#525)"Anchored on name and evr on purpose: the gap measurement already stops asking the factory to rebuild a name upstream ships, so an entry here only ever matches the exact stale build that earned it. A future rebuild of the same name at a different evr -- upstream dropping the package, or this factory legitimately catching back up -- is a different glob and is served normally; nothing here can suppress it by accident.
-
Open a PR.
tests/test_withdrawn_packages.pyvalidates the manifest (declared targets are real, no duplicate entries, no glob metacharacters inname/evr);scripts/check-upstream-shadowing.pystops failing for that(name, evr)as soon as the entry is merged -- it is reported aswithdrawn, pending publishinstead of an unactioned failure, so the drift check does not stay red waiting on a decision that has already been made. -
The withdrawal takes effect on the next publish through whichever path reaches that target's prefix. For hummingbird that is both
build-chain-fanout.yml's dedicated publish job andpublish-build-chain-rpms.yml's matrix publisher (thehummingbird-x86_64/hummingbird-aarch64cells inmanifests/package-builds.yamlreach the same prefix the fanout workflow does) -- both plan their excludes from this manifest viascripts/plan-withdrawn-excludes.py, so withdrawal holds regardless of which path a maintainer dispatches next.
No one needs to do anything to make withdrawal happen beyond merging the
manifest entry: publication was already a deliberate, dispatch-only human
action before this (docs/PACKAGE_FACTORY.md's "adding a publisher trigger
that is not a deliberate human dispatch is out of contract"), and this
rides that same dispatch rather than adding a new one.
Removing an entry
Delete it once check-upstream-shadowing.py's served_evr for the package
no longer matches: either the next publish indexed a build no longer older
than upstream's, or the name dropped out of the overlap entirely (upstream
stopped shipping it, and the gap measurement is rebuilding it again).
What this does not cover
- Reverse dependencies are not simulated for a withdrawal. The NEVER
BREAK RDEPS gate in
publish-rpm-wave.shsimulates whether a staged wave breaks an existing package's resolution; it does not simulate removing a name from repodata. In practice this is low risk -- the premise of a withdrawal is that upstream's repository (enabled alongside ours, without priority) still serves the name, so a consumer that needed it resolves it from there instead -- but an exact-versionRequires:on a withdrawn NEVRA from some other still-served package in the same repository would not be caught before publish. The mitigation is the same one that makes withdrawal safe to begin with: it is reversible in one manifest edit and the next publish. - This is per target, not per architecture.
check-upstream-shadowing.pymeasures the architecture implied by a target'sr2_path(x86_64 for hummingbird today). An entry's glob has no architecture in it on purpose, so it excludes the same(name, evr)on every architecture that happens to have published it, but nothing here measures aarch64 separately to confirm there is anything to exclude there.