Skip to main content

ci and iso pipeline

How xfce-linux gets from .bst elements to a bootable, installable live ISO — and how to debug it when it breaks. (Same architecture as tuna-os/xfce-linux; patterns originate in projectbluefin/dakota and dakota-iso.)

Build chain​

elements/** ──► Build xfce-linux (Multi-Runner) ──► ghcr.io/tuna-os/xfce-linux:latest
│ (workflow_run)
▼
Build and Publish xfce-linux Live ISO ──► R2: xfce-linux/xfce-linux-live-*.iso
│
▼ (boot gate: XFCE_LINUX_LIVE_READY on serial + screenshot)
LUKS Install End-to-End Test (PR round-robin / weekly / dispatch)

Multi-runner build (build-multirunner.yml)​

The plan, core, and parallel dependency chunks (steps 1-3) use the shared tuna-os/bst-ci reusable workflow. Each BuildStream desktop repository uses this workflow, with different image names, targets, and chunk counts. On main and stable, the local build_final job (step 4) signs the image digest with a cosign key from the SIGNING_SECRET secret. If that secret has no value, the job skips the sign step and logs a warning. For the key setup and the current state, see signing.md. The keyless instructions in README.md apply only to the live ISOs that build-iso.yml signs.

Free GitHub runners can't hold the whole GNOME+XFCE build, so the workflow splits it:

  1. plan — The scripts/ci-build-matrix.py script from bst-ci runs bst show. It splits uncached elements into a core set (first CORE_SPLIT) and NUM_CHUNKS round-robin chunks. Each chunk has a composite cache key. The job checks out the script from bst-ci at run time. This repository no longer contains a copy.
  2. build_core — builds the bootstrap set, pushes the CAS as ghcr.io/…/cache-xfce-linux-core:latest (zstd tarball via oras).
  3. build_deps (matrix) — each chunk restores core + its own previous CAS, builds, and pushes cache-xfce-linux-<chunk>:{latest,<cache-key>}. The job skips a chunk when GHCR already has its exact cache key.
  4. build_final — merges all chunk CAS tarballs, builds the final target, just export (squash + OCI labels + chunkify), just lint (bootc container lint), pushes latest + date + sha tags (main only).

BuildStream settings CI uses live in the checked-in buildstream-ci.conf.

Cache-key invalidation warning: a change to the cache key of every element causes a full world rebuild. A change to name: in project.conf is one example. Expect chunk jobs to run for hours or reach their six-hour limit once. They recover after they refresh the GHCR caches.

Live ISO (iso.justfile + xfce-linux/)​

just iso-sd-boot xfce-linux (see iso.justfile, imported from Justfile):

  1. just container xfce-linux — 3-stage xfce-linux/Containerfile: ghcr payload (kernel modules) → Debian stage builds a dmsquash-live initramfs (incl. the 95xfce-linux-isofile Ventoy dracut module) → final stage installs flatpaks (src/install-flatpaks.sh) and configures the live env (src/configure-live.sh).
  2. The recipe squashes the payload image and puts it in the squashfs as an OCI layout. /etc/bootc-installer/media-input.json binds fisherman to this layout through the read-only squashfs mount at /run/rootfsbase. This process enables the offline installation.
  3. xfce-linux/src/build-iso.sh assembles a systemd-boot UEFI ISO.

The live session automatically logs in to XFCE on Wayland through xfwl4 as liveuser. It starts org.tunaos.InstallerXfce from the OCI flatpak remote for tuna-os. A symbolic link at /usr/local/bin/fisherman points to fisherman. It uses the shared org.tunaos.Installer.install polkit action. See INSTALLER-FRONTENDS.md in the organization workspace.

LUKS end-to-end test (test-luks-install.yml)​

Local equivalent:

just debug=1 iso-sd-boot xfce-linux # debug=1 enables SSH (liveuser/live)
just luks-test-qemu xfce-linux # boot → fisherman LUKS install → reboot → unlock

xfce-linux/src/luks-unlock.py drives the QEMU monitor. It checks screen dumps until Plymouth appears, types the passphrase with sendkey, and verifies the installed system boot. The workflow publishes screenshots to the ci-screenshots branch and PR comments. They show the live desktop, Plymouth prompt, and installed desktop.

Source updates​

  • Renovate (renovate.json) — GitHub Actions, container tags. Automerge on green CI, majors included.
  • track-bst-sources.yml — Renovate can't parse .bst; this runs bst source track daily. Local elements (elements/xfce-linux, elements/core) go into one automergeable PR; the freedesktop-sdk.bst + gnome-build-meta.bst junctions get a separate review-required PR (a junction bump can rebuild the world). PRs made with the default GITHUB_TOKEN don't trigger CI — set a BOT_TOKEN secret to fix that.

Troubleshooting log (symptom → root cause → fix)​

DateSymptomRoot causeFix
2026-11-12zizmor dangerous-triggers: if the image build gets a pull_request trigger, a fork PR branch named main can start the Live ISO job with its R2 secretsThe workflow_run branches: filter checks only the branch name, not the repository or the eventThe ISO job accepts only successful schedule or workflow_dispatch runs from this repository. test_chained_iso_requires_successful_same_repo_image_build guards it.
2026-10-25Failed Multi-Runner runs immediately launch failing Live ISO builds (#131)workflow_run: completed includes failed and cancelled buildsRequire a successful upstream conclusion at the ISO job boundary; push and manual triggers remain available. This suppresses failure cascades; it does not verify image publication or gate independent install E2E runs.
2026-07-19Every just call in CI fails: "multiple candidate justfiles"justfile + Justfile both at root; just ≥1.30 hard-errorsISO recipes moved to iso.justfile, imported from Justfile
2026-07-19All 10 chunk jobs building for 5+ hproject.conf name: (tromso example) renaming project changes every cache key → world rebuildexpected one-time cost; caches repopulate
2026-07-19Installer flatpak never launched in live sessioninstall-flatpaks.sh baked org.bootcinstaller.Installer while configure-live.sh wired org.xfceinstaller.Installerboth sides now use org.tunaos.InstallerXfce
2026-07-19Multi-runner never went green since May; every run "cancelled" at ~6.5 hchunk jobs killed by job-level timeout-minutes — a cancelled job never reaches the CAS-push step, so 6 h × 10 chunks of build work was discarded daily (≈720 runner-hours; zero chunk cache packages ever existed on GHCR)build bounded inside the step (timeout 270m), push steps if: always() — partial CAS salvaged, builds converge across days
2026-07-19Failed chunks could publish their exact-cache-key tag and be skipped foreverfor i in 1 2 3 … done retry loop exits 0 on total failure (status of last sleep)retry loop removed (bst retry-failed/network-retries already cover it); rc propagated
2026-07-19LUKS e2e / ISO jobs die in seconds: "Unknown attribute group" at Justfile:5Ubuntu 24.04 apt ships just 1.21 (predates [group()]); old runs survived because ancient just silently picked the group-free lowercase justfile we removedworkflows install just via extractions/setup-just and invoke sudo "$(command -v just)"
2026-07-21Superseded push checks and image/ISO builds remained queued for hourshigh-frequency workflows lacked concurrency groups or explicitly queued stale same-ref worksame-ref concurrency groups cancel superseded image publishes; Test and automatic ISO builds now preserve every run (see #140 below)
2026-10-25Merged commits retain cancelled required checks (#140); ISO triggers can also evict each otherRef-based concurrency groups cancel running work and replace pending work during merge trainsTest uses PR numbers only for PRs and unique run IDs otherwise; ISO uses unique run IDs for automatic triggers and branch groups only for manual reruns. Event-scoped cancellation preserves automatic verdicts; test_automatic_checks_and_iso_builds_preserve_every_run guards both policies
2026-09-10build_final dies with no job log and no uploaded build log; :latest last published ~2026-07-19, so every LUKS/Plain E2E run fails with manifest unknownRunner disk exhaustion, not a build error: the check-run annotation on runs 34072176433, 34176106533 and 34298830711 is the runner agent's own System.IO.IOException: No space left on device on its _diag log. Core plus ten chunk CAS tarballs measured ~266 GB compressed on 2026-09-10 (chunks 20-29 GB each) against ~100-110 GB of runner disk, because each chunk tarball embeds a full copy of core, nothing expires (no cache.quota), and oras pull -o . staged each tarball on disk before extracting. Run 34298830711 is decisive: the build itself succeeded in 18 min and the runner died in Export OCI imageStream restores via scripts/cas-restore-stream.sh (oras blob fetch --output -); cache: quota: 60G in buildstream-ci.conf; check the artifact out and delete the CAS before just export (EXPORT_REUSE_CHECKOUT=1); log df -h / around every merge. Invariants in tests/pytest/test_ci_disk_invariants.py
2026-10-25Live ISO builds but signing fails with create bundle file: open : no such file or directory (run 40116760576)cosign 3 defaults to signing-config/bundle mode and ignores detached signature/certificate output flagsExplicitly disable signing-config and new-bundle modes for the existing detached-output contract; require nonempty .sig and .cert before upload. OCI publishing already recovered in merged PR #241 (successful multi-runner run 40051281113). Invariant: test_iso_signing_preserves_detached_outputs_with_cosign3

Add rows to this table as you change CI. See the organization ci-fix-loop skill. The docs/ci-troubleshooting/failure-catalog.md file in tuna-os/tunaos shows the format.

Channels: nightly (main) and stable​

  • main is the nightly trunk: the daily scheduled multi-runner build publishes :latest, :nightly, :nightly-YYYYMMDD, :<sha>; the ISO lands at R2 xfce-linux/.
  • stable is a release bookmark branch. A weekly schedule or manual dispatch starts promote-stable.yml; set force=true for an override. The workflow verifies the newest nightly build and ISO. It force-pushes that commit to stable and starts the stable build. The build creates the :stable and :stable-YYYYMMDD tags, plus an ISO under R2 xfce-linux/stable/. The stable ISO embeds the :stable payload. The build-iso.yml workflow changes payload_ref for each channel.
  • Update PRs from source trackers or Renovate target only main. Stable moves only through promotion.

Release-linked sources​

Local elements use upstream release tags (globs such as v[0-9]*) instead of development branches. Thus, the daily bst source track selects releases. Some content repositories use branches, such as aurora common, docs, wallpapers, and the xfwl4 development repositories. Junctions also use a fixed branch. Do not set track: to one exact tag, because the tracker cannot move it.

Guard rails (what stops a bad commit)​

Branch protection needs the pre-merge checks on main. Automerge starts when they pass. The checks include shellcheck, yamllint, actionlint, BATS, and pytest. They also include the 52-test luks-unlock suite and test_iso_invariants.py. Each invariant represents a defect that reached users.

The BuildStream graph gate runs bst show --deps all on the release target and resolves junctions. Just Parse is also a check. pr-build-changed.yml builds the elements that a PR changes against the core CAS from GHCR. It is not a required check, and it runs only when a PR changes a .bst element.

Post-merge: salvage-enabled nightly world build → ISO boot gate (ready-marker + screenshot artifact) → weekly LUKS install e2e (screenshots on the ci-screenshots branch + PR comments). A cloud routine ("tromso + xfce-linux CI babysitter", every 3 h) diagnoses completed failures from logs and pushes fixes.

Rules that keep this healthy: do not add paths: filters to workflows with required jobs. An absent required check prevents automerge. If you rename a required job, update the branch protection contexts in the same PR. Do not wrap a gate in || echo. Such a wrapper hid failures in bst-validate and pytest for months.

Pull request CI (test.yml)​

All pre-merge checks are in one workflow, test.yml. Its changes job lists the changed files and starts only the jobs that those files can affect. A skipped job reports success, so each required check always reports. If the changes job fails, all checks run. A change to test.yml runs all checks.

JobRuns when the change touches
shellcheck*.sh, *.bash
yamllint*.yml, *.yaml
actionlint.github/workflows/, .github/actions/
Renovate Config Validaterenovate.json
lint .bst files*.bst
BuildStream Grapha file outside docs, tests, scripts, tools, xfce-linux/, .github/ and the justfiles
Unit Testsa file that is not *.md or LICENSE*
Just ParseJustfile, iso.justfile, just/

The plain and LUKS install tests build the same ISO and use different install paths. A change to their shared inputs (xfce-linux/, the justfiles, scripts/lib-install-qemu.sh, scripts/fisherman-install.sh) runs one of the two tests. The run number selects the test. Set the repository variable E2E_ROUND_ROBIN_SIZE to 2 to run both. A change to the install script or workflow of one test always runs that test. The two tests run fully on their weekly schedules and on manual dispatch.

Rollback​

rollback-stable.yml runs only through manual dispatch and defaults dry_run to true. It reverses a promotion. First, it verifies that the target :<sha> image exists. Then, skopeo copy --preserve-digests sets :stable and a dated stable-rollback-* tag.

The workflow force-pushes the stable branch to the same commit so the branch and tag cannot differ. It shares the concurrency group of the promotion workflow, so both cannot run at once. Dakota-pattern note: add a cosign verification step before the retag after cosign support lands.

Serial log permissions in install E2E (#135)​

SymptomCauseFix / invariant
Plain installed boot times out, then tail reports Permission denied (run 39399656661)QEMU runs under sudo and can create root-owned 0600 serial logs; unprivileged grep hides the read error and never sees readiness or emergency markersRead live and plain-installed serial logs with sudo, matching QEMU and the LUKS unlock helper; test_qemu_serial_readers_use_qemu_privileges guards both readiness and diagnostic reads.

The earlier missing :latest publication failure is distinct: merged #147 unblocked the build and #241 restored scheduled publication. The October 20 Plain run successfully built its ISO and booted QEMU before this log-access failure. This fix does not resolve all guest boot failures.