Skip to content

sci: ship the SCI layer as verity-signed systemd-sysext extensions - #401

Draft
fwiesel wants to merge 6 commits into
mainfrom
sysext-virt
Draft

fwiesel wants to merge 6 commits into
mainfrom
sysext-virt

Conversation

@fwiesel

@fwiesel fwiesel commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

SCI as verity-signed systemd-sysext extensions

Draft, based on main. This pull request adds the sysext build, the delivery step with its version gate, dm-verity signing with a boot test, and the CI that proves them. No release, key, or readiness change is claimed.

What it adds

Seven mkosi-built SCI sysext images on the pinned published base:

  • sci-10-base: base identities
  • sci-20-compute: compute
  • sci-21-network: network
  • sci-22-storage: storage
  • sci-40-libvirt: libvirt
  • sci-50-virt-ch: Cloud Hypervisor flavour
  • sci-51-virt-qemu: QEMU flavour

Each image is a standalone EROFS-root GPT raw with version-keyed timestamps and a per-image seed, so builds are byte-reproducible. The numeric prefixes state the merge order that systemd-sysext applies.

The delivery step verifies SHA256SUMS before anything lands, refuses a base-version mismatch through the manifest TARGET_BASE gate, reconciles idempotently, and records pending activation for replay.

Every raw gets a dm-verity hash tree and a signature partition. The committed certificate is the anchor that a merging host checks, and the private key never enters the repository. A three-scenario VM boot test requires the signed set to boot the merged system and requires tampered or unsigned layers to be refused.

CI proves the result on push: the quick suite, the extension build, the full suite with the VM boot test, and one sysext-raws artifact. The integration job consumes that artifact like a test consumer would, exports only the manifest-selected raws over virtiofs, and runs the existing QEMU and Cloud Hypervisor live-migration tests against them.

Test distribution

The test distribution comes from the existing upstream gardenlinux/tests makefile target .build/dist.tar.gz. This pull request adds no second builder:

make -C gardenlinux/tests -f util/build.makefile .build/dist.tar.gz
install -d .build/sysext-runtime
cp gardenlinux/tests/.build/dist.tar.gz .build/sysext-runtime/dist.tar.gz

One make invocation produces both recipes (building python runtime, building test framework distribution). The cp hands the archive to the suite and to the delivery step before the full suite starts. No cache step is involved.

Base

This change now sits directly on main; the wait-cap change it used to stack on was squash-merged as 902e315.

Commits

  • 0a5db6a feat: sysext build engine, seven extensions, byte-reproducible builds
  • ced5b89 feat: sysext delivery step with version gate
  • 4dd15d8 feat: dm-verity signing and boot test
  • 44586db ci: prove the sysext artifacts on GitHub
  • 8350140 refactor: one hypervisor preparation pipeline — the pipeline generates either variant into HV_DIR/{HV1.xml,HV2.xml,metadata.env}, and setup consumes only hv_dir
  • 955dad1 docs: document the sysext build and its delivery step

Against main (902e315): 81 files, +6485/−122.

Local evidence

Full suite on this exact tree 5378c3c (955dad1), no filter and no deselection:

  • 130 passed in 179.28s, exit 0. JUnit tests=130, failures=0, errors=0, skipped=0. Composition by marker: 118 quick, 11 build, 1 VM. The collected count rose from 120 with the five workflow tests that the refactor adds and the five variant cases added here.
  • All three VM scenarios passed (signed, tamper, policy); the one collected vm test runs them. The tamper console carries device-mapper: verity: 259:2: data block 1 is corrupted.
  • Reproducibility: test_reproducible passes in the same run; 7 images with build1 == build2.
  • Archive sha256:6c2fc8c02d51653f7acf7ccbfc9fc68f2e62fad0610593871dc4219afe16de59, built earlier by the cold upstream-make/Podman path. This hash is LOCAL only.

CI

Local full suite at tree 5378c3c (955dad1): 130 passed in 179.28s, exit 0, JUnit tests=130, failures=0, errors=0, skipped=0, no marker filter and no deselection; three VM scenarios green, tamper console device-mapper: verity: 259:2: data block 1 is corrupted.

Observed from gh run list --branch sysext-virt --limit 5 --json databaseId,event,status,conclusion,headSha at 2026-11-11T16:20Z, after the push:

  • Push 43056806812 at this tip: failure — quick tier success, then Build the extensions failed with Temporary failure resolving 'security.ubuntu.com' / 'archive.ubuntu.com' and E: Unable to locate package base-files at 16:19:38Z, exit 100; integration and publish skipped.
  • Pull-request 43056818644 at this tip: success, quick tier only; the PR guard skips the producer, integration and publish.
  • dev 43056819742 at this tip: failure — Build and cache test distribution failed on Temporary failure resolving 'azure.archive.ubuntu.com', exit 100; Test PR image skipped.

At the previous tip 7738fcd (tree 20d65de), push 43050828563 and dev 43050845141 failed on runner DNS (security.ubuntu.com, azure.archive.ubuntu.com), and pull-request 43050844157 was quick-tier success. Those are old-tree runs only.

Prior-tree green, not this tree: push 43015191117 at 0cf70f6 (tree 370994e) had 120 passed in 221.16s with both migrations successful, and it ran the same sysext guard jq filter on the real base, so the systemd-sysext status --json shape is proven there.

The tip commit 955dad1 touches only delivery/README.md and docs/sysext-virt.md. No CI run at this tree has reported the producer's guest-distribution step, the full suite in CI, either migration, the sysext-raws artifact, or publish.

Limits

  • Two overlaps exist in the built set: libparted ships in sci-10-base and sci-40-libvirt, and systemd-container ships in both virt flavours. The copies are byte-identical, so the merge does not conflict, but only the libvirt-to-virt overlap is asserted at build time; the other pairs are not.
  • Production signing-key custody and rotation, and a host anchor that the base image carries, remain unproven; this work is tracked in gardenlinux/gardenlinux-sci#402.
  • Release execution, policy, and the base re-pin remain unproven open work.
  • The guest mounts /run/extensions read-only over virtiofs, but the host export is not read-only.
  • The pre-marker activation interruption window is documented, not fixed.

Status: draft at 955dad1 (tree 5378c3c), 6 commits, local full suite 130/130 at this tree. CI after the push: pull-request quick tier success, push producer and dev both failed on runner DNS, so the integration stage is unverified in CI.

@fwiesel
fwiesel force-pushed the sysext-virt branch 19 times, most recently from 2cf3634 to 00a1df9 Compare November 3, 2026 13:01
@fwiesel
fwiesel force-pushed the sysext-virt branch 7 times, most recently from 1824943 to 819dade Compare November 10, 2026 17:13
@fwiesel
fwiesel changed the base branch from main to fix/migration-windows November 11, 2026 11:48
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from 6663e50 to 3be7ac7 Compare November 11, 2026 13:17
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from 3be7ac7 to a219dad Compare November 11, 2026 13:21
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from a219dad to 1fdb750 Compare November 11, 2026 13:27
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from 1fdb750 to e8b2292 Compare November 11, 2026 13:30
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from e8b2292 to 3118048 Compare November 11, 2026 13:35
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from 3118048 to 2f9b972 Compare November 11, 2026 13:38
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from 2f9b972 to c193230 Compare November 11, 2026 14:03
@fwiesel
fwiesel force-pushed the fix/migration-windows branch from c193230 to fe830a8 Compare November 11, 2026 14:05
Base automatically changed from fix/migration-windows to main November 11, 2026 14:41
Seven SCI extensions ship as standalone EROFS GPT images on the pinned
published base, so a package change rebuilds only its own extension and
the base image stops churning for every host. Builds are byte-reproducible
from version-keyed timestamps, per-image seeds and three conscious-bump
pins, and the tools tree is accepted only when its manifest digest matches
the committed pin.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
A delivery is safe only when its raws are intact and were built for the
host's base version. The step verifies SHA256SUMS, enforces the manifest's
TARGET_BASE gate, reconciles idempotently and records pending activation
for replay. Fetch mode verifies the detached inventory signature and
downloads only files whose staged hash differs.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
An unverified layer widens the boot root's trust boundary. Every raw gets a
dm-verity hash tree and a signature partition, anchored in the committed
certificate; the private key never enters the repository. A three-scenario
boot test proves the signed set boots and that tampered or unsigned layers
are refused with the kernel's dm-verity corruption verdict.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
Local-only proof dies with the machine that ran it. On push the workflow
builds the extensions, runs the full suite with the VM boot test and
uploads the raws as a plain-file artifact; the quick suite runs on every
change.

The test distribution comes from the upstream tests build.makefile, so one
recipe replaces the local Docker orchestration and the archive cannot drift
from the upstream builder.

The integration job consumes a prior run's artifact like a test consumer:
it boots two Garden Linux hypervisors from a read-only virtiofs share,
activates the seven extensions and reruns the existing live migrations.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
@fwiesel
fwiesel force-pushed the sysext-virt branch 4 times, most recently from 582b745 to 16a58e4 Compare November 11, 2026 15:18
The integration test carried two exclusive branches: the image_tag path
prepared the HyperVisor pair inline, the sysext path consumed a prepared
directory, and the guest configuration was written twice. The pipeline now
generates either variant into one directory, and setup consumes only that,
so the guest configuration has one home and a missing piece fails in one
place. The test mode input disappears with the last caller that needed it.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
Record the design and the delivery contract while they are current, so the
docs stay the single source of truth: docs/sysext-virt.md states the
invariants and open items, delivery/README.md the operational contract and
the product boundary.

Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant