Repository navigation
✨ feat: generate PEP 770 SBOMs for Python wheels - #2
Draft
gaborbernat wants to merge 33 commits into
Draft
gaborbernat wants to merge 33 commits into
gaborbernat wants to merge 33 commits into
Conversation
Port the CycloneDX SBOM generator from pypa/virtualenv's hatch_build.py into a reusable hatchling plugin, with the project-specific parts (supplier, repository, bundled wheels) moved into hook options and a subclass method for per-wheel properties. The SPDX rendering and the validator ship as the hatch-pep770 spdx and validate commands.
Generate the SBOM from the wheel's METADATA so the hatchling hook and the new embed command share one code path, and expose the hook, the Project settings, the building blocks and the SPDX and validation helpers from the package root. Raise the floor to Python 3.11.
gaborbernat
changed the title
The distribution, import package, console script and hatch hook are now python-sbom (python_sbom for imports), and the build hook class is SbomBuildHook.
gaborbernat
changed the title
Join the build script path as a string so a drive letter survives on Windows, and admit simplint 0.7.1 past the seven-day cutoff so the cooled hook job can install it.
A pure-Python package needs every supported Python on Linux, the oldest and newest on Windows, the newest on macOS, and the tooling once. ty checks every platform in its single job instead.
The committed wheel tripped Scorecard's Binary-Artifacts check and could not be reviewed; a fixture now writes the same small wheel with fixed timestamps, so its hashes stay stable.
The core (collect, model, formats, wheel, validate, cli) no longer imports hatchling; the build hook moves to python_sbom.hatch behind a Backend ABC, with hatchling as the hatch extra. Build facts are gathered once at the edge and passed in. Also fixes five review findings: unique SPDX ids for conditional requirements, distinct refs for bundled wheel variants, marker-aware runtime dependencies, structural validation before deeper checks, and SPDX rendering of versionless roots.
Split settings and build facts, the document model, wheel archives, formats and validation into subpackages that import in one direction, with build_sbom as the only pipeline. A Backend now just returns the request, project content comes from a ContentProvider passed in, and the hatch hook writes the SBOM in finalize from the built wheel, so a later hook can no longer make it stale. Address the first review's correctness and security findings: bundled wheels are checked against their RECORD, embedded entries are the same on every OS and keep the wheel's mode, repeated requirements collapse, other tools' SBOMs get the schema check alone, and bad input raises an error the CLI reports instead of a traceback.
A custom hook loads the shipped hook from the source tree, since the package cannot be a build requirement of itself, and pkg_meta fails when the built wheel lacks .dist-info/sboms/.
Cover the content provider, the backends and the pipeline, the source revision chain, embedding after auditwheel, and why build tools are listed down to their files. Redraw the architecture diagram from the import graph, render type aliases and the TypedDicts, and keep hatchling's lifecycle methods out of the API reference.
Trim .gitignore and CODEOWNERS, run simplint between pyproject-fmt and ruff, point Homepage at the documentation and add the Python modules classifier.
gaborbernat
changed the title
The public API is now the content provider surface the hatchling hook calls, plus the types its signatures reach; embedding, archives, SPDX and validation run through the command line. A provider receives a context with the project root, settings, identity and a reader for the artifact, so one provider serves the wheel and an archive. The new archive command describes a zipapp with the same bundled-wheel identities the wheel SBOM uses. Address the second review: build facts survive egg-info and broken requirements, git is used only for a tracked project without running a parent repository's hooks, an sdist carries its commit, bundled wheels must match every RECORD row, decoy dist-info and duplicate members are refused, and validation reports hostile documents instead of raising.
Work in progress, split across agents from here: - core uses installer, packageurl-python, dep-logic, license-expression and packaging.direct_url; build-system requires them for the self-SBOM - validate caps components at 5,000, raised with --max-components - the sdist revision file is written as bytes, so it holds LF on Windows - bench/ benchmark harness scaffolding
for more information, see https://pre-commit.ci
A fixed project, wheel and build environment must produce the same SBOM and wheel members, so optimisations prove they change nothing. The test wheels are written with Unix attributes, and the pinned ones stored, so the bytes match on every OS and zlib build.
dep-logic rejects extra comparisons other than == and != through assert, which python -O strips, losing requirements that apply, and on import it replaces packaging's marker grammar for the whole build process. The own evaluation returns, and an extra compared as a version now stays unknown instead of raising. A test runs the verdicts under python -O.
Distribution.files stats every recorded file and builds a path object per row; reading RECORD once through installer's parser halves the collection time. A file recorded but missing on disk is now listed on every Python, as 3.11 already did, and a malformed RECORD warns instead of raising.
write_sbom rewrote every member through zlib. zipremove, the backport of the ZipFile.remove and repack that CPython 3.16 adds, drops the old RECORD and SBOM in place, so the other members keep their bytes as the backend wrote them and only the SBOM and RECORD get compressed.
Wheel, bundled, vendored and build-environment metadata now goes through packaging.metadata.parse_email, unvalidated so spec-breaking metadata is still described. A field packaging cannot take, such as a repeated label or single-use field, is read from unparsed as written. The build environment parses each METADATA once instead of twice. Across 7,620 distinct METADATA files the components, licenses and project identity match the old output except one: Keywords come back joined by commas, so spacing around them is no longer kept. A Project-URL without a URL is left out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Projects that ship wheels have no reusable way to put a software bill of materials where PEP 770 tells installers to keep it, in
.dist-info/sboms/. virtualenv carries its own generator, and the other tox-dev projects carry none. python-sbom turns that generator into a library: a hatchling project enables it with one table inpyproject.toml, and a project on any other backend, such as meson-python, runspython-sbom embedon the built wheel. 📦The CycloneDX document records the project, the runtime dependencies it declares, any wheels it bundles, and the build toolchain down to file hashes. Nothing about the build machine goes in, so a rebuild with the same inputs reproduces the wheel byte for byte. The library also renders the document as SPDX 2.3 and validates either format, including an allow-list of licenses.
Architecture
The adapters at the edge, the hatchling hook and the command line, gather what they know and hand it to one pipeline function with explicit inputs: what the backend reports about the wheel, the source tree, the build facts (commit,
SOURCE_DATE_EPOCH, build environment), and a content provider. The model builds the document from those inputs with pure functions and imports nothing else of the package.Three contracts mark the extension points. A build system plugs in as a
Backend, a project adds its own components and properties through aContentProvider, and another version control system supplies the commit as aRevisionReader. The content provider is how virtualenv describes the pip and setuptools wheels it seeds, without copying any internals. Nothing below the adapters imports hatchling, so the command line and the library install without it.flowchart LR hook[hatch hook] --> pipeline cli[python-sbom CLI] --> pipeline collect[collect: settings, build facts, source tree] --> pipeline provider[ContentProvider] --> pipeline pipeline[build_sbom] --> model[model: pure document building] pipeline --> formats[CycloneDX JSON / SPDX 2.3] pipeline --> wheel[wheel: RECORD + archive rewrite] classDef edge fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a; classDef core fill:#d8f3dc,stroke:#2d6a4f,color:#081c15; class hook,cli,collect,provider edge; class pipeline,model,formats,wheel core;The package embeds its own SBOM, so its wheels carry the same record they produce for others.