Skip to content

[heft] Add shard result contracts and output ownership - #6205

Open
Yashwanth Reddy Verupaka (yashwanth195) wants to merge 3 commits into
microsoft:mainfrom
yashwanth195:user/yverupaka/heft-shard-contracts
Open

Yashwanth Reddy Verupaka (yashwanth195) wants to merge 3 commits into
microsoft:mainfrom
yashwanth195:user/yverupaka/heft-shard-contracts

Conversation

@yashwanth195

Copy link
Copy Markdown
Contributor

Summary

Add reusable Heft contracts for workloads that need to publish isolated shard results safely. This standalone
foundation provides identities, versioned manifests, atomic publication and exclusive output claims; it does
not enable managed sharding or change existing task execution.

Details

  • Strict 1-based shard parsing and count-aware temp paths; schema-validated manifest reads/writes distinguish
    missing results from corrupt ones and leave workload payloads opaque.
  • Canonical path validation, alias/overlap checks, short-lived multi-output registration and attempt-fenced
    publication preserve exclusive ownership through cancellation and cleanup. Abandoned claims require verified
    manual recovery, never PID/age-based stealing.
  • Add focused regression suites, the manifest type exports/API report, a minor changefile and contract docs.

Ported only the generic foundation from 6af2a03e5461da3e2ac8bd91ce2d0f5fd91310c9 onto upstream main, with
additional standalone regressions and a trailing-newline parser correction. No unmerged PR is a prerequisite.
Collation/coherence validation, adapter registration, CLI mode wiring, Jest execution/coverage/JUnit and Rush
identity/cache integration remain separate. Callers must keep outputs disjoint from source and payload inputs;
these cooperative filesystem helpers are not a sandbox. Existing unsharded behavior is unchanged.

How it was tested

Validated this standalone port on Windows with Node 24.22.1 and the repository's Rush 5.182.0/Heft/Jest setup.
The integrated source's earlier validation is not used as evidence for this PR.

Command Result
node common\scripts\install-run-rush.js install --to @rushstack/heft Passed; locked scoped install after the initial build reported missing dependencies.
node common\scripts\install-run-rush.js build --to @rushstack/heft Prerequisites passed; Heft generated the new API report with the expected update warning, resolved by the production build below.
node common\scripts\install-run-rush.js build --only @rushstack/heft --production Passed: TypeScript, lint, schema copy and API report validation.
From apps\heft: node ..\..\common\scripts\install-run-rushx.js _phase:test --test-path-pattern sharding --disable-code-coverage Passed: four suites, 100 tests passed, two Windows 8.3-alias cases skipped because this volume exposes no short-path alias.
node common\scripts\install-run-rush.js prettier Passed; staged files formatted.
node common\scripts\install-run-rush.js change --verify --target-branch upstream/main Passed, including after commit.
git diff --cached --check Passed before commit.

Coverage includes identity/schema domains, invalid/corrupt manifests, cancellation boundaries, atomic failure
cleanup, symlinked roots versus nested junctions, filesystem name equivalence, overlapping/concurrent claims,
partial acquisition cleanup, replaced/hardlinked claims and abandoned-gate diagnostics. No POSIX run or broad
Jest adapter/coverage/collator/acceptance sweep was performed.

Impacted documentation

Added docs/heft/shard-contracts.md for implemented guarantees and caller obligations. No existing website
behavior changes. The new schema describes generated result files, so no init template applies.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

1 participant