Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Upgrade notes

- **KMS state moves to `/var/lib/kms`.** The root keys of every epoch move out
of `cert_dir` into the new `core.keyring_dir` (default
`/var/lib/kms/keyring`), as `epochs/<n>/` with `root-epoch` marking the
newest. `cert_dir` and the default listener TLS paths move from
`/etc/kms/certs` to `/var/lib/kms/certs`. A deployment that persists
`/etc/kms/certs` and relies on the defaults must persist `/var/lib/kms` or
set both paths. The on-disk layout is not read from older versions, so
upgrade by onboarding a new KMS, as before.

### Changed

- KMS: each root epoch keeps the attestation produced when its keys were
generated (`epochs/<n>/root-attestation`), and key handover carries it in
`KmsKeys.attestation`, so joined and rotated nodes keep the evidence of every
epoch. The receiver archives it without verifying it; relying parties verify
it. `bootstrap-info.json` is gone: `GetMeta.bootstrap_info` reports the served
epoch's attestation. Auto-bootstrap now records the attestation too (#539).

## [0.6.5] - 2026-11-15

### Upgrade notes
Expand Down
6 changes: 3 additions & 3 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,8 +522,8 @@ port = 8001
auth_token = "<source-admin-token>"

[listeners.default.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
key = "/var/lib/kms/certs/rpc.key"
certs = "/var/lib/kms/certs/rpc.crt"
# No `mutual` section: client certificates are verified by their attestation.
```

Expand Down Expand Up @@ -577,7 +577,7 @@ governance made current. See the [rotation design](./kms-emergency-key-rotation-
for the full procedure; the KMS steps are:

1. If epochs were never registered, publish the attestation the KMS produced at
bootstrap (kept in `bootstrap-info.json` on the bootstrapped node) and
bootstrap (served as `bootstrap_info` by `KMS.GetMeta`) and
register the current root public keys as epoch 0:
`DstackKms.registerRootEpoch(caPubkey, k256Pubkey, evidenceUrl)`, via
`script/Manage.s.sol:RegisterRootEpoch`, or a `rootEpochs` entry in the
Expand Down
6 changes: 4 additions & 2 deletions docs/kms-emergency-key-rotation-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,10 @@ latest two. Keeping old roots does not authorize issuing old derived keys. There
is no local epoch configuration: a KMS serves the current epoch on chain,
provided its keyring holds it and reproduces its chain hash, and waits
otherwise. When the current epoch moves, the KMS restarts to follow it and
re-signs its RPC certificate under the new root CA. A `root-epoch` marker records
the newest stored epoch.
re-signs its RPC certificate under the new root CA. Each epoch is stored under
`epochs/<n>/` with the attestation produced when its keys were generated, and
onboarding hands both over; a `root-epoch` marker records the newest stored
epoch.

Historical epochs are not time-limited. Their roots are already exposed, so
withholding derived keys from authorized CVMs would not reduce what an attacker
Expand Down
2 changes: 1 addition & 1 deletion docs/security/security-best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ The VMM RPC is a management interface for trusted operators and front-end servic

## Keep private material owner-only

Secret-bearing files should be owner-only (`0600`) wherever possible, including app keys, decrypted env files, KMS root keys, gateway WireGuard/TLS keys, and ACME credentials. Preserve restrictive permissions when copying volumes, backing up `/etc/kms/certs`, or moving gateway and certbot state between hosts. Public issue [#606](https://github.com/Dstack-TEE/dstack/issues/606) tracks the remaining low-cost hardening work in dstack-managed file writes.
Secret-bearing files should be owner-only (`0600`) wherever possible, including app keys, decrypted env files, KMS root keys, gateway WireGuard/TLS keys, and ACME credentials. Preserve restrictive permissions when copying volumes, backing up the KMS `keyring_dir`, or moving gateway and certbot state between hosts. Public issue [#606](https://github.com/Dstack-TEE/dstack/issues/606) tracks the remaining low-cost hardening work in dstack-managed file writes.

## docker logs is public available by default

Expand Down
29 changes: 16 additions & 13 deletions docs/tutorials/kms-build-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,8 @@ log_level = "info"

# TLS certificate for the listeners (not the plain-HTTP onboard page)
[listeners.default.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
key = "/var/lib/kms/certs/rpc.key"
certs = "/var/lib/kms/certs/rpc.crt"

# KMS RPCs for CVMs
[listeners.guest]
Expand All @@ -187,7 +187,8 @@ port = 9100

# Core KMS Configuration
[core]
cert_dir = "/etc/kms/certs"
cert_dir = "/var/lib/kms/certs"
keyring_dir = "/var/lib/kms/keyring"
subject_postfix = ".dstack"
# Intel PCCS URL for TDX quote verification
pccs_url = "https://pccs.phala.network/sgx/certification/v4"
Expand Down Expand Up @@ -231,6 +232,7 @@ EOF
| `[listeners.admin]` | `address`, `port`, `auth_token_hash` | Admin listener (9101), which serves the key handover; the SHA-256 of the admin token. A serving KMS requires it with TLS |
| `[listeners.onboard]` | `address`, `port` | Plain-HTTP onboarding page (9100, before the KMS has keys) |
| `[core]` | `cert_dir` | Directory for certificates |
| `[core]` | `keyring_dir` | Directory for the root keys of every epoch (must be persistent) |
| `[core]` | `pccs_url` | PCCS endpoint for quote verification |
| `[core.auth_api]` | `url` | auth-eth webhook service URL |
| `[core.onboard]` | `enabled` | Enable bootstrap/onboard mode |
Expand Down Expand Up @@ -356,7 +358,7 @@ RUN apt-get update && \
COPY --from=bun /usr/local/bin/bun /usr/local/bin/bun

# Create directories
RUN mkdir -p /etc/kms/certs /etc/kms/images /var/run/kms /var/log/kms
RUN mkdir -p /etc/kms/images /var/lib/kms/certs /var/lib/kms/keyring /var/run/kms /var/log/kms

# Copy KMS binary
COPY dstack-kms /usr/local/bin/dstack-kms
Expand Down Expand Up @@ -417,8 +419,8 @@ log_level = "info"

# TLS certificate for the listeners (not the plain-HTTP onboard page)
[listeners.default.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
key = "/var/lib/kms/certs/rpc.key"
certs = "/var/lib/kms/certs/rpc.crt"

# KMS RPCs for CVMs
[listeners.guest]
Expand All @@ -433,7 +435,8 @@ port = 9100

# Core KMS Configuration
[core]
cert_dir = "/etc/kms/certs"
cert_dir = "/var/lib/kms/certs"
keyring_dir = "/var/lib/kms/keyring"
subject_postfix = ".dstack"
pccs_url = "https://pccs.phala.network/sgx/certification/v4"

Expand All @@ -459,12 +462,12 @@ port = 9100
# Onboarding Configuration
[core.onboard]
enabled = true
# Empty domain = manual bootstrap mode (ensures bootstrap-info.json is written)
# Empty domain = manual bootstrap mode
auto_bootstrap_domain = ""
EOF
```

> **Why empty `auto_bootstrap_domain`?** With an empty domain, KMS starts in "onboard mode" — a plain HTTP server that waits for you to trigger bootstrap via an RPC call. This ensures `bootstrap-info.json` is written to disk, which is required for on-chain KMS registration. You'll provide the domain during the bootstrap step in [KMS CVM Deployment](/tutorial/kms-cvm-deployment).
> **Why empty `auto_bootstrap_domain`?** With an empty domain, KMS starts in "onboard mode" — a plain HTTP server that waits for you to trigger bootstrap via an RPC call. Its response carries the attestation and public keys required for on-chain KMS registration. You'll provide the domain during the bootstrap step in [KMS CVM Deployment](/tutorial/kms-cvm-deployment).

### Copy build artifacts and configuration

Expand Down Expand Up @@ -542,15 +545,15 @@ services:
volumes:
# Mount config file from local directory
- ./kms.toml:/etc/kms/kms.toml:ro
# Named volume for persistent certificates
- kms-certs:/etc/kms/certs
# Named volume for persistent state: certificates and root keys
- kms-data:/var/lib/kms
environment:
- RUST_LOG=info
restart: unless-stopped

volumes:
kms-certs:
# Certificates persist across container restarts
kms-data:
# Certificates and root keys persist across container restarts
EOF
```

Expand Down
37 changes: 22 additions & 15 deletions docs/tutorials/kms-cvm-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,16 +65,22 @@ When you deploy KMS as a CVM, the following happens:

### Generated Artifacts

Inside the CVM at `/etc/kms/certs/`:
Inside the CVM, in the `kms-data` volume at `/var/lib/kms/`. The root keyring under `keyring/` (`keyring_dir`):

| File | Purpose |
|------|---------|
| `epochs/<n>/root-ca.crt` | Root Certificate Authority of epoch `n` (self-signed) |
| `epochs/<n>/root-ca.key` | Root CA signing key of epoch `n` (P256 ECDSA) |
| `epochs/<n>/root-k256.key` | Ethereum signing key of epoch `n` (secp256k1) |
| `epochs/<n>/root-attestation` | Attestation produced when the epoch `n` keys were generated; absent if there is none |
| `root-epoch` | Newest root epoch held |

And the RPC certificates under `certs/` (`cert_dir`):

| File | Purpose |
|------|---------|
| `root-ca.crt` | Root Certificate Authority (self-signed) |
| `root-ca.key` | Root CA signing key (P256 ECDSA) |
| `rpc.crt` | TLS certificate for RPC server |
| `rpc.key` | RPC server private key |
| `root-k256.key` | Ethereum signing key (secp256k1) |
| `bootstrap-info.json` | Public keys and TDX attestation quote |

---

Expand Down Expand Up @@ -138,7 +144,7 @@ services:
- "9101:9101"
volumes:
- /var/run/dstack.sock:/var/run/dstack.sock
- kms-certs:/etc/kms/certs
- kms-data:/var/lib/kms
environment:
- RUST_LOG=info
- KMS_DOMAIN=kms.yourdomain.com
Expand All @@ -151,14 +157,14 @@ services:
restart: unless-stopped

volumes:
kms-certs:
kms-data:

configs:
kms_config:
content: |
[listeners.default.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
key = "/var/lib/kms/certs/rpc.key"
certs = "/var/lib/kms/certs/rpc.crt"

[listeners.guest]
address = "0.0.0.0"
Expand All @@ -183,7 +189,8 @@ configs:
# certificate signing verifies CSR signature and attestation.

[core]
cert_dir = "/etc/kms/certs"
cert_dir = "/var/lib/kms/certs"
keyring_dir = "/var/lib/kms/keyring"
pccs_url = "https://pccs.phala.network/sgx/certification/v4"

[core.image]
Expand Down Expand Up @@ -234,7 +241,7 @@ This docker-compose uses a Docker `configs` section to inject a complete `kms.to
| `[core.auth_api]` | Authentication via auth-eth webhook on localhost:9200 |
| `[core.onboard]` | Bootstrap settings — `auto_bootstrap_domain` is empty so KMS enters onboard mode for manual bootstrap |

> **Why manual bootstrap?** With `auto_bootstrap_domain` left empty, KMS starts in "onboard mode" — a plain HTTP server on port 9100 that waits for you to trigger bootstrap via an RPC call. This ensures `bootstrap-info.json` (containing the TDX attestation quote and public keys) is written to disk. You'll need this file later to register the KMS on-chain.
> **Why manual bootstrap?** With `auto_bootstrap_domain` left empty, KMS starts in "onboard mode" — a plain HTTP server on port 9100 that waits for you to trigger bootstrap via an RPC call. The response carries the TDX attestation quote and public keys you'll need later to register the KMS on-chain.

**Environment variables explained:**

Expand Down Expand Up @@ -356,7 +363,7 @@ Onboarding

### Step 6: Bootstrap KMS

With KMS in onboard mode, trigger key generation by calling the Bootstrap RPC endpoint. This generates root keys, a TDX attestation quote, and writes `bootstrap-info.json`:
With KMS in onboard mode, trigger key generation by calling the Bootstrap RPC endpoint. This generates root keys and a TDX attestation quote over them:

```bash
# Inspect the KMS measurement before bootstrap
Expand Down Expand Up @@ -499,7 +506,7 @@ The TDX quote can be verified by:
│ │ │ │ └────┬────┘ └──────┬───────┘ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ ▼ ▼ │ │ │ │
│ │ │ │ /etc/kms/certs Ethereum RPC │ │ │ │
│ │ │ │ /var/lib/kms Ethereum RPC │ │ │ │
│ │ │ └──────────────────────────────────┘ │ │ │
│ │ │ │ │ │
│ │ │ guest-agent (/var/run/dstack.sock) │ │ │
Expand Down Expand Up @@ -544,9 +551,9 @@ For detailed solutions, see the [KMS Deployment Troubleshooting Guide](/tutorial

### Understanding Storage

CVM certificates are stored in a Docker named volume (`kms-certs`). This provides:
CVM certificates and root keys are stored in the Docker named volume `kms-data`, while `/etc/kms` holds only configuration. This provides:

- **Container restart persistence** - Certificates survive container restarts
- **Container restart persistence** - Certificates and root keys survive container restarts
- **CVM restart consideration** - Depending on VMM configuration, volumes may or may not persist

### Backup Recommendations
Expand Down
1 change: 1 addition & 0 deletions dstack/kms/dstack-app/compose-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ configs:

[core]
cert_dir = "/kms/certs"
keyring_dir = "/kms/keyring"

[core.image]
cache_dir = "/kms/images"
Expand Down
1 change: 1 addition & 0 deletions dstack/kms/dstack-app/compose-simple.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ configs:

[core]
cert_dir = "/kms/certs"
keyring_dir = "/kms/keyring"

[core.image]
cache_dir = "/kms/images"
Expand Down
9 changes: 6 additions & 3 deletions dstack/kms/kms.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ proxy_protocol = false
# No `mutual` section: client certificates are verified by the attestation they
# carry, not by an issuer CA (see ra-rpc's ratls_client_verifier).
[listeners.default.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
key = "/var/lib/kms/certs/rpc.key"
certs = "/var/lib/kms/certs/rpc.crt"

# For CVMs: all KMS RPCs, including those that release keys.
[listeners.guest]
Expand Down Expand Up @@ -68,7 +68,10 @@ address = "0.0.0.0"
port = 8000

[core]
cert_dir = "/etc/kms/certs"
cert_dir = "/var/lib/kms/certs"
# Root keys of every epoch, the only state the KMS cannot regenerate. Keep it on
# persistent storage.
keyring_dir = "/var/lib/kms/keyring"
subject_postfix = ".dstack"
site_name = ""
# Whether trusted RPCs require the KMS to first attest itself to its own
Expand Down
3 changes: 3 additions & 0 deletions dstack/kms/rpc/proto/kms_rpc.proto
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ message GetKmsKeyRequest {
message KmsKeys {
string ca_key = 1;
bytes k256_key = 2;
// Attestation produced when this epoch's keys were generated, as registered
// with governance; empty if none. Archived by the receiver, not verified.
bytes attestation = 3;
}

message KmsKeyResponse {
Expand Down
Loading
Loading