Unlocking nodes after a reboot

A PowerCloud node keeps its identity, certificates and app data on an encrypted state volume, and after every reboot that volume is locked until something supplies its key. By default an admin unlocks it with pcctl node unlock; you can instead let a node unlock itself with its TPM (under Secure Boot) or with key shares held by its peers. On AMD SEV-SNP confidential VMs, pcctl also checks the chip's attestation before it hands over a key.

How the state volume is protected

The state volume is encrypted with a random volume key, with an HMAC-SHA256 tag on every sector, so a disk changed while the node was off reads as an error, not as altered data. The volume key is stored only wrapped: by a key derived from the cluster's root key, which only someone who can unlock the cluster can compute, and optionally by a TPM slot and a peer slot that you turn on per node. The node never holds the cluster's root key.

The admin unlock always stays available: if TPM or peer unlock can't open the volume, the node waits for an admin.

In the commands on this page, LOC,... is your cluster's storage locations (you can set them once in the POWERCLOUD_REPO environment variable), or use --via NODE to reach the storage through a node. These commands ask for the cluster passphrase. See Storage.

Unlock a node by hand

A locked node prints a message on its console (the serial console, a cloud's console output, or a monitor attached to the machine):

  PowerCloud node home-1 (NODE-ID) is locked: its state volume is encrypted.
  On your admin machine run:
    pcctl --repo ... node unlock home-1 <code>
  unlock code: CODE

Run the command it shows, with the code:

pcctl --repo LOC,... node unlock home-1 CODE

node home-1 unlocked

If the node came back on a different address, give it as a third argument, HOST or HOST:PORT (the node API's port is 7443 by default):

pcctl --repo LOC,... node unlock home-1 CODE 203.0.113.9

To record the new address for later commands, use pcctl --repo LOC,... node addr NAME HOST (see Managing nodes).

What pcctl checks before it sends anything:

• The code pins the node. A locked node has no access to its identity, so it makes a temporary TLS key and shows that key's fingerprint as the unlock code. pcctl refuses a node whose key doesn't match, so a machine in the middle can't pose as the node.
• The IDs come from your registry. The node must report the node ID and volume ID that the cluster's encrypted registry holds for that name. pcctl never trusts the disk's own header, so a forged disk can't obtain another volume's key.
• Attestation, where it applies. See "Attested unlock on confidential VMs" below.

Once unlocked, the node loads its identity and certificates from the volume and goes back to its normal, mutually authenticated API.

Read the code from a console you trust. On a cloud VM, the provider can see the console too.

During a key rotation (see Admins, keys and revocation), a node that hasn't moved to the new keys yet is unlocked with the previous keys, which pcctl tries second. It then says node NAME unlocked with the previous cluster keys (run 'pcctl --repo ... rotate' to move it to the new ones).

Secure Boot with your own key

Secure Boot is the base for TPM unlock, and worth having on its own. The PowerCloud kernel, its built-in initramfs and its fixed command line are one EFI file, so one signature covers everything up to the init system and its services. With your certificate enrolled and Secure Boot on, the firmware boots only images you signed: not a modified PowerCloud on a stolen disk, and not an image swapped by a cloud provider.

1. Create a signing key

Do this once, on a machine you trust:

scripts/sb-keygen.sh ~/powercloud-sb

It writes an RSA-2048 key and certificate:

The script refuses to overwrite an existing db.key. The certificate is valid for 20 years, since firmware doesn't check expiry.

2. Build signed images

The images in a PowerCloud release are not signed with your key, so build your own:

POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image

This signs each EFI image with sbsign (from sbsigntool) and checks the signature with sbverify. The resulting disk images and .efi files are signed. Sign every later upgrade with the same key too, or the firmware refuses it and the node rolls back (see Upgrades and releases).

3. Enroll your certificate

Secure Boot on AWS has run on real AWS instances, x86 and Graviton. The bare-metal and Google Cloud steps haven't been tested yet, and Google Cloud images haven't been booted there at all.

4. Check it

pcctl node info home-1

The boot line reads UEFI, Secure Boot on or UEFI, Secure Boot off, read from the firmware. The node can't change its own boot settings.

Rollback is a limit of Secure Boot itself: an older image signed with your key still boots. To retire old images, use a new key for a new generation of images, and add old images' hashes to the firmware's dbx.

TPM unlock

With Secure Boot on and a TPM 2.0, a node can unlock its state volume at boot by itself, but only on the same machine, booting an image signed with your key.

pcctl --repo LOC,... node tpm home-1 on
pcctl --repo LOC,... node tpm home-1 off

node home-1: TPM unlock on: unlocks at boot while PCR [7] (Secure Boot state) matches

Turning it on sends the node its volume's wrapping key over the admin API. The node checks it opens the volume, and refuses unless Secure Boot is on and the machine has a TPM 2.0. It then seals a new random key to the TPM, checks that it unseals, and adds a TPM slot to the volume's header. At each boot it unseals that key and opens the volume; if the TPM refuses, it waits for an admin unlock as usual.

What is measured

The key is sealed to PCR 7, where the firmware records the Secure Boot configuration: whether Secure Boot is on, the PK, KEK, db and dbx, and which certificate verified the boot. Sealing to it means "this machine, booting anything signed with this key", so upgrades signed with the same key keep working.

Why Secure Boot matters: without it, PCR 7 would vouch for nothing, and any image could ask the TPM for the key. That is why the node refuses to turn TPM unlock on while Secure Boot is off.

pcctl node info NAME shows the tpm unlock line: on: ..., off (TPM 2.0 available) or no TPM.

Good to know

• Nothing is stored in the TPM itself: the sealed key lives in the volume header, so no TPM setup or ownership is needed.
• The key never crosses the bus to the TPM in plaintext, which matters against someone probing a stolen machine.
• The volume header has a backup copy, written first, so a crash while adding or removing a slot can't lose the admin slot.
• The trade-off. With TPM unlock on, a stolen machine boots into a running, enrolled node. Its data is then protected by what the running system enforces (an API that requires cluster certificates, no shell, kernel lockdown), not by a key the thief lacks. Use it on machines in places you trust (your home, a colo cage) or on cloud VMs. Remove a stolen node at once with pcctl node rm (see Admins, keys and revocation).
• Cloud TPMs trust the provider. On AWS the TPM is the NitroTPM, AWS's virtual TPM; Google Cloud's Shielded VMs and Azure's Trusted Launch VMs also have virtual TPMs. A virtual TPM protects against a stolen disk snapshot, not against the provider itself. On AWS, TPM unlock with Secure Boot has run on real instances, unlocking by itself after a stop and start. On Google Cloud it is untested, and on Azure it isn't available, since Secure Boot stays off there.

Peer unlock

Peer unlock lets a node unlock itself after a reboot without a TPM, from key shares its peers hold, as long as it can reach enough of them. A disk taken away from its network stays locked.

pcctl --repo LOC,... node peer-unlock home-1 on                 # a majority of its peers must answer
pcctl --repo LOC,... node peer-unlock home-1 on --threshold 2   # any 2 of them
pcctl --repo LOC,... node peer-unlock home-1 off

node home-1: peer unlock on: unlocks itself after a reboot while 2 of colo-1, aws-1 answer

--threshold K is how many peers must answer. Without it, K is a majority of the holders.

How it works:

• The key is split k of n. The node makes a random key, wraps its volume key under it in a peer slot, and splits that key with Shamir's secret sharing. Each of its peers (every other node in the cluster that isn't revoked and has an address) gets one share over the encrypted mesh, and keeps it on its own encrypted state volume. Then the node forgets the key. Any K shares rebuild it; fewer reveal nothing.
• Asking while locked. A locked node has neither its certificate nor its mesh key, so it asks the holders directly at their API address, checks each holder's certificate against the cluster CA recorded in the slot, and presents a token. It asks again every 10 seconds until K shares open the slot. The admin unlock stays available meanwhile.
• What a holder checks. It gives its share only for that node's volume and token, and never to a node on the cluster's revocation list.

A holder's API address is where pcctl reaches it, or otherwise its mesh endpoint's host on the default API port. If peers must reach a node's API somewhere else, set it before turning peer unlock on:

pcctl --repo LOC,... mesh api colo-1 198.51.100.4:7443

pcctl node info NAME shows peer unlock on: K of HOLDERS while it is on.

Limits

• Network-bound, not tamper-proof. The token is on the disk in the clear, so a disk that reaches K holders unlocks. That is the point: a rebooted node back on its network comes up by itself. A stolen machine still on your network must be removed with pcctl node rm, after which the holders refuse it.
• Shares follow the cluster as it was. The holders, their addresses and the CA certificate are copied when you turn it on. After adding nodes, moving them, or rotating the cluster keys, turn it on again.
• Every holder must be reachable over the mesh when you turn it on, so every share is handed out. There must be at least as many holders as the threshold.

Attested unlock on confidential VMs

On an ordinary VM, the host can read the guest's memory, and with it any key the node holds. On an AMD SEV-SNP VM the CPU encrypts that memory, and the chip signs reports that say so. pcctl checks such a report before it sends a volume key.

It works without any setup:

1. The locked node offers a report along with its unlock code. The report is bound to the node's temporary key, the one the unlock code pins, so another machine can't replay it.
2. pcctl verifies it before sending anything. The report must be signed by the chip's endorsement key, that key's certificate must chain to AMD's root, the guest policy must forbid debugging, and the report must be bound to the key this connection is pinned to. pcctl uses certificates the platform supplies with the report first, and otherwise fetches them from AMD's key distribution service and caches them in its configuration directory, so the admin machine may need to reach AMD's service over HTTPS.
3. Once attested, always attested. After the first verified unlock, pcctl records the requirement in the node's registry entry and refuses from then on to unlock it without a valid report. A disk image copied to an ordinary VM can't get its key.

AMD SEV-SNP verified (...); launch measurement ...
node aws-snp-1 unlocked
node aws-snp-1: SEV-SNP attestation required at every unlock from now on

Set or drop the requirement yourself:

pcctl --repo LOC,... node attest aws-snp-1 sev-snp   # require it before the first unlock
pcctl --repo LOC,... node attest aws-snp-1 off       # the node moved to a machine without SEV-SNP on purpose

Check a running node at any time; it returns a fresh report bound to the node's key and a random value:

pcctl node attest aws-snp-1

pcctl node info NAME shows confidential AMD SEV-SNP on such nodes.

On AWS, SEV-SNP is available on c6a, m6a and r6a instances; this has run by hand on AWS. For enrolling a confidential VM, prefer console enrollment over a join bundle in user data, since the provider can read user data but not the VM's memory (see Enroll a node).

Limits

• What is measured is the cloud's firmware, not PowerCloud. On AWS, the launch measurement covers AWS's UEFI firmware; the kernel it boots is vouched for by Secure Boot with your key, not by the report. pcctl shows the measurement but doesn't pin it.
• Only AMD SEV-SNP. Intel TDX and Arm CCA are planned (see Roadmap).
• AMD's revocation lists aren't checked, and AMD's key distribution service is trusted over HTTPS for the certificate chain.
• Unattended unlock skips it. A node that unlocks itself with its TPM or its peers attests to nobody; use pcctl node attest NAME to check it.

Choosing a method

Whatever you choose, removing a node with pcctl node rm stops it unlocking again: the registry forgets it, and peers refuse it shares.

Limits common to all methods

• No rollback protection for a whole disk. Every sector's tag binds it to its position, but someone with the disk could put back an older copy of the whole disk unnoticed.
• The volume key is in kernel memory while the volume is open, as with any disk encryption. Kernel lockdown blocks the usual ways of reading it; on someone else's VM only confidential computing protects it.
• A node without a state disk keeps its state in RAM, loses it at reboot, and has nothing to unlock.

For what each protection does and doesn't cover, see Security model.