Admins, keys and revocation

Everything in a PowerCloud cluster follows from one secret, the cluster root key, which is never stored in plaintext and is unlocked with your passphrase or your recovery key. Whoever can unlock it is an admin: they can read the storage, issue certificates and unlock nodes. This page covers those secrets, the short-lived admin certificates that day-to-day commands use, and what to do when a laptop, a node or the passphrase itself may be in the wrong hands.

The three admin credentials

Commands that need the cluster's keys take --repo LOC,... (your storage locations, which you can set once in the POWERCLOUD_REPO environment variable) or --via NODE (the storage reached through a node), and ask for the passphrase. Commands that only talk to nodes, such as pcctl status, pcctl node info NAME and the pcctl app commands, use the admin certificate alone. See Storage and pcctl command reference.

The passphrase and the recovery key

pcctl init creates the cluster. It asks for a new passphrase twice and prints a recovery key once:

pcctl --repo LOC,... init

Initialized cluster CLUSTER-ID

RECOVERY KEY (write it down and store it offline — it is shown only once):

    RECOVERY-KEY

Without your passphrase or this recovery key, no one — including you — can read this data.

• Both open the cluster on their own. They unwrap the root key from a key header stored next to your data in every storage location. The header holds no plaintext key; the passphrase is stretched with Argon2id (256 MiB of memory per try), which makes guessing slow.
• Choose a strong passphrase. Anyone who has it and can read the key header from any one of your storage locations holds the whole cluster.
• Store the recovery key offline, on paper or in a safe. It is your way back if the passphrase is lost.
• Nothing else can recover the data. If you lose both the passphrase and the recovery key, there is no backdoor and the data is gone.

For scripts, pcctl reads the passphrase from POWERCLOUD_PASSPHRASE instead of the terminal. To unlock with the recovery key, set POWERCLOUD_RECOVERY_KEY to it; pcctl then uses it in place of the passphrase for that command.

There is no command to change only the passphrase. Setting a new one is part of rotating the cluster keys (below), which also replaces the recovery key. If you have lost the passphrase but kept the recovery key, run pcctl rotate with POWERCLOUD_RECOVERY_KEY set to choose a new passphrase.

Admin certificates

Nodes accept only connections that present a certificate from your cluster's certificate authority (CA). The CA's key is derived from the root key and never stored, so only someone who can unlock the cluster can issue certificates.

Issue yourself an admin certificate on each machine you manage the cluster from:

pcctl --repo LOC,... login

admin certificate for cluster CLUSTER-ID written to /home/you/.config/powercloud (valid 30 days)

pcctl writes admin.key, admin.crt and the cluster's ca.crt to its configuration directory, readable only by you. The directory is $POWERCLOUD_CONFIG if set, otherwise ~/.config/powercloud on Linux and ~/Library/Application Support/powercloud on macOS. It also keeps nodes.json there, the nodes this machine knows by name and address.

• Renewal is automatic while you use it. Any command that unlocks the cluster (with --repo or --via) reissues the saved certificate once it has a week or less left, and says so.
• Once it has expired, commands to nodes fail with admin certificate expired (run 'pcctl --repo ... login'). Run login again.
• Enrolling a node also writes a certificate if this machine has none for the cluster.
• On another admin machine, run login there with the passphrase. To have pcctl on that machine know the nodes by name, run pcctl --repo LOC,... node addr NAME HOST for each one (see Managing nodes).

Roles

Every certificate carries a role, and each node API endpoint checks it. There are two roles: admin, for the people managing the cluster, and node, for the nodes themselves, which use their certificates to talk to each other. (The S3 gateway has a server certificate of its own; see S3 gateway.)

There are no finer-grained roles. Every admin certificate can do everything an admin can, and everyone who knows the passphrase is equally an admin.

Revoke admin certificates

If a laptop holding an admin certificate is lost or stolen, revoke every admin certificate issued so far:

pcctl --repo LOC,... revoke-admins

admin certificates issued before 2026-01-01T12:00:00Z revoked (revocation list serial 3)
  NODE    REVOCATION LIST
  colo-1  serial 3
  home-1  serial 3
new admin certificate written to /home/you/.config/powercloud; other admins must run 'pcctl --repo ... login' again

Every admin certificate issued before that moment stops working, including those of other admins, who then run login again. A revoked certificate gets the error the node refused the admin certificate; if admin certificates were revoked, run 'pcctl --repo ... login'.

Revoking certificates does not help if the passphrase may have leaked too, since anyone with it can issue new certificates. In that case rotate the cluster keys.

Renew node certificates

Node certificates last a year. Only an admin can renew them, because only admins hold the CA's key:

pcctl --repo LOC,... node renew              # every node due within 30 days, or already expired
pcctl --repo LOC,... node renew home-1       # the named nodes, whether due or not

NODE    CERTIFICATE
colo-1  not due, expires 2027-03-02 10:15 UTC
home-1  renewed, expires 2027-10-28 09:00 UTC

A node whose certificate has expired drops out of the mesh, because its peers and pcctl refuse it, but it can still be renewed: the renewal proves the node's identity with its key, which an expired certificate shows as well as a current one.

Watch for expiry in pcctl node info NAME (the certificate line, with a reminder within 30 days) and pcctl status, which lists a node under "needs attention" in its last week. A node also logs a warning daily from 30 days before expiry. A node can't renew itself, so run node renew from a scheduled job or as part of routine admin work.

Remove a node

Remove a machine that is stolen, compromised or retired:

pcctl --repo LOC,... node rm colo-2
pcctl --repo LOC,... node rm --erase colo-2   # wipe it first

node colo-2 (NODE-ID) removed: its certificate is revoked (revocation list serial 4)
  NODE    REVOCATION LIST
  colo-1  serial 4
  home-1  serial 4

What removal does:

• Revokes the node on a cluster-wide revocation list signed by the CA. Revoking the node's ID revokes every certificate for its key, including any issued later. Nodes enforce it in every connection, drop the node from their encrypted mesh, and refuse to add it back.
• Forgets the node in the cluster's registry, so pcctl node unlock can no longer give it its volume key, and in this machine's node list. Peers holding its peer-unlock shares refuse it.
• Publishes a membership list without the node.

--erase first has the node crypto-erase itself: it destroys the only wrapped copies of its volume key, then powers off. Its identity, keys and data are unrecoverable, and at its next boot the disk reads as blank and waits to be enrolled as a new node. Erasing needs the node to be reachable; if it isn't, pcctl removes nothing and says so, and you can run node rm without --erase.

How revocation spreads

pcctl pushes the new list to every node it reaches. A node it can't reach shows not updated (it will get it from a mesh peer). Every mesh message carries the sender's list serial, so a node that was off or cut off fetches the newer list from a peer once it is back, and checks the CA's signature. Nodes refuse a list older than the one they have, so a removal can't be rolled back. A node that finds itself on the list leaves the mesh.

Limits

• A stolen node keeps what is on its own disk. Removal cuts it off from the cluster, and --erase works only while it is reachable. With TPM unlock on, a stolen machine still boots and unlocks its own volume, which holds its own state but never the cluster's keys (see Unlocking nodes after a reboot).
• Two admins removing nodes at the same moment can produce two lists with the same serial. Run node rm again for a node that is missing from the list.

Rotate the cluster keys

If the passphrase or the recovery key may have leaked, revoking certificates isn't enough: whoever holds either can read the store, issue certificates and unlock nodes. pcctl rotate replaces the root key, and with it every key derived from it.

Before you start, stop other admin work on the cluster, and make sure every storage location and as many nodes as possible are reachable. rotate needs --repo (not --via), and every location in it.

pcctl --repo LOC,... rotate

It asks for the current passphrase, then a new one twice (POWERCLOUD_NEW_PASSPHRASE for scripts). The new one must differ from the old. Then:

1. The store is re-encrypted. Every object is read, verified and written again under the new keys. If rotate stops here, nothing has changed yet: run it again.
2. A new key header is written to every location. This is the point of no return: from now on the old passphrase and recovery key open nothing. pcctl prints the new recovery key once; store it offline as before.
3. The old data is deleted from every location.
4. Every reachable node moves to the new CA. Each one rewraps its volume key under the new keys and from then on accepts admins only from the new CA. Your admin certificate is reissued from the new CA.

NODE    KEYS
colo-1  moved to the new keys (CA ...)
home-1  not moved yet: ...

1 node(s) still on the old keys: home-1
Run 'pcctl --repo ... rotate' again once they are up; meanwhile they can't reach the moved nodes.

Nodes that were offline

A node that was off stays on the old keys, and it and the moved nodes can't reach each other until it moves. If it rebooted, pcctl node unlock still unlocks it with the old keys. Then run pcctl --repo LOC,... rotate again, with the new passphrase: it moves whatever is reachable now. Provisioned nodes that never joined are listed as not located yet: their join bundles hold the old keys, so remove them and provision them again (see Enroll a node).

If a node reports NOT MOVED: it presents a certificate from neither CA, someone else holding the old keys may have moved it first. That node is lost to you: remove it and enroll it again.

Finish the rotation

Once every node has moved:

pcctl --repo LOC,... rotate finish

Each node stops accepting the old CA, and the old root key is dropped from the key header. It ends with key rotation finished: the old root key is gone from the key header.

rotate finish refuses while some nodes are still on the old keys. rotate finish --force gives up on them; they must then be removed and enrolled again. A moved node that misses the finish stops accepting the old CA on its own a week after the rotation started.

Limits

• Data already copied stays readable. Rotation stops the old keys from opening what is stored now. Copies an attacker already made, or a provider's backups of old blobs, still open with the old key.
• No other admin work during a rotation. An object written meanwhile under the old keys is deleted.
• It reads and writes all the data. A large store takes a long time and costs egress at every location.
• Finish promptly. Until rotate finish, a moved node still accepts peers with a certificate from the old CA, so someone holding the old keys could pose as a node to it (not as an admin).
• Peer unlock shares are tied to the old CA. Turn peer unlock on again for each node that uses it.

What each measure protects against

For the full threat model, including what providers and hypervisors can and can't see, see Security model.