# Admins, keys and revocation > Who can manage a PowerCloud cluster and how, from the passphrase and recovery key to admin certificates, revoking admins, renewing node certificates, removing nodes and rotating the cluster keys after a leak. Everything in a PowerCloud cluster follows from one secret, the cluster root key, which is never stored in plaintext and is unlocked with your passphrase or your recovery key. Whoever can unlock it is an admin: they can read the storage, issue certificates and unlock nodes. This page covers those secrets, the short-lived admin certificates that day-to-day commands use, and what to do when a laptop, a node or the passphrase itself may be in the wrong hands. ## The three admin credentials | Credential | What it opens | Where it lives | Lifetime | | --- | --- | --- | --- | | Passphrase | The cluster root key, and so everything | In your head or your password manager | Until you rotate the cluster keys | | Recovery key | The cluster root key, like the passphrase | Written down and stored offline | Until you rotate the cluster keys | | Admin certificate | The nodes' API: status, apps, reboots, upgrades | pcctl's configuration directory on your machine | 30 days | How the keys hang together: the passphrase and the recovery key each unwrap the root key, and every other key and certificate follows from it. ```mermaid %% alt: The passphrase or the recovery key unwraps the cluster root key, which derives the storage keys, the node volume keys and the cluster CA key; the CA signs admin certificates, node certificates and the membership and revocation lists. flowchart TB subgraph you ["Only you hold"] direction LR pass["Passphrase"] rec["Recovery key"] end you -->|"unwraps"| root["Cluster root key"] root -->|derives| store["Storage keys"] root -->|derives| vol["Node volume keys"] root -->|derives| ca["Cluster CA key"] ca -->|signs| adm["Admin certificates
30 days"] ca -->|signs| nodec["Node certificates
1 year"] ca -->|signs| lists["Membership and
revocation lists"] ``` Commands that need the cluster's keys take `--repo LOC,...` (your storage locations, which you can set once in the `POWERCLOUD_REPO` environment variable) or `--via NODE` (the storage reached through a node), and ask for the passphrase. Commands that only talk to nodes, such as `pcctl status`, `pcctl node info NAME` and the `pcctl app` commands, use the admin certificate alone. See [Storage](https://powercloud.unidatum.com/docs/storage.md) and [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md). ## The passphrase and the recovery key `pcctl init` creates the cluster. It asks for a new passphrase twice and prints a recovery key once: ```sh pcctl --repo LOC,... init ``` ```text Initialized cluster CLUSTER-ID RECOVERY KEY (write it down and store it offline — it is shown only once): RECOVERY-KEY Without your passphrase or this recovery key, no one — including you — can read this data. ``` - **Both open the cluster on their own.** They unwrap the root key from a key header stored next to your data in every storage location. The header holds no plaintext key; the passphrase is stretched with Argon2id (256 MiB of memory per try), which makes guessing slow. - **Choose a strong passphrase.** Anyone who has it and can read the key header from any one of your storage locations holds the whole cluster. - **Store the recovery key offline**, on paper or in a safe. It is your way back if the passphrase is lost. - **Nothing else can recover the data.** If you lose both the passphrase and the recovery key, there is no backdoor and the data is gone. For scripts, pcctl reads the passphrase from `POWERCLOUD_PASSPHRASE` instead of the terminal. To unlock with the recovery key, set `POWERCLOUD_RECOVERY_KEY` to it; pcctl then uses it in place of the passphrase for that command. There is no command to change only the passphrase. Setting a new one is part of rotating the cluster keys (below), which also replaces the recovery key. If you have lost the passphrase but kept the recovery key, run `pcctl rotate` with `POWERCLOUD_RECOVERY_KEY` set to choose a new passphrase. ## Admin certificates Nodes accept only connections that present a certificate from your cluster's certificate authority (CA). The CA's key is derived from the root key and never stored, so only someone who can unlock the cluster can issue certificates. Issue yourself an admin certificate on each machine you manage the cluster from: ```sh pcctl --repo LOC,... login ``` ```text admin certificate for cluster CLUSTER-ID written to /home/you/.config/powercloud (valid 30 days) ``` pcctl writes `admin.key`, `admin.crt` and the cluster's `ca.crt` to its configuration directory, readable only by you. The directory is `$POWERCLOUD_CONFIG` if set, otherwise `~/.config/powercloud` on Linux and `~/Library/Application Support/powercloud` on macOS. It also keeps `nodes.json` there, the nodes this machine knows by name and address. - **Renewal is automatic while you use it.** Any command that unlocks the cluster (with `--repo` or `--via`) reissues the saved certificate once it has a week or less left, and says so. - **Once it has expired**, commands to nodes fail with `admin certificate expired (run 'pcctl --repo ... login')`. Run `login` again. - **Enrolling a node** also writes a certificate if this machine has none for the cluster. - **On another admin machine**, run `login` there with the passphrase. To have pcctl on that machine know the nodes by name, run `pcctl --repo LOC,... node addr NAME HOST` for each one (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). ### Roles Every certificate carries a role, and each node API endpoint checks it. There are two roles: `admin`, for the people managing the cluster, and `node`, for the nodes themselves, which use their certificates to talk to each other. (The S3 gateway has a server certificate of its own; see [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md).) There are no finer-grained roles. Every admin certificate can do everything an admin can, and everyone who knows the passphrase is equally an admin. ## Revoke admin certificates If a laptop holding an admin certificate is lost or stolen, revoke every admin certificate issued so far: ```sh pcctl --repo LOC,... revoke-admins ``` ```text admin certificates issued before 2026-01-01T12:00:00Z revoked (revocation list serial 3) NODE REVOCATION LIST colo-1 serial 3 home-1 serial 3 new admin certificate written to /home/you/.config/powercloud; other admins must run 'pcctl --repo ... login' again ``` Every admin certificate issued before that moment stops working, including those of other admins, who then run `login` again. A revoked certificate gets the error `the node refused the admin certificate; if admin certificates were revoked, run 'pcctl --repo ... login'`. Revoking certificates does not help if the passphrase may have leaked too, since anyone with it can issue new certificates. In that case rotate the cluster keys. ## Renew node certificates Node certificates last a year. Only an admin can renew them, because only admins hold the CA's key: ```sh pcctl --repo LOC,... node renew # every node due within 30 days, or already expired pcctl --repo LOC,... node renew home-1 # the named nodes, whether due or not ``` ```text NODE CERTIFICATE colo-1 not due, expires 2027-03-02 10:15 UTC home-1 renewed, expires 2027-10-28 09:00 UTC ``` A node whose certificate has expired drops out of the mesh, because its peers and pcctl refuse it, but it can still be renewed: the renewal proves the node's identity with its key, which an expired certificate shows as well as a current one. Watch for expiry in `pcctl node info NAME` (the `certificate` line, with a reminder within 30 days) and `pcctl status`, which lists a node under "needs attention" in its last week. A node also logs a warning daily from 30 days before expiry. A node can't renew itself, so run `node renew` from a scheduled job or as part of routine admin work. ## Remove a node Remove a machine that is stolen, compromised or retired: ```sh pcctl --repo LOC,... node rm colo-2 pcctl --repo LOC,... node rm --erase colo-2 # wipe it first ``` ```text node colo-2 (NODE-ID) removed: its certificate is revoked (revocation list serial 4) NODE REVOCATION LIST colo-1 serial 4 home-1 serial 4 ``` What removal does: - **Revokes the node** on a cluster-wide revocation list signed by the CA. Revoking the node's ID revokes every certificate for its key, including any issued later. Nodes enforce it in every connection, drop the node from their encrypted mesh, and refuse to add it back. - **Forgets the node** in the cluster's registry, so `pcctl node unlock` can no longer give it its volume key, and in this machine's node list. Peers holding its peer-unlock shares refuse it. - **Publishes a membership list** without the node. `--erase` first has the node crypto-erase itself: it destroys the only wrapped copies of its volume key, then powers off. Its identity, keys and data are unrecoverable, and at its next boot the disk reads as blank and waits to be enrolled as a new node. Erasing needs the node to be reachable; if it isn't, pcctl removes nothing and says so, and you can run `node rm` without `--erase`. ### How revocation spreads pcctl pushes the new list to every node it reaches. A node it can't reach shows `not updated (it will get it from a mesh peer)`. Every mesh message carries the sender's list serial, so a node that was off or cut off fetches the newer list from a peer once it is back, and checks the CA's signature. Nodes refuse a list older than the one they have, so a removal can't be rolled back. A node that finds itself on the list leaves the mesh. ```mermaid %% alt: pcctl pushes the signed revocation list to every node it reaches; a node that was offline learns of the newer serial from a mesh peer, fetches the list and checks the CA signature. sequenceDiagram participant P as pcctl participant A as colo-1 participant B as home-1 P->>A: signed list, serial 4 P-xB: not reached (offline) Note over B: back online A->>B: mesh message, serial 4 B->>A: fetch the newer list A-->>B: list serial 4 Note over B: checks signature ``` ### Limits - **A stolen node keeps what is on its own disk.** Removal cuts it off from the cluster, and `--erase` works only while it is reachable. With TPM unlock on, a stolen machine still boots and unlocks its own volume, which holds its own state but never the cluster's keys (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). - **Two admins removing nodes at the same moment** can produce two lists with the same serial. Run `node rm` again for a node that is missing from the list. ## Rotate the cluster keys If the passphrase or the recovery key may have leaked, revoking certificates isn't enough: whoever holds either can read the store, issue certificates and unlock nodes. `pcctl rotate` replaces the root key, and with it every key derived from it. Before you start, stop other admin work on the cluster, and make sure every storage location and as many nodes as possible are reachable. `rotate` needs `--repo` (not `--via`), and every location in it. ```sh pcctl --repo LOC,... rotate ``` It asks for the current passphrase, then a new one twice (`POWERCLOUD_NEW_PASSPHRASE` for scripts). The new one must differ from the old. Then: 1. **The store is re-encrypted.** Every object is read, verified and written again under the new keys. If `rotate` stops here, nothing has changed yet: run it again. 2. **A new key header is written to every location.** This is the point of no return: from now on the old passphrase and recovery key open nothing. pcctl prints the new recovery key once; store it offline as before. 3. **The old data is deleted** from every location. 4. **Every reachable node moves to the new CA.** Each one rewraps its volume key under the new keys and from then on accepts admins only from the new CA. Your admin certificate is reissued from the new CA. ```text NODE KEYS colo-1 moved to the new keys (CA ...) home-1 not moved yet: ... 1 node(s) still on the old keys: home-1 Run 'pcctl --repo ... rotate' again once they are up; meanwhile they can't reach the moved nodes. ``` The whole rotation, including nodes that were offline: ```mermaid %% alt: Key rotation re-encrypts the store, writes the new key header (the point of no return), deletes the old data and moves reachable nodes; rotate is run again until every node has moved, then rotate finish drops the old root key. flowchart TB subgraph run ["pcctl rotate"] direction LR s1["1. Re-encrypt
the store"] --> s2["2. New key header:
no way back"] s2 --> s3["3. Delete
old data"] --> s4["4. Move
reachable nodes"] end run --> q{"Every node moved?"} q -->|no| again["Start them,
run rotate again"] again --> q q -->|yes| fin["rotate finish:
old root key dropped"] ``` ### Nodes that were offline A node that was off stays on the old keys, and it and the moved nodes can't reach each other until it moves. If it rebooted, `pcctl node unlock` still unlocks it with the old keys. Then run `pcctl --repo LOC,... rotate` again, with the new passphrase: it moves whatever is reachable now. Provisioned nodes that never joined are listed as `not located yet`: their join bundles hold the old keys, so remove them and provision them again (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). If a node reports `NOT MOVED: it presents a certificate from neither CA`, someone else holding the old keys may have moved it first. That node is lost to you: remove it and enroll it again. ### Finish the rotation Once every node has moved: ```sh pcctl --repo LOC,... rotate finish ``` Each node stops accepting the old CA, and the old root key is dropped from the key header. It ends with `key rotation finished: the old root key is gone from the key header`. `rotate finish` refuses while some nodes are still on the old keys. `rotate finish --force` gives up on them; they must then be removed and enrolled again. A moved node that misses the finish stops accepting the old CA on its own a week after the rotation started. ### Limits - **Data already copied stays readable.** Rotation stops the old keys from opening what is stored now. Copies an attacker already made, or a provider's backups of old blobs, still open with the old key. - **No other admin work during a rotation.** An object written meanwhile under the old keys is deleted. - **It reads and writes all the data.** A large store takes a long time and costs egress at every location. - **Finish promptly.** Until `rotate finish`, a moved node still accepts peers with a certificate from the old CA, so someone holding the old keys could pose as a node to it (not as an admin). - **Peer unlock shares** are tied to the old CA. Turn peer unlock on again for each node that uses it. ## What each measure protects against | Threat | What to do | How far it goes | | --- | --- | --- | | A lost or stolen admin laptop | `pcctl revoke-admins` | Covered, unless the passphrase was on it too. | | A stolen or compromised node | `pcctl node rm` (with `--erase` if it's reachable) | The node is cut off everywhere; it keeps its own local state. | | A leaked passphrase or recovery key | `pcctl rotate`, then `rotate finish` | Covered from now on; data copied before stays readable with the old key. | | A node certificate about to expire | `pcctl node renew` | Covered, also after expiry. | | Losing every passphrase and the recovery key | Nothing | By design, the data is gone. | For the full threat model, including what providers and hypervisors can and can't see, see [Security model](https://powercloud.unidatum.com/security).