# Troubleshooting
> Common PowerCloud problems by symptom - pcctl errors about passphrases, storage locations, certificates and key rotation, nodes that won't boot, join, unlock or mesh - with the cause and the fix for each.
This page lists problems you may meet with PowerCloud, organised by what you see: an error from `pcctl`, a message on a node's console, or a warning in `pcctl status`. Each entry gives the likely cause and the fix. Error messages are quoted as `pcctl` and `pcd` print them, with names and IDs replaced by placeholders such as `NAME`.
A good first step for any node problem is `pcctl status`, which lists every node with a "needs attention" section, and `pcctl node info NAME` for one node in detail.
To find the right section quickly:
```mermaid
%% alt: Triage: an error from pcctl points to the passphrase or certificate sections; in pcctl status, an unreachable node leads to the locked, booting or reaching-nodes sections depending on its console, and a mesh warning leads to the mesh section.
flowchart TB
start["Something is wrong"] --> q1{"Did pcctl print
an error?"}
q1 -->|"passphrase, storage"| pr["Passphrase and repository"]
q1 -->|"certificate"| cert["Admin or node certificates"]
q1 -->|no| q2{"pcctl status:
what needs attention?"}
q2 -->|"unreachable"| q3{"Node's console shows?"}
q3 -->|"unlock code"| lock["Node locked after a reboot"]
q3 -->|"no boot, errors"| boot["Booting and the state disk"]
q3 -->|"running"| reach["Reaching nodes and the mesh"]
q2 -->|"reaches N of M peers"| reach
```
## Passphrase and repository
### "keys: wrong passphrase or recovery key"
**Cause.** The passphrase (or `POWERCLOUD_PASSPHRASE`, or `POWERCLOUD_RECOVERY_KEY`) doesn't open the key header. After a key rotation (`pcctl rotate`), only the new passphrase and the new recovery key work.
**Fix.** Check for a stale `POWERCLOUD_PASSPHRASE` in your environment. If the passphrase is lost, unlock with the recovery key (`POWERCLOUD_RECOVERY_KEY=... pcctl --repo LOC,... ...`) and then rotate to a new passphrase. If both are lost, the data can't be recovered: there is no backdoor.
### "no terminal for passphrase; set POWERCLOUD_PASSPHRASE (and POWERCLOUD_NEW_PASSPHRASE for rotate)"
**Cause.** `pcctl` was run without a terminal (from a script, a cron job, a pipe) and needs the passphrase.
**Fix.** Set `POWERCLOUD_PASSPHRASE` for that run, from a secret store rather than a file in plain sight. `pcctl rotate` also needs `POWERCLOUD_NEW_PASSPHRASE`.
### "repository not initialized; run 'pcctl init'"
**Cause.** No location in `--repo` holds a key header. Usually the list is wrong: a typo in a path, a relative path run from another directory, or an S3 URL that the shell cut short at `&` because it wasn't quoted.
**Fix.** Check `--repo` (or `POWERCLOUD_REPO`) and quote any value containing `&` or `;`. Run `pcctl init` only for a genuinely new cluster: it refuses to run over an existing one ("repository already initialized"), and if any location is unreachable it refuses too ("checking for an existing cluster"), since that location might hold an existing header.
### A write fails with "N/M backends succeeded, need Q"
**Cause.** Fewer locations than the write quorum (by default a majority) accepted the write: some are down, full, or refusing your credentials.
**Fix.** Bring the locations back and write again. Reads keep working from any location that has the data. As a stopgap, `--quorum N` lowers how many locations must accept a write; run `pcctl --repo LOC,... repair` afterwards so every location catches up. See [Storage](https://powercloud.unidatum.com/docs/storage.md).
### "locations differ; run 'pcctl repair'"
**Cause.** `pcctl check` found locations missing, stale or corrupt copies, typically one that was down during writes, or one newly added.
**Fix.** `pcctl --repo LOC,... repair`. It authenticates every copy before writing it, so it can't spread corruption. "DAMAGED" objects have no good copy on any reachable location; bring back the missing location if you can.
### "store: gc needs every backend, and LOC is unreachable"
**Cause.** `pcctl gc` refuses to delete anything unless every location answers, since an unreachable one might hold a manifest that uses chunks that look unused elsewhere. Key rotation's purge has the same rule.
**Fix.** Wait until every location is reachable, or remove a location you've retired from `--repo` for good.
### "store: rollback: ... the storage serves an old version"
**Cause.** A location served an older, authentic version of an object (or dropped it) than the nodes' consensus has recorded. This is rollback protection catching a provider, or a location restored from an old backup.
**Fix.** Run `pcctl --repo LOC,... check` to see which location is behind. Writing the object again (`put`) gives it a newer time and repairs it. See [Storage](https://powercloud.unidatum.com/docs/storage.md).
### "warning: no rollback protection this time"
**Cause.** The cluster has a consensus group, but `pcctl` couldn't reach any node to read the rollback marks, so it went on without checking.
**Fix.** Nothing is wrong with the data; the check was skipped. Make sure at least one node is reachable for the next command.
### "init works on each storage location and needs --repo, not --via"
**Cause.** `init`, `check`, `repair`, `gc`, `rotate` and `node storage` work on each location separately, so they can't go through a node.
**Fix.** Run them with `--repo` and the full list of locations (and their credentials in the environment).
## Admin certificates
### "admin certificate expired (run 'pcctl --repo ... login')" or "no admin credentials"
**Cause.** Admin certificates last 30 days. Any command that unlocks the cluster renews yours when a week or less is left, but if you only used day-to-day commands (`status`, `node info`), it ran out. "No admin credentials" means this machine never had one, or `POWERCLOUD_CONFIG` points elsewhere.
**Fix.** `pcctl --repo LOC,... login` issues a fresh 30-day certificate.
### "the node refused the admin certificate; if admin certificates were revoked, run 'pcctl --repo ... login'"
**Cause.** The node rejected your certificate during the TLS handshake. Usually someone ran `pcctl revoke-admins`, which revokes every admin certificate issued before that moment. It also happens after a finished key rotation, when nodes accept only certificates from the new CA.
**Fix.** `pcctl --repo LOC,... login`. Every other admin must do the same on their machines.
### "unknown node "NAME" (see 'pcctl node ls')"
**Cause.** Day-to-day commands use this machine's own list of nodes (`nodes.json` in the configuration directory). A node enrolled from another admin machine, or provisioned without `--addr`, isn't on it.
**Fix.** `pcctl --repo LOC,... node addr NAME HOST[:PORT]` records where the node is, on this machine and in the cluster's registry.
## Node certificates
### "certificate has expired (renew the node's certificate with 'pcctl --repo ... node renew NAME')"
**Cause.** Node certificates last a year, and only an admin can renew them. An expired node is refused by `pcctl` and by its peers, so it drops out of the mesh. `pcctl node info` shows `certificate EXPIRED`, and `pcctl status` warns a week ahead.
**Fix.** `pcctl --repo LOC,... node renew NAME`. Expired certificates can still be renewed. Without names, `node renew` renews every node due within 30 days; run it from a scheduled job or with routine admin work so it never comes to this.
### "not renewed: no address yet (run 'pcctl --repo ... node addr NAME HOST')"
**Cause.** A provisioned node that `pcctl` has never been told the address of.
**Fix.** Record its address with `node addr`, then run `node renew` again.
## Key rotation
### "storage locations hold different key headers, left by an interrupted key rotation"
**Cause.** `pcctl rotate` wrote the new key header to some locations but not all (one was unreachable, or the run stopped). `pcctl` refuses other work until the rotation completes, so nothing is written under mixed keys.
**Fix.** Run `pcctl --repo LOC,... rotate` again with the **new** passphrase once every location is reachable.
### "N node(s) not on the new keys yet: ..."
**Cause.** `pcctl rotate finish` found nodes still on the old cluster CA, usually because they were off during the rotation. Moved and unmoved nodes can't reach each other until they are all moved.
**Fix.** Start the nodes (a rebooted node still unlocks with the old keys, and `pcctl` says "unlocked with the previous cluster keys"), run `pcctl --repo LOC,... rotate` again to move them, then `rotate finish`. `rotate finish --force` gives up on the rest, which must then be removed and enrolled again. Provisioned nodes that never joined hold the old keys in their join bundle: remove them and provision them again. See [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md).
## Booting and the state disk
### Nothing boots, or the firmware says "Access Denied"
**Cause.** The machine isn't booting with UEFI (legacy BIOS or CSM mode), or Secure Boot is on and the image isn't signed with a key the firmware trusts. Release images aren't signed for Secure Boot.
**Fix.** Boot in UEFI mode. Either turn Secure Boot off, or build images signed with your own key and enroll it. See [Install a node](https://powercloud.unidatum.com/docs/install.md).
### "state disk ... holds data that is not a PowerCloud volume; refusing to format it"
**Cause.** At enrollment, the node found something other than zeros in the first MiB of its state partition, and won't format a disk that might hold data. The usual reason is writing the image with `dd conv=sparse`, which skips zero blocks and leaves the disk's old contents in place.
**Fix.** Write the image again without `conv=sparse` (with `conv=fsync`, as in [Install a node](https://powercloud.unidatum.com/docs/install.md)), or erase the disk first with `blkdiscard`. The `powercloud.state.wipe=1` boot option the message mentions can't be given to a released image, whose kernel command line is fixed.
### "several PowerCloud state partitions"
**Cause.** More than one disk in the machine carries a PowerCloud state partition, for example an old PowerCloud disk left in a second bay, and the node won't guess which is its own.
**Fix.** Remove or erase the disk you don't want, then boot again.
### "no state disk (powercloud.state=); node state is kept in RAM and lost at reboot"
**Cause.** The node runs without an encrypted state disk: a kernel booted directly rather than from the disk image, or `pcd` running outside PowerCloud OS. `pcctl status` shows "state is in RAM, lost at reboot".
**Fix.** Boot the disk image (see [Install a node](https://powercloud.unidatum.com/docs/install.md)). A node in RAM loses its identity at every reboot.
### The node is locked after a reboot
**Cause.** This is normal. A node's state volume is encrypted, and after every reboot it shows an unlock code and waits, unless TPM unlock or peer unlock is on:
```text
PowerCloud node NAME (ID) is locked: its state volume is encrypted.
On your admin machine run:
pcctl --repo ... node unlock NAME
unlock code: ...
```
**Fix.** `pcctl --repo LOC,... node unlock NAME CODE`. If the node came back on a different address, add it: `node unlock NAME CODE NEW_HOST`, then `node addr NAME NEW_HOST`. "node key does not match the unlock code" means a mistyped code or something intercepting the connection; read the code again. To have nodes unlock themselves, see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md).
### "TPM unlock failed; waiting for an admin to unlock"
**Cause.** TPM unlock is sealed to the machine's Secure Boot state. Secure Boot was turned off, its keys changed, or the disk moved to another machine. This is TPM unlock refusing as designed.
**Fix.** Unlock with `pcctl --repo LOC,... node unlock NAME CODE`. If the change was intended (new keys), turn TPM unlock on again with `pcctl --repo LOC,... node tpm NAME on`.
### "peer unlock: not yet; an admin can still unlock"
**Cause.** The node is asking its peers for key shares and fewer than the threshold have answered: peers are down or locked themselves, or their addresses changed since peer unlock was turned on. "the shares don't open the volume (holders with old shares?)" means the shares are from before a change in the node set or a key rotation.
**Fix.** Bring the peers up, or unlock the node yourself. After adding or moving nodes or rotating keys, turn peer unlock on again with `pcctl --repo LOC,... node peer-unlock NAME on`. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md).
## Enrolling and joining
### "node key does not match the enrollment code (wrong code or interception)"
**Cause.** The code was mistyped, it's from an earlier boot (an unenrolled node may have a new key and code after a reboot), or something between you and the node is posing as it.
**Fix.** Read the code from the node's current console output and try again. If it still fails, check that the address really reaches that machine.
### "node is already enrolled in a cluster"
**Cause.** The node already has an identity: you enrolled it before, it joined from a join bundle, or someone else enrolled it first. On a cloud VM, whoever reads the console first (including the provider) can enroll it.
**Fix.** If it's in your cluster, use it (`pcctl node ls`). If it isn't, don't trust the machine: create a fresh one. A node of yours that you want to start over with can be wiped with `pcctl --repo LOC,... node rm --erase NAME`.
### "a node named "NAME" is already enrolled"
**Cause.** Names are unique in the cluster's registry.
**Fix.** Pick another name, or remove the old node first with `node rm`.
### "metadata service not answering" and then "could not read user data; waiting for console enrollment"
**Cause.** The node is on a cloud (AWS, Google Cloud or Azure) and couldn't read its user data after five tries. On AWS this happens when the instance metadata service is turned off.
**Fix.** Turn the metadata service on (on AWS, `aws ec2 modify-instance-metadata-options --instance-id ID --http-endpoint enabled`; the node uses IMDSv2, so `HttpTokens=required` is fine), then reboot: a node with no identity looks for a join bundle again at every boot. Or enroll it from the console.
### "invalid join bundle; waiting for console enrollment" or "join bundle not installed"
**Cause.** The user data or fw_cfg file looks like a bundle but is damaged (cut short, or edited), or it doesn't check out against its own CA, or the node couldn't create its state volume (see the state disk entries above). Bundles from before a key rotation are stale too.
**Fix.** Remove the node (`pcctl --repo LOC,... node rm NAME`), provision it again, and start a fresh machine with the new bundle, passed as a file (`file://` on AWS) rather than pasted.
### "node enrolled, but its mesh key is unknown (run 'pcctl mesh sync' later)"
**Cause.** The node enrolled, but `pcctl` couldn't read its WireGuard key straight away.
**Fix.** `pcctl --repo LOC,... mesh sync` once the node answers; it re-reads every node's key and publishes the membership list again.
## Reaching nodes and the mesh
### A node is "unreachable" in pcctl status, or commands time out
**Cause.** `pcctl` can't reach the node's API on TCP 7443. Common reasons: a firewall or security group doesn't allow TCP 7443 from your machine, the node is off or still booting, or it came back with a new IP address (a cloud VM after a stop and start).
**Fix.** Check the node's console. Allow TCP 7443 from your admin machines. If its address changed, run `pcctl --repo LOC,... node addr NAME NEW_IP`, which also moves its mesh endpoint and tells the other nodes. An Elastic IP or other reserved address avoids this on clouds.
### "NAME: reaches N of M mesh peers"
**Cause.** WireGuard between nodes isn't getting through. Common reasons:
- UDP 51820 isn't open on a public node, or UDP 51821-51920 on a node that relays for others.
- A node's endpoint is wrong, for example a private LAN address that nodes elsewhere can't use, or an address that changed.
- Two nodes changed IP addresses at once and nobody ran `node addr`, so they can't find each other.
- Two nodes behind NAT have no third node that both can reach.
**Fix.** Open the ports, then check endpoints in `pcctl node info NAME` (each peer's endpoint and handshake age). Fix one with `pcctl --repo LOC,... mesh endpoint NAME IP:PORT`, or clear it with `mesh endpoint NAME ''` for a node behind NAT. Keep at least one node with a stable public address. Nodes behind symmetric NATs connect through a relay automatically, which `node info` shows as `(relay NAME)`.
### "node NAME didn't come back on the new image within ..."
**Cause.** After `pcctl node upgrade`, the node didn't answer on the new image in time. It may be waiting to be unlocked, or the image may not have booted.
**Fix.** If it's locked, unlock it and run `pcctl node upgrade --commit NAME`. Otherwise do nothing: an image that isn't committed reverts at the next boot, and `pcctl node info` then shows `last upgrade ROLLED BACK`. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md).
### Replica writes fail with "this node's state volume is full"
**Cause.** A node holding a replica (`node://NAME`) refuses writes that would leave less than 256 MiB free on its state volume, so the replica never starves the node's own state.
**Fix.** Give the node a bigger disk (it grows its state volume to fill a resized disk at the next boot), or move the replica to another location.
## pcctl on macOS
### macOS won't open pcctl, or says it can't check it for malicious software
**Cause.** Gatekeeper holds back programs downloaded from the internet that Apple hasn't notarized. The macOS `pcctl` in a PowerCloud release is signed and notarized, and the release notes say so; a build without notarization, or one you built yourself and copied over, is held back.
**Fix.** Check the file against the release's `SHA256SUMS`, make it executable with `chmod +x`, and if the release notes say it isn't notarized, remove the quarantine flag: `xattr -d com.apple.quarantine pcctl-VERSION-darwin-ARCH`. There is no Windows build of `pcctl`.
## Still stuck
- `pcctl node info NAME` shows a node's state, certificate, Secure Boot, TPM and mesh status in one place.
- The node's console log shows what `pcd` is doing, with every warning and error.
- [Concepts](https://powercloud.unidatum.com/docs/overview.md) explains the terms used here, and [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md) lists every command.