# PowerCloud documentation > PowerCloud is an operating system for running your own encrypted cloud across home servers, colocated machines and any public cloud at once. Storage providers only ever hold ciphertext, and data is replicated or erasure-coded across them, so no single provider can read it, change it or take it away. Nodes run a minimal Linux kernel with an all-Go userland and have no shell or login: everything is done with the pcctl command-line tool, over mutually authenticated TLS. Source: https://powercloud.unidatum.com/docs (each page also at https://powercloud.unidatum.com/docs/SLUG.md) --- ## Concepts URL: https://powercloud.unidatum.com/docs/overview > What PowerCloud is and the ideas every other page builds on - trusted and untrusted nodes, the cluster and its storage locations, the passphrase and recovery key, the cluster CA, nodes, pcctl, the mesh and apps - with a glossary. PowerCloud is an operating system for running your own cloud on machines you own, rented servers and VMs on any public cloud, all at once. Data is encrypted before it leaves a machine you trust, so storage providers and plain cloud VMs only ever hold ciphertext, and it is spread over several providers so that no single one can read it, change it unnoticed, or take it away. This page explains the parts of a PowerCloud cluster and the words the rest of the documentation uses. ### What PowerCloud is PowerCloud OS is a minimal, hardened Linux kernel with a userland written entirely in Go, for x86-64 (amd64) and arm64 machines. - **No shell, no SSH, no login.** The userland is a handful of static Go binaries: `pcinit` (PID 1, which starts and supervises services) and `pcd` (the node daemon). There is no package manager and no way to log in. - **Managed only through an API.** Everything is done with the `pcctl` command-line tool on your own computer, which talks to nodes over mutually authenticated TLS. - **One image file.** The kernel, its initramfs and a fixed kernel command line are a single EFI application that UEFI firmware boots directly, with no boot loader. Nobody at the console or at the cloud provider can change the command line, and one signature covers the whole OS for Secure Boot. - **Hardened kernel.** Loadable modules are disabled entirely and kernel lockdown is forced to confidentiality mode. PowerCloud has two halves that you can use separately: an encrypted, replicated object store you drive from `pcctl` alone, and nodes that run the OS, join a mesh, hold storage and run apps. ### Trusted and untrusted Every place your data touches is either trusted or untrusted. | | Examples | What it may do | |---|---|---| | **Trusted node** | Your own hardware (home server, colo machine), or a confidential VM (AMD SEV-SNP) that passed attestation | Holds keys, decrypts data, runs apps | | **Untrusted location or node** | An S3 bucket, a plain cloud VM | Stores and relays ciphertext only | The reason is the threat model: on someone else's ordinary VM, the host can read the VM's memory, and with it any key the VM holds. Only a confidential VM, whose memory the CPU encrypts, closes that gap; PowerCloud verifies AMD SEV-SNP attestation before it gives such a VM a key (Intel TDX and Arm CCA are not supported yet). See [Security model](https://powercloud.unidatum.com/security) for the full threat model and [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md) for attested unlock. ### The cluster A cluster is one set of keys, one store and the nodes enrolled in it. #### Storage locations: the repository The cluster's data lives in an encrypted store spread over **storage locations**. Together they are called the **repository**, given to `pcctl` with `--repo` as a comma-separated list: ```sh pcctl --repo '/srv/pc,s3://my-bucket/pc?region=us-east-2,node://home-1' ls ``` A location is a local directory, an S3-compatible bucket (AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and others), the encrypted disk of one of your nodes (`node://NAME`), or an erasure-coded set of several of these (`ec:K:LOC;LOC;...`). You can mix them freely. Instead of typing `--repo` every time, you can set the `POWERCLOUD_REPO` environment variable to the same list. - **Everything is encrypted on your machine first.** Files are split into 4 MiB chunks, each sealed with XChaCha20-Poly1305. Names, sizes and chunk lists are in encrypted manifests, so a provider sees neither file names nor contents. - **Each write goes to several locations.** A write must reach a **quorum** of locations (by default a majority) to succeed. Reads fall back to any location that has the data, so losing one provider loses nothing. - **`pcctl check`, `repair` and `gc`** compare the locations, bring one that was down (or newly added) up to date, and reclaim chunks nothing uses any more. [Storage](https://powercloud.unidatum.com/docs/storage.md) covers locations, quorum, erasure coding and repair in detail. #### The passphrase and the recovery key The cluster has one random 256-bit **root key**, which is never stored in plaintext. It is wrapped, in the style of LUKS keyslots, in a small **key header** stored at every location: - a **passphrase** slot (at least 12 characters; stretched with Argon2id), which you choose at `pcctl init` - a **recovery key** slot: `pcctl init` prints a 160-bit recovery key once. Write it down and store it offline. It opens the cluster on its own, if the passphrase is forgotten. Every other key is derived from the root key for its purpose: the store's encryption keys, the cluster CA, the keys for each node's disk. So whoever has the passphrase or the recovery key, plus a copy of the header from any location, holds the whole cluster. If either leaks, `pcctl rotate` replaces the root key and re-encrypts the store (see [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md)). There is no backdoor. If every passphrase and the recovery key are lost, the data cannot be recovered by anyone. `pcctl` reads the passphrase from the terminal, or from `POWERCLOUD_PASSPHRASE` for scripts. To unlock with the recovery key instead, set `POWERCLOUD_RECOVERY_KEY`. #### The cluster CA Nodes and admins prove who they are with certificates from the **cluster CA**. Its Ed25519 key is derived from the root key and is never stored anywhere, so anyone who can unlock the cluster can issue certificates, and nobody else can, including a provider holding every blob. The CA signs everything nodes accept from admins: node and admin certificates, the membership list, the revocation list, releases, cluster apps and the consensus founding document. Nodes check the CA's signature, so these documents can be passed from node to node safely. ### Nodes and pcd A **node** is a machine (physical or virtual) running PowerCloud OS. Its daemon, `pcd`, holds the node's identity and serves the **node API** on TCP port 7443. - **Identity.** On first start, `pcd` generates an Ed25519 key. The node ID is a hash of that key (shown as 16 hex digits, such as `fa8adc53910a6a79`). - **Enrollment.** A new node prints a one-time **enrollment code** on its console. `pcctl node enroll` uses it to check it is talking to that exact node and to prove you read the console, then gives the node a certificate from the cluster CA. Alternatively, a node can be issued its identity ahead of time as a **join bundle** and join with nobody at its console. See [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md). - **After enrollment** the node accepts only clients with a certificate from its cluster CA, and checks each certificate's role: `admin` (you) or `node` (its peers). Node certificates last a year and are renewed with `pcctl node renew`. - **Encrypted state volume.** A node keeps its identity and state on an encrypted partition of its disk (dm-crypt, with an HMAC-SHA256 tag on every sector, so a disk changed while the node is off reads as an error rather than as altered data). The key for it can only be derived by someone who can unlock the cluster. After a reboot, the node stays **locked** until an admin runs `pcctl node unlock`, unless TPM unlock or peer unlock is on; see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). - **The registry.** Every enrolled node is recorded in the cluster's encrypted store, so every admin of the cluster sees the same nodes, their addresses and their disk volume IDs. See [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md) for managing nodes day to day. ### pcctl and admin certificates `pcctl` is the only management tool. It runs on Linux and macOS (amd64 and arm64); there is no Windows build. Commands come in two kinds: - **Commands that need the cluster keys** take `--repo` (or `--via NODE`, below) and ask for the passphrase: `init`, `put`, `get`, `node enroll`, `node unlock`, `node rm`, `rotate` and others. - **Day-to-day commands** need only your **admin certificate**: `pcctl status`, `pcctl node info NAME`, `pcctl node reboot NAME`, `pcctl app ...`, `pcctl console`. An admin certificate lasts 30 days. `pcctl --repo ... login` issues one, and any command that unlocks the cluster renews it by itself when a week or less is left. It is kept, with the CA certificate and the list of nodes this machine knows (`nodes.json`), in the configuration directory: `$POWERCLOUD_CONFIG` if set, otherwise `~/.config/powercloud` on Linux or `~/Library/Application Support/powercloud` on macOS. `pcctl revoke-admins` revokes every admin certificate at once, for example after losing a laptop. Once a node serves the storage (`pcctl node storage`), `--via NODE` can replace `--repo`: the node holds the storage locations and their credentials and relays encrypted blobs, so your laptop needs only the passphrase and its admin certificate. The node never gets the keys. `init`, `check`, `repair`, `gc` and `rotate` still need `--repo`. [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md) is the full command reference. ### The mesh Enrolled nodes connect to each other over an encrypted **mesh** of kernel WireGuard tunnels, on UDP port 51820. - **Addresses come from identity.** Each node's mesh address is a private IPv6 address made from the cluster ID and the node ID, so there is nothing to allocate. - **Peers are distributed automatically.** A **membership list**, signed by the cluster CA, names every node with its WireGuard key and **endpoint** (the public `IP:PORT` where peers can reach it). `pcctl` publishes a new list whenever a node is added or removed, and nodes pass newer lists on to each other, so a node that was offline catches up from any peer. - **NAT traversal.** Nodes behind NAT find each other's public address through a node both can reach and connect directly. When their NATs prevent that (symmetric NAT), they connect through a **relay**: a public node that forwards their WireGuard packets, which it cannot read. - **Authenticated twice.** Inside the tunnel, nodes still call each other's API with their node certificates. `pcctl node info NAME` shows each peer's handshake age and whether its API answers over the mesh. ### Apps Nodes run services you deploy, called **apps**: static Linux binaries or container images from a registry. - **Apps on one node** (`pcctl app deploy NODE NAME ...`): each runs in a sandbox (chrooted on the node's encrypted disk, as its own user, in its own namespaces and cgroup with memory, CPU and process limits, under a seccomp filter, with its own network reached only through ports it publishes), and is restarted when it exits. With `--vm`, an app runs in a **microVM** with a kernel of its own. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). - **Cluster apps** (`pcctl --repo ... cluster app deploy ...`): you publish the app for the whole cluster, and the nodes decide among themselves where it runs, starting it elsewhere when a node goes away. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). ### Consensus For state the nodes must agree on themselves, such as which node runs an exclusive app or the marks that detect a provider rolling back an object, the nodes form a Raft group: `pcctl --repo ... consensus init`, once. It keeps working while a majority of its voters are up. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md) and [Storage](https://powercloud.unidatum.com/docs/storage.md). ### Glossary | Term | Meaning | |---|---| | **Admin certificate** | A 30-day client certificate from the cluster CA, kept in pcctl's configuration directory, that lets you manage nodes without the passphrase. | | **Attestation** | A report signed by the CPU (AMD SEV-SNP) proving a VM's memory is encrypted; pcctl checks it before giving such a node its disk key. | | **Cluster** | One root key, the store it encrypts, and the nodes enrolled with its CA. | | **Cluster app** | An app published for the whole cluster, which the nodes place and move among themselves. | | **Cluster CA** | The certificate authority whose key is derived from the root key; it signs node and admin certificates and every document nodes accept. | | **Consensus** | The nodes' Raft group, which keeps a key-value state they agree on while a majority is up. | | **Endpoint** | Where peers reach a node's WireGuard port, as `IP:PORT` or `HOST:PORT`. | | **Enrollment code** | The one-time code a new node shows on its console: a fingerprint of its key plus a secret, used by `pcctl node enroll`. | | **Join bundle** | A file from `pcctl node provision` holding a node's whole identity, given to a machine as cloud user data or a QEMU fw_cfg file so it joins unattended. | | **Key header** | The one unencrypted blob at each location: the keyslots that wrap the root key, holding no plaintext key. | | **Location** | One place the store is kept: a directory, an S3-compatible bucket, a node's disk (`node://NAME`) or an erasure-coded set (`ec:K:...`). | | **Membership list** | The CA-signed list of nodes with their WireGuard keys and endpoints, passed between nodes. | | **Mesh** | The WireGuard network between a cluster's nodes, with addresses derived from node identity. | | **MicroVM** | A small virtual machine with its own kernel that an app can run in (`--vm`), for a stronger boundary than the shared-kernel sandbox. | | **Node** | A machine running PowerCloud OS, enrolled in a cluster. | | **Node API** | The mutual-TLS API `pcd` serves on TCP 7443, used by pcctl and by peers. | | **pcctl** | The command-line tool that manages a cluster from your own computer. | | **pcd** | The node daemon: identity, enrollment, unlock, mesh, storage and apps. | | **Quorum** | How many locations must accept a write for it to succeed; by default a majority (`--quorum N` overrides it). | | **Recovery key** | A 160-bit key printed once at `pcctl init` (and at `pcctl rotate`) that unlocks the cluster without the passphrase. | | **Registry** | The cluster's record of its nodes, kept in the encrypted store. | | **Relay** | A public node that forwards WireGuard packets between two nodes whose NATs block a direct connection; it sees only ciphertext. | | **Release** | A CA-signed set of node images that nodes roll out themselves, one at a time; a PowerCloud release is also the set of downloadable files named `vYYYY.MM.DD`. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). | | **Repository** | All of a cluster's storage locations together, given with `--repo` or `POWERCLOUD_REPO`. | | **Revocation list** | The CA-signed list of removed nodes and revoked admin certificates, enforced by every node. | | **Root key** | The cluster's 256-bit master key, from which every other key is derived; never stored in plaintext. | | **State volume** | The encrypted, integrity-checked partition where a node keeps its identity and state; locked after each reboot until unlocked. | | **Trusted node** | A node that may hold keys and run apps: your own hardware or an attested confidential VM. | | **Unlock code** | The code a locked node shows on its console after a reboot, used by `pcctl node unlock`. | | **Untrusted** | A bucket or plain VM that may only store or relay ciphertext. | --- ## Quickstart URL: https://powercloud.unidatum.com/docs/quickstart > Try PowerCloud on one computer with no hardware - create a cluster over three local directories, store and read a file, lose a provider and repair it, then run the node daemon locally and enroll it. This page walks through PowerCloud on a single computer, with no spare hardware, cloud account or VM. Three local directories stand in for three storage providers, so you can see the encrypted store survive one of them disappearing; then you run the node daemon, `pcd`, as an ordinary program and enroll it as a node. Everything here takes a few minutes, and nothing is installed system-wide. ### What you need - `pcctl`, the PowerCloud command-line tool, for Linux or macOS (amd64 or arm64). Either take `pcctl-VERSION-OS-ARCH` from a PowerCloud release (check it against the release's `SHA256SUMS`, then `chmod +x` it and rename it `pcctl`), or build it from the PowerCloud source with a Go toolchain. - For the second half (running a node locally), a Linux machine and the PowerCloud source, because `pcd` runs only on Linux and isn't shipped as a separate download. To build both from a checkout of the source: ```sh go build -o pcctl ./cmd/pcctl go build -o pcd ./cmd/pcd # Linux only ``` Keep this trial separate from any real cluster: `pcctl` stores admin credentials in its configuration directory, so point it at a scratch one for now. ```sh mkdir -p ~/pc-try && cd ~/pc-try # put pcctl (and pcd) here export POWERCLOUD_CONFIG=$PWD/config ``` ### 1. Create a cluster over three locations A repository is a comma-separated list of storage locations. Here they are three directories, `./home`, `./colo` and `./cloud`; in real use they would be a disk at home, a server in a colo and a cloud bucket. ```sh ./pcctl --repo ./home,./colo,./cloud init ``` `pcctl` asks for a new passphrase twice (at least 12 characters), creates the cluster's root key, and writes a key header to every location: ```text Initialized cluster 66770d7cd1f5522f114c0381e8fe93e1 RECOVERY KEY (write it down and store it offline — it is shown only once): LJRF-5NB4-BMPX-PAB2-CZVD-JRKL-XB6L-TF52 Without your passphrase or this recovery key, no one — including you — can read this data. ``` The long hex string is your cluster ID. The recovery key unlocks the cluster without the passphrase; for a real cluster, write it down and keep it offline, because it is never shown again. For what the passphrase and recovery key protect, see [Concepts](https://powercloud.unidatum.com/docs/overview.md). Every later command asks for the passphrase. For a trial you can set it in the environment instead, and you can set the repository once too, so that `pcctl ls` means `pcctl --repo ./home,./colo,./cloud ls`: ```sh export POWERCLOUD_PASSPHRASE='your passphrase' # trial only: it ends up in shell history export POWERCLOUD_REPO=./home,./colo,./cloud # optional; this page keeps --repo explicit ``` ### 2. Store and read a file `put NAME FILE` stores a file under a name of your choosing; `get NAME [FILE]` reads it back (to standard output if no file is given). ```sh printf 'hello, PowerCloud\n' > notes.txt ./pcctl --repo ./home,./colo,./cloud put docs/notes.txt notes.txt ./pcctl --repo ./home,./colo,./cloud ls ./pcctl --repo ./home,./colo,./cloud get docs/notes.txt ``` ```text stored docs/notes.txt (18 bytes, 1 chunks) 18 2026-10-29 01:16 docs/notes.txt hello, PowerCloud ``` Look inside `./home`: there are no file names and nothing readable, only blobs with opaque IDs. The file was split into chunks, each encrypted on your machine, and its name lives in an encrypted manifest. All three directories got a copy, because a write must reach a quorum (by default a majority) of the locations. ### 3. Lose a provider Delete one location entirely, as if a provider closed your account: ```sh rm -rf ./colo ./pcctl --repo ./home,./colo,./cloud get docs/notes.txt out.txt cat out.txt ``` ```text hello, PowerCloud ``` The read still works: `pcctl` falls back to the locations that have the data, and checks every chunk it reads against its MAC, so a location that returned altered data would be caught. `check` shows what each location is missing: ```sh ./pcctl --repo ./home,./colo,./cloud check ``` ```text 1 objects, 0 deleted, 1 chunks in use, 0 unreferenced (reclaimed by gc) LOCATION BLOBS MISSING STALE CORRUPT REPAIRED STATUS ./home 3 0 0 0 0 ok ./colo 0 3 0 0 0 needs repair ./cloud 3 0 0 0 0 ok pcctl: locations differ; run 'pcctl repair' ``` The three blobs per location are the key header, the file's manifest and its one chunk. ### 4. Repair `repair` copies every missing or outdated blob back, authenticating each copy before it writes it, so it can't spread corruption: ```sh ./pcctl --repo ./home,./colo,./cloud repair ``` ```text 1 objects, 0 deleted, 1 chunks in use, 0 unreferenced (reclaimed by gc) LOCATION BLOBS MISSING STALE CORRUPT REPAIRED STATUS ./home 3 0 0 0 0 ok ./colo 0 3 0 0 3 repaired ./cloud 3 0 0 0 0 ok ``` Adding a new provider works the same way: add its location to `--repo` and run `repair`. Real locations are usually S3-compatible buckets; see [Storage](https://powercloud.unidatum.com/docs/storage.md). ### 5. Run a node locally On Linux, start `pcd` with a state directory and a local listen address. In the OS, `pcd` starts at boot; here it is an ordinary program. ```sh ./pcd --state ./node-state --listen 127.0.0.1:7443 ``` It logs a few lines and prints an enrollment code: ```text level=WARN msg="no state disk (powercloud.state=); node state is kept in RAM and lost at reboot" svc=pcd level=WARN msg="apps unavailable" svc=pcd err="apps: enabling cgroup controllers in /sys/fs/cgroup: open /sys/fs/cgroup/cgroup.subtree_control: permission denied" PowerCloud node 262c2fdc861a9cf2 is waiting to be enrolled. On your admin machine run: pcctl node enroll enrollment code: EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO level=INFO msg="API listening" svc=pcd addr=127.0.0.1:7443 ``` The warnings are expected when `pcd` runs as an ordinary user outside PowerCloud OS: there is no encrypted state disk, and no permission to run apps. `262c2fdc861a9cf2` is the node ID, a hash of the key `pcd` just generated. ### 6. Enroll it In a second terminal, in the same directory and with the same `POWERCLOUD_CONFIG` (and `POWERCLOUD_PASSPHRASE`, if you set it), enroll the node under a name of your choosing. Use your own code: ```sh ./pcctl --repo ./home,./colo,./cloud node enroll 127.0.0.1 EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO my-node ``` ```text enrolled node my-node (262c2fdc861a9cf2) at 127.0.0.1:7443 mesh membership: 1 node (serial 1) NODE MEMBERSHIP my-node serial 1 ``` `pcctl` checked that the key it reached matches the fingerprint in the code (so nothing in between could pose as the node), issued the node a certificate from the cluster CA, recorded it in the cluster's registry, wrote you an admin certificate, and published a membership list. From now on the node accepts only clients with a certificate from this cluster. The port defaults to 7443 when you give only a host. ### 7. Look at the node Day-to-day commands need only the admin certificate, not the passphrase or the repository: ```sh ./pcctl node ls ./pcctl node info my-node ``` ```text NAME ID ADDRESS my-node 262c2fdc861a9cf2 127.0.0.1:7443 name my-node id 262c2fdc861a9cf2 cluster 66770d7cd1f5522f114c0381e8fe93e1 arch amd64 kernel ... boot UEFI, Secure Boot off lockdown none state RAM (lost at reboot) version dev uptime 1530h17m52s certificate expires 2027-10-29 01:14 UTC cluster CA 8471c8d7afafd600 membership list serial 1 mesh not configured yet ``` Running locally, the node reports your computer's kernel, uptime and lockdown setting. A real node shows `lockdown confidentiality`, its encrypted volume under `state`, and its mesh address and peers. The mesh isn't configured here because creating a WireGuard interface needs privileges `pcd` doesn't have as an ordinary user. `pcctl status` shows every node on one line, then anything that needs attention: ```sh ./pcctl status ``` ```text NODE STATE ARCH VERSION UPTIME CPU MEMORY MESH APPS STORAGE CERT my-node ok amd64 dev 1530h17m52s 0% of 32 52864/192933 MiB 0/0 peers 0/0 - 364d needs attention: my-node: state is in RAM, lost at reboot ``` The warning is the one you'd want on a real node: without an encrypted state disk, its identity would be lost at reboot. ### 8. Clean up ```sh ./pcctl node poweroff my-node # or Ctrl-C in pcd's terminal ./pcctl --repo ./home,./colo,./cloud node rm my-node ``` `node poweroff` on a node outside PowerCloud OS just makes `pcd` exit. `node rm` revokes the node's certificate and removes it from the registry and from this machine's list of nodes. Then delete the trial directory (`~/pc-try`), which holds the three locations, the node's state and the trial's admin credentials. ### Next steps - [Install a node](https://powercloud.unidatum.com/docs/install.md): run a real node on a physical machine, a VM or a cloud. - [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md): enroll nodes from the console or join them unattended. - [Storage](https://powercloud.unidatum.com/docs/storage.md): use S3-compatible buckets and your nodes' disks as locations. - [Concepts](https://powercloud.unidatum.com/docs/overview.md): the concepts and a glossary. --- ## Install a node URL: https://powercloud.unidatum.com/docs/install > How to run PowerCloud OS on a physical machine, a QEMU, libvirt or Proxmox VM, AWS, Google Cloud or Azure - the image files, writing and booting them, cloud upload scripts, firewall ports, supported hardware and Secure Boot. A PowerCloud node is any x86-64 or arm64 machine, physical or virtual, that boots the PowerCloud disk image with UEFI firmware. This page covers getting the image onto a machine and booting it; once it is up, it waits to be enrolled, which [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md) covers. AWS is tested on real instances; Google Cloud and Azure images and upload scripts exist but have not yet been booted on those clouds. ### The images Each PowerCloud release carries, for `amd64` and `arm64` (`VERSION` is the release name, `vYYYY.MM.DD`): | File | What it is | Used for | |---|---|---| | `powercloud-VERSION-ARCH.img.xz` | The bootable disk image, compressed | Physical machines, VMs, and AWS (decompressed) | | `powercloud-VERSION-ARCH.gce.tar.gz` | The same disk image packed for Google Cloud | `scripts/gcp-image.sh` | | `powercloud-VERSION-ARCH.vhd.xz` | The same disk image as a fixed VHD, compressed | `scripts/azure-image.sh` | | `powercloud-VERSION-ARCH.efi` | The OS alone, as one EFI file | Upgrading running nodes (`pcctl release publish`; see [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md)) | | `SHA256SUMS` | Checksums of every file | Checking downloads | Check what you download with `sha256sum -c SHA256SUMS --ignore-missing`. To build the images from the PowerCloud source instead, `make image` writes `out/powercloud-amd64.img` and `out/powercloud-arm64.img` (and the `.efi` files), and `make cloud-images` packs them for Google Cloud and Azure. The upload scripts named on this page are in the source's `scripts/` directory. The disk image is a 1 GiB GPT disk with two partitions: - **An EFI system partition** (256 MiB) holding the whole OS as one file at the firmware's default boot path: the kernel with its initramfs and a fixed command line built in. There is no boot loader and no boot menu, and nothing outside can add kernel options. - **The state partition**, empty until the node is enrolled. At enrollment the node creates its encrypted state volume there. On every boot it grows the partition to fill the disk, so a disk larger than the image (or a cloud disk resized later) is used in full. ### What you see when it boots The kernel logs to the serial console and to a monitor, if one is attached. `pcd` prints its message to both: ```text PowerCloud node 262c2fdc861a9cf2 is waiting to be enrolled. On your admin machine run: pcctl node enroll enrollment code: EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO ``` The node gets its network address by DHCP. Its API listens on TCP port 7443, which your admin machine must be able to reach. After enrollment, every reboot shows an unlock code instead, unless TPM or peer unlock is on (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). A machine given a join bundle skips the code and joins by itself (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). ### Physical machines 1. **Decompress and write the image** to the disk the machine will boot from. On Linux, with the target disk as `/dev/sdX` (double-check the name: everything on it is overwritten): ```sh xz -d powercloud-VERSION-amd64.img.xz sudo dd if=powercloud-VERSION-amd64.img of=/dev/sdX bs=4M conv=fsync ``` 2. **Boot it with UEFI.** Legacy BIOS boot is not supported; turn off CSM or legacy mode if the firmware has it. The firmware finds the OS at the default path, so no boot entry is needed. 3. **Secure Boot.** Release images are not signed for Secure Boot. On a machine with Secure Boot on (the default on most PCs), either turn it off in the firmware setup, or build images signed with your own key and enroll it (below). 4. **Read the enrollment code** from the monitor or serial console, and enroll the node from your admin machine. Caveats when writing the disk: - **Don't use `conv=sparse`.** The state partition must start out zeroed, and `pcd` refuses to format a state partition whose first MiB holds anything else. A sparse write leaves whatever the disk held before. - **`dd` overwrites only the image's size** (1 GiB). Erase a disk that held sensitive data first, with `blkdiscard` or the drive's own secure erase. - **One PowerCloud disk per machine.** If two disks in the same machine carry a PowerCloud state partition, `pcd` stops with "several PowerCloud state partitions". Erase the one you don't want. - **arm64 machines need UEFI firmware with ACPI**, as Arm servers have. The arm64 image boots through UEFI and describes the machine with ACPI, not a device tree. ### Virtual machines: QEMU, libvirt and Proxmox Any KVM host works: give the VM UEFI firmware (OVMF on x86, AAVMF on arm64), the disk image as a virtio disk, and a virtio NIC. On a Debian or Ubuntu host with the `ovmf` package, a QEMU node with its API forwarded to the host's port 7443 looks like this: ```sh cp /usr/share/OVMF/OVMF_VARS_4M.fd node-vars.fd qemu-img resize -f raw powercloud-VERSION-amd64.img 8G # optional: more room for the state volume qemu-system-x86_64 -machine q35,accel=kvm -cpu max -smp 2 -m 1G -nographic \ -drive if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd \ -drive if=pflash,format=raw,unit=1,file=node-vars.fd \ -drive file=powercloud-VERSION-amd64.img,format=raw,if=virtio \ -netdev user,id=n0,hostfwd=tcp::7443-:7443,hostfwd=udp::51820-:51820 \ -device virtio-net-pci,netdev=n0 -device virtio-rng-pci ``` With `-nographic`, the serial console (and the enrollment code) is in the terminal. Enroll it at `127.0.0.1`. The automated tests run nodes like this with 1 GiB of memory. For a VM to join with nobody at its console, give it a join bundle as a QEMU **fw_cfg** file: ```sh pcctl --repo LOC,... node provision vm-1 vm-1.join ## add to the qemu command line: -fw_cfg name=opt/powercloud/join,file=vm-1.join ``` libvirt (a `fwcfg` sysinfo entry) and Proxmox (the VM's `args` option) can pass the same fw_cfg file; the plain QEMU path is the one covered by automated tests. See [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md) for what the node does with the bundle. The same image runs on VMware (vmxnet3 network, PVSCSI or SATA disk) with UEFI firmware. ### AWS `scripts/aws-image.sh` turns a disk image into an AMI in about a minute. It writes the image straight into an EBS snapshot (skipping all-zero blocks, so no S3 bucket or import role is needed) and registers a UEFI AMI with ENA networking and NitroTPM. It uses the AWS CLI's usual credentials and region, and prints the AMI ID. ```sh xz -d powercloud-VERSION-arm64.img.xz scripts/aws-image.sh arm64 powercloud-VERSION-arm64.img # amd64 for x86 instances ``` Launch it on any Nitro instance type, x86 (t3, m6i, ...) or Graviton (t4g, m7g, ...), with a join bundle as user data so it joins unattended: ```sh pcctl --repo LOC,... node provision node-1 node-1.join aws ec2 run-instances --image-id AMI_ID --instance-type t4g.micro \ --metadata-options HttpTokens=required --user-data file://node-1.join ... pcctl --repo LOC,... node addr node-1 PUBLIC_IP ``` - **Security group.** Open TCP 7443 (the node API) to your admin machines only, and UDP 51820 (WireGuard) to the internet. A public node that relays for nodes behind strict NATs also needs UDP 51821-51920. - **Without user data**, the node waits for console enrollment, and `aws ec2 get-console-output --instance-id ID --latest` shows the code. - **New public IP after a stop and start.** Run `pcctl --repo LOC,... node addr NAME NEW_IP`, which also updates the node's mesh endpoint for its peers. An Elastic IP avoids this. - **Secure Boot.** With `POWERCLOUD_SB_CERT` set to the certificate your image was signed with, the AMI turns Secure Boot on with your key as its only trusted key, so instances boot only images you signed. Then `pcctl --repo LOC,... node tpm NAME on` lets the node unlock itself with its NitroTPM after a stop and start. This needs the PowerCloud source and a Go toolchain on the machine running the script. - **Confidential VMs.** On AMD SEV-SNP instances (c6a, m6a, r6a, offered by AWS on shared tenancy in US East (Ohio) and Europe (Ireland)), pcctl verifies the CPU's attestation before it unlocks the node; see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). Limits: older Xen instance types (t2, m4, ...) are not supported. NitroTPM is AWS's virtual TPM, so TPM unlock protects against a stolen EBS snapshot, not against AWS itself. ### Google Cloud (not yet tried) The Google Cloud image and script are built to Google's documented import format and CLI, but have **not yet been booted on Google Cloud**. Expect to be among the first to try them. `scripts/gcp-image.sh` copies the image into a Cloud Storage bucket you own, creates a Compute Engine image from it (UEFI, gVNIC, and SEV-SNP capable on x86), deletes the uploaded file and prints the image name. It uses gcloud's usual account and project, and takes either the release's `.gce.tar.gz` or a disk image (which needs the PowerCloud source and Go to pack). ```sh GCP_BUCKET=my-bucket scripts/gcp-image.sh arm64 powercloud-VERSION-arm64.gce.tar.gz gcloud compute instances create node-1 --image IMAGE --machine-type t2a-standard-1 \ --network-interface nic-type=GVNIC --shielded-vtpm \ --metadata-from-file user-data=node-1.join ... ``` Launch it as a Shielded VM (which has a vTPM), x86 (n2d, c3, ...) or Arm (t2a, c4a). Allow TCP 7443 from your admin machines and UDP 51820 from anywhere in the VPC firewall. With `POWERCLOUD_SB_CERT`, the image carries your certificate as its Secure Boot keys, and `--shielded-secure-boot` boots only images you signed; without it, leave Secure Boot off, since Google's default keys don't trust PowerCloud images. ### Azure (not yet tried) The Azure image and script are built to Azure's documented import format and CLI, but have **not yet been booted on Azure**. `AZURE_RESOURCE_GROUP=GROUP scripts/azure-image.sh` uploads the VHD into a managed disk with `azcopy` and makes it a version of the image `powercloud-ARCH` in an Azure Compute Gallery named `powercloud` in that group, creating both if missing. It uses the Azure CLI's usual account and subscription, takes the release's `.vhd.xz` or a disk image, and prints the image version's ID. ```sh AZURE_RESOURCE_GROUP=my-group scripts/azure-image.sh amd64 powercloud-VERSION-amd64.vhd.xz az vm create -g my-group -n node-1 --image IMAGE_VERSION_ID --specialized --size Standard_D2as_v6 \ --security-type TrustedLaunch --enable-secure-boot false --enable-vtpm true \ --user-data node-1.join --nsg-rule NONE ... ``` The images are Generation 2 (UEFI) and specialized, so Azure provisions nothing in them; the node joins from its user data. On x86 they are Trusted Launch, which gives instances a vTPM. Arm sizes (Standard_D2ps_v6, ...) take the same command without the three security options. Secure Boot stays off, because Azure's default keys don't trust PowerCloud images. `--nsg-rule NONE` opens nothing, so add network security group rules for TCP 7443 from your admin machines and UDP 51820. ### Supported hardware Every driver is built into the kernel, which loads no modules. Network cards whose drivers need a separate firmware file are not supported. | Where | Network | Disk | |---|---|---| | KVM: Proxmox, libvirt, Hetzner, DigitalOcean, Vultr, ... | virtio-net | virtio-blk, virtio-scsi | | AWS (Nitro instances, x86 and Graviton) | ENA | NVMe | | Google Cloud (x86 and Arm) | gVNIC, virtio-net | NVMe, virtio-scsi | | Azure (x86 and Cobalt) | Hyper-V netvsc, MANA | Hyper-V storvsc, NVMe | | VMware | vmxnet3 | PVSCSI (amd64), SATA | | Physical servers | Intel e1000e, igb, igc, ixgbe, i40e, ice; Mellanox ConnectX-4 and later; Broadcom NetXtreme-C/E; Realtek r8169 | NVMe, SATA (AHCI), Broadcom MegaRAID, SAS3 HBAs | What has actually run: the drivers QEMU can emulate (virtio, e1000e, vmxnet3, SATA, MegaRAID, NVMe) are checked in automated tests, with one node's disk moved between controllers; ENA and NVMe have run on real AWS instances, x86 and Graviton. gVNIC, Hyper-V and MANA are built in but have not run yet. Physical machines are expected to work with the drivers above but are not part of the automated tests. ### Secure Boot with your own key Because the whole OS is one EFI file, one signature covers everything up to the services. With Secure Boot on and your key enrolled, the firmware boots only images you signed: not a modified image, and not one a cloud provider swaps in. 1. Create a key once, and keep `db.key` offline: ```sh scripts/sb-keygen.sh ~/powercloud-sb ``` It writes `db.key` (private), `db.crt` (for signing) and `db.cer` (for enrolling in firmware). 2. Build images signed with it, from the PowerCloud source: ```sh POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image ``` 3. Enroll the certificate: on a physical machine, in the firmware setup (Secure Boot custom mode, add `db.cer` to `db`); on AWS and Google Cloud, by passing `POWERCLOUD_SB_CERT` to the upload script. Enrolling on real hardware has not been tested yet. `pcctl node info NAME` shows `boot UEFI, Secure Boot on` once it works. Sign every later image with the same key, or the firmware refuses it (an upgrade then rolls back by itself). Secure Boot is also what TPM unlock builds on; see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). ### Next steps - [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md): enroll the node from its console, or join it unattended. - [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md): a node that doesn't boot, can't be reached or won't format its disk. --- ## Enroll a node URL: https://powercloud.unidatum.com/docs/enroll > How to bring a freshly booted node into a PowerCloud cluster - from its console with an enrollment code, or unattended with a join bundle given as cloud user data or a QEMU fw_cfg file - and what the node does on its first boot and when it joins the mesh. A freshly booted PowerCloud node belongs to no cluster until you enroll it. There are two ways: read the one-time enrollment code from its console and run `pcctl node enroll`, or issue the node's identity ahead of time with `pcctl node provision` and hand the machine the resulting join bundle, so it joins at first boot with nobody at its console. Either way, the node ends up with a certificate from your cluster CA, an encrypted state volume on its disk, and a place in the mesh. To get a machine booted in the first place, see [Install a node](https://powercloud.unidatum.com/docs/install.md). Both ways need the cluster keys, so the commands below take `--repo` (or `--via NODE`) and ask for your passphrase. ### Enroll from the console When a new node boots, `pcd` prints on its serial console and monitor: ```text PowerCloud node 262c2fdc861a9cf2 is waiting to be enrolled. On your admin machine run: pcctl node enroll enrollment code: EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO ``` On a cloud VM, the console is the provider's console log, for example `aws ec2 get-console-output --instance-id ID --latest` on AWS. From your admin machine, enroll it at an address where you can reach its API (TCP 7443 by default), under a name you choose: ```sh pcctl --repo LOC,... node enroll 192.0.2.10 EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO home-1 ``` ```text enrolled node home-1 (262c2fdc861a9cf2) at 192.0.2.10:7443 mesh membership: 2 nodes (serial 5) NODE MEMBERSHIP colo-1 serial 5 home-1 serial 5 ``` The address may be `HOST` or `HOST:PORT`; without a port, 7443 is used. Node names must be unique in the cluster. #### What enrollment does 1. **Checks it's the right machine.** The code holds an 80-bit fingerprint of the node's key. `pcctl` connects over TLS and aborts unless the key matches, so nobody in between can pose as the node. 2. **Proves you read the console.** The code also holds an 80-bit secret. `pcctl` sends the node its certificates with a MAC under that secret, and the node accepts them only if the MAC checks out. 3. **Issues the node certificate** from the cluster CA (valid one year), and sends a new volume ID and the key for the node's encrypted state volume. 4. **Records the node** in the cluster's registry (in the encrypted store, so every admin sees it) and in this machine's list of nodes, and writes you an admin certificate if you don't have a current one. 5. **Joins it to the mesh.** `pcctl` reads the node's WireGuard key, sends it the current revocation list, and publishes a new membership list to every node it can reach. From then on, the node accepts only clients with a certificate from your cluster. ### What happens on first boot A node booted from the disk image has a state partition, empty until enrollment. When the node is enrolled (from the console or a join bundle), it: - **Creates its encrypted state volume** on that partition: dm-crypt with AES-256-XTS, and an HMAC-SHA256 tag on every sector (dm-integrity), so a disk changed while the node is off reads as an error, not as altered data. The volume key is wrapped by a key only someone who can unlock the cluster can derive. - **Formats only a blank disk.** `pcd` creates the volume only if the first MiB of the state partition is all zeros, so it never formats a disk that holds something else. - **Moves its identity onto the volume**, then keeps its certificates, mesh configuration, apps and any storage there. - **Grows into the disk** in the background: it writes zeros over the space beyond the initial filesystem (an authenticated volume can't read blocks never written), then grows the filesystem. `pcctl node info NAME` then shows, for example: ```text state encrypted volume, 7811 MiB (7690 MiB free) volume authenticated: AES-256-XTS with HMAC-SHA256 per sector (dm-integrity) ``` After every later reboot, the volume is locked: the node shows an **unlock code** on its console and waits for `pcctl --repo LOC,... node unlock NAME CODE`, unless TPM unlock or peer unlock is on. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). ### Join unattended with a join bundle A cloud VM, or a machine started by automation, has nobody to read its console. Only admins hold the CA key, so you issue the node's whole identity ahead of time: ```sh pcctl --repo LOC,... node provision web-1 web-1.join ``` ```text provisioned node web-1 (92bb5bdf55e3932f) mesh membership: 2 nodes (serial 2) NODE MEMBERSHIP my-node serial 2 web-1 no address yet (it will get it from a mesh peer) join bundle written to web-1.join. Give it to the machine as user data: AWS: aws ec2 run-instances --user-data file://web-1.join ... GCP: gcloud compute instances create --metadata-from-file user-data=web-1.join ... Azure: az vm create --user-data web-1.join ... QEMU: -fw_cfg name=opt/powercloud/join,file=web-1.join It holds the node's private keys and disk key: delete the file once the node is up, and the user data too where the provider allows it. Once the node is up, tell pcctl where it is: pcctl --repo ... node addr web-1 HOST[:PORT] ``` Flags, given before the name: | Flag | Meaning | |---|---| | `--addr HOST[:PORT]` | Where `pcctl` reaches the node's API, if you already know it (for example a reserved IP). Otherwise set it later with `node addr`. | | `--endpoint IP:PORT` | Where peers reach the node's WireGuard port, if it is public. Leave it out for a node behind NAT. | The bundle is a text file: a `#powercloud-join v1` line followed by base64-encoded JSON. It holds the node's key and certificate, the CA certificate, a WireGuard key, the state volume's ID and key, and the signed membership and revocation lists. Provisioning also adds the node to the registry and publishes a membership list that includes it, so its peers accept it the moment it comes up. `pcctl` refuses to overwrite an existing file. #### Giving the bundle to the machine | Where | How | |---|---| | AWS | `aws ec2 run-instances ... --metadata-options HttpTokens=required --user-data file://web-1.join` | | Google Cloud | `gcloud compute instances create ... --metadata-from-file user-data=web-1.join` | | Azure | `az vm create ... --user-data web-1.join` | | QEMU | `-fw_cfg name=opt/powercloud/join,file=web-1.join` | | libvirt, Proxmox | Pass the same fw_cfg file (a libvirt `fwcfg` sysinfo entry, or Proxmox's `args` option) | On first boot, a node with no identity yet looks for a bundle: first a QEMU fw_cfg file, then the user data of the cloud its firmware's SMBIOS strings name (AWS through IMDSv2, Google Cloud's `user-data` attribute, or Azure). A machine anywhere else never waits on a metadata service. A just-started instance's metadata service may need a moment, so the node tries five times before giving up. The node then checks the bundle: it must hold a node certificate from the bundle's CA for the bundle's key, and membership and revocation lists signed by that CA that name the node. If all is well, it creates its encrypted state volume as at enrollment, installs its identity and starts enrolled, logging `joined the cluster with a join bundle`. It never offers console enrollment. If anything is wrong, it logs why and falls back to waiting for console enrollment. - **User data that isn't a bundle** (a cloud-init script, say) is ignored. - **It joins once.** A node that already has an identity ignores the bundle, which cloud user data keeps serving. After a reboot it is locked like any enrolled node. - **Tested where.** AWS user data and the QEMU fw_cfg path run end to end in automated tests and on real AWS instances. The Google Cloud and Azure metadata readers are tested only against stand-in metadata servers, and the images have not been booted on those clouds yet. #### Tell pcctl where the node is Unless you gave `--addr`, `pcctl` doesn't know the new node's address. Once the machine is up, record it: ```sh pcctl --repo LOC,... node addr web-1 203.0.113.7 ``` ```text node web-1 is at 203.0.113.7:7443 mesh endpoint of web-1 set to 203.0.113.7:51820 mesh membership: 2 nodes (serial 3) NODE MEMBERSHIP my-node serial 3 web-1 serial 3 ``` `node addr` also derives the node's mesh endpoint from the address (port 51820) if none was set, or if the old one was derived from the old address, and tells every node. Run it again whenever the node's address changes, such as a cloud VM that comes back with a new public IP after a stop and start. `node addr` is also how a second admin machine learns about a node it didn't enroll itself. #### Keep the bundle secret The bundle holds the node's private keys and its disk key in the clear: whoever reads it can pose as that node and decrypt that node's disk. A cloud provider can read user data, just as it can read the console used for enrollment, so this adds no exposure on an untrusted node, but user data stays readable for as long as the instance keeps it. - Delete the bundle file once the node is up, and the user data too where the provider allows it. - If a bundle leaks, `pcctl --repo LOC,... node rm NAME` revokes that node. - For a confidential VM, prefer console enrollment: the provider can read user data but not the VM's memory. - Bundles made before a key rotation (`pcctl rotate`) carry the old keys. Remove such a node and provision it again. ### Joining the mesh Enrolled nodes connect to each other over WireGuard (UDP 51820). A node's **endpoint** is where its peers send to: - **At console enrollment**, the endpoint defaults to the enrollment address's host on port 51820. - **With a join bundle**, it is what `--endpoint` gave, or what `node addr` derives later. - **To set it yourself**: `pcctl --repo LOC,... mesh endpoint NAME IP:PORT` (or `HOST:PORT`, which nodes resolve again every minute, for dynamic DNS). For a node behind NAT that nobody can reach first, clear it with `pcctl --repo LOC,... mesh endpoint NAME ''`; it then connects outwards and its peers learn its public address. Each change publishes a new CA-signed membership list. Nodes pass newer lists to each other, so a node that was offline picks up new members from any peer. Two nodes behind NAT find each other through a node both can reach, or relay through a public node when their NATs prevent a direct connection. Public nodes need UDP 51820 open, and UDP 51821-51920 for relaying. Check the mesh with `pcctl node info NAME`, which lists each peer, its endpoint, the WireGuard handshake age and whether its API answers over the mesh: ```text mesh fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820) PEER MESH ADDRESS ENDPOINT HANDSHAKE API OVER MESH colo-1 fdd5:f552:ce70::4c1e:92b0:7d3a:15e2 198.51.100.4:51820 12s ago ok, 3.1 ms ``` `pcctl status` flags any node that reaches fewer peers than it has. If enrollment reports that the node's mesh key is unknown, run `pcctl --repo LOC,... mesh sync` later; it re-reads every node's key and publishes the list again. ### Good to know - **Whoever reads the console first can enroll the node.** On a cloud VM that includes the provider. If someone races you, your own enrollment fails with "already enrolled", so the attack is visible; nodes on plain cloud VMs are untrusted anyway (see [Concepts](https://powercloud.unidatum.com/docs/overview.md)). - **Node certificates last a year.** Renew them with `pcctl --repo LOC,... node renew` (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). - **Removing a node** (`pcctl --repo LOC,... node rm [--erase] NAME`) revokes its certificate on every node; `--erase` also has it crypto-erase its disk, after which it boots as a blank machine ready to enroll again. - Problems while enrolling are collected in [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md). --- ## Deployment designs URL: https://powercloud.unidatum.com/docs/designs > The architectures you can build with PowerCloud, from storage-only to multi-site clusters on confidential VMs, what each protects against, how it unlocks, and how to choose between them. This page describes six ways to deploy PowerCloud and when to choose each. Every design rests on one split: trusted machines hold keys and run apps, and everything else (buckets, plain cloud VMs, relays) only ever holds ciphertext. What changes between designs is how many trusted machines you have, where they are, and how many independent providers hold your encrypted data. ### The building blocks Every design is assembled from the same pieces. For definitions of node, cluster and repository, see [Concepts](https://powercloud.unidatum.com/docs/overview.md). - **Trusted node.** A PowerCloud node on hardware you own (a home server, a machine in a colocation rack) or on a confidential VM that passed attestation. It holds keys and runs apps. - **Untrusted location.** An S3-compatible bucket (AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and similar), a local directory, or a plain cloud VM. It stores and relays ciphertext only. A provider can delete or corrupt its copy, but it cannot read it, and authentication catches any change. - **Storage locations.** The cluster's encrypted store is written to every location in `--repo`: directories, `s3://` buckets, `node://NAME` replicas on a node's encrypted disk, or an erasure-coded group `ec:K:LOC;LOC;...`. A write succeeds once the write quorum accepts it (by default a majority of the locations). See [Storage](https://powercloud.unidatum.com/docs/storage.md). - **Consensus.** Three or more nodes form a Raft group (`pcctl consensus init`) that keeps state they agree on. It is needed for exclusive apps, app volumes, the S3 gateway and rollback protection. Two voters tolerate no loss, so failover needs three. - **Unlock.** A node's state volume is locked after every reboot. It is opened by an admin (`pcctl node unlock`), by the node's TPM (`pcctl node tpm NAME on`), by shares held by its peers (`pcctl node peer-unlock NAME on`), or, on an AMD SEV-SNP VM, by an admin unlock that first verifies the chip's attestation. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). - **Mesh and relays.** Nodes talk over WireGuard (UDP 51820). Nodes behind NAT find each other through a public node, and nodes behind symmetric NAT relay through one. Throughout, `pcctl --repo LOC,...` names the storage locations. You can set them once in `POWERCLOUD_REPO` instead of repeating `--repo`, and once a node serves the storage (`pcctl node storage`), `pcctl --via NAME` replaces `--repo` for everyday commands. ### 1. One home server, encrypted copies in cloud buckets ```text home untrusted providers +-------------------+ +------------------------+ | home-1 (trusted) | -- ciphertext ->| bucket on provider A | | node://home-1 | -- ciphertext ->| bucket on provider B | | apps | +------------------------+ +-------------------+ ^ mTLS, TCP 7443 admin laptop (pcctl, passphrase) ``` **Who it suits.** A person or a small household that wants its own server at home, with off-site copies that no single cloud can read or take away. **How it is set up.** Create the repository on the buckets, enroll the home server, add its disk as a location and copy everything onto it, then let it serve the storage: ```sh export POWERCLOUD_REPO='s3://pc-home?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com®ion=auto&creds=r2,s3://pc-home?endpoint=https://s3.REGION.backblazeb2.com®ion=REGION&creds=b2' pcctl init ## enroll home-1 (see /docs/enroll), then: export POWERCLOUD_REPO="node://home-1,$POWERCLOUD_REPO" pcctl repair pcctl node storage home-1 pcctl --via home-1 put photos/cat.jpg ./cat.jpg ``` With three locations, a write needs two of them, so it still succeeds while one bucket provider is down. The home server's scheduled sync fills in what that provider missed. **What it protects against.** A bucket provider reading the data, tampering with it, cutting off the account or going out of business. A stolen disk from the home server (the state volume is encrypted, with an integrity tag on every sector). **What it doesn't.** The home server is a single point of failure for apps: when it is down, nothing runs. One node can't form consensus, so there is no rollback protection (pcctl warns), no exclusive apps, no app volumes and no S3 gateway. A burglar who takes the running machine with TPM unlock on gets a running, enrolled node, protected only by its API and lockdown. **Unlock.** TPM unlock, if the machine has a TPM 2.0 and boots with Secure Boot under your own key; otherwise unlock by hand after each reboot. Peer unlock needs peers, so it doesn't apply. **Cost class.** Lowest: your own hardware and electricity, plus bucket storage. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** Restoring from the buckets after losing the server sends every byte out of those providers, so check their egress fees. Run `pcctl check` now and then; it compares every location. ### 2. Home, colo and buckets across several providers ```text home colocation rack +-----------------+ WG +-----------------+ | home-1 trusted |<----->| colo-1 trusted | +-----------------+ +-----------------+ \ / \ ciphertext only / +-----------+ +-----------+ +-----------+ | provider A| | provider B| | provider C| buckets, or one +-----------+ +-----------+ +-----------+ ec:4: group over six ``` **Who it suits.** Someone who wants no single provider, building or internet connection to be able to take the data away, and who can place a second machine somewhere else: a colo rack, an office, a relative's house. **How it is set up.** Both machines are enrolled and hold `node://` replicas, next to buckets on several providers. Full replicas cost the data times the number of locations. Erasure coding spreads each blob as data and parity shards, so `ec:4:` over six locations (four buckets and the two nodes) serves reads with any two lost, at 1.5 times the data (writes need five of the six). The nodes' disks can only join once the nodes are enrolled, so start from the repository you enrolled them with, add the erasure-coded group next to it, copy everything across, then use the group on its own: ```sh EC='ec:4:s3://...&creds=r2;s3://...&creds=b2;s3://...&creds=wasabi;s3://...;node://home-1;node://colo-1' pcctl --repo "OLD_LOC,OLD_LOC,$EC" repair pcctl --repo "$EC" check ``` **What it protects against.** Everything design 1 does, plus losing a whole site or a whole provider. Either node can serve the storage, so admins keep working with one site down. **What it doesn't.** Two nodes can't form a consensus group that survives losing either one, so this design still lacks failover for exclusive apps and dependable rollback protection. Add a third node (design 4's reasoning applies) to get them. Ordinary cluster apps do fail over between the two nodes. **Unlock.** TPM unlock on both, or peer unlock: each node holds the other's share, so a node that reboots alone unlocks itself. If both reboot together (a shared power cut is unlikely across two sites, but possible), neither can answer the other, and one needs an admin unlock first. **Cost class.** Low to moderate: two machines, a colo fee, and storage on several providers. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** Each write leaves its site once per remote location, and `pcctl repair` after adding a location copies everything, so egress and request charges add up. GC refuses to run while any location is unreachable. ### 3. All cloud: confidential VMs plus ciphertext-only storage ```text trusted (attested AMD SEV-SNP) untrusted (ciphertext only) +----------------+ +----------------+ +-------------------------+ | cvm-1, cloud X | | cvm-2, cloud Y | | buckets on X, Y, Z | +----------------+ +----------------+ | plain VMs, relays | \ / +-------------------------+ +----------------+ | cvm-3, cloud Z | three voters: consensus survives one loss +----------------+ ``` **Who it suits.** Teams with no hardware of their own who still want no cloud to be able to read their data or shut them down. **Why confidential VMs.** On someone else's VM the host can read the VM's memory, and with it the keys. On an AMD SEV-SNP VM the CPU encrypts that memory and signs a report saying so. pcctl verifies the report (signed by the chip's endorsement key, chained to AMD's root, debugging off, bound to the node's unlock key) before it sends a volume key, and requires it at every unlock from then on. A plain VM should never hold keys. **What is built and what is tested.** - AWS: images boot on x86 and Graviton. SEV-SNP attested unlock has been run on AWS on a c6a.large with SEV-SNP on. AWS offers SEV-SNP on shared tenancy only in US East (Ohio) and Europe (Ireland). - Google Cloud and Azure: each release carries their images, with upload scripts, but the images have not been booted on either cloud yet. The Google Cloud image is created SEV-SNP capable on x86. The Azure script creates Trusted Launch images, not confidential VM images. - Intel TDX and Arm CCA attestation are not built. Only AMD SEV-SNP reports are verified. So today a three-provider confidential cluster is a design you can plan for, but only its AWS part has been run. **What it protects against.** Any one cloud reading memory or storage (on attested nodes), and any one cloud pulling the plug: two trusted nodes and a consensus majority remain. **What it doesn't.** The attestation report measures the cloud's own firmware, not PowerCloud's kernel, which is vouched for by Secure Boot instead; pcctl shows the measurement but doesn't pin it. AMD's revocation lists aren't checked. A node that unlocks itself with its TPM attests to no one (check it with `pcctl node attest NAME`). Most cloud VMs don't offer virtualisation to the guest, so apps with `--vm` won't run there. **Unlock.** Admin unlock with attestation, after each reboot, keeps the guarantee whole. TPM unlock and peer unlock are available, but neither checks attestation. **Cost class.** Highest: about $210 a month for three small SEV-SNP VMs on three clouds, compute only, at prices checked on 2026-10-24. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** Replicating between clouds is egress on every write. On AWS, a VM that restarts gets a new public IP unless it has an Elastic IP; run `pcctl node addr NAME NEW_IP` after. ### 4. A small business office cluster ```text office LAN off-site +-----------+ +-----------+ +-----------+ +----------------------+ | office-1 |---| office-2 |---| office-3 | | buckets on 2+ | | voter | | voter | | voter |-->| providers, or ec:K: | +-----------+ +-----------+ +-----------+ +----------------------+ | cluster apps fail over; S3 gateway on all three (TCP 9443) staff machines, backup software, any S3 client ``` **Who it suits.** An office that wants its own servers for files, backups and internal apps, with one machine allowed to fail. **How it is set up.** Three nodes, consensus founded on all three, the storage served by the nodes, cluster apps placed by the nodes, and the S3 gateway for tools that already speak S3: ```sh pcctl --repo ... consensus init --nodes office-1,office-2,office-3 pcctl --repo ... node storage office-1 # likewise office-2 and office-3 pcctl --repo ... cluster app deploy --exclusive --volume --mem 512M --port 8080:8080 wiki ./wiki pcctl --repo ... cluster app deploy --replicas 2 --image nginx:alpine --port 80:80 site pcctl --repo ... s3 enable --name s3.office.example office-1 office-2 office-3 pcctl --repo ... s3 key add --bucket backups backup-server ``` An exclusive app runs on at most one node and moves when its node goes away; with `--volume` its data directory follows it. Clients use whichever gateway node is up, or a DNS name for all three. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md) and [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md). **What it protects against.** One node failing (apps restart elsewhere after about a minute, consensus keeps a majority), a provider holding the off-site copies, a stolen disk, a provider serving an old version of an object (rollback protection). **What it doesn't.** One building: fire, theft of all three machines or a long power cut takes everything local down; the off-site locations hold the data, not the running apps. An exclusive app's volume is saved every 30 seconds, so a node that dies loses up to 30 seconds of its changes, and databases should keep their state in `--storage` or checkpoint it. Two nodes lost at once stop consensus, and with it exclusive apps and the gateway's key changes. **Unlock.** Peer unlock fits well: a node that reboots alone unlocks itself from its peers' shares, and one carried out of the office stays locked. The default threshold is a majority of a node's peers, which is both of the other two in a three-node cluster; `--threshold 1` lets a node unlock with one peer up. After a power cut that reboots all three, the locked nodes can't answer each other, so unlock enough of them by hand for the rest to follow, or combine it with TPM unlock. **Cost class.** Moderate: three small machines you own, plus off-site storage. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** Placement counts apps' memory limits, not their use, so an app that fits nowhere runs nowhere and `pcctl status` says so. Leave each node room for the apps of a failed peer. ### 5. Edge and behind-NAT nodes with relays ```text +-----------------------+ | relay-1: public node | UDP 51820 and 51821-51920 open +-----------------------+ / ciphertext \ +---------------------+ +---------------------+ | edge-1: home router | direct | edge-2: CGNAT, 4G, | | (most home NATs) |<--------->| corporate firewall | +---------------------+ or via +---------------------+ relay-1 ``` **Who it suits.** Machines in places with no public address: a home behind a router, a shop or site on a mobile connection behind carrier-grade NAT, a branch office behind a corporate firewall. **How it works.** A node behind NAT has no endpoint, but connects outwards. A public node learns its NAT's address from its packets and tells other nodes, so two nodes behind ordinary NATs connect directly. When their NATs pick a new port for every destination (symmetric NAT), they relay through a public node instead, which sees only WireGuard ciphertext. Relayed pairs try a direct path again every 10 minutes, backing off up to about 10 hours. A home node with a changing address can use a DNS name as its endpoint: `pcctl mesh endpoint home-1 home.example.net:51820`. **What it protects against.** The same as the designs the edge nodes belong to. The relay can drop traffic but can't read or change it. **What it doesn't.** Every pair of NATed nodes needs a third node both can reach. Relayed traffic costs the relay's bandwidth and a detour. The mesh carries traffic between nodes; pcctl reaches each node's API (TCP 7443) at the address set with `pcctl node addr`, so admins still need a network path to a NATed node. **Unlock.** A locked node asks its peer-unlock holders outside the mesh, at their API addresses, so holders behind NAT can't be reached from elsewhere. Make public nodes the holders, set a reachable address with `pcctl mesh api NAME HOST:PORT`, or use TPM unlock. **Cost class.** Adds one public node (it can be small) to whatever design the edge nodes join. Its outbound traffic is billed by its provider. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** If the first peer to report a missing node is on that node's LAN, it reports a LAN address that nodes elsewhere can't use, until that peer's session lapses. ### 6. Storage only: pcctl and buckets, no nodes ```text admin laptop (pcctl, passphrase) | encrypts and decrypts locally +--> /srv/pc (local disk) +--> bucket on provider A +--> bucket on provider B ``` **Who it suits.** Encrypted, multi-provider backup and file storage with nothing to run: no servers, no images. **How it is set up.** ```sh pcctl --repo /srv/pc,'s3://pc-b?endpoint=https://s3.REGION.wasabisys.com®ion=REGION&creds=wasabi','s3://pc-c?region=us-east-2' init pcctl --repo ... put docs/report.pdf ./report.pdf pcctl --repo ... check pcctl --repo ... gc ``` Erasure coding works here too. Every machine that uses the repository needs every location's credentials, since no node holds them. **What it protects against.** Providers reading, altering or losing the data. **What it doesn't.** With no nodes there is no consensus, so no rollback protection (pcctl warns on every command): a provider can serve an older authentic version of an object. Two admins writing the same name at once: the later write wins. There are no apps, no `--via`, no scheduled sync and no S3 gateway; run `pcctl repair` yourself after a provider was down. **Unlock.** Only the passphrase or recovery key; there are no node volumes. **Cost class.** Storage and egress only. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md). **What to watch.** `check --verify` downloads every chunk copy, which is egress on every provider. Start here and add nodes later: enrolling a node and adding `node://NAME` to `--repo` turns this into design 1. ### Choosing a design | You need | Design | | --- | --- | | Encrypted off-site backup, nothing to run | 6. Storage only | | A home server with off-site copies | 1. Home plus buckets | | No single site or provider can take data away | 2. Home, colo and buckets | | No hardware of your own | 3. Confidential VMs (AWS tested today) | | Apps that survive a machine failing, an S3 endpoint for existing tools | 4. Three-node cluster | | Nodes with no public address | 5. Edge nodes, plus one public node | | Rollback protection | Any design with three nodes and consensus | The designs combine: an office cluster (4) with a confidential VM off-site (3) and an edge node at a branch (5) is one cluster. ### Trade-offs **Replication or erasure coding.** Replicas cost the data times the number of locations, and any one copy serves a read. Erasure coding with k data shards over n locations costs n/k times the data and survives losing n−k locations, but a read needs k providers to answer and a write needs k+1. Use replicas for a few locations or small data; erasure coding when you spread over many providers. **Write quorum.** The default is a majority of locations (`--quorum` changes it). A higher quorum means a write is on more locations before it is acknowledged; a lower one lets writes go on with more locations down, leaving more for sync or `pcctl repair` to fill in. **One node or three.** One node is simplest and cheapest but has no failover and no consensus. Two nodes give a second site but no consensus that survives a loss: two voters need both. Three or more give failover for exclusive apps, the S3 gateway and rollback protection, with one node allowed to fail. **Trusted or untrusted placement.** Put a node that holds keys only on hardware you control or on an attested SEV-SNP VM. A plain VM enrolled as a node never gets the cluster keys, but it is still a full member: it votes in consensus, may relay, and is eligible for cluster apps unless you name the allowed nodes with `--nodes`. Buckets are the simplest untrusted placement. **TPM, peer or manual unlock.** | Method | Unlocks by itself | Stays locked when | Watch | | --- | --- | --- | --- | | Manual (`pcctl node unlock`) | Never | Always, until an admin unlocks | Someone must act after every reboot | | TPM (`node tpm NAME on`) | Same machine, Secure Boot on with your key | Disk moved, Secure Boot off or keys changed | A stolen running machine is a running node | | Peer (`node peer-unlock NAME on`) | Enough peers reachable | Taken away from its peers, or removed | A cluster-wide reboot needs a manual start | | Attested (SEV-SNP, manual) | Never | Not on a verified SEV-SNP VM | TPM and peer unlock skip attestation | ### Good to know - Google Cloud and Azure images are built and have upload scripts but have not been booted on those clouds yet. Intel TDX and Arm CCA attestation are planned; see [Roadmap](https://powercloud.unidatum.com/roadmap). - Losing every passphrase and the recovery key loses the data in every design. There is no backdoor. - Moving between designs is adding and removing locations and nodes: add a location to `--repo`, run `pcctl repair`, then drop the old one. - For the security model behind these designs, see [Security model](https://powercloud.unidatum.com/security). --- ## Costs and sizing URL: https://powercloud.unidatum.com/docs/sizing > How to size PowerCloud nodes, storage and network, what confidential cloud VMs and object storage cost, worked monthly estimates for common designs, and how to spend less. This page helps you work out how big your nodes need to be, how much storage your data takes once it is replicated or erasure-coded, and what a deployment costs each month. The key idea is that only trusted nodes need expensive machines: storage locations and relays only ever hold ciphertext, so they can be the cheapest option that meets your durability needs. For the designs referred to here, see [Deployment designs](https://powercloud.unidatum.com/docs/designs.md). ### Sizing a node #### What the node keeps for itself A node runs from RAM (its root filesystem is the initramfs built into its image) and keeps its state on an encrypted disk. When the nodes place cluster apps, each node offers its apps what is left after a reserve for itself: - **Memory:** the node's RAM less the larger of 256 MiB and one eighth. - **CPU:** the node's CPUs less the larger of a quarter of a CPU and one eighth. | Node | Memory for apps | CPU for apps | | --- | --- | --- | | 1 GiB, 2 CPUs | 768 MiB | 1.75 | | 2 GiB, 2 CPUs | 1792 MiB | 1.75 | | 4 GiB, 2 CPUs | 3584 MiB | 1.75 | | 8 GiB, 2 CPUs | 7168 MiB | 1.75 | | 16 GiB, 4 CPUs | 14336 MiB | 3.5 | | 32 GiB, 4 CPUs | 28672 MiB | 3.5 | The reserve is what placement sets aside, not a measurement of what the node uses. The smallest machines PowerCloud is checked on in automated tests have 1 GiB of memory: AWS t3.micro (x86) and t4g.micro (Graviton), and 1 GiB, 2-CPU virtual machines for the emulated tests. A node with no apps, used only to hold a storage replica or to relay, fits on machines of that size. #### What each app takes An app's memory limit (`--mem`, default 256 MiB, no swap) and CPU cap (`--cpu`, default none) are what placement counts, on every node the app runs on. Placement counts limits, not use, and an app without `--cpu` takes no CPU in placement and shares what is left over. So: - Add up the `--mem` of every app a node may run. With failover, a node must also have room for the apps of a peer that goes away, or those apps run nowhere until it returns (`pcctl status` reports them). - Exclusive apps are placed first; the others follow in name order. - Apps deployed by hand with `pcctl app deploy` are not counted by placement. Leave room for them yourself. ```sh pcctl app deploy --mem 128M --cpu 0.5 home-1 web ./web pcctl --repo ... cluster app deploy --replicas 2 --mem 512M --cpu 1 --image nginx:alpine --port 80:80 site ``` See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md) and [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). #### MicroVM apps An app deployed with `--vm` runs under a kernel of its own: - Placement counts it at its memory limit plus 96 MiB. - The cgroup's limit covers the guest's memory (the app's limit, its files and the guest kernel's 64 MiB) and the monitor's. - The guest's root is in its memory, so the app's files count against it: images of a few hundred MiB are fine, multi-GiB ones are not (2 GiB at most). - It gets one vCPU for each CPU of its `--cpu` limit, rounded up (`--cpu 1.5` gives two), or one per node CPU, up to 32, without a limit. MicroVMs need hardware virtualisation on the node: bare metal, or a VM with nested virtualisation. Most cloud VMs don't offer it (on AWS, `.metal` instances do), so microVM apps are cheapest on hardware you own. #### Disk - **The disk image is 1 GiB** by default: a 256 MiB EFI system partition holding the signed kernel, and the state partition. A fresh image is about 15 MiB of data. Give the machine a disk of any size: the state partition, the encrypted volume and its filesystem grow to fill it, and grow again on the next boot after you enlarge a cloud disk. - **Integrity tags take about a tenth of the disk.** New state volumes store an HMAC-SHA256 tag for every sector. - **Writes cost about double.** The integrity journal writes data twice. That is fine for node state and replicas, but not tuned for heavy write loads. - **A replica keeps 256 MiB free.** A `node://` write that would leave less is refused, so a full replica never starves the node's own state. - **Apps' data and logs** live on the state volume. Logs rotate at 1 MiB. For a node that holds a `node://` replica, budget the replica's share of the data (see below), plus a tenth for tags, plus the apps' data. ### Sizing storage Objects are split into fixed 4 MiB chunks, each encrypted with a few tens of bytes of overhead, plus an encrypted manifest per object. How much you store in total depends on how the locations are combined: | Layout | Stored | Reads survive losing | Writes survive losing (default quorum) | | --- | --- | --- | --- | | 3 full replicas | 3 × data | 2 locations | 1 location (a majority, 2 of 3, must accept) | | `ec:2:` over 3 locations | 1.5 × data | 1 location | none (k+1 = 3 shards must be stored) | | `ec:4:` over 6 locations | 1.5 × data | 2 locations | 1 location (k+1 = 5 shards) | With erasure coding, each location holds one shard of every blob: at 4+2, the shards of a 4 MiB chunk are 1 MiB each, so each of six locations holds a quarter of the data. 1 TiB at `ec:4:` over six locations is 256 GiB per location and 1.5 TiB in all, against 3 TiB for three replicas. - **Deduplication.** A chunk's ID is a keyed MAC of its contents, so identical chunks are stored once: the same file stored twice, or the unchanged 4 MiB pieces of a file overwritten in place. Because chunks are fixed-size, inserting bytes near the start of a file changes the chunks after it. App storage, app volumes and the S3 gateway each have their own keys and namespace, so they don't deduplicate against each other or the cluster's own objects. - **Small objects.** Each object is at least one chunk and one manifest, so many small files cost more in requests than their size suggests. - **Deleted data costs until GC.** `pcctl gc` deletes a chunk only after it has been unreferenced for the grace period (default 24 hours) across two runs, and only when every location is reachable. Overwritten versions and deleted objects keep using space until then. Deletes leave small tombstones that are kept for good. App storage and app volumes are reclaimed by nodes serving storage during their sync (hourly by default), also with a 24-hour grace. See [Storage](https://powercloud.unidatum.com/docs/storage.md) and [Use cases with the Unidatum Data Engine](https://powercloud.unidatum.com/docs/data-engine.md). ### Network | Port | Open to | For | | --- | --- | --- | | TCP 7443 | Your admin machines only | The node API, which pcctl uses | | UDP 51820 | The internet, on public nodes | WireGuard between nodes | | UDP 51821-51920 | The internet, on public nodes that relay | Relays for nodes behind symmetric NAT | | TCP 9443 | Your S3 clients | The S3 gateway, on nodes where it is enabled | Traffic to plan for: - **Replication is egress.** Every write is sent to each location: with replicas, the whole object to each; with erasure coding, one shard to each, n/k times the data in all. Writes from a cloud VM to buckets on other providers leave that cloud once per provider. - **Reads, restores and checks are egress from the provider.** Restoring after losing a site, `pcctl repair` after adding a location, and `pcctl check --verify` (which downloads every copy of every chunk) all pull data out of providers. - **Relayed traffic passes through the relay.** The relay receives and sends every packet of a relayed pair, so its provider bills that traffic. Pairs try a direct path again every 10 minutes, then more rarely. - **Background traffic is small but constant.** Every node checks each peer every 5 seconds over the mesh, and the consensus leader sends heartbeats. ### Cloud costs #### Confidential VMs for trusted nodes A trusted node on someone else's hardware needs a confidential VM, because the host can otherwise read the VM's memory and the keys in it. PowerCloud verifies attestation on AMD SEV-SNP. Prices checked on 2026-10-24, on-demand, USD, Linux, compute only, 730 hours a month. Prices change; check them again before you commit to a budget. | Cloud | Region | Size | vCPU / RAM | Plain VM | SEV-SNP VM | | --- | --- | --- | --- | --- | --- | | AWS | us-east-2 | c6a.large | 2 / 4 GiB | $55.84 | $61.43 | | AWS | us-east-2 | m6a.large | 2 / 8 GiB | $63.07 | $69.38 | | Google Cloud | us-central1 | n2d-standard-2 | 2 / 8 GiB | $61.68 | $67.85 | | Azure | eastus | DC2as_v6 (vs D2as_v6) | 2 / 8 GiB | $66.28 | $72.93 | | Azure | eastus | DC2as_v5 (vs D2as_v5) | 2 / 8 GiB | $62.78 | $62.78 | | AWS | us-east-2 | m6a.xlarge | 4 / 16 GiB | $126.14 | $138.76 | | Google Cloud | us-central1 | n2d-standard-4 | 4 / 16 GiB | $123.36 | $135.69 | | Azure | eastus | DC4as_v6 (vs D4as_v6) | 4 / 16 GiB | $132.86 | $146.00 | | AWS | us-east-2 | r6a.xlarge | 4 / 32 GiB | $165.56 | $182.12 | | Google Cloud | us-central1 | n2d-highmem-4 | 4 / 32 GiB | $166.41 | $183.05 | | Azure | eastus | EC4as_v5 (vs E4as_v5) | 4 / 32 GiB | $164.98 | $164.98 | - **Confidential VMs cost about 10% more** than the same plain VM on all three clouds. Some Azure sizes (the v5 DCas and ECas series above) cost the same. - **How the premium is charged differs.** AWS adds 10% of the instance's on-demand rate, and Reserved Instances and Savings Plans don't reduce that fee. Google Cloud adds a surcharge per vCPU-hour and per GiB-hour, so memory counts. Azure sells confidential sizes at their own prices. - **Use SEV-SNP, not plain AMD SEV.** Plain SEV lacks SNP's memory integrity protection; on Google Cloud it is also the dearest option, about 20% extra. - **Not included:** disks, public IPs, taxes, network egress and object storage. What has been run on each cloud matters as much as the price: - **AWS:** images boot on x86 and Graviton. SEV-SNP attested unlock has been run on a c6a.large. AWS offers SEV-SNP on shared tenancy only in US East (Ohio) and Europe (Ireland), and it can't be turned on after launch. - **Google Cloud and Azure:** each release carries images for both, with upload scripts, but they have not been booted on either cloud yet. - **Intel TDX** sizes (Google Cloud C3, Azure DCesv6 and ECesv6) cost about the same premium, but TDX attestation is not built yet. Treat their prices as planning figures only. See [Roadmap](https://powercloud.unidatum.com/roadmap). #### Object storage Bucket costs depend on three things, which vary a lot between providers: the price per GB stored each month, the price of each request (PUT, GET, LIST), and the price of data leaving the provider (egress). For PowerCloud: - **Storage** is the stored total from the layout table above, split across providers. - **Requests** come from every chunk and manifest written, the hourly sync of each node serving storage (which lists every location), app volumes (saved every 30 seconds while files change), and `check`, `repair` and `gc`. - **Egress** comes from reads, restores, `repair` and `check --verify`. Providers with free or cheap egress make good replication targets. Check each provider's current price list; no bucket prices are quoted here. ### Worked examples Compute prices are the on-demand figures above, checked on 2026-10-24. Storage is shown as the amount billed, since bucket prices vary by provider. #### Home server with two buckets (design 1) | Item | Monthly | | --- | --- | | Home server | Your hardware and electricity, no cloud cost | | 500 GiB of data, one replica in each of two buckets | 1000 GiB billed across two providers, plus requests | | Home server disk | At least about 560 GiB (the replica plus a tenth for tags), plus apps' data | Cloud compute: $0. Egress is small day to day, and a full restore pulls 500 GiB out of one provider. #### Home first, one confidential VM off-site (designs 1 and 2) | Item | Monthly | | --- | --- | | Home server | No cloud cost | | One SEV-SNP node, AWS c6a.large, us-east-2 | $61.43 | | Buckets on two providers | Storage, requests and egress at their prices | About $62 to $73 a month in compute for the clouds' 2 vCPU, 8 GiB confidential sizes; the AWS c6a.large (2 vCPU, 4 GiB), the size SEV-SNP unlock has been run on, is $61.43. Two trusted nodes give a second site, but consensus needs a third node to survive losing either. #### Three clouds, one SEV-SNP node on each (design 3) | Item | Monthly | | --- | --- | | Google Cloud n2d-standard-2 | $67.85 | | AWS m6a.large | $69.38 | | Azure DC2as_v6 | $72.93 | | Compute total | about $210 | Losing any one provider leaves two trusted nodes and a consensus majority. Only the AWS part has been run today. Committed-use discounts can lower this a lot on Google Cloud and Azure; on AWS they lower the instance but not the SEV-SNP fee. Egress between the three clouds comes on top and, for write-heavy use, can exceed compute. #### Storage only (design 6) No compute. 1 TiB at `ec:4:` over six buckets is 1.5 TiB billed, 256 GiB with each provider, against 3 TiB for three replicas. ### How to spend less - **Keep untrusted roles cheap.** Buckets and plain VMs only see ciphertext. Use the cheapest tier that meets your durability needs for storage, and a small plain VM for a relay or a `node://` replica. Don't run cluster apps on plain VMs: name the allowed nodes with `--nodes`. - **Run trusted nodes on hardware you own.** A home or office server, or bare metal in a colo, costs no confidential-VM premium and can run microVM apps. - **Use arm64 where it fits.** Images boot on arm64 machines and AWS Graviton. Arm cloud VMs are plain VMs to PowerCloud (Arm CCA attestation is not built), so use them for untrusted roles, or as trusted nodes only on your own hardware. - **Prefer erasure coding over many replicas** once you spread over four or more providers. - **Pick replication targets with cheap egress**, and avoid `check --verify` more often than you need it. - **Discounts.** Google Cloud and Azure committed-use discounts and reservations apply to confidential VMs. On AWS the SEV-SNP fee stays 10% of the on-demand rate whatever you pay for the instance. Spot capacity is cheaper but can be reclaimed at any time, which suits replicas and relays better than trusted nodes. - **Right-size apps.** Placement counts `--mem` limits, not use, so generous limits leave nodes looking full. --- ## Use cases with the Unidatum Data Engine URL: https://powercloud.unidatum.com/docs/data-engine > How the Unidatum Data Engine (document collections, SQL over parquet, content-addressed files) could run on PowerCloud nodes and keep its data in PowerCloud's encrypted storage, what works today, what is untested, and designs for concrete use cases. The Unidatum Data Engine is a sibling Unidatum product: a peer-to-peer data engine with MongoDB-style document collections, distributed SQL over parquet files (a lakehouse), a content-addressed file layer built on BitTorrent v2, and SQL wire-protocol front ends. PowerCloud supplies what the engine leaves to its host: encrypted disks, a sandbox that keeps services running, an encrypted mesh between your machines, and encrypted storage spread across providers. This page says exactly what of that combination works today, then describes use cases as designs: no combined deployment has been tested yet. The engine's own product site is https://www.unidatum.ie/en. It is licensed separately from PowerCloud. Its executable is called `unidatum`, and some of its settings are still environment variables named `P2PFS_*`. ### Status Each product works on its own and has its own tests. Nothing on this page has been run end to end, so treat every command below as a starting point to verify on your own cluster. #### What is established today | Fact | Where it comes from | | --- | --- | | PowerCloud apps run a static Linux binary or a container image, sandboxed, kept running, with their data in `/data` on the node's encrypted state volume. | PowerCloud, tested in its automated tests ([Apps on a node](https://powercloud.unidatum.com/docs/apps.md)). | | The engine's Go executable builds as a static Linux binary for x86-64 and arm64, which is what a PowerCloud binary app requires. The engine's standard Linux releases are built without cgo and run DuckDB as a separate program; a build with DuckDB linked in (`-tags duckdb_inproc`, against a static DuckDB from the engine's `scripts/linux-duckdb.sh`) is one binary that queries on its own. Fully static (musl), that build has run on arm64 Linux and Android. | The engine's source and its in-process DuckDB measurements. The cgo-free build was checked by building it; neither build has run on a PowerCloud node. | | The engine's blob tier talks to S3 with path-style requests signed with AWS Signature Version 4, using put, get, head, delete and ListObjectsV2. PowerCloud's app storage and S3 gateway serve exactly that subset. | Both projects' source. Not run together. | | The engine finds an S3 endpoint from `P2PFS_S3_ENDPOINT` and credentials from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. PowerCloud's `--storage` sets the credentials, and the endpoint can be passed with `--env`. | Both projects' source. | | On Linux the engine trusts the CA certificates in `SSL_CERT_FILE`, as every Go program does, so it can be made to trust the PowerCloud cluster CA for the S3 gateway. | Go's standard behaviour; the engine sets no CA list of its own. Not tried against the gateway. | | Engine peers sync over one TCP port (47800 by default), optionally with QUIC over UDP on the same number, and can be told the address peers should use (`--advertise`). PowerCloud links carry TCP and UDP between apps by name, and two peer-to-peer apps may link to each other on the port they both serve. | Both projects' source and docs. | #### Should work, but untested - **The engine as a single static binary app** with DuckDB linked in: one file, nothing beside it, queries included. This needs the fully static (musl) in-process build; the engine's standard releases don't ship it, so you build it. - **The engine as a container-image app** (`pcctl app deploy --image`), with the companion programs it needs, and either the in-process build or the DuckDB command-line program in the image. There is no published engine image, so you build one (see "Packaging the engine" below). - **The engine's blob tier on PowerCloud app storage**, with the engine as a cluster app deployed with `--storage`, and `P2PFS_S3_ENDPOINT=http://127.0.0.1:9000`. - **The engine's blob tier on the PowerCloud S3 gateway**, from an engine running anywhere, trusting the cluster CA. - **Engine peers on several PowerCloud nodes**, each a cluster app pinned to its node with `--nodes`, linked to each other by name over the encrypted mesh. - **SQL front ends** (PostgreSQL, MySQL and SQL Server wire protocols, Arrow Flight SQL) and the MongoDB wire protocol served from an engine app through published ports or links. - **The engine's Node.js edition in an image with Node.js.** PowerCloud runs any image whose programs tolerate its sandbox, but the Go executable is the one this page assumes: it needs no Node.js. #### What doesn't work yet, and why - **The standard release binary can write but not query as a binary app.** The engine reads documents and runs SQL through DuckDB. Its standard Linux releases are built without cgo and run the `duckdb` program beside them, but a PowerCloud binary app's root holds only its one binary and `/data`. Such a node can write documents, sync, hold and serve files, but cannot read documents back or run SQL. Use the in-process build (DuckDB linked in) or an image for anything that queries. - **Companion programs need an image.** `unidatum-sqld` (SQL wire serving), `unidatum-kafka` (the Kafka sink) and `unidatum-xfer` (fast bulk transfer) are separate programs that must sit beside `unidatum`, which a binary app can't do. - **No shell in an app.** PowerCloud has no command to run a one-off command inside an app. The engine's setup commands (`unidatum init`, `unidatum store attach`, `unidatum sync PEER`) must run from the image's entrypoint script, or as separate deploys of a binary app (each deploy under the same name keeps `/data`). - **No LAN or internet discovery in an app's own network.** An app's network holds only itself, so the engine's mDNS finds nothing, and its DHT needs `--egress` to reach anyone. Peers must be named explicitly, by link name. - **The engine can't tell clients apart.** PowerCloud passes each connection to a published port or link from inside the app's network, so the engine sees it coming from loopback (in a microVM, from the node's side of the guest's link). The engine always admits loopback, so its own address allow-lists and per-address limits (`.p2pfs/access.json`) don't work behind PowerCloud. Rules by node ID still apply. - **Memory is sized from the node, not the app.** The engine sizes DuckDB's memory from the machine's total RAM, which in an app is the node's, not the app's `--mem`. Set `P2PFS_DUCKDB_MEMORY` and `P2PFS_DUCKDB_RSS_MAX` yourself, or the app's memory limit kills large queries. - **A Kafka broker isn't part of the engine.** The engine ingests from an existing Kafka cluster (its Kafka sink), and has a Kafka-shaped producer of its own. It doesn't accept Kafka clients as a broker. ### How the two fit together | Need | PowerCloud provides | The engine provides | | --- | --- | --- | | Data model | | Document collections, tables in parquet, files, version history, sync between peers | | Encryption at rest on a machine | The node's encrypted state volume holds every app's `/data` | Signed, verified data; on its own it keeps documents and files on disk in plaintext | | Encryption at rest at a provider | Storage encrypted on the node before it leaves, replicated or erasure-coded across providers | The blob tier: every blob kept in an S3 bucket, with a local cache | | Network between machines | WireGuard mesh, NAT traversal, links between apps by name | Peer sync, transfer, distributed (scatter) queries | | Keeping it running | Sandbox, restarts, placement and failover of cluster apps | Peers that converge after any of them comes back | The main gains from running them together: - **Nothing in plaintext on any disk or bucket you don't hold the key to.** The engine's metadata, documents and file bytes sit on the node's encrypted state volume. Its bulk data, through the blob tier, lands in PowerCloud storage, where providers see only ciphertext without names. - **No single provider.** The bucket behind the blob tier isn't one provider's bucket: PowerCloud replicates or erasure-codes it across the locations your nodes serve (directories, S3-compatible buckets, your own nodes), so losing a provider or moving off one doesn't touch the engine. - **Two layers of failover.** The engine replicates between peers, so a peer that goes away is one copy less, not an outage. PowerCloud restarts a crashed engine, and can move an engine app to another node. - **Encrypted links between peers.** Links between apps on different nodes ride the WireGuard mesh, so the engine's own first contact with a peer, which it makes in plaintext before switching to its secure wire, is still inside an encrypted tunnel. ### Packaging the engine #### As a container image (recommended) An image can hold the engine, its companion programs, DuckDB (linked into the engine, or as the command-line program) and a start-up script. It is pulled by `pcctl` on your machine from a registry over HTTPS (private registries work with `docker login`). A sketch with the standard release build, so with `unidatum`, `unidatum-sqld` and a `duckdb` program for the image's architecture in the build directory (with the in-process build, leave out `duckdb` and `P2PFS_DUCKDB`): ```dockerfile FROM debian:stable-slim RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/* COPY unidatum unidatum-sqld duckdb /usr/local/bin/ COPY start.sh /start.sh ENV P2PFS_DUCKDB=/usr/local/bin/duckdb HOME=/data ENTRYPOINT ["/start.sh"] ``` The script prepares the library on first start, attaches the blob tier once, and runs the node. `POWERCLOUD_APP` is the app's name, which PowerCloud sets: ```sh #!/bin/sh set -e cd /data [ -f .p2pfs/config.json ] || unidatum init --library "$LIBRARY" --node-name "$POWERCLOUD_APP" if [ -n "$BLOB_ROOT" ] && [ ! -f .blob-tier-attached ]; then unidatum store attach "$BLOB_ROOT" && touch .blob-tier-attached fi for peer in $PEERS; do unidatum sync "$peer" || true; done exec unidatum ui --bind "" --port 47800 --no-mdns --no-dht --rendezvous off \ --advertise "$POWERCLOUD_APP:47800" "$@" ``` Attach only once: attaching again makes the engine forget which blobs the bucket holds. A peer that isn't up yet when the script runs isn't registered; restart the app later with `pcctl app restart NODE NAME`. Arguments after `--` in `pcctl app deploy` replace the image's command, which here are passed on to `unidatum ui` (such as `--sql` or `--mongo-port 27017`). Good to know about images: - PowerCloud ignores the image's `User`: the app runs as its own unprivileged user, which owns the image's files. - `/data` is the app's data directory, kept across restarts and redeploys. - The engine's auto-update (off by default) replaces its own binary; leave it off and update by deploying a new image. - In a microVM (`--vm`), the image's files count against the app's memory, and `/data` is shared into the guest from the node. The engine's metadata database, SQLite in write-ahead-log mode, hasn't been tried on that shared directory. #### As a static binary The engine runs as a binary app with no image. Which build decides what it can do: - **With DuckDB linked in** (`-tags duckdb_inproc`), it does everything a node does, queries included, from one file. A binary app must be fully static, so build DuckDB with the engine's `scripts/linux-duckdb.sh` for a musl toolchain with `PIC=OFF`, then build the engine against it. Linking DuckDB in also removes the two DuckDB process starts the standard build pays on every query. - **The standard release build** (no cgo) writes, syncs and serves files, but can't read documents back or run SQL, because its `duckdb` program can't sit beside it. Each step is a deploy under the same name, which keeps `/data`: ```sh pcctl app deploy --mem 512M edge-1 ud ./unidatum -- init --library plant --node-name edge-1 pcctl app logs edge-1 ud pcctl app deploy --mem 512M edge-1 ud ./unidatum -- ui --port 47800 --no-mdns --no-dht --rendezvous off ``` The first deploy creates the library, exits, and is restarted until the next deploy replaces it; a second `init` refuses to overwrite the library, so the restarts do no harm. A binary is for one architecture: build or fetch the one that matches the node. ### Use cases Each use case below is a design. The names `ud-home`, `ud-colo` and `ud-cloud` are engine apps; `home-1`, `colo-1` and `cloud-1` are nodes. #### Document database for your own apps **Setup.** One engine image app, and your application as another app linked to it. Your app talks to the engine's HTTP API, or to its MongoDB wire protocol with a MongoDB driver: ```sh pcctl app deploy --image registry.example.com/ud:latest --mem 2G --cpu 2 \ --env LIBRARY=apps --env P2PFS_DUCKDB_MEMORY=1GB --env P2PFS_DUCKDB_RSS_MAX=1600MB \ home-1 ud-home -- --mongo-port 27017 pcctl app deploy --link ud-home:7480 --link ud-home:27017 --port 8080:8080 home-1 api ./api ``` `api` then reaches `http://ud-home:7480` and `mongodb://ud-home:27017`. The engine's MongoDB users come from `P2PFS_MONGO_USERS`. **Why together.** The engine needs no published port at all, so its API is reachable only by the apps you linked. Its files are on the node's encrypted disk. **Sizing.** Writes need little memory; reads and queries run in DuckDB. 1 to 2 GiB is a reasonable start for a small database. Set `P2PFS_DUCKDB_MEMORY` to about 60% of `--mem` and `P2PFS_DUCKDB_RSS_MAX` to about 80%. **Caveats.** A link to an app deployed by hand works only on the same node. For a database that survives the node, add peers (below). #### Bulk data encrypted across providers **Setup.** The engine as a cluster app with `--storage`, its blob tier pointed at the app's own S3 endpoint: ```sh pcctl --repo LOC,... cluster app deploy --nodes colo-1 --storage --mem 2G \ --env P2PFS_S3_ENDPOINT=http://127.0.0.1:9000 --env BLOB_ROOT=s3://ud/colo \ --env P2PFS_BLOB_CACHE=20G --env LIBRARY=archive \ --image registry.example.com/ud:latest ud-colo ``` In a microVM the endpoint is `http://10.0.2.2:9000`. `P2PFS_BLOB_CACHE` bounds the blobs kept on the node; the rest are fetched back from storage when read. An engine outside PowerCloud (a workstation, a server you already run) can use the S3 gateway instead: ```sh pcctl --repo LOC,... s3 key add --bucket ud workstation pcctl s3 ca > powercloud-ca.pem cat /etc/ssl/certs/ca-certificates.crt powercloud-ca.pem > ca-bundle.pem export SSL_CERT_FILE=$PWD/ca-bundle.pem P2PFS_S3_ENDPOINT=https://s3.example.com:9443 export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... unidatum store attach s3://ud/workstation ``` `SSL_CERT_FILE` replaces the system's list, hence the combined bundle. On macOS and Windows, Go programs use the system's trust store instead, so the cluster CA would have to be trusted there; that hasn't been tried. See [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md). **Why together.** The engine's blobs (every file and every parquet member) are content-addressed and verified on fetch; PowerCloud encrypts them before any provider sees them, spreads them over several, and keeps bucket and object names private. With `--storage`, the blobs follow the app if it moves to another node. **Sizing.** Every engine blob is one object, sent in a single put; PowerCloud splits it into 4 MiB encrypted chunks, each stored once per replica. Provider request counts and bytes therefore multiply by the replication factor. Plan the local cache to hold the working set: a cache smaller than what queries read makes every query fetch again. **Caveats.** The blob tier holds blobs only. The engine's metadata (its database of documents' history, manifests and catalog) stays in `/data` on the node, so an app moved to another node finds its blobs but not its library; peers are what keep the library itself. `--volume` saves `/data` every 30 seconds as whole files, which isn't a safe copy of a live database. The blob tier only adds: run `unidatum store prune` to delete what nothing references. #### One library across home, colo and cloud **Setup.** One engine cluster app per node, each pinned with `--nodes`, each linked to the others on the sync port: ```sh pcctl --repo LOC,... cluster app deploy --nodes home-1 --mem 4G --cpu 2 \ --link ud-colo:47800 --link ud-cloud:47800 \ --env LIBRARY=fleet --env "PEERS=ud-colo:47800 ud-cloud:47800" \ --image registry.example.com/ud:latest ud-home -- --sql ``` Repeat for `ud-colo` on `colo-1` and `ud-cloud` on `cloud-1`, each linking the other two. Add `--link NAME:47800/udp` to carry the engine's QUIC too, or start it with `--quic-port -1` to use TCP only. Each engine advertises its app name, which the others resolve through their links. **Why together.** Links between nodes run over the mesh, encrypted, using the mesh's own NAT traversal, so no engine port is opened on a router or the internet. Every peer holds the library, so a node that goes away loses nothing; when it comes back, its engine catches up. **Sizing.** Each peer holds what its library holds, unless its blobs live in the blob tier. Links carry the engine's traffic through the nodes' own processes and the mesh; expect bulk transfer between peers to be slower than over a direct connection (not measured). **Caveats.** Use one app per node rather than `--replicas`: replicas share a name, so they can't be linked to each other. Registration in the engine goes one way; each script syncs to its peers at start, so start all three and restart any that started before the others were up. #### Analytics over private data in several places **Setup.** The peer group above, started with `--sql` so each peer answers its share of a distributed query, plus the SQL front ends in the image. A start script can start `unidatum sql-serve --pg-port 5433 --user U --password P` beside the node, and the app publishes or links that port. **Why together.** Data stays where it was produced (a home server, a colo box, a cloud VM), each copy on an encrypted disk, and queries go to the data. BI tools connect with their own PostgreSQL, MySQL or SQL Server drivers. **Sizing.** From the engine's own measurements: about 1 GB per node for a `GROUP BY` on an integer or a short text key, and about 3 GB plus 5.7 bytes per input row for text keys longer than 12 characters, with 8 DuckDB threads. Thread count changes memory a lot; set `P2PFS_DUCKDB_THREADS` and measure. DuckDB in the engine doesn't spill to disk: a query fits or runs out of memory. Give PowerCloud's placement the real figure with `--mem`. See [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md) for the nodes. **Caveats.** Querying parquet that sits directly in a bucket (the engine's store path) goes through DuckDB's own S3 support, which needs its httpfs extension and its own endpoint settings; that hasn't been checked against PowerCloud storage. #### File sharing and sync **Setup.** Files added to the library (through the engine's API, or on any other peer) sync to every peer as content-addressed blobs. Publish the sync port to let devices outside the cluster join the library, or let ordinary BitTorrent clients download from it: ```sh pcctl app deploy --image registry.example.com/ud:latest --port 47800:47800 --env LIBRARY=family home-1 ud-home ``` **Why together.** Every copy is on an encrypted disk, and with the blob tier the bulk lives in encrypted storage across providers rather than on any one machine. **Sizing.** A peer without the blob tier needs disk for the whole library. The engine's fast bulk transfer (`unidatum-xfer`) is a separate program that needs the C crypto library and a port of its own; without it, transfers use the engine's slower built-in path. **Caveats.** The sync port is the engine's front door: limit who can join by node ID in its access policy, since every connection reaches it from loopback. #### Event streams **Setup.** The engine's Kafka sink (`unidatum kafka-sink`, needing `unidatum-kafka` in the image) consumes topics from a Kafka cluster and lands each batch with its offsets in one atomic commit. With `--egress` it reaches an external Kafka; a Kafka broker run as another app is reached by a link. Without Kafka, producers write through `unidatum produce` or the HTTP API. **Why together.** Topics become queryable, encrypted tables, and the sink is restarted by PowerCloud if it stops. **Caveats.** One sink per topic. The engine isn't a broker, and a Kafka broker as a PowerCloud app hasn't been tried. #### Edge nodes **Setup.** A small node (an arm64 board or a branch-office box) runs an engine that writes locally and syncs to the peers at home or in a colo. It runs as one static binary: the in-process build if it should answer queries too, or the standard release build if it only produces data and leaves queries to bigger peers. Your collector app links to it on port 7480. **Why together.** The edge disk is encrypted, the edge node needs no open port (it reaches its peers through the mesh), and a stolen disk holds nothing readable. How much a stolen machine gives away depends on how it unlocks (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). **Sizing.** A write-only engine is light; leave room for DuckDB only on nodes that query. **Caveats.** For the edge node to reach peers on other nodes, both must be cluster apps (links to hand-deployed apps work only on the same node). ### Good to know - **Keep the engine's UI private.** Reach it through links from your own apps. If you publish port 7480 to use it from a browser, do it only on a node whose network you trust. - **Name peers explicitly.** Start the engine with `--no-mdns --no-dht --rendezvous off` in an app's own network, and give it `--advertise NAME:47800` so peers dial the link name. - **Set memory by hand.** `--mem` for PowerCloud, `P2PFS_DUCKDB_MEMORY` and `P2PFS_DUCKDB_RSS_MAX` for the engine. - **Consistency.** The engine's sync is conflict-free between its peers. PowerCloud storage under it is last-writer-wins per object, which suits the engine's content-addressed blobs: a blob's name is its content. Related pages: [Apps on a node](https://powercloud.unidatum.com/docs/apps.md), [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md), [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md), [Storage](https://powercloud.unidatum.com/docs/storage.md), [Costs and sizing](https://powercloud.unidatum.com/docs/sizing.md), [Security model](https://powercloud.unidatum.com/security). --- ## Storage URL: https://powercloud.unidatum.com/docs/storage > How PowerCloud's encrypted store works, how to set up storage locations on directories, S3-compatible buckets and your own nodes, and how to keep them in step with check, repair and gc. PowerCloud keeps the cluster's data in an encrypted, deduplicating object store spread over several storage locations: local directories, S3-compatible buckets, and the encrypted disks of your own nodes. Everything is encrypted by `pcctl` on your machine before it leaves, so every location holds only ciphertext, and losing any one location loses nothing because each object is replicated (or erasure-coded) across them. ### How the store works - **Objects are split into 4 MiB chunks**, each sealed with XChaCha20-Poly1305 under a key derived from the cluster root key. A chunk's ID is an HMAC of its content, so identical chunks are stored once (deduplication) without letting a provider confirm what is inside. - **Names live in encrypted manifests.** An object's name, size and chunk list are sealed too, under an ID that is an HMAC of the name. Providers see no names, only opaque blobs. - **Every read is checked.** Each chunk is decrypted and its MAC compared with its ID, so a provider that changes or swaps a blob is caught. - **The cluster header** is the one unencrypted blob. It holds the keyslots (passphrase and recovery key), which wrap the root key but contain no plaintext key material. The cluster root key never leaves your machine in the clear. Nodes that relay or hold storage never get it. For the key hierarchy and what a passphrase protects, see [Security model](https://powercloud.unidatum.com/security) and [Concepts](https://powercloud.unidatum.com/docs/overview.md). ### Storage locations A repository is a comma-separated list of locations, given with `--repo` or the `POWERCLOUD_REPO` environment variable: ```sh export POWERCLOUD_REPO='/srv/pc,s3://my-bucket/pc?region=us-east-2,node://home-1' pcctl ls # same as pcctl --repo "$POWERCLOUD_REPO" ls ``` Each location is one of: | Kind | Form | Notes | |---|---|---| | Local directory | `/srv/pc`, `./home` | Any path on the machine running `pcctl`. | | S3-compatible bucket | `s3://BUCKET[/PREFIX][?endpoint=URL®ion=REGION&creds=NAME&path_style=BOOL]` | AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and others. | | A node's disk | `node://NAME` | A replica on an enrolled node's encrypted state volume. | | Erasure-coded set | `ec:K:LOC;LOC;...` | Several of the above as one location; see "Erasure coding". | Quote `--repo` values in the shell whenever they contain `&` or `;`. #### S3 bucket options | Option | Meaning | Default | |---|---|---| | `endpoint` | `http(s)://HOST[:PORT]` of the service | AWS: `https://s3.REGION.amazonaws.com` | | `region` | Region used for request signing | `us-east-1` | | `creds` | Name of the credentials to use from the environment | `AWS` | | `path_style` | `true` for `https://HOST/BUCKET/...` requests, `false` for `https://BUCKET.HOST/...` | `true` when `endpoint` is given, otherwise `false` | A bucket name containing dots needs `path_style=true` over HTTPS. Unknown options are refused. #### Credentials Credentials never go in the location string. For `creds=NAME`, `pcctl` reads them from the environment: - `NAME_ACCESS_KEY_ID` - `NAME_SECRET_ACCESS_KEY` - `NAME_SESSION_TOKEN` (optional) `NAME` is upper-cased, so `creds=r2` reads `R2_ACCESS_KEY_ID`. Without `creds=`, the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are used. Each location can use its own provider's keys. A location whose credentials are missing fails with a message naming the variables to set. Use one bucket and one key scoped to that bucket per provider. Stolen storage keys let an attacker delete or corrupt that provider's copy, but not read it, and the other locations cover the loss. Where a provider offers bucket versioning or object lock, it protects that copy as well. #### Provider examples | Provider | Location | |---|---| | AWS S3 | `s3://my-bucket/pc?region=us-east-2` | | Cloudflare R2 | `s3://my-bucket/pc?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com®ion=auto&creds=r2` | | Backblaze B2 | `s3://my-bucket/pc?endpoint=https://s3.us-west-004.backblazeb2.com®ion=us-west-004&creds=b2` | | Wasabi | `s3://my-bucket/pc?endpoint=https://s3.eu-central-1.wasabisys.com®ion=eu-central-1&creds=wasabi` | | MinIO, Ceph RGW, Garage | `s3://my-bucket/pc?endpoint=https://minio.lan:9000&creds=minio` | A repository over a local disk, R2 and B2: ```sh export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... export B2_ACCESS_KEY_ID=... B2_SECRET_ACCESS_KEY=... pcctl --repo "/srv/pc,s3://my-r2-bucket/pc?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com®ion=auto&creds=r2,s3://my-b2-bucket/pc?endpoint=https://s3.us-west-004.backblazeb2.com®ion=us-west-004&creds=b2" init ``` The S3 client in `pcctl` needs no SDK: it uses PUT, GET, DELETE and ListObjectsV2, signed with AWS Signature Version 4, and signs the SHA-256 of every request body so a body changed in transit is rejected. Network errors, throttling (429) and server errors (5xx) are retried with backoff, up to 4 attempts. Authentication errors and missing buckets fail at once, and a missing bucket is never mistaken for a missing object. It is checked against real S3 servers (RustFS) in automated tests, and against AWS S3. ### Replication and the write quorum With several locations, every write goes to all of them and succeeds once a write quorum has accepted it. Reads try each location until one serves an authentic copy. - The default quorum is a majority: 1 of 1, 2 of 2, 2 of 3, 3 of 4 or 5. - `--quorum N` changes it for one command, for example to keep writing while a provider is down: ```sh pcctl --repo /srv/pc,s3://a/pc,s3://b/pc --quorum 1 put docs/a.pdf a.pdf ``` A location that missed writes is brought up to date later with `pcctl repair`, or by a serving node's scheduled sync (see "Storage through a node"). - **Newest version wins.** Reads fetch the manifest from every reachable location and use the newest one that authenticates, so a location that missed an overwrite never serves the old version. - **Deletes are tombstones.** `rm` writes an encrypted "deleted" manifest rather than removing the object, so a location that missed the delete can't bring the object back. - **Unreachable is not "not found".** An object missing from some locations while others are unreachable is reported as an error, since it may exist on the unreachable ones. ### Erasure coding Full replicas cost n times the data. An erasure-coded location spreads each blob over n providers as k data shards and n−k parity shards (Reed-Solomon). Any k shards rebuild it, so it survives losing n−k providers while storing n/k times the data: ```sh pcctl --repo 'ec:4:s3://a/pc?endpoint=...&creds=r2;s3://b/pc?endpoint=...&creds=b2;s3://c/pc?endpoint=...&creds=wasabi;s3://d/pc?region=us-east-2;node://home-1;node://colo-1' init ``` That is 4 data and 2 parity shards over six providers: any two can be lost, at 1.5 times the data instead of 3 times for three replicas. - `ec:K:LOC;LOC;...` separates its parts with `;`. Parts can be directories, S3 buckets or `node://` replicas. K must be at least 1 and less than the number of parts. Erasure-coded sets can't be nested. - The whole set is one location to everything else (replication, quorum, check, repair, gc, node sync), so it can stand alone or sit next to plain locations: `--repo 'ec:2:/a;/b;/c,/srv/pc'`. - A write must store k+1 shards (or all n, if fewer), so a fresh write survives losing one more provider; a write that stores fewer fails. A delete must reach every part. - No single provider is trusted: a reader decodes a version only when k shards agree on it, so a provider forging its shard is outvoted. - Lost shards rebuild themselves: a read rewrites the shard of any part that answered without a usable one, and `pcctl repair` and a serving node's sync heal every blob of each erasure-coded location first. ### Creating a repository and using it #### init ```sh pcctl --repo /srv/pc,s3://a/pc,s3://b/pc init ``` `init` asks for a new passphrase twice, writes the cluster header to every location and prints the cluster ID and a one-time recovery key: ```text Initialized cluster ... RECOVERY KEY (write it down and store it offline — it is shown only once): ... ``` Store the recovery key offline. Without the passphrase or the recovery key no one, including you, can read the data. `init` refuses a repository that is already initialised, and refuses to run if any location can't be reached, so it never writes a new header over an existing cluster it can't see. Every other command asks for the passphrase. For automation, set `POWERCLOUD_PASSPHRASE`, or `POWERCLOUD_RECOVERY_KEY` to unlock with the recovery key instead. #### put, get, ls, rm ```sh pcctl --repo "$POWERCLOUD_REPO" put docs/a.pdf ./a.pdf # NAME then FILE pcctl --repo "$POWERCLOUD_REPO" get docs/a.pdf ./out.pdf # FILE optional: stdout without it pcctl --repo "$POWERCLOUD_REPO" ls docs/ # PREFIX optional pcctl --repo "$POWERCLOUD_REPO" rm docs/a.pdf ``` `put` prints what it stored; `ls` prints size, modification time and name: ```text stored docs/a.pdf (482113 bytes, 1 chunks) ``` ```text 482113 2026-10-28 14:02 docs/a.pdf ``` Writing a name that exists replaces it. When two admins write the same name at the same time, the later one wins. ### check, repair and gc These commands work on each location separately, so they always need `--repo` (not `--via`). #### check ```sh pcctl --repo /srv/pc,s3://a/pc,s3://b/pc check [--verify] ``` `check` lists every location and reports, per location, missing, stale (older than the newest version) and corrupt copies: ```text 12 objects, 3 deleted, 40 chunks in use, 2 unreferenced (reclaimed by gc) LOCATION BLOBS MISSING STALE CORRUPT REPAIRED STATUS /srv/pc 55 0 0 0 0 ok s3://a/pc 51 4 0 0 0 needs repair s3://b/pc 0 0 0 0 0 unreachable: ... ``` It exits with an error when locations differ. `--verify` also downloads and authenticates every copy of every chunk, catching bit rot or tampering at rest; without it, chunks are checked by listing only (manifests are always read and authenticated). An object reported as `DAMAGED` has data missing from every reachable location. A `WARNING: ... has a different cluster header` means a location holds another cluster's header (or one left by an interrupted key rotation); it is reported, never changed. #### repair ```sh pcctl --repo /srv/pc,s3://a/pc,s3://b/pc repair [--verify] ``` `repair` writes the newest manifest, every chunk a live object uses, and the header wherever they are missing or outdated. Every copy is authenticated before it is written, so repair can't spread corruption. With `--verify`, corrupt chunk copies are found and replaced too. **Adding a provider** is the same operation: add its location to `--repo` and run `repair`. **Dropping a provider** means removing it from `--repo` (and from any node serving the storage). ```sh pcctl --repo /srv/pc,s3://a/pc,s3://new/pc repair # copies everything onto s3://new/pc ``` #### gc ```sh pcctl --repo /srv/pc,s3://a/pc,s3://b/pc gc [--grace 24h] [--dry-run] ``` `gc` deletes chunks no live object uses: from deleted objects, overwritten versions, or a `put` that failed partway. - A chunk is deleted only after it has been unreferenced for the grace period (default `24h`) across two runs. The first run records it, a later run deletes it if it is still unused. This protects chunks that a `put` in progress has uploaded but not yet listed. - `--dry-run` reports what would be deleted. - `gc` refuses to run unless every location is reachable, since a manifest on an unreachable one might use chunks that look unused elsewhere. ```text 40 chunks in use, deleted 2, 1 unreferenced waiting out the 24h0m0s grace period ``` Use a longer `--grace` if a single `put` can take longer than a day. ### Rollback protection Encryption stops a provider from forging or swapping objects, but not from serving an older, authentic version of an object, or hiding it. Because every location is untrusted, PowerCloud records the newest version of each object with your nodes instead. Once the nodes run their consensus group (`pcctl --repo ... consensus init`, see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)): - After every write, `pcctl` records the object's version time with the nodes. The record is keyed by the object's encrypted ID, so the nodes learn nothing about names. - On every read, `pcctl` compares what storage returned with that record. An object older than its record, or a recorded object that no location has, fails with `store: rollback`, naming both times. Deletions are covered the same way. - To recover, write the object again: the new version replaces the rolled-back one. - If no node with the consensus answers, `pcctl` warns `no rollback protection this time` and carries on. If only one node answers without a majority, it warns that the check uses that node's copy, which may miss recent writes. Chunks are covered through the manifest that lists them. The cluster header and the node registry are read before `pcctl` can find a node, so they are not covered. Before `consensus init` there is no rollback protection at all. ### Storage through a node With `--repo`, every admin machine needs every location and the credentials for each. An enrolled node can hold them instead, so other admins (or you on another machine) need only the passphrase and an admin certificate: ```sh export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... pcctl --repo "s3://...&creds=r2,s3://...&creds=b2,node://home-1" node storage home-1 # once pcctl --via home-1 put docs/a.pdf ./a.pdf # from then on ``` - **Syntax:** `pcctl --repo LOC,... node storage [--sync 1h|off] NAME [LOC,...]`. The node's locations default to `--repo`'s. Give them explicitly when the node reaches them by other addresses, such as a private endpoint or a LAN address. `--quorum` sets the node's write quorum. - **Only `s3://`, `node://` and `ec:` sets of them.** A node's root filesystem is in RAM, so it does not serve local directories. - **Credentials** named by `creds=` are taken from `pcctl`'s environment and stored on the node's encrypted state volume. They survive a reboot, a crypto-erase destroys them, and `pcctl node info` shows the locations but never the credentials. - **Checked before use.** The node refuses a location it can't reach, or one that holds another cluster's header, so a typo can't point admins at the wrong bucket. - **The node only relays ciphertext.** `pcctl` still unlocks the keys with the passphrase and encrypts and decrypts locally. A node serving storage sees what a storage provider sees. Only admin certificates may use these endpoints, so a compromised node can't read or overwrite storage through another node. - `POWERCLOUD_VIA` can stand in for `--via`. `--via` works for every command except `init`, `check`, `repair`, `gc`, `rotate` and `node storage`, which work on each location separately and need `--repo`. #### Scheduled sync A serving node copies every blob that some of its locations lack from one that has it: one minute after it starts, then every hour by default. A provider that was down during writes catches up without an admin. ```sh pcctl --repo ... node storage --sync 30m home-1 # sync every 30 minutes pcctl --repo ... node storage --sync off home-1 # no scheduled sync pcctl node sync home-1 # sync now pcctl node info home-1 # shows the storage and the last sync ``` Sync needs no keys: it works on opaque IDs and only fills gaps. Replacing a stale or corrupt copy takes decrypting to tell which copy is right, so that remains the job of `pcctl repair`. A location that can't be listed is left alone. ### Nodes as storage locations An enrolled node's encrypted state volume can hold a replica of the cluster's blobs, so your own home or colo machines are providers alongside the clouds: ```sh pcctl --repo /srv/pc,s3://a/pc,node://home-1,node://colo-1 repair # copy everything onto the nodes pcctl --repo node://home-1,node://colo-1,s3://a/pc put docs/a.pdf ./a.pdf pcctl --repo node://home-1,node://colo-1 node storage home-1 # then: pcctl --via home-1 ... ``` - `node://NAME` is a location like any other, reached with your admin certificate, under the same replication, quorum, check, repair and gc rules. - The blobs are encrypted twice: by the cluster's keys, which the node never has, and by the node's disk encryption. A crypto-erase of the node destroys the replica. A node whose state is only in RAM refuses to hold one. - A node serving storage reaches other nodes' replicas over the mesh with its own node certificate. - Nodes may read and write each other's replicas, but only admins may delete. A compromised node could overwrite blobs with garbage, which is caught on read and replaced by `repair`, but it can't erase another node's replica. - A write that would leave less than 256 MiB free on the node's state volume is refused, so a full replica never starves the node. - Connections to a node give up after 10 seconds, so a node that is down costs reads seconds, not minutes. ### What a provider can and can't see | A storage provider (or a node relaying storage) | | |---|---| | Can see | That blobs exist, their count, their encrypted sizes, when they are written, read and deleted, and the cluster header (which holds no plaintext keys). | | Can't see | Object names, file contents, which chunks belong to which file, or whether a chunk matches a file it already knows. | | Can't do undetected | Change, forge or swap a blob (caught on read), or roll back or hide an object once rollback protection is on. | | Can do | Delete or corrupt its own copy, or stop serving. Other locations cover that loss, and `repair` restores the copy. | Because each blob's size is visible, a provider can estimate the total amount of data stored, and the size of a small file that fits in one chunk. ### Limits - **Last writer wins.** Two admins writing the same name at once can't both win; the later modification time does. - **Tombstones are kept forever.** They are small, and removing them safely would need to know that every provider has seen them. - **gc assumes no `put` runs longer than the grace period.** Raise `--grace` for very slow uploads. - **Rollback protection needs the nodes' consensus** and a reachable node; otherwise `pcctl` warns and goes on without it. The cluster header and node registry are not covered. - **check, repair, gc, init and rotate need `--repo`** with every location, not `--via`. - **Responses over 256 MiB are refused** from S3 services, as a guard against a hostile service; chunks are far smaller. - **Losing the passphrase and the recovery key loses the data.** There is no backdoor. For an S3 endpoint over this storage that any S3 client can use, see [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md). For troubleshooting unreachable locations and rollback errors, see [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md). --- ## S3 gateway URL: https://powercloud.unidatum.com/docs/s3-gateway > How to serve the cluster's encrypted, replicated storage as an S3-compatible HTTPS endpoint from your nodes, manage its access keys, and use it from the AWS CLI and boto3. The S3 gateway gives the whole cluster one S3-compatible endpoint, served over HTTPS by the nodes you enable it on, so backups, media tools and programs written for S3 can keep their data in PowerCloud's encrypted, replicated storage. Every request must be signed with an access key you create with `pcctl`, each key can be limited to certain buckets or to reading, and storage providers still see only ciphertext. ### Before you start The gateway builds on two things a cluster may not have yet: 1. **Nodes that serve the storage.** Each gateway node reads and writes through the storage locations it serves, set with `pcctl --repo ... node storage NAME` (see [Storage](https://powercloud.unidatum.com/docs/storage.md)). A node that serves no storage can be enabled, but `s3 enable` warns and its gateway answers every request with `ServiceUnavailable`. 2. **The nodes' consensus group.** Access keys live in the nodes' replicated consensus state, so `pcctl --repo ... consensus init` must have been run (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). A gateway node whose consensus isn't running refuses requests with `ServiceUnavailable`. You also need the repository (`--repo` or `POWERCLOUD_REPO`) and the passphrase to enable the gateway and to create access keys, because the gateway's own storage key is kept in the cluster's encrypted store. ### Enable the gateway ```sh pcctl --repo LOC,... s3 enable [--port 9443] [--name DNS]... [--ip ADDR]... NODE... ``` | Flag | Meaning | Default | |---|---|---| | `--port` | Port the gateway listens on (HTTPS, all interfaces) | `9443` | | `--name` | A DNS name clients use to reach the gateway; put in its certificate. Repeatable. | none | | `--ip` | An address clients use, besides the node's own addresses; put in its certificate. Repeatable. Useful behind NAT or a load balancer. | none | For example, two nodes behind one DNS name: ```sh pcctl --repo "$POWERCLOUD_REPO" s3 enable --name s3.example.com home-1 colo-1 ``` ```text home-1: S3 gateway on port 9443, for s3.example.com, 192.0.2.10, ... (certificate until 2027-10-28) colo-1: S3 gateway on port 9443, for s3.example.com, 198.51.100.4, ... (certificate until 2027-10-28) Clients trust the cluster CA ('pcctl s3 ca > ca.pem', then AWS_CA_BUNDLE=ca.pem) and sign with an access key ('pcctl --repo ... s3 key add NAME'). ``` `s3 enable` issues each node a TLS server certificate from the cluster CA, valid for one year, for the node's own addresses plus the `--name`s and `--ip`s given. The certificate is good for serving TLS only: it grants nothing on the node API. Each node gets the gateway's storage key, which only gateway nodes receive. - **Renewing:** run `s3 enable` again on the same nodes before the certificate expires (`pcctl s3 status` shows days left). Give the same `--name` and `--ip` flags again, since the new certificate covers only what this run names. - **After `pcctl rotate`:** the cluster CA changes, so run `s3 enable` again and give clients the new CA certificate. - Any gateway node serves the same objects, so clients may use whichever node is up, or one DNS name pointing at several. To stop the gateway on nodes: ```sh pcctl s3 disable home-1 colo-1 ``` ```text home-1: S3 gateway stopped colo-1: S3 gateway stopped ``` ### Access keys #### Create a key ```sh pcctl --repo LOC,... s3 key add [--bucket B]... [--read-only] NAME ``` | Flag | Meaning | |---|---| | `--bucket` | A bucket the key may use. Repeatable. Without it, the key may use every bucket. | | `--read-only` | The key may only read (GET and HEAD requests). | `NAME` is a label for you, unique among the keys. For example, a read-write key for one bucket: ```sh pcctl --repo "$POWERCLOUD_REPO" s3 key add --bucket photos laptop ``` ```text access key PCXXXXXXXXXXXXXXXXXX for laptop; its secret isn't shown again: AWS_ACCESS_KEY_ID=PCXXXXXXXXXXXXXXXXXX AWS_SECRET_ACCESS_KEY=... ``` The secret (40 characters) is shown once; store it where the client will use it. The key is written to the nodes' consensus state with its secret sealed under a key derived from the gateway's, so every node keeps the entry but only gateway nodes can open it. It works on every gateway node as soon as that node applies the change, without `pcctl` reaching each one. A key limited to some buckets sees only those buckets when it lists, and gets `AccessDenied` for others. A read-only key gets `AccessDenied` for any write. #### List and remove keys ```sh pcctl s3 key ls pcctl s3 key rm laptop # by name or by access key ID ``` ```text NAME ACCESS KEY BUCKETS ACCESS CREATED backup PCYYYYYYYYYYYYYYYYYY all read-only 2026-10-20 laptop PCXXXXXXXXXXXXXXXXXX photos read-write 2026-10-28 ``` Removal takes effect on every gateway node as soon as it applies the change. A node that was down gets the change when it rejoins its peers. These commands need only your admin certificate, not the repository. ### Check the gateway ```sh pcctl s3 status ``` ```text NODE GATEWAY NAMES KEYS CERT colo-1 port 9443 s3.example.com 2 364d home-1 port 9443 s3.example.com 2 364d web-1 - ``` `GATEWAY` shows `-` for nodes without a gateway, `unreachable` for nodes that didn't answer, and `failed: ...` when a node's gateway couldn't start (for example, the port is taken). `KEYS` counts the access keys usable on that node. `CERT` shows the days until its certificate expires, or `expired`. ### Trust the CA certificate The gateway's certificate comes from your cluster CA, not a public authority, so clients must be told to trust it: ```sh pcctl s3 ca > ca.pem export AWS_CA_BUNDLE=$PWD/ca.pem ``` `pcctl s3 ca` prints the cluster CA certificate (PEM) from your admin credentials. `AWS_CA_BUNDLE` is read by the AWS CLI and boto3; most other tools take a CA file option. The CA certificate is public and safe to copy to client machines. After `pcctl rotate` it changes, so fetch it again. ### Use it from clients The endpoint is `https://HOST:9443`, where `HOST` is one of the names or addresses in the gateway's certificate (a `--name`, an `--ip`, or one of the node's own addresses). Requests are path-style: `https://HOST:9443/BUCKET/KEY`. The region can be any value; `us-east-1` is a safe choice. Requests must be within 15 minutes of the node's clock. #### AWS CLI ```sh export AWS_ACCESS_KEY_ID=PCXXXXXXXXXXXXXXXXXX AWS_SECRET_ACCESS_KEY=... export AWS_CA_BUNDLE=$PWD/ca.pem AWS_DEFAULT_REGION=us-east-1 aws --endpoint-url https://s3.example.com:9443 s3 cp --recursive ~/Pictures s3://photos/ aws --endpoint-url https://s3.example.com:9443 s3 ls s3://photos/ --recursive aws --endpoint-url https://s3.example.com:9443 s3 cp s3://photos/cat.jpg ./cat.jpg aws --endpoint-url https://s3.example.com:9443 s3 presign s3://photos/cat.jpg --expires-in 3600 ``` Uploads above the CLI's multipart threshold (8 MiB) go as multipart uploads, which the gateway supports. `--ca-bundle ca.pem` can replace `AWS_CA_BUNDLE`. #### boto3 ```python import boto3 s3 = boto3.client( "s3", endpoint_url="https://s3.example.com:9443", region_name="us-east-1", verify="ca.pem", aws_access_key_id="PCXXXXXXXXXXXXXXXXXX", aws_secret_access_key="...", ) s3.put_object(Bucket="photos", Key="a.jpg", Body=open("a.jpg", "rb")) s3.upload_file("big.mov", "photos", "big.mov") # multipart for large files data = s3.get_object(Bucket="photos", Key="a.jpg")["Body"].read() keys = [o["Key"] for o in s3.list_objects_v2(Bucket="photos").get("Contents", [])] s3.delete_object(Bucket="photos", Key="a.jpg") ``` #### Other clients The AWS CLI and boto3 are the clients checked in automated tests; others are untested. Any client you try must sign with AWS Signature Version 4 (presigned GET and HEAD may also use Signature Version 2) and use path-style addressing. The AWS CLI and boto3 use path style for a custom endpoint by themselves; other SDKs need it set, for example `UsePathStyle` in the Go SDK or `forcePathStyle` in the JavaScript SDK. Virtual-hosted style (`https://BUCKET.HOST/KEY`) is not supported. ### What is supported | Operation | Supported | |---|---| | ListBuckets | Yes: the buckets that hold objects, limited to a key's buckets | | Create bucket (PUT), HEAD bucket | Accepted; a bucket exists once an object is in it | | Delete bucket | Accepted, but does nothing; a bucket disappears when its last object is deleted | | PutObject, GetObject, HeadObject, DeleteObject | Yes | | ListObjectsV2 | Yes, with `prefix`, `delimiter`, `start-after`, `continuation-token` and `max-keys` (up to 1000 per page) | | Ranged GET | Yes: `bytes=A-B`, `bytes=A-` and `bytes=-N`. A request for several ranges gets the whole object | | Multipart uploads | Yes: create, upload part (parts 1 to 10000), complete, abort | | Presigned URLs | Yes, valid for up to 7 days; for GET and HEAD, also as version 1 of the AWS CLI presigns them (Signature Version 2) | | CopyObject | No (`NotImplemented`) | | DeleteObjects (several keys in one request) | No | | Versioning, ACLs, bucket policies, listing in-progress multipart uploads | No | #### Integrity checks on uploads The gateway checks every body as the client declared it before storing anything of it: its signed SHA-256, an unsigned payload, or `aws-chunked` uploads with each chunk signed or not and checksums (CRC32, CRC32C, CRC64NVME, SHA-1, SHA-256) in trailing headers, which is how current AWS SDKs upload. `Content-MD5` and `x-amz-checksum-*` headers are checked too. A body that doesn't match fails, and nothing of it is kept. A request without a valid signature gets `403` and nothing else, not even whether a bucket exists. A wrong secret gets `SignatureDoesNotMatch`. ### How the data is stored - **A namespace of its own.** Gateway objects are encrypted under the gateway's own key and stored under blob IDs of their own. They don't appear in `pcctl ls`, and files written with `pcctl put` don't appear in the gateway. The cluster's own `check` and `gc`, and the purge of old blobs after `pcctl rotate`, leave them alone. The gateway's key itself is kept in the cluster's store. - **Same storage, same guarantees.** Objects go through the storage the gateway node serves, so they are replicated or erasure-coded over the same locations, with rollback protection through the nodes' consensus. - **Providers see ciphertext only**, with neither bucket nor object names. - Chunks of deleted or overwritten gateway objects are reclaimed by the gateway nodes during their storage sync. ### Limits - **Path-style requests only.** Virtual-hosted style isn't supported. - **No copy, multi-object delete, versioning, ACLs, policies, or listing of in-progress multipart uploads.** Abort an unfinished multipart upload from the client that started it. - **Revocation follows the consensus.** A gateway node cut off from a majority of the other nodes keeps the access keys it last had, a removed one included, until it rejoins. If a key leaked and a node may be cut off, stop that node's gateway with `pcctl s3 disable NODE`. - **Last writer wins.** Two gateways writing the same key at the same moment leave one of the two versions. - **Certificates last a year.** Run `s3 enable` again to renew, and after every `pcctl rotate`. - **Clock skew.** Signed requests must be within 15 minutes of the node's clock. For node setup, consensus and the mesh, see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md). For an S3 endpoint that an app gets privately on its own node, see [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). --- ## Apps on a node URL: https://powercloud.unidatum.com/docs/apps > How to deploy, inspect and remove sandboxed apps (static binaries or container images) on one PowerCloud node, with their limits, ports, links, egress and microVMs. An app is a service you deploy to a PowerCloud node with `pcctl app`: a static Linux binary or a container image, run in a sandbox and kept running across crashes and reboots. This page covers apps deployed by hand to one node; to let the nodes choose where an app runs and move it when a node goes away, see [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). ### What an app is An app is one of two things: - **A static Linux binary.** The app's root holds only the binary and its data directory, with no libraries or dynamic loader, so the node refuses dynamically linked ELF files and anything that is not ELF. For Go, build with `CGO_ENABLED=0`. The binary must match the node's architecture (x86-64 or arm64). - **A container image** (`--image`). `pcctl` pulls the image from its registry on your machine, not on the node. It takes the image for the node's architecture, checks every manifest, config and layer against its sha256 digest, applies the layers in order and sends the node the flattened files. The command is the image's entrypoint and command, the image's environment and working directory apply, and the node records the image by digest. Public images (Docker Hub, GHCR and others) are pulled with an anonymous token. Private images use the login that `docker login` saved in `~/.docker/config.json` (or `$DOCKER_CONFIG`). Docker credential helpers are not used. Each app lives in a directory on the node's encrypted state volume, holding its spec, its files, its data and its log. Only admins can deploy, remove, restart or read the logs of an app: the commands use your admin certificate, like every other change to a node. The `pcctl app` commands talk to the node directly and need no `--repo`. ### Deploy an app The command has two forms, one for a binary and one for an image: ```sh pcctl app deploy [options] NODE NAME FILE [-- ARGS...] pcctl app deploy --image NAME[:TAG][@DIGEST] [options] NODE NAME [-- COMMAND...] ``` `NODE` is a node's name as `pcctl node ls` shows it. `NAME` is the app's name: 1 to 32 lowercase letters, digits and dashes, starting with a letter or digit. Options go before `NODE`. A binary with a memory limit, a CPU cap, an environment variable and its own arguments: ```sh pcctl app deploy --mem 128M --cpu 0.5 --env MODE=prod home-1 web ./web -- --port 8080 ``` ```text app web deployed to home-1 (6815744 bytes) NAME STATE UPTIME RESTARTS OOM KILLS MEMORY CPU SECCOMP NETWORK LAST EXIT web running 0s 0 0 0/128 MiB 0.00/0.50 on own ``` An image, with node port 80 published to the app's port 80: ```sh pcctl app deploy --image nginx:alpine --port 80:80 home-1 site ``` For an image, arguments after `--` replace the image's command (the entrypoint stays). For a binary, everything after `FILE` (with or without `--`) is passed to the binary as arguments. Deploying again under the same name replaces the app. It keeps its data directory and its user, so its files stay its own. #### Options | Option | Default | What it does | | --- | --- | --- | | `--mem SIZE` | `256M` | Memory limit, with `K`, `M` or `G`; at least 4 MiB. There is no swap. | | `--cpu N` | none | CPU limit in CPUs, such as `0.5` or `2`. Without it the app may use any CPU left over. | | `--procs N` | 256 | Limit on processes and threads. | | `--env K=V` | | An environment variable. Repeatable. | | `--port NODE:APP` | | Publish the app's TCP port `APP` on the node's port `NODE`. `NODE:APP/udp` publishes a UDP port. Repeatable. | | `--link NAME:PORT` | | Let the app reach app `NAME`'s TCP port `PORT` by its name. `NAME:PORT/udp` for UDP. Repeatable, up to 32. | | `--egress` | off | Let the app connect out through the node. | | `--net host` | own network | Share the node's network instead of having a network of its own. | | `--vm` | off | Run the app in a microVM, under a kernel of its own (nodes with KVM). | | `--image REF` | | Deploy a container image instead of a binary. | Some options cannot be combined: `--net host` excludes `--port`, `--link`, `--egress` and `--vm`, since all of those work on the app's own network. #### What the app is told Besides its `--env` variables, every app gets `POWERCLOUD_APP` (its name) and `POWERCLOUD_NODE` (its node's name). The replicas of a cluster app can use these to tell themselves apart. An `--env` of the same name wins over both. #### Its data directory An app's data directory is `/data` as the app sees it. What the app writes there is on the node's encrypted state volume and is kept across restarts, redeploys and reboots. Everything outside `/data` (the binary, or the image's files) is replaced on each deploy. ### The sandbox Every app runs in a sandbox of its own. In brief: - **Filesystem.** The app is chrooted into its own root, so it sees only its binary (or its image's files) and its data: no node state, keys or devices. It gets its own `/proc` (showing only its processes), a `/dev` with `null`, `zero`, `full`, `random` and `urandom`, and `/dev/shm` and `/tmp`. The apps directory is mounted `nosuid` and `nodev`, and setuid bits, device nodes and FIFOs are dropped from images. - **User.** It runs as a user of its own (numbered from 20000, assigned at deploy) with no capabilities and no supplementary groups, so it cannot write its own binary or another app's data. An image's `User` setting is ignored, so images that expect to run as root must tolerate running as an ordinary user. - **Processes.** It gets new PID, mount and UTS namespaces. It is PID 1 of its own process tree, its hostname is its name, and everything it starts dies with it. - **Limits.** It is placed in its own cgroup as it starts, which caps its memory, CPU time and number of processes. - **System calls.** It runs under a seccomp filter with `no_new_privs`. The filter refuses calls from another CPU ABI, and returns `EPERM` for mounts, `chroot`, `pivot_root`, `unshare`, `setns` and namespace-creating `clone`; `ptrace` and `process_vm_*`; `bpf`, `perf_event_open`, keyrings, `userfaultfd` and `io_uring`; module loading, kexec, `reboot` and swap; setting the clock; and open-by-handle. `clone3` gets `ENOSYS`, so C libraries fall back to `clone`. - **Network.** By default it has a network of its own, with only loopback: it cannot reach the node's network, the mesh or the internet, and nothing reaches it but its published ports and the links other apps were given to it. It may bind ports below 1024. `pcctl app ls` shows `on` in the `SECCOMP` column for an app running under the filter. ### Networking #### Publish ports `--port NODE:APP` makes the node listen on port `NODE` and pass each TCP connection to port `APP` on the app's loopback. `--port NODE:APP/udp` does the same for UDP: each client address gets its own session to the app's port, closed after two minutes idle, with at most 1024 open at once. ```sh pcctl app deploy --image coredns/coredns --port 53:53/udp --port 53:53 home-1 dns ``` The app has no address of its own. Users reach it at the node's address on the published port. #### Connect out with --egress `--egress` gives the app a way out while keeping its own network. Its traffic goes out through the node, masqueraded as the node, to anything outside it. It still cannot reach the node itself (its API, its storage, other apps' published ports) or other apps. The app gets the node's name servers in `/etc/resolv.conf`. Egress is IPv4 only. #### Share the node's network with --net host `--net host` puts the app on the node's own network, with all of its access. It is the coarse option: there is nothing in between, such as reaching only some hosts. Prefer `--port`, `--egress` and `--link`. #### Link apps by name `--link NAME:PORT` lets the app reach another app, `NAME`, on its TCP port `PORT`, by that name, without the other app publishing anything. Inside the app, `NAME` resolves (through its `/etc/hosts`) to an address in the app's own network where the node picks up each connection and passes it to the target's port. ```sh pcctl app deploy --image postgres:alpine --mem 512M home-1 db pcctl app deploy --link db:5432 --port 8080:8080 home-1 api ./api ``` The `api` app then connects to `db:5432`. Add `/udp` for UDP (`--link dns:53/udp`); each of the app's sockets gets a session, closed after two minutes idle, with at most 1024 at once. UDP to a link address's other ports is refused at once, so an app trying QUIC to a peer it has only a TCP link to falls back to TCP. Good to know about links: - An app reaches only the apps and ports it was linked to. - The app may serve the same port itself, so two peer-to-peer apps can be linked to each other on the port they both use. - A link to an app deployed by hand works only on the same node. A link to a cluster app works wherever the cluster placed it, over the encrypted mesh, and follows it when it moves (see [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md)). ### See, restart and remove apps ```sh pcctl app ls home-1 pcctl app logs home-1 web pcctl app logs home-1 web 1048576 pcctl app restart home-1 web pcctl app rm home-1 web pcctl app rm --keep-data home-1 web ``` - `app ls NODE` lists the node's apps: state, uptime, restarts, memory kills, memory used against the limit, CPU used against the cap, whether seccomp is on, the network (`own`, `host` or `microvm`, with `+egress`, `->NAME:PORT` for links and `NODE->APP` for published ports) and the last exit. - `app logs NODE NAME [BYTES]` prints the end of the app's output, 64 KiB by default. Standard output and standard error both go to the log, which is rotated at 1 MiB. Programs that reopen `/dev/stdout` or `/dev/stderr` (nginx, for one) work. - `app restart NODE NAME` stops the app and starts it again. - `app rm NODE NAME` stops the app and deletes it with its data. With `--keep-data`, the data directory is kept, and the app gets its user and its files back if deployed again on that node under the same name. The web console (`pcctl console`) shows the same per node, and can deploy, restart and remove apps from a form; see [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md). ### Kept running An app that exits, or is killed (by its memory limit too), starts again after a delay that doubles from 1 second up to a minute. The delay starts again from 1 second once the app has run for more than a minute. `RESTARTS` and `OOM KILLS` in `pcctl app ls` count these, and `LAST EXIT` says how it last ended. Apps start again by themselves when the node boots and is unlocked; until then the state volume, and so every app, is locked (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). ### MicroVMs By default, apps share the node's kernel, so a kernel bug an app can reach is a way out of its sandbox. `--vm` runs the app under a kernel of its own, in a microVM, with the CPU's hardware virtualization as the boundary. Use it for code you trust less than the node. ```sh pcctl app deploy --vm --cpu 2 --mem 512M --image nginx:alpine --port 80:80 home-1 site ``` Nothing else about the app changes: it is deployed, listed, logged, restarted and removed like any app, from a binary or an image, and its microVM runs inside the same sandbox (namespaces, chroot, cgroup, its own user, seccomp). A guest escape lands in that sandbox. Inside the guest, the app again runs as its own user with no capabilities, under the same seccomp filter. A guest boots, runs its app and reports its exit in about a second. What is different: - **It needs KVM.** A node runs microVMs where its CPU offers virtualization to it: bare metal (on AWS, x86 and Graviton `.metal` instances), or a VM with nested virtualization. Most cloud VMs do not. On a node without KVM, the deploy fails with "this node can't run microVM apps (it needs KVM)". - **vCPUs come from `--cpu`.** The guest gets a vCPU for each CPU of its limit, rounded up (`--cpu 1.5` gives two), or, with no `--cpu`, one for each of the node's CPUs, up to 32. The CPU cap applies to the vCPUs together. - **Memory includes its files.** The guest's root filesystem is in its memory, so the app's files (but not its data directory) count against `--mem`, as does the guest kernel's 64 MiB. Images of a few hundred MiB are fine; multi-GiB ones are not (2 GiB at most). Files the app writes outside `/data` are in that memory too, and are gone when the guest stops. - **Data stays on the node.** The data directory is shared with the guest at `/data`, not copied in, so what the app writes there is on the node and survives the guest. - **Network.** The app always has its own network (not `--net host`). The guest is at 10.0.2.15 on a private link to the node, which is at 10.0.2.2. Published ports, `--egress` and links work as for any app; linked apps get addresses from 10.0.2.100 onwards. - `app ls` shows `microvm` in the `NETWORK` column. The guest kernel's serious messages go to the app's log, without its boot messages. ### Limits - **One node, by hand.** An app runs where you deploy it. For placement, replicas and failover, use [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). - **No outbound rules.** Egress reaches anything outside the node; there is no list of allowed hosts. Egress is IPv4 only. - **A denylist.** The seccomp filter refuses known-dangerous calls and allows the rest, rather than allowing only a known-safe set. - **The node's kernel.** Apps share it unless deployed with `--vm`. Images run without user namespaces, as one unprivileged user. - **One user per app.** An app runs as its own user, whatever its image asks for. --- ## Cluster apps URL: https://powercloud.unidatum.com/docs/cluster-apps > How to publish apps for the whole PowerCloud cluster, how the nodes place them and fail them over, and how to use exclusive apps, app storage, app volumes, the nodes' consensus and the kv commands. A cluster app is an app you publish for the whole cluster rather than deploy to one node: the nodes decide among themselves where it runs, within each node's memory and CPU, and start it elsewhere when a node goes away. This page covers `pcctl cluster app`, exclusive apps, app storage and volumes, and the nodes' consensus with its `pcctl kv` commands. What an app is and the sandbox it runs in are described in [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). ### Publish a cluster app Cluster app commands change the cluster's signed state, so they need the cluster's keys: pass the storage locations with `--repo` (or set `POWERCLOUD_REPO` once in your shell), or reach the storage through a node with `--via NODE`. See [Storage](https://powercloud.unidatum.com/docs/storage.md). ```sh export POWERCLOUD_REPO=s3://my-bucket/cluster,/mnt/backup/cluster ``` The command has two forms, one for a static binary and one for a container image: ```sh pcctl --repo LOC,... cluster app deploy [--replicas N | --exclusive] [--nodes a,b] [--storage] [--volume] [app options] NAME FILE [-- ARGS...] pcctl --repo LOC,... cluster app deploy [--replicas N] [--nodes a,b] [app options] --image REF NAME [-- COMMAND...] ``` Two replicas of an image, on any nodes, with node port 80 published: ```sh pcctl --repo LOC,... cluster app deploy --replicas 2 --image nginx:alpine --port 80:80 site ``` A binary that may run only on two named nodes: ```sh pcctl --repo LOC,... cluster app deploy --nodes home-1,colo-1 --port 8080:8080 api ./api ``` ```text cluster apps 4 published (2 apps) home-1: updated colo-1: updated edge-1: not reached; it gets the change from its peers ``` Each publish is numbered. Nodes that `pcctl` cannot reach catch up from their peers over the mesh, so a node that was off picks up the change when it comes back. Publishing again under the same name replaces the app's definition everywhere. #### Options The cluster options: | Option | Default | What it does | | --- | --- | --- | | `--replicas N` | 1 | How many nodes run the app. | | `--nodes a,b` | any | The nodes that may run it, by name, comma-separated. | | `--exclusive` | off | Run the app on at most one node at any time, through the nodes' consensus. Needs `consensus init`; has one replica. | | `--storage` | off | Give the app an S3 endpoint backed by the cluster's encrypted storage. | | `--volume` | off | Keep the app's data directory in the cluster's storage, so it follows the app. Needs `--exclusive`. | | `--image REF` | | Deploy a container image (`NAME[:TAG][@DIGEST]`) instead of a binary. | The app options are the same as for `pcctl app deploy`: `--mem`, `--cpu`, `--procs`, `--env`, `--port`, `--link`, `--egress`, `--net host` and `--vm`. They are described in [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). Each node that runs the app applies them. #### Binaries, images and architectures An app's files are stored as artifacts named by their SHA-256: the static binary, or the image flattened to one tar for each architecture. `pcctl` sends each node it reaches its architecture's artifact; any other node fetches it from a peer and checks the hash. Artifacts nothing uses any more are deleted. - For an image, `pcctl` pulls one copy for each architecture among the nodes it reaches (only the `--nodes` ones, if given), on your machine. - A binary is for one architecture only. In a cluster that mixes x86-64 and arm64 nodes, use `--nodes` to keep a binary app on nodes that can run it, or publish an image. ### See and remove cluster apps ```sh pcctl --repo LOC,... cluster app ls pcctl --repo LOC,... cluster app rm site ``` ```text NAME REPLICAS NODES RUNNING ON SOURCE api 1 home-1,colo-1 colo-1 (running) binary site 2 any home-1 (running), edge-1 (running) registry-1.docker.io/library/nginx@sha256:... ``` `SOURCE` is `binary` or the image by digest, followed by `+storage`, `+volume` and `+exclusive` where they apply. `cluster app rm NAME` takes the app away from every node. On each node, a cluster app also shows in `pcctl app ls NODE`, with its logs in `pcctl app logs NODE NAME`. The node marks it as managed: change it with `pcctl cluster app`, not by hand. Apps deployed by hand with `pcctl app deploy` live alongside cluster apps, and the cluster leaves them alone. `pcctl status` reports cluster apps that are not running where they should, including an app that no node has the memory or CPU for. ### Placement and failover - **What should run is signed by an admin.** The cluster's apps are one CA-signed document, kept in the cluster's storage and passed between nodes over the mesh. Only admins write it, so nodes need no agreement on what should run. - **Where it runs is computed by every node alike.** For each app, every node ranks the eligible nodes (`--nodes`, or all) it considers alive, by a hash of the app's name and the node's ID (rendezvous hashing), and runs the app if it is among the first `--replicas`. Losing a node moves only that node's apps. - **Within each node's memory and CPU.** Each node offers apps its RAM less a reserve for itself (the larger of 256 MiB and an eighth), and its CPUs less a reserve (the larger of a quarter CPU and an eighth). An app takes its `--mem` and its `--cpu` on every node it runs on; an app without `--cpu` takes no CPU in placement. Exclusive apps are placed first, then the others in name order, each skipping nodes without room. An app that fits nowhere runs nowhere, rather than overcommitting a node, and `pcctl status` says so. - **MicroVM apps** (`--vm`) go only to nodes with KVM, and count at their memory limit plus 96 MiB. - **Failover.** A node that its peers cannot reach for a minute stops counting, and its apps start on the next nodes in line. When it comes back it takes its apps back, and the others stop their copies. - **Local data stays behind.** An app moved away keeps its data directory on the node it left (and its user, so the files stay its own), in case it returns. To have data follow the app, use `--storage` or, for an exclusive app, `--volume`. - **After a reboot**, a node leaves cluster apps stopped until it has checked again where they should run. Every app is told `POWERCLOUD_NODE`, so replicas can tell themselves apart. Links to a cluster app (`--link NAME:PORT` on another app) reach it on whichever node it was placed, over the encrypted mesh, and follow it when it moves. ### Exclusive apps Rendezvous placement can briefly run an app twice when nodes see each other differently, for example during a network split. `--exclusive` runs an app on at most one node at a time, decided through the nodes' consensus: ```sh pcctl --repo LOC,... consensus init pcctl --repo LOC,... cluster app deploy --exclusive --mem 512M --port 5432:5432 --image postgres:alpine db ``` - **The assignment is agreed.** The consensus state records each exclusive app's node, and only the Raft leader changes it. It moves an app off a node it has not reached for a minute, to the eligible node first in rendezvous order, and the app stays there when the old node returns. - **Fencing.** A node runs an exclusive app only while its copy of the agreed state assigns it the app and was confirmed current within the last 15 seconds. A node cut off from the majority stops the app within about 20 seconds, long before the leader moves it. - **Moving off a live node** (when you change `--nodes`) first clears the assignment, so the node stops the app, and assigns it elsewhere 15 seconds later. Good to know: a node that freezes (a paused VM, a long stall) and wakes does not know how long it was away. It keeps its copy until it hears the new assignment, a few seconds after it wakes, so two copies can overlap that long. Apps that must never overlap need their own fencing on top, such as conditional writes to their storage. ### App storage (--storage) A cluster app deployed with `--storage` gets an S3 endpoint inside its own network, at `127.0.0.1:9000` (at `10.0.2.2:9000` in a microVM). `AWS_ENDPOINT_URL` and dummy credentials are set for it, so AWS SDKs and the AWS CLI find it unchanged. Behind it is the cluster's storage: encrypted, and replicated or erasure-coded across the locations the nodes serve (see [Storage](https://powercloud.unidatum.com/docs/storage.md)). When the app moves, it finds its objects again on the new node. ```sh pcctl --repo LOC,... cluster app deploy --storage --port 8080:8080 counter ./counter ``` - **Its own key and namespace.** Each app has a random 256-bit key, kept in the cluster's encrypted store. Its objects are encrypted under keys derived from it, under IDs of their own, so another app cannot read them and storage providers do not see app names. - **The key goes only where the app may run.** `pcctl` sends it to the eligible nodes it reaches; another eligible node can fetch it from a peer, which gives it only to a node the signed document allows. Eligible nodes fetch the key and the app's files ahead of time, so a failover does not depend on the node that just went away. - **What it speaks.** Path-style buckets and objects (put, get with ranges, head, delete), ListObjectsV2 with prefixes, delimiters and paging, and multipart uploads, with S3's ETags. That is enough for `aws s3 mb`, `cp`, `ls` and `rm` and for SDKs. There is no copy, versioning or ACLs. Only the app's own network reaches the endpoint, so request signatures are not checked. - **Cleanup.** Nodes that serve storage reclaim the chunks of an app's deleted objects during their storage sync (hourly by default), with a 24-hour grace for uploads in progress. A cluster where no node serves storage does not reclaim them. `--storage` cannot be combined with `--net host`. ### App volumes (--volume) An exclusive app deployed with `--volume` keeps its data directory (`/data`) in the cluster's storage, so its ordinary files follow it to another node: ```sh pcctl --repo LOC,... cluster app deploy --exclusive --volume --port 8080:8080 counter ./counter ``` - **Saved as it runs.** Every 30 seconds, the node running the app saves the directory, uploading only files whose size, time or mode changed. A save cut short leaves the previous one whole. - **And as it stops.** A node stopping the app saves once more after it exits, unless another node has taken the app since. Moving an app off a live node loses nothing; a node that goes away loses at most its last 30 seconds. - **Restored before it starts.** A node starting the app downloads the directory if its own copy is older, into a new directory that it swaps in. Files belong to the app's user, with their modes and times. - **Only the holder writes.** Saves come only from the node assigned the app while it may run it, so a node waking from a freeze does not overwrite newer data. - **Encrypted and rollback-protected.** The volume is encrypted under a key derived from the app's storage key, and protected against a provider serving an older version through the nodes' consensus. - **Symlinks are saved as links**, never followed, so an app cannot make the node copy its own files into the volume. Limits of volumes: - **Snapshots, not replication.** A node that goes away loses up to 30 seconds of changes, and files are not saved at one instant together. Apps whose files must stay consistent with each other, such as databases, should keep their state in `--storage`, or checkpoint it. - **Whole files.** A changed file is uploaded again entire (chunks already stored are deduplicated). - Owners, extended attributes, hard links and special files (sockets, FIFOs) are not kept. ### Consensus among the nodes Some state must be agreed by the nodes themselves, in one order, and survive any minority of them going away: exclusive apps' assignments, and the marks that catch storage rolled back to an older version. For that the nodes form a Raft group replicating a key-value state, with each node's log on its encrypted state volume. #### Found the group An admin founds the group once: ```sh pcctl --repo LOC,... consensus init pcctl --repo LOC,... consensus init --nodes home-1,colo-1,edge-1 ``` ```text consensus founded with 3 voters: colo-1, edge-1, home-1 colo-1: started edge-1: started home-1: started ``` `--nodes` names the first voters; by default every node is one. `consensus init` signs a document naming them, which spreads over the mesh like the other signed documents. It can run only once: afterwards it says the group was founded already. After that, voters follow the membership list by themselves. The leader adds a member it has reached within 30 seconds, and removes voters that are no longer members or were revoked, one change at a time. A node enrolled later becomes a voter by itself and catches up from the leader. #### Check the group ```sh pcctl consensus status ``` ```text NODE ROLE TERM LEADER APPLIED KEYS VOTERS colo-1 Follower 3 home-1 1842 12 colo-1,edge-1,home-1 edge-1 Follower 3 home-1 1842 12 colo-1,edge-1,home-1 home-1 Leader 3 home-1 1842 12 colo-1,edge-1,home-1 ``` Each node shows its role, term, leader, applied index, number of keys and voters, or `unreachable`, or `not started` if it has no group yet. `consensus status` needs only the admin certificate, no `--repo`. #### Read and write keys `pcctl kv` reads and changes the agreed state through a node. Flags go before the subcommand: ```sh pcctl kv put config/mode prod pcctl kv get config/mode pcctl kv ls config/ pcctl kv rm config/mode pcctl kv --node home-1 --stale get config/mode ``` - `put KEY VALUE` returns once the change is applied on the node asked, after a majority of voters have it. Without a majority it fails after 10 seconds. - `get KEY` prints the value; `ls [PREFIX]` prints `KEY=VALUE` lines. Reads are linearizable: they see every change made before they started, through whichever node. Without a majority, a read fails like a change. - `--stale` answers from the node's own copy instead, which works without a majority but may miss recent changes. - `--node NAME` picks the node to ask; by default it is the first that answers with a group. Keys and values are plain strings in one flat map. The nodes themselves keep keys there too (exclusive apps' assignments under `assign/`), so do not change keys you did not write. #### Limits of consensus - **Two voters tolerate no loss.** A majority of two is both, so a two-node cluster stops taking changes, and exclusive apps stop, when either node is down. Three or more voters tolerate a loss. `consensus init` warns about two. - **The group cannot be refounded** with `pcctl` once founded. Losing a majority of voters for good needs a manual recovery. ### Limits - **No consensus on liveness, unless exclusive.** For ordinary cluster apps, each node judges liveness from its own view. A node cut off from the others, but not from users, may run an app another node also runs. That is fine for stateless apps; use `--exclusive` for the others. - **No locking across nodes.** Two copies of an app (during a split, or with several replicas) see the same `--storage` objects, last writer wins, as with S3 itself. - **Limits, not use.** Placement counts apps' memory limits and CPU caps, not what they use. It ignores apps deployed by hand with `pcctl app`, and apps without a CPU cap share the CPU left over. - **Local data moves only with a volume.** Without `--volume`, an app's `/data` stays on the node it ran on. --- ## Managing nodes URL: https://powercloud.unidatum.com/docs/nodes > Day-to-day operation of PowerCloud nodes with pcctl, including status, node info, the web console, power, storage sync, changing addresses, the mesh, NAT traversal, relays and how pcctl finds and authenticates to nodes. This page covers running PowerCloud nodes once they are enrolled: seeing their health, opening the web console, rebooting them, and keeping the mesh working when addresses change. Most of these commands need only your admin certificate, not the cluster passphrase. The ones that change the cluster's records (addresses, endpoints) unlock the cluster with `--repo` or `--via`. A node is a machine running PowerCloud OS that has joined your cluster (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). Terms such as cluster, repository and admin certificate are defined in [Concepts](https://powercloud.unidatum.com/docs/overview.md). Every command and flag is listed in [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md). ### How pcctl finds and authenticates to nodes `pcctl` keeps its credentials and its list of nodes in a local configuration directory: - `$POWERCLOUD_CONFIG` if that variable is set. - Otherwise the operating system's user configuration directory plus `powercloud`: `~/.config/powercloud` on Linux (or `$XDG_CONFIG_HOME/powercloud`), and `~/Library/Application Support/powercloud` on macOS. The directory is created with mode 0700 and each file with mode 0600. It holds: | File | What it is | | --- | --- | | `admin.key`, `admin.crt` | Your admin client certificate and its private key, issued by the cluster CA. Valid for 30 days. | | `ca.crt` | The cluster CA certificate, used to check that a node is really one of yours. | | `nodes.json` | The nodes this machine knows: each node's name, node ID and API address (`HOST:PORT`). | | `amd-kds/` | A cache of AMD's certificates, used when verifying SEV-SNP attestation. | Every connection to a node is mutual TLS. `pcctl` presents the admin certificate, and checks that the node's certificate comes from your cluster CA and names the node ID recorded for it in `nodes.json`. A node accepts only certificates from its own cluster CA, and checks the role (`admin` or `node`) on every request. Nodes have no SSH and no login: the node API on TCP port 7443 is the only way in. #### Getting and renewing the admin certificate `pcctl --repo LOC,... login` unlocks the cluster with the passphrase and writes a fresh admin certificate: ```sh pcctl --repo /srv/pc,s3://my-bucket/pc login ``` ```text admin certificate for cluster 92283421c94b924005629e1405c47c54 written to /home/alex/.config/powercloud (valid 30 days) ``` You rarely need to run it by hand. Any command that unlocks the cluster (with `--repo` or `--via`) reissues the saved certificate when it has a week or less left, and prints a note on stderr. `node enroll` and `node provision` write one if none is saved. If the certificate has expired, commands that use it say so and ask you to run `login`. To revoke every admin certificate, for example one on a lost laptop, see [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md). To save typing, set the storage locations once in the environment. `POWERCLOUD_REPO` stands in for `--repo` on every command: ```sh export POWERCLOUD_REPO=/srv/pc,s3://my-bucket/pc ``` #### Which nodes pcctl knows `nodes.json` is filled in by this machine's own commands: `node enroll`, `node provision --addr`, and `node addr` add or update a node, and `node rm` removes it. The cluster's shared record of nodes (the registry, kept in the encrypted storage) is what `--repo` and `--via` commands use, so every admin sees the same nodes there. On a second admin machine, `login` writes the certificate but no node list. Record each node's address on that machine with `node addr` (it needs the cluster passphrase), after which `status`, `node info` and the other certificate-only commands work there too: ```sh pcctl --repo LOC,... login pcctl --repo LOC,... node addr home-1 192.0.2.10 ``` ### Cluster status `pcctl status` asks every known node about itself, all at once, and prints one line per node followed by what needs attention. It uses only the admin certificate. ```sh pcctl status ``` ```text NODE STATE ARCH VERSION UPTIME CPU MEMORY MESH APPS STORAGE CERT cloud-1 ok arm64 vYYYY.MM.DD 72h4m10s 3% of 2 412/3891 MiB 2/2 peers 1/1 2 loc 341d home-1 ok amd64 vYYYY.MM.DD 5h12m3s 12% of 4 1204/7820 MiB 1/2 peers 3/3 - 200d vps-1 unreachable - - - - - - - - - needs attention: home-1: reaches 1 of 2 mesh peers vps-1: unreachable (dial tcp 198.51.100.4:7443: i/o timeout) ``` The columns: | Column | Meaning | | --- | --- | | `STATE` | `ok`, plus `AMD SEV-SNP` when the node runs on a confidential VM; `unreachable` when it doesn't answer within 10 seconds, or answers with an error. | | `ARCH`, `VERSION`, `UPTIME` | The node's architecture, the PowerCloud version it runs and how long since it booted. | | `CPU` | How busy its CPUs are, of how many. | | `MEMORY` | Memory in use of the total, in MiB. | | `MESH` | How many of its mesh peers it can reach over the mesh. | | `APPS` | Apps running, of apps deployed. | | `STORAGE` | How many storage locations it serves (`-` if it serves none; see [Storage](https://powercloud.unidatum.com/docs/storage.md)). | | `CERT` | Days until its node certificate expires. | #### Needs attention When nothing is wrong, the last line is `all nodes healthy` (it also says so when `nodes.json` lists no nodes at all). Otherwise each line names a node and a problem: | Message | What to do | | --- | --- | | `unreachable (...)` | The node is down, locked after a reboot, or its address changed. Check it with `pcctl node info NAME`; unlock it ([Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)) or update its address with `node addr`. | | `reaches N of M mesh peers` | Some peers can't be reached over the mesh. See [the mesh](#the-mesh) below. | | `app NAME is STATE (...)` | An app isn't running; the last exit is shown. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). | | `last storage sync incomplete` | The node's last copy between its storage locations failed for some blobs, or a location couldn't be listed. Run `pcctl node sync NAME` once the location is back. | | `certificate expires in N days` | Fewer than 7 days left. Run `pcctl --repo LOC,... node renew`. | | `release V rejected` or `halted` | A published release failed on this node. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). | | `runs an upgrade on trial` | The node booted a new image that isn't committed yet. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). | | `cluster app NAME: no node has the memory or CPU for it` | A cluster app runs on fewer nodes than it should. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). | | `state is in RAM, lost at reboot` | The node has no encrypted state disk, so its identity and apps don't survive a reboot. | ### Listing nodes and node details `pcctl node ls` prints the nodes in this machine's `nodes.json`, without contacting them: ```sh pcctl node ls ``` ```text NAME ID ADDRESS cloud-1 3c91e07d5b2a8f14 203.0.113.7:7443 home-1 fa8adc53910a6a79 192.0.2.10:7443 ``` `pcctl node info NAME` asks one node for its details: ```sh pcctl node info home-1 ``` It prints these fields, the later ones only when they apply: | Field | Meaning | | --- | --- | | `name`, `id`, `cluster` | The node's name, node ID and cluster ID. | | `arch`, `kernel`, `version`, `uptime` | Architecture, kernel release, PowerCloud version, time since boot. | | `boot` | How it booted, including whether Secure Boot is on. | | `lockdown` | The kernel lockdown mode. | | `state` | `encrypted volume, N MiB (N MiB free)`, or `RAM (lost at reboot)`. | | `certificate` | When the node certificate expires, with a reminder to run `node renew` from 30 days before. | | `cluster CA` | The CA fingerprint; during a key rotation, also the old CA it still accepts and until when. | | `confidential` | The confidential computing kind, with a hint to verify it with `pcctl node attest NAME`. | | `image` | The running image's sha256, and whether an upgrade is on trial; `last upgrade` reports a rollback. | | `release` | The published release's serial, name and this node's state on it. | | `storage`, `last sync` | Locations served, write quorum, sync interval, and the last sync's result. | | `volume`, `tpm unlock`, `peer unlock` | How the state volume is encrypted and how it unlocks at boot ([Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). | | `membership`, `revocations` | The serials of the signed membership and revocation lists the node holds. | | `mesh` | Its mesh address and WireGuard port, then a table of peers. | The peer table shows, for each peer, its mesh address, the endpoint WireGuard sends to (with `(relay NAME)` when the pair is relayed), the age of the last WireGuard handshake, and whether the peer's API answers over the mesh, with the round-trip time: ```text mesh fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820) PEER MESH ADDRESS ENDPOINT HANDSHAKE API OVER MESH cloud-1 fdd5:f552:ce70::3c91:e07d:5b2a:8f14 203.0.113.7:51820 12s ago ok, 18.4 ms home-2 fdd5:f552:ce70::77e2:1ab:9c3d:4e55 203.0.113.7:51822 (relay cloud-1) 31s ago ok, 41.0 ms ``` If the node refuses the admin certificate, `pcctl` says so and suggests `login` (admin certificates may have been revoked). If the node's own certificate has expired, it suggests `node renew NAME`. ### The web console `pcctl console` serves a web page on your own machine that shows the cluster as `pcctl status` sees it, live, and lets you read app logs and deploy, restart and remove apps. The nodes serve no web page: the console runs inside `pcctl` and reaches the nodes with your admin certificate, like every other command, so no key leaves your machine. ```sh pcctl console ``` ```text PowerCloud console at http://127.0.0.1:41377/?t=5b0c... (for one browser, the first to open it; Ctrl-C to stop) ``` It opens your browser at that link and runs until you press Ctrl-C. | Flag | Default | What it does | | --- | --- | --- | | `--port N` | any free port | The port to listen on. | | `--listen ADDR` | `127.0.0.1` | The IP address to listen on. Any address other than loopback lets other machines in; `0.0.0.0` listens on every address. Must be an IP address, not a name. | | `--host NAME` | this machine's name and addresses | A name or address other machines use to reach the console. Repeatable. Only used with `--listen`. | | `--no-open` | off | Don't open a browser; just print the link. | #### What it shows - **A summary bar**: nodes up, apps running, CPU busy across all nodes, memory in use, how much memory apps' limits promise of what the nodes set aside for apps, and the number of issues. - **Needs attention**: the same list as `pcctl status`. - **Nodes**: one row per node with state, CPU, memory, apps, architecture, version, uptime, mesh peers reached, storage locations served, consensus role and leader, release state, and days until the certificate expires. - **A node's detail** (select its row): bars for CPU, memory, app memory and app CPU promised and in use, and the state disk; its addresses (where `pcctl` reaches it, the node's own addresses, and its mesh address with each peer's round-trip time or relay); and facts such as node ID, kernel, boot, state encryption, how it unlocks, what apps get, the image and lockdown. - **The node's apps**: state, uptime, restarts, memory, CPU, network, where each published port is reached (as a link for TCP ports), whether it is a cluster app, and the last exit. - **Logs**: the last 64 KiB of an app's output, refreshed every 2 seconds while Follow is ticked. The page refreshes every 5 seconds and shows how old its data is. A short guided tour starts on a browser's first visit, and again from the Guide button. #### What you can do from it - **Restart** an app, or **Remove** it (with an option to keep its data directory). Removing a cluster app removes it from the whole cluster. - **Deploy an app** to the selected node, from a container image (pulled on your machine and sent to the node) or a static Linux binary you upload (up to 512 MiB), with memory, CPU cap, published ports, environment, links, microVM and egress options. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md) for what these mean. - **Deploy a cluster app**, which the nodes place among themselves, with replicas, allowed nodes, exclusive, storage and volume options. This needs the cluster's keys, so it is offered only when you start the console with the repository: `pcctl --repo LOC,... console`. A console started with `--via` or with neither flag can't publish cluster apps. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). #### Who can open it - The link carries a random token. The first browser to open it trades the token for a cookie (HttpOnly, SameSite=Strict), and the link lets no other browser in after that. To open it in another browser, stop and restart `pcctl console` for a new link. - Requests must use one of the console's own host names, which blocks DNS rebinding, and changes must come from the console's own page. - The page loads nothing from elsewhere (a strict Content-Security-Policy). #### Reaching it from another machine By default the console listens on loopback only. The safest way to use it from another machine is an SSH tunnel to a loopback console. On the machine that has `pcctl` and your admin certificate (here `admin-box`), start the console on a fixed port without a browser: ```sh pcctl console --port 8080 --no-open ``` On the machine with the browser, forward the port, then open the link the console printed (it uses `127.0.0.1:8080`, which the tunnel carries): ```sh ssh -L 8080:127.0.0.1:8080 admin-box ``` `--listen` puts it on the network over plain HTTP, and `pcctl` warns that anyone who can see the traffic can take the link or the session. Use it only on a network you trust: ```sh pcctl console --listen 0.0.0.0 --port 8080 --host admin-box.lan --no-open ``` Pass `--no-open` whenever the browser that should use the console is on another machine. Without it, `--listen 0.0.0.0` opens a browser on this machine, which then takes the one-time link. ### Power and reboots ```sh pcctl node reboot home-1 pcctl node poweroff home-1 ``` ```text node home-1 is rebooting ``` Both need only the admin certificate. After a reboot a node's encrypted state stays locked until it is unlocked, by you with `pcctl --repo LOC,... node unlock NAME CODE`, or by itself with TPM or peer unlock. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). Until then `pcctl status` reports it as unreachable. A powered-off node has to be started again at the machine or at its provider. ### Storage sync on a node A node can serve the cluster's storage locations so that admins can use `--via NAME` instead of `--repo` (set up with `node storage`, described in [Storage](https://powercloud.unidatum.com/docs/storage.md)). Such a node copies any blob that some of its locations lack from one that has it, one minute after it starts and then every hour by default. `pcctl node sync NAME` runs that copy now: ```sh pcctl node sync home-1 ``` ```text node home-1 synced its storage: 2026-10-28 14:02 UTC: 18234 blobs, 12 copied ``` It exits with an error (`sync incomplete`) if any copy failed or a location couldn't be listed. Sync only fills gaps and needs no keys; replacing stale or corrupt copies is `pcctl --repo LOC,... repair`. ### When an address changes There are two addresses per node, and they can change separately: - **The API address** (`HOST:PORT`, port 7443 by default) is where `pcctl` reaches the node. - **The mesh endpoint** (`IP:PORT` or `HOST:PORT`, port 51820 by default) is where the other nodes send the node's WireGuard traffic. #### A node's API address: node addr When a node comes back with a new IP (a cloud VM restarted without a fixed IP, a home router handing out a new lease), tell `pcctl` where it is: ```sh pcctl --repo LOC,... node addr cloud-1 203.0.113.50 ``` ```text node cloud-1 is at 203.0.113.50:7443 mesh endpoint of cloud-1 set to 203.0.113.50:51820 ``` This updates the cluster registry and this machine's `nodes.json`. If the node's mesh endpoint was empty or had been derived from its old address, `node addr` also sets it to the new address on port 51820 and publishes a new membership list to every node, so the other nodes find it again. An endpoint you set by hand with `mesh endpoint` is left alone. A DNS name works too (`node addr home-1 home.example.net`), for a node behind dynamic DNS. #### A node's mesh endpoint: mesh endpoint Set the endpoint explicitly when the other nodes must reach the node at a different address from the one `pcctl` uses, for example a public address with a port forwarded on the router while `pcctl` uses the LAN address: ```sh pcctl --repo LOC,... mesh endpoint home-1 198.51.100.7:51820 pcctl --repo LOC,... mesh endpoint home-1 home.example.net:51820 pcctl --repo LOC,... mesh endpoint home-1 '' ``` The last form clears the endpoint, for a node behind NAT that nobody can send to first. Every change publishes a new signed membership list to the nodes. With a DNS name, the nodes resolve it themselves, again every minute, and apply a new address only to a peer with no handshake in the last three minutes. #### Re-reading keys: mesh sync Enrolling, removing and changing endpoints publish the membership list on their own. `pcctl --repo LOC,... mesh sync` asks every reachable node for its WireGuard key, records any that changed, and publishes the membership and revocation lists to every node again. Use it when a node was missed or its key changed: ```sh pcctl --repo LOC,... mesh sync ``` Nodes that can't be reached keep the key on record (with a warning), and get the new lists from their mesh peers later. ### The mesh Nodes talk to each other over an encrypted WireGuard mesh. Each node gets a private IPv6 address derived from the cluster ID and its node ID, so there are no addresses to manage. The peer list is the membership list, signed by the cluster CA, which `pcctl` pushes to every node it can reach whenever it changes; nodes that were offline get it from any peer. Inside the tunnel, nodes authenticate each other again with their node certificates. Every node checks each peer's API through the mesh every 5 seconds, which is what `pcctl node info` and the `MESH` column of `pcctl status` report. #### NAT traversal: what's automatic A node behind NAT (a home router, a cloud VM without a public IP) needs no configuration beyond a clear or unreachable endpoint: - It connects outwards to the nodes that have a public endpoint, which learn its NAT's public address from its packets. - Nodes share what they see. A node with a peer it has no session with asks the peers it can reach every 5 seconds, and tries the address they report. Two nodes behind NAT do this at the same time, which opens both NATs, and they get a direct session. Traffic between them doesn't pass through the public node. - When two nodes still can't connect after 20 seconds (symmetric NAT, as with carrier-grade NAT and many corporate firewalls), they relay through a public node. Both pick the same relay; if it doesn't connect them within another 20 seconds, they move on to the next. The relay forwards WireGuard packets it can't read or change: the session is end to end between the two nodes. - Relayed pairs try a direct path again every 10 minutes, backing off up to about every 10 hours after repeated failures, in case the network changed. `pcctl node info` shows a relayed peer as `(relay NAME)`. These paths are checked in automated tests with emulated NATs. #### What you configure - **At least one node with a public endpoint.** Two nodes behind NAT find each other only through a third node both can reach. Relays are the nodes with an endpoint in the membership list. - **Firewall ports** on public nodes: | Port | Protocol | Who connects | Needed on | | --- | --- | --- | --- | | 7443 | TCP | `pcctl` (and, for peer unlock, locked peers) | Every node `pcctl` manages | | 51820 | UDP | Other nodes' WireGuard | Every node with a public endpoint | | 51821 to 51920 | UDP | Relayed WireGuard traffic | Public nodes that may act as relays | | 9443 | TCP | S3 clients | Nodes running the S3 gateway ([S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md)) | - **The right endpoint.** Enrolling a node sets its endpoint from the address you enrolled it at, on port 51820. If that is a LAN address the other nodes can't reach, set the public one with `mesh endpoint` (with a port forward on the router), or clear it with `mesh endpoint NAME ''` so the node is treated as behind NAT. Note that `node addr` sets an empty endpoint again from the new address; clear it afterwards if the node is behind NAT. #### Limits - Mesh changes start from an admin: nodes only pass on membership lists signed by the CA. - Relaying costs the relay's bandwidth and a detour. Pairs behind truly symmetric NATs keep relaying. - When the first peer to report a missing node's address is on the same LAN as it, it reports the LAN address, which a node elsewhere can't use; the two connect once that peer loses its session. - The mesh carries traffic between the nodes' APIs (health checks, consensus, replicas on other nodes). Your admin machine is not on the mesh: `pcctl` reaches each node's API directly, on its API address. ### Node storage A node's encrypted disk can hold a replica of the cluster's storage (`node://NAME` locations), and a node can serve the storage locations to admins so they use `--via NAME` with only the passphrase and admin certificate. Both are covered in [Storage](https://powercloud.unidatum.com/docs/storage.md). ### Good to know - Commands that only read or act on one node (`status`, `node ls`, `node info`, `node reboot`, `node poweroff`, `node sync`, `node upgrade`, `node attest NAME`, `app ...`, `console`) need only the admin certificate. - Commands that change the cluster's records (`node addr`, `mesh endpoint`, `mesh sync`, `node renew`, `node rm`) unlock the cluster with the passphrase, through `--repo` or `--via`. - `pcctl status` and the console ask each node for at most 10 seconds, so one slow node doesn't hold up the rest. - For problems not covered here, see [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md). --- ## Upgrades and releases URL: https://powercloud.unidatum.com/docs/upgrades > How to keep PowerCloud nodes and pcctl current, from checking a release's files to rolling it out to every node one at a time, upgrading a single node by hand, and what happens when an image fails to boot. A PowerCloud node runs one file, an EFI image holding the kernel, the init system and every service, so upgrading a node means booting a new image. Every upgrade is an A/B upgrade: the node boots the new image once, on trial, and keeps it only once it proves it works, otherwise it falls back to the image it ran before. You can roll a release out to the whole cluster with `pcctl release publish`, which lets the nodes upgrade themselves one at a time, or upgrade one node by hand with `pcctl node upgrade`. ### What a release holds A PowerCloud release is named for the day it was built, `vYYYY.MM.DD`, with `.1`, `.2` and so on added for a later release on the same day. It holds these files, where `VERSION` is the release name and `ARCH` is `amd64` or `arm64`: | File | What it is for | | --- | --- | | `powercloud-VERSION-ARCH.efi` | The node image. This is what `pcctl release publish` and `pcctl node upgrade` send to running nodes. | | `powercloud-VERSION-ARCH.img.xz` | A bootable disk image for new machines (decompress it, then write it to a disk or upload it to AWS). See [Install a node](https://powercloud.unidatum.com/docs/install.md). | | `powercloud-VERSION-ARCH.gce.tar.gz` | The disk image packed for Google Cloud. | | `powercloud-VERSION-ARCH.vhd.xz` | The disk image packed for Azure, as a fixed VHD. | | `pcctl-VERSION-OS-ARCH` | The `pcctl` command-line tool for `linux` and `darwin` (macOS), each for `amd64` and `arm64`. The macOS builds are signed and notarized. | | `SHA256SUMS` | The SHA-256 checksum of every other file. | The Google Cloud and Azure files are built to each cloud's documented import format and have upload scripts, but they have not been booted on those clouds yet. To upgrade nodes that are already running, you need only the `.efi` files. The disk images are for adding new machines. #### Check the files Check every file you download against `SHA256SUMS` before you use it. In the directory holding the downloads: ```sh sha256sum -c --ignore-missing SHA256SUMS # Linux shasum -a 256 -c --ignore-missing SHA256SUMS # macOS ``` Each file you have should report `OK`. Don't use a file that doesn't. #### Building from source instead You can also build the same files yourself. `make image` writes the disk images `out/powercloud-amd64.img` and `out/powercloud-arm64.img` and, beside each, the node image `out/powercloud-amd64.efi` and `out/powercloud-arm64.efi`. `go build -o pcctl ./cmd/pcctl` builds `pcctl`. A `pcctl` built this way reports its version as `dev`. If your machines boot with Secure Boot, build with your signing key (see "Upgrades and Secure Boot" below). ### See what your nodes run ```sh pcctl version # the release this pcctl was built for pcctl node info home-1 # one node in detail pcctl status # every node, and what needs attention ``` `pcctl node info NAME` shows, among other lines: - `version`: the build the node's software reports. - `image`: the first 16 hex digits of the SHA-256 of the image it runs, with `(upgrade on trial: ...)` while an upgrade hasn't been committed. - `last upgrade`: `ROLLED BACK: image ... didn't boot or wasn't committed`, if the last upgrade failed. - `release`: the release serial, its version name, and the node's state on it (see "Follow the rollout" below). `pcctl status` lists under "needs attention" any node that runs an upgrade on trial, and any node that rejected or halted on a release. These commands use only your admin certificate, so they need no `--repo`. See [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md) for admin certificates. ### Roll out a release to every node Publishing a release is the normal way to upgrade a cluster. Give `pcctl release publish` one `.efi` image per architecture you run; pcctl reads each image's architecture from the file itself. ```sh pcctl --repo LOC,... release publish --version vYYYY.MM.DD \ powercloud-vYYYY.MM.DD-amd64.efi powercloud-vYYYY.MM.DD-arm64.efi ``` `LOC,...` is your cluster's storage locations; you can set them once in the `POWERCLOUD_REPO` environment variable instead of passing `--repo`, or reach the storage through a node with `--via NODE` (see [Storage](https://powercloud.unidatum.com/docs/storage.md)). The command asks for the cluster passphrase, because it signs the release with the cluster's keys. `--version` is the name nodes show for the release. Use the release's own name. Without it, the name is the current date and time in UTC. ```text release 4 (vYYYY.MM.DD) published home-1: release and amd64 image sent colo-1: release and arm64 image sent aws-1: not reached (...); it gets the release from its peers nodes upgrade one at a time; follow with 'pcctl release status' ``` The number after `release` is the release's serial, which goes up by one with every publish. #### What happens next 1. **The release is signed and stored.** pcctl signs a release document with the cluster's certificate authority (CA): its serial, its name, and each image's SHA-256 and size. Nodes accept only a release signed by your cluster. 2. **Every node gets it.** pcctl sends the document and the image for each node's architecture to every node it reaches. A node that was off or unreachable fetches both from its peers over the encrypted mesh once it is back. Every copy is checked against the signed hash and size. 3. **Nodes upgrade one at a time, in node ID order.** A node installs the release only when every reachable peer before it already runs it, and no peer is installing or trying it. A peer that went quiet in the middle of its upgrade (rebooting, or waiting to be unlocked) holds the rollout for up to 30 minutes. 4. **Each node commits itself.** On its trial boot, a node keeps the new image once it is healthy: it reaches a peer on the mesh again (or, if it had no peers, it has run for 30 seconds). If it isn't healthy within 5 minutes, it reboots to the previous image. 5. **A bad release stops at the first node.** A release whose image doesn't boot, doesn't become healthy, or can't be installed is rejected on that node and never retried there. Every node that sees a peer reject or halt on it halts too, so the rest of the cluster keeps running the old image. To recover from a bad release, publish a new one. Publishing clears the halt. #### Follow the rollout ```sh pcctl release status ``` ```text NAME ARCH IMAGE RELEASE STATE aws-1 arm64 1f0c9e2a7b3d4c5e 4 (vYYYY.MM.DD) pending colo-1 arm64 9a8b7c6d5e4f3a2b 4 (vYYYY.MM.DD) current home-1 amd64 3c4d5e6f7a8b9c0d 4 (vYYYY.MM.DD) trial ``` | State | Meaning | | --- | --- | | `pending` | Waiting for the image or for its turn. | | `installing` | The image is installed; the node is about to reboot to try it. | | `trial` | Running the new image on trial; it commits itself once healthy. | | `current` | Running the release's image. | | `rejected` | It tried the release, and it failed on this node. | | `halted: failed on NAME` | Not trying it, because it failed on the node named. | | `not upgradable` | The node didn't boot from a PowerCloud disk image (for example, a kernel booted directly by a hypervisor), so it can't be upgraded this way. | | `unreachable` | pcctl couldn't reach the node. | ### Upgrade one node by hand `pcctl node upgrade` upgrades a single node, and waits while you watch: ```sh pcctl node upgrade home-1 powercloud-vYYYY.MM.DD-amd64.efi ``` ```text uploading powercloud-vYYYY.MM.DD-amd64.efi (24 MiB, sha256 3c4d5e6f7a8b9c0d) to home-1 node home-1 is rebooting into the new image; waiting for it (up to 10m0s) node home-1 runs 3c4d5e6f7a8b9c0d, committed: it is now the default image ``` What happens: 1. The node checks the image's hash and that it is an EFI image for its architecture, and writes it next to the running image on its EFI system partition. 2. It asks the firmware to boot the new image on the next boot only, and reboots. No boot loader is involved: the firmware's one-time boot setting does the counting. 3. pcctl waits for the node to answer on the new image, then commits it. The old image is kept as the previous one, and the new one becomes the default. If the node already runs that image, pcctl says `node NAME already runs ...` and does nothing. #### When the new image fails Anything short of a commit reverts to the old image: an image the firmware won't load, a kernel that panics, or a trial nobody commits. The next boot is the old image, the node reports the rollback, and `pcctl node upgrade` fails with: ```text pcctl: the new image didn't boot on home-1; it is back on 9a8b7c6d5e4f3a2b ``` #### A node that must be unlocked A node without unattended unlock waits at every boot, the trial boot included, until its encrypted state volume is unlocked (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). If that takes longer than pcctl waits, pcctl stops with a message saying so. Unlock the node, then commit the trial by hand: ```sh pcctl --repo LOC,... node unlock home-1 CODE pcctl node upgrade --commit home-1 ``` If you don't commit it, the node goes back to the old image at its next boot. pcctl waits 10 minutes by default; set `POWERCLOUD_UPGRADE_WAIT` (a duration such as `20m`) to wait longer. A trial started with `pcctl node upgrade` is never committed by the node itself; only pcctl commits it. ### Upgrades and Secure Boot With Secure Boot on, the firmware runs only images signed with your key, and that includes upgrades. An unsigned image, or one signed with another key, is refused by the firmware, which shows up as a rollback (and, during a release, as `rejected` on the first node, which halts the rest). The images in a PowerCloud release are not signed with your key. If your nodes boot with Secure Boot, build your images signed with your key, as described in [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md): ```sh POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image ``` Then publish `out/powercloud-amd64.efi` and `out/powercloud-arm64.efi`. Images signed with the same key keep TPM unlock working across the upgrade. ### Keep pcctl current `pcctl` has no self-update. To update it, download the `pcctl-VERSION-OS-ARCH` file for your system from a PowerCloud release, check it against `SHA256SUMS`, make it executable and put it in place of the old one: ```sh sha256sum -c --ignore-missing SHA256SUMS chmod +x pcctl-vYYYY.MM.DD-linux-amd64 mv pcctl-vYYYY.MM.DD-linux-amd64 /usr/local/bin/pcctl pcctl version ``` Your admin certificate, node list and other settings live in pcctl's configuration directory, not in the binary, so they carry over. It is simplest to keep pcctl at the same release as your nodes. pcctl for Windows doesn't exist yet. ### Limits - **A hang isn't a failure.** A trial image that hangs instead of panicking stays hung until someone resets the machine (a cloud's reboot button, a power cycle); then it reverts. There is no watchdog. - **Health means "back on the mesh".** A release that boots and reaches its peers, but breaks something else, is committed. - **Whoever can unlock the cluster can publish a release.** Releases are signed with the cluster CA; there is no separate release-signing key yet. Secure Boot, if on, still requires images signed with your own key. - **A locked node needs its admin.** Without TPM or peer unlock, every trial boot waits to be unlocked. If that takes more than 30 minutes, the next node in the rollout goes ahead. - **Only nodes booted from a disk image can be upgraded.** A kernel booted directly by a hypervisor has no EFI system partition. - **Room on the EFI system partition.** It holds three images during an upgrade (default, previous and next). Current disk images have plenty of room; machines installed from much older images have a smaller partition, and there the node drops the previous image if the new one wouldn't fit. - **Where it has run.** A/B upgrades are checked in automated tests on the standard virtual-machine UEFI firmware for both architectures, and have run on AWS's firmware on x86. They have not yet run on Graviton, Google Cloud, Azure or physical machines. Firmware that ignores the one-time boot setting never boots the trial, which shows as a rollback. --- ## Unlocking nodes after a reboot URL: https://powercloud.unidatum.com/docs/unlock > How a PowerCloud node's encrypted state volume is unlocked after a reboot, by an admin, by the node's TPM under Secure Boot, by key shares held by its peers, or with AMD SEV-SNP attestation on confidential VMs, and how to choose between them. A PowerCloud node keeps its identity, certificates and app data on an encrypted state volume, and after every reboot that volume is locked until something supplies its key. By default an admin unlocks it with `pcctl node unlock`; you can instead let a node unlock itself with its TPM (under Secure Boot) or with key shares held by its peers. On AMD SEV-SNP confidential VMs, pcctl also checks the chip's attestation before it hands over a key. ### How the state volume is protected The state volume is encrypted with a random volume key, with an HMAC-SHA256 tag on every sector, so a disk changed while the node was off reads as an error, not as altered data. The volume key is stored only wrapped: by a key derived from the cluster's root key, which only someone who can unlock the cluster can compute, and optionally by a TPM slot and a peer slot that you turn on per node. The node never holds the cluster's root key. | Method | Who acts at boot | Needs | Command | | --- | --- | --- | --- | | Admin unlock | An admin, reading the node's console | The cluster passphrase | `pcctl --repo LOC,... node unlock NAME CODE` | | TPM unlock | Nobody | Secure Boot on, a TPM 2.0 | `pcctl --repo LOC,... node tpm NAME on` | | Peer unlock | Nobody | Enough reachable peers | `pcctl --repo LOC,... node peer-unlock NAME on` | | Attested unlock | An admin, as for admin unlock | An AMD SEV-SNP VM | Automatic; `pcctl --repo LOC,... node attest NAME sev-snp` | The admin unlock always stays available: if TPM or peer unlock can't open the volume, the node waits for an admin. In the commands on this page, `LOC,...` is your cluster's storage locations (you can set them once in the `POWERCLOUD_REPO` environment variable), or use `--via NODE` to reach the storage through a node. These commands ask for the cluster passphrase. See [Storage](https://powercloud.unidatum.com/docs/storage.md). ### Unlock a node by hand A locked node prints a message on its console (the serial console, a cloud's console output, or a monitor attached to the machine): ```text PowerCloud node home-1 (NODE-ID) is locked: its state volume is encrypted. On your admin machine run: pcctl --repo ... node unlock home-1 unlock code: CODE ``` Run the command it shows, with the code: ```sh pcctl --repo LOC,... node unlock home-1 CODE ``` ```text node home-1 unlocked ``` If the node came back on a different address, give it as a third argument, `HOST` or `HOST:PORT` (the node API's port is 7443 by default): ```sh pcctl --repo LOC,... node unlock home-1 CODE 203.0.113.9 ``` To record the new address for later commands, use `pcctl --repo LOC,... node addr NAME HOST` (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). What pcctl checks before it sends anything: - **The code pins the node.** A locked node has no access to its identity, so it makes a temporary TLS key and shows that key's fingerprint as the unlock code. pcctl refuses a node whose key doesn't match, so a machine in the middle can't pose as the node. - **The IDs come from your registry.** The node must report the node ID and volume ID that the cluster's encrypted registry holds for that name. pcctl never trusts the disk's own header, so a forged disk can't obtain another volume's key. - **Attestation, where it applies.** See "Attested unlock on confidential VMs" below. Once unlocked, the node loads its identity and certificates from the volume and goes back to its normal, mutually authenticated API. Read the code from a console you trust. On a cloud VM, the provider can see the console too. During a key rotation (see [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md)), a node that hasn't moved to the new keys yet is unlocked with the previous keys, which pcctl tries second. It then says `node NAME unlocked with the previous cluster keys (run 'pcctl --repo ... rotate' to move it to the new ones)`. ### Secure Boot with your own key Secure Boot is the base for TPM unlock, and worth having on its own. The PowerCloud kernel, its built-in initramfs and its fixed command line are one EFI file, so one signature covers everything up to the init system and its services. With your certificate enrolled and Secure Boot on, the firmware boots only images you signed: not a modified PowerCloud on a stolen disk, and not an image swapped by a cloud provider. #### 1. Create a signing key Do this once, on a machine you trust: ```sh scripts/sb-keygen.sh ~/powercloud-sb ``` It writes an RSA-2048 key and certificate: | File | Use | | --- | --- | | `db.key` | The private key. Keep it offline; anyone holding it can sign images your machines will boot. | | `db.crt` | The certificate in PEM form, for signing. | | `db.cer` | The certificate in DER form, for enrolling in firmware or a cloud image. | The script refuses to overwrite an existing `db.key`. The certificate is valid for 20 years, since firmware doesn't check expiry. #### 2. Build signed images The images in a PowerCloud release are not signed with your key, so build your own: ```sh POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image ``` This signs each EFI image with `sbsign` (from sbsigntool) and checks the signature with `sbverify`. The resulting disk images and `.efi` files are signed. Sign every later upgrade with the same key too, or the firmware refuses it and the node rolls back (see [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md)). #### 3. Enroll your certificate | Where | How | | --- | --- | | Bare metal | In the firmware setup, put Secure Boot in custom mode and add `db.cer` to `db`. That keeps the vendor's and Microsoft's keys. To trust only your own images, replace PK, KEK and db instead (with a tool such as `sbctl`); then other operating systems won't boot on that machine. | | AWS | Set `POWERCLOUD_SB_CERT` to `db.crt` when you run `scripts/aws-image.sh`. The AMI then carries UEFI variables that enroll your certificate as PK, KEK and db, so instances boot only images you signed. | | Google Cloud | Set `POWERCLOUD_SB_CERT` when you run `scripts/gcp-image.sh`; the image then carries your certificate as PK, KEK and db. Create instances with `--shielded-secure-boot`. | | Azure | Not available: Azure's default keys don't trust PowerCloud images, so its images run with Secure Boot off. | Secure Boot on AWS has run on real AWS instances, x86 and Graviton. The bare-metal and Google Cloud steps haven't been tested yet, and Google Cloud images haven't been booted there at all. #### 4. Check it ```sh pcctl node info home-1 ``` The `boot` line reads `UEFI, Secure Boot on` or `UEFI, Secure Boot off`, read from the firmware. The node can't change its own boot settings. Rollback is a limit of Secure Boot itself: an older image signed with your key still boots. To retire old images, use a new key for a new generation of images, and add old images' hashes to the firmware's `dbx`. ### TPM unlock With Secure Boot on and a TPM 2.0, a node can unlock its state volume at boot by itself, but only on the same machine, booting an image signed with your key. ```sh pcctl --repo LOC,... node tpm home-1 on pcctl --repo LOC,... node tpm home-1 off ``` ```text node home-1: TPM unlock on: unlocks at boot while PCR [7] (Secure Boot state) matches ``` Turning it on sends the node its volume's wrapping key over the admin API. The node checks it opens the volume, and refuses unless Secure Boot is on and the machine has a TPM 2.0. It then seals a new random key to the TPM, checks that it unseals, and adds a TPM slot to the volume's header. At each boot it unseals that key and opens the volume; if the TPM refuses, it waits for an admin unlock as usual. #### What is measured The key is sealed to PCR 7, where the firmware records the Secure Boot configuration: whether Secure Boot is on, the PK, KEK, db and dbx, and which certificate verified the boot. Sealing to it means "this machine, booting anything signed with this key", so upgrades signed with the same key keep working. | Boot | TPM unlock | | --- | --- | | Same machine, Secure Boot on, image signed with your key | Unlocks itself | | Secure Boot turned off, or keys changed in firmware | Refused (PCR 7 differs); waits for an admin | | Disk moved to another machine | Refused (another TPM can't load the sealed key); waits for an admin | | Unsigned or modified image | Never runs: the firmware refuses it | Why Secure Boot matters: without it, PCR 7 would vouch for nothing, and any image could ask the TPM for the key. That is why the node refuses to turn TPM unlock on while Secure Boot is off. `pcctl node info NAME` shows the `tpm unlock` line: `on: ...`, `off (TPM 2.0 available)` or `no TPM`. #### Good to know - Nothing is stored in the TPM itself: the sealed key lives in the volume header, so no TPM setup or ownership is needed. - The key never crosses the bus to the TPM in plaintext, which matters against someone probing a stolen machine. - The volume header has a backup copy, written first, so a crash while adding or removing a slot can't lose the admin slot. - **The trade-off.** With TPM unlock on, a stolen machine boots into a running, enrolled node. Its data is then protected by what the running system enforces (an API that requires cluster certificates, no shell, kernel lockdown), not by a key the thief lacks. Use it on machines in places you trust (your home, a colo cage) or on cloud VMs. Remove a stolen node at once with `pcctl node rm` (see [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md)). - **Cloud TPMs trust the provider.** On AWS the TPM is the NitroTPM, AWS's virtual TPM; Google Cloud's Shielded VMs and Azure's Trusted Launch VMs also have virtual TPMs. A virtual TPM protects against a stolen disk snapshot, not against the provider itself. On AWS, TPM unlock with Secure Boot has run on real instances, unlocking by itself after a stop and start. On Google Cloud it is untested, and on Azure it isn't available, since Secure Boot stays off there. ### Peer unlock Peer unlock lets a node unlock itself after a reboot without a TPM, from key shares its peers hold, as long as it can reach enough of them. A disk taken away from its network stays locked. ```sh pcctl --repo LOC,... node peer-unlock home-1 on # a majority of its peers must answer pcctl --repo LOC,... node peer-unlock home-1 on --threshold 2 # any 2 of them pcctl --repo LOC,... node peer-unlock home-1 off ``` ```text node home-1: peer unlock on: unlocks itself after a reboot while 2 of colo-1, aws-1 answer ``` `--threshold K` is how many peers must answer. Without it, K is a majority of the holders. How it works: - **The key is split k of n.** The node makes a random key, wraps its volume key under it in a peer slot, and splits that key with Shamir's secret sharing. Each of its peers (every other node in the cluster that isn't revoked and has an address) gets one share over the encrypted mesh, and keeps it on its own encrypted state volume. Then the node forgets the key. Any K shares rebuild it; fewer reveal nothing. - **Asking while locked.** A locked node has neither its certificate nor its mesh key, so it asks the holders directly at their API address, checks each holder's certificate against the cluster CA recorded in the slot, and presents a token. It asks again every 10 seconds until K shares open the slot. The admin unlock stays available meanwhile. - **What a holder checks.** It gives its share only for that node's volume and token, and never to a node on the cluster's revocation list. A holder's API address is where pcctl reaches it, or otherwise its mesh endpoint's host on the default API port. If peers must reach a node's API somewhere else, set it before turning peer unlock on: ```sh pcctl --repo LOC,... mesh api colo-1 198.51.100.4:7443 ``` `pcctl node info NAME` shows `peer unlock on: K of HOLDERS` while it is on. #### Limits - **Network-bound, not tamper-proof.** The token is on the disk in the clear, so a disk that reaches K holders unlocks. That is the point: a rebooted node back on its network comes up by itself. A stolen machine still on your network must be removed with `pcctl node rm`, after which the holders refuse it. - **Shares follow the cluster as it was.** The holders, their addresses and the CA certificate are copied when you turn it on. After adding nodes, moving them, or rotating the cluster keys, turn it on again. - **Every holder must be reachable over the mesh** when you turn it on, so every share is handed out. There must be at least as many holders as the threshold. ### Attested unlock on confidential VMs On an ordinary VM, the host can read the guest's memory, and with it any key the node holds. On an AMD SEV-SNP VM the CPU encrypts that memory, and the chip signs reports that say so. pcctl checks such a report before it sends a volume key. It works without any setup: 1. **The locked node offers a report** along with its unlock code. The report is bound to the node's temporary key, the one the unlock code pins, so another machine can't replay it. 2. **pcctl verifies it before sending anything.** The report must be signed by the chip's endorsement key, that key's certificate must chain to AMD's root, the guest policy must forbid debugging, and the report must be bound to the key this connection is pinned to. pcctl uses certificates the platform supplies with the report first, and otherwise fetches them from AMD's key distribution service and caches them in its configuration directory, so the admin machine may need to reach AMD's service over HTTPS. 3. **Once attested, always attested.** After the first verified unlock, pcctl records the requirement in the node's registry entry and refuses from then on to unlock it without a valid report. A disk image copied to an ordinary VM can't get its key. ```text AMD SEV-SNP verified (...); launch measurement ... node aws-snp-1 unlocked node aws-snp-1: SEV-SNP attestation required at every unlock from now on ``` Set or drop the requirement yourself: ```sh pcctl --repo LOC,... node attest aws-snp-1 sev-snp # require it before the first unlock pcctl --repo LOC,... node attest aws-snp-1 off # the node moved to a machine without SEV-SNP on purpose ``` Check a running node at any time; it returns a fresh report bound to the node's key and a random value: ```sh pcctl node attest aws-snp-1 ``` `pcctl node info NAME` shows `confidential AMD SEV-SNP` on such nodes. On AWS, SEV-SNP is available on c6a, m6a and r6a instances; this has run by hand on AWS. For enrolling a confidential VM, prefer console enrollment over a join bundle in user data, since the provider can read user data but not the VM's memory (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). #### Limits - **What is measured is the cloud's firmware, not PowerCloud.** On AWS, the launch measurement covers AWS's UEFI firmware; the kernel it boots is vouched for by Secure Boot with your key, not by the report. pcctl shows the measurement but doesn't pin it. - **Only AMD SEV-SNP.** Intel TDX and Arm CCA are planned (see [Roadmap](https://powercloud.unidatum.com/roadmap)). - **AMD's revocation lists aren't checked,** and AMD's key distribution service is trusted over HTTPS for the certificate chain. - **Unattended unlock skips it.** A node that unlocks itself with its TPM or its peers attests to nobody; use `pcctl node attest NAME` to check it. ### Choosing a method | Situation | Suggested | | --- | --- | | You can be there for every reboot, or reboots are rare | Admin unlock only. Strongest: a rebooted or stolen machine stays locked. | | Your own machine in a place you trust, with Secure Boot and a TPM | TPM unlock. | | Several nodes on one network, no TPM, or you want a disk taken away to stay locked | Peer unlock. | | An AWS instance that should survive a stop and start on its own | TPM unlock with Secure Boot on the NitroTPM, accepting that it trusts AWS. | | A VM on someone else's hardware that will hold keys | An AMD SEV-SNP confidential VM, with admin unlock, so every unlock is attested. | | A plain cloud VM | Don't let it hold keys at all: use it only to store or relay ciphertext. | Whatever you choose, removing a node with `pcctl node rm` stops it unlocking again: the registry forgets it, and peers refuse it shares. ### Limits common to all methods - **No rollback protection for a whole disk.** Every sector's tag binds it to its position, but someone with the disk could put back an older copy of the whole disk unnoticed. - **The volume key is in kernel memory** while the volume is open, as with any disk encryption. Kernel lockdown blocks the usual ways of reading it; on someone else's VM only confidential computing protects it. - **A node without a state disk** keeps its state in RAM, loses it at reboot, and has nothing to unlock. For what each protection does and doesn't cover, see [Security model](https://powercloud.unidatum.com/security). --- ## Admins, keys and revocation URL: https://powercloud.unidatum.com/docs/access > Who can manage a PowerCloud cluster and how, from the passphrase and recovery key to admin certificates, revoking admins, renewing node certificates, removing nodes and rotating the cluster keys after a leak. Everything in a PowerCloud cluster follows from one secret, the cluster root key, which is never stored in plaintext and is unlocked with your passphrase or your recovery key. Whoever can unlock it is an admin: they can read the storage, issue certificates and unlock nodes. This page covers those secrets, the short-lived admin certificates that day-to-day commands use, and what to do when a laptop, a node or the passphrase itself may be in the wrong hands. ### The three admin credentials | Credential | What it opens | Where it lives | Lifetime | | --- | --- | --- | --- | | Passphrase | The cluster root key, and so everything | In your head or your password manager | Until you rotate the cluster keys | | Recovery key | The cluster root key, like the passphrase | Written down and stored offline | Until you rotate the cluster keys | | Admin certificate | The nodes' API: status, apps, reboots, upgrades | pcctl's configuration directory on your machine | 30 days | Commands that need the cluster's keys take `--repo LOC,...` (your storage locations, which you can set once in the `POWERCLOUD_REPO` environment variable) or `--via NODE` (the storage reached through a node), and ask for the passphrase. Commands that only talk to nodes, such as `pcctl status`, `pcctl node info NAME` and the `pcctl app` commands, use the admin certificate alone. See [Storage](https://powercloud.unidatum.com/docs/storage.md) and [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md). ### The passphrase and the recovery key `pcctl init` creates the cluster. It asks for a new passphrase twice and prints a recovery key once: ```sh pcctl --repo LOC,... init ``` ```text Initialized cluster CLUSTER-ID RECOVERY KEY (write it down and store it offline — it is shown only once): RECOVERY-KEY Without your passphrase or this recovery key, no one — including you — can read this data. ``` - **Both open the cluster on their own.** They unwrap the root key from a key header stored next to your data in every storage location. The header holds no plaintext key; the passphrase is stretched with Argon2id (256 MiB of memory per try), which makes guessing slow. - **Choose a strong passphrase.** Anyone who has it and can read the key header from any one of your storage locations holds the whole cluster. - **Store the recovery key offline**, on paper or in a safe. It is your way back if the passphrase is lost. - **Nothing else can recover the data.** If you lose both the passphrase and the recovery key, there is no backdoor and the data is gone. For scripts, pcctl reads the passphrase from `POWERCLOUD_PASSPHRASE` instead of the terminal. To unlock with the recovery key, set `POWERCLOUD_RECOVERY_KEY` to it; pcctl then uses it in place of the passphrase for that command. There is no command to change only the passphrase. Setting a new one is part of rotating the cluster keys (below), which also replaces the recovery key. If you have lost the passphrase but kept the recovery key, run `pcctl rotate` with `POWERCLOUD_RECOVERY_KEY` set to choose a new passphrase. ### Admin certificates Nodes accept only connections that present a certificate from your cluster's certificate authority (CA). The CA's key is derived from the root key and never stored, so only someone who can unlock the cluster can issue certificates. Issue yourself an admin certificate on each machine you manage the cluster from: ```sh pcctl --repo LOC,... login ``` ```text admin certificate for cluster CLUSTER-ID written to /home/you/.config/powercloud (valid 30 days) ``` pcctl writes `admin.key`, `admin.crt` and the cluster's `ca.crt` to its configuration directory, readable only by you. The directory is `$POWERCLOUD_CONFIG` if set, otherwise `~/.config/powercloud` on Linux and `~/Library/Application Support/powercloud` on macOS. It also keeps `nodes.json` there, the nodes this machine knows by name and address. - **Renewal is automatic while you use it.** Any command that unlocks the cluster (with `--repo` or `--via`) reissues the saved certificate once it has a week or less left, and says so. - **Once it has expired**, commands to nodes fail with `admin certificate expired (run 'pcctl --repo ... login')`. Run `login` again. - **Enrolling a node** also writes a certificate if this machine has none for the cluster. - **On another admin machine**, run `login` there with the passphrase. To have pcctl on that machine know the nodes by name, run `pcctl --repo LOC,... node addr NAME HOST` for each one (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). #### Roles Every certificate carries a role, and each node API endpoint checks it. There are two roles: `admin`, for the people managing the cluster, and `node`, for the nodes themselves, which use their certificates to talk to each other. (The S3 gateway has a server certificate of its own; see [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md).) There are no finer-grained roles. Every admin certificate can do everything an admin can, and everyone who knows the passphrase is equally an admin. ### Revoke admin certificates If a laptop holding an admin certificate is lost or stolen, revoke every admin certificate issued so far: ```sh pcctl --repo LOC,... revoke-admins ``` ```text admin certificates issued before 2026-01-01T12:00:00Z revoked (revocation list serial 3) NODE REVOCATION LIST colo-1 serial 3 home-1 serial 3 new admin certificate written to /home/you/.config/powercloud; other admins must run 'pcctl --repo ... login' again ``` Every admin certificate issued before that moment stops working, including those of other admins, who then run `login` again. A revoked certificate gets the error `the node refused the admin certificate; if admin certificates were revoked, run 'pcctl --repo ... login'`. Revoking certificates does not help if the passphrase may have leaked too, since anyone with it can issue new certificates. In that case rotate the cluster keys. ### Renew node certificates Node certificates last a year. Only an admin can renew them, because only admins hold the CA's key: ```sh pcctl --repo LOC,... node renew # every node due within 30 days, or already expired pcctl --repo LOC,... node renew home-1 # the named nodes, whether due or not ``` ```text NODE CERTIFICATE colo-1 not due, expires 2027-03-02 10:15 UTC home-1 renewed, expires 2027-10-28 09:00 UTC ``` A node whose certificate has expired drops out of the mesh, because its peers and pcctl refuse it, but it can still be renewed: the renewal proves the node's identity with its key, which an expired certificate shows as well as a current one. Watch for expiry in `pcctl node info NAME` (the `certificate` line, with a reminder within 30 days) and `pcctl status`, which lists a node under "needs attention" in its last week. A node also logs a warning daily from 30 days before expiry. A node can't renew itself, so run `node renew` from a scheduled job or as part of routine admin work. ### Remove a node Remove a machine that is stolen, compromised or retired: ```sh pcctl --repo LOC,... node rm colo-2 pcctl --repo LOC,... node rm --erase colo-2 # wipe it first ``` ```text node colo-2 (NODE-ID) removed: its certificate is revoked (revocation list serial 4) NODE REVOCATION LIST colo-1 serial 4 home-1 serial 4 ``` What removal does: - **Revokes the node** on a cluster-wide revocation list signed by the CA. Revoking the node's ID revokes every certificate for its key, including any issued later. Nodes enforce it in every connection, drop the node from their encrypted mesh, and refuse to add it back. - **Forgets the node** in the cluster's registry, so `pcctl node unlock` can no longer give it its volume key, and in this machine's node list. Peers holding its peer-unlock shares refuse it. - **Publishes a membership list** without the node. `--erase` first has the node crypto-erase itself: it destroys the only wrapped copies of its volume key, then powers off. Its identity, keys and data are unrecoverable, and at its next boot the disk reads as blank and waits to be enrolled as a new node. Erasing needs the node to be reachable; if it isn't, pcctl removes nothing and says so, and you can run `node rm` without `--erase`. #### How revocation spreads pcctl pushes the new list to every node it reaches. A node it can't reach shows `not updated (it will get it from a mesh peer)`. Every mesh message carries the sender's list serial, so a node that was off or cut off fetches the newer list from a peer once it is back, and checks the CA's signature. Nodes refuse a list older than the one they have, so a removal can't be rolled back. A node that finds itself on the list leaves the mesh. #### Limits - **A stolen node keeps what is on its own disk.** Removal cuts it off from the cluster, and `--erase` works only while it is reachable. With TPM unlock on, a stolen machine still boots and unlocks its own volume, which holds its own state but never the cluster's keys (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). - **Two admins removing nodes at the same moment** can produce two lists with the same serial. Run `node rm` again for a node that is missing from the list. ### Rotate the cluster keys If the passphrase or the recovery key may have leaked, revoking certificates isn't enough: whoever holds either can read the store, issue certificates and unlock nodes. `pcctl rotate` replaces the root key, and with it every key derived from it. Before you start, stop other admin work on the cluster, and make sure every storage location and as many nodes as possible are reachable. `rotate` needs `--repo` (not `--via`), and every location in it. ```sh pcctl --repo LOC,... rotate ``` It asks for the current passphrase, then a new one twice (`POWERCLOUD_NEW_PASSPHRASE` for scripts). The new one must differ from the old. Then: 1. **The store is re-encrypted.** Every object is read, verified and written again under the new keys. If `rotate` stops here, nothing has changed yet: run it again. 2. **A new key header is written to every location.** This is the point of no return: from now on the old passphrase and recovery key open nothing. pcctl prints the new recovery key once; store it offline as before. 3. **The old data is deleted** from every location. 4. **Every reachable node moves to the new CA.** Each one rewraps its volume key under the new keys and from then on accepts admins only from the new CA. Your admin certificate is reissued from the new CA. ```text NODE KEYS colo-1 moved to the new keys (CA ...) home-1 not moved yet: ... 1 node(s) still on the old keys: home-1 Run 'pcctl --repo ... rotate' again once they are up; meanwhile they can't reach the moved nodes. ``` #### Nodes that were offline A node that was off stays on the old keys, and it and the moved nodes can't reach each other until it moves. If it rebooted, `pcctl node unlock` still unlocks it with the old keys. Then run `pcctl --repo LOC,... rotate` again, with the new passphrase: it moves whatever is reachable now. Provisioned nodes that never joined are listed as `not located yet`: their join bundles hold the old keys, so remove them and provision them again (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). If a node reports `NOT MOVED: it presents a certificate from neither CA`, someone else holding the old keys may have moved it first. That node is lost to you: remove it and enroll it again. #### Finish the rotation Once every node has moved: ```sh pcctl --repo LOC,... rotate finish ``` Each node stops accepting the old CA, and the old root key is dropped from the key header. It ends with `key rotation finished: the old root key is gone from the key header`. `rotate finish` refuses while some nodes are still on the old keys. `rotate finish --force` gives up on them; they must then be removed and enrolled again. A moved node that misses the finish stops accepting the old CA on its own a week after the rotation started. #### Limits - **Data already copied stays readable.** Rotation stops the old keys from opening what is stored now. Copies an attacker already made, or a provider's backups of old blobs, still open with the old key. - **No other admin work during a rotation.** An object written meanwhile under the old keys is deleted. - **It reads and writes all the data.** A large store takes a long time and costs egress at every location. - **Finish promptly.** Until `rotate finish`, a moved node still accepts peers with a certificate from the old CA, so someone holding the old keys could pose as a node to it (not as an admin). - **Peer unlock shares** are tied to the old CA. Turn peer unlock on again for each node that uses it. ### What each measure protects against | Threat | What to do | How far it goes | | --- | --- | --- | | A lost or stolen admin laptop | `pcctl revoke-admins` | Covered, unless the passphrase was on it too. | | A stolen or compromised node | `pcctl node rm` (with `--erase` if it's reachable) | The node is cut off everywhere; it keeps its own local state. | | A leaked passphrase or recovery key | `pcctl rotate`, then `rotate finish` | Covered from now on; data copied before stays readable with the old key. | | A node certificate about to expire | `pcctl node renew` | Covered, also after expiry. | | Losing every passphrase and the recovery key | Nothing | By design, the data is gone. | For the full threat model, including what providers and hypervisors can and can't see, see [Security model](https://powercloud.unidatum.com/security). --- ## Troubleshooting URL: https://powercloud.unidatum.com/docs/troubleshooting > Common PowerCloud problems by symptom - pcctl errors about passphrases, storage locations, certificates and key rotation, nodes that won't boot, join, unlock or mesh - with the cause and the fix for each. This page lists problems you may meet with PowerCloud, organised by what you see: an error from `pcctl`, a message on a node's console, or a warning in `pcctl status`. Each entry gives the likely cause and the fix. Error messages are quoted as `pcctl` and `pcd` print them, with names and IDs replaced by placeholders such as `NAME`. A good first step for any node problem is `pcctl status`, which lists every node with a "needs attention" section, and `pcctl node info NAME` for one node in detail. ### Passphrase and repository #### "keys: wrong passphrase or recovery key" **Cause.** The passphrase (or `POWERCLOUD_PASSPHRASE`, or `POWERCLOUD_RECOVERY_KEY`) doesn't open the key header. After a key rotation (`pcctl rotate`), only the new passphrase and the new recovery key work. **Fix.** Check for a stale `POWERCLOUD_PASSPHRASE` in your environment. If the passphrase is lost, unlock with the recovery key (`POWERCLOUD_RECOVERY_KEY=... pcctl --repo LOC,... ...`) and then rotate to a new passphrase. If both are lost, the data can't be recovered: there is no backdoor. #### "no terminal for passphrase; set POWERCLOUD_PASSPHRASE (and POWERCLOUD_NEW_PASSPHRASE for rotate)" **Cause.** `pcctl` was run without a terminal (from a script, a cron job, a pipe) and needs the passphrase. **Fix.** Set `POWERCLOUD_PASSPHRASE` for that run, from a secret store rather than a file in plain sight. `pcctl rotate` also needs `POWERCLOUD_NEW_PASSPHRASE`. #### "repository not initialized; run 'pcctl init'" **Cause.** No location in `--repo` holds a key header. Usually the list is wrong: a typo in a path, a relative path run from another directory, or an S3 URL that the shell cut short at `&` because it wasn't quoted. **Fix.** Check `--repo` (or `POWERCLOUD_REPO`) and quote any value containing `&` or `;`. Run `pcctl init` only for a genuinely new cluster: it refuses to run over an existing one ("repository already initialized"), and if any location is unreachable it refuses too ("checking for an existing cluster"), since that location might hold an existing header. #### A write fails with "N/M backends succeeded, need Q" **Cause.** Fewer locations than the write quorum (by default a majority) accepted the write: some are down, full, or refusing your credentials. **Fix.** Bring the locations back and write again. Reads keep working from any location that has the data. As a stopgap, `--quorum N` lowers how many locations must accept a write; run `pcctl --repo LOC,... repair` afterwards so every location catches up. See [Storage](https://powercloud.unidatum.com/docs/storage.md). #### "locations differ; run 'pcctl repair'" **Cause.** `pcctl check` found locations missing, stale or corrupt copies, typically one that was down during writes, or one newly added. **Fix.** `pcctl --repo LOC,... repair`. It authenticates every copy before writing it, so it can't spread corruption. "DAMAGED" objects have no good copy on any reachable location; bring back the missing location if you can. #### "store: gc needs every backend, and LOC is unreachable" **Cause.** `pcctl gc` refuses to delete anything unless every location answers, since an unreachable one might hold a manifest that uses chunks that look unused elsewhere. Key rotation's purge has the same rule. **Fix.** Wait until every location is reachable, or remove a location you've retired from `--repo` for good. #### "store: rollback: ... the storage serves an old version" **Cause.** A location served an older, authentic version of an object (or dropped it) than the nodes' consensus has recorded. This is rollback protection catching a provider, or a location restored from an old backup. **Fix.** Run `pcctl --repo LOC,... check` to see which location is behind. Writing the object again (`put`) gives it a newer time and repairs it. See [Storage](https://powercloud.unidatum.com/docs/storage.md). #### "warning: no rollback protection this time" **Cause.** The cluster has a consensus group, but `pcctl` couldn't reach any node to read the rollback marks, so it went on without checking. **Fix.** Nothing is wrong with the data; the check was skipped. Make sure at least one node is reachable for the next command. #### "init works on each storage location and needs --repo, not --via" **Cause.** `init`, `check`, `repair`, `gc`, `rotate` and `node storage` work on each location separately, so they can't go through a node. **Fix.** Run them with `--repo` and the full list of locations (and their credentials in the environment). ### Admin certificates #### "admin certificate expired (run 'pcctl --repo ... login')" or "no admin credentials" **Cause.** Admin certificates last 30 days. Any command that unlocks the cluster renews yours when a week or less is left, but if you only used day-to-day commands (`status`, `node info`), it ran out. "No admin credentials" means this machine never had one, or `POWERCLOUD_CONFIG` points elsewhere. **Fix.** `pcctl --repo LOC,... login` issues a fresh 30-day certificate. #### "the node refused the admin certificate; if admin certificates were revoked, run 'pcctl --repo ... login'" **Cause.** The node rejected your certificate during the TLS handshake. Usually someone ran `pcctl revoke-admins`, which revokes every admin certificate issued before that moment. It also happens after a finished key rotation, when nodes accept only certificates from the new CA. **Fix.** `pcctl --repo LOC,... login`. Every other admin must do the same on their machines. #### "unknown node "NAME" (see 'pcctl node ls')" **Cause.** Day-to-day commands use this machine's own list of nodes (`nodes.json` in the configuration directory). A node enrolled from another admin machine, or provisioned without `--addr`, isn't on it. **Fix.** `pcctl --repo LOC,... node addr NAME HOST[:PORT]` records where the node is, on this machine and in the cluster's registry. ### Node certificates #### "certificate has expired (renew the node's certificate with 'pcctl --repo ... node renew NAME')" **Cause.** Node certificates last a year, and only an admin can renew them. An expired node is refused by `pcctl` and by its peers, so it drops out of the mesh. `pcctl node info` shows `certificate EXPIRED`, and `pcctl status` warns a week ahead. **Fix.** `pcctl --repo LOC,... node renew NAME`. Expired certificates can still be renewed. Without names, `node renew` renews every node due within 30 days; run it from a scheduled job or with routine admin work so it never comes to this. #### "not renewed: no address yet (run 'pcctl --repo ... node addr NAME HOST')" **Cause.** A provisioned node that `pcctl` has never been told the address of. **Fix.** Record its address with `node addr`, then run `node renew` again. ### Key rotation #### "storage locations hold different key headers, left by an interrupted key rotation" **Cause.** `pcctl rotate` wrote the new key header to some locations but not all (one was unreachable, or the run stopped). `pcctl` refuses other work until the rotation completes, so nothing is written under mixed keys. **Fix.** Run `pcctl --repo LOC,... rotate` again with the **new** passphrase once every location is reachable. #### "N node(s) not on the new keys yet: ..." **Cause.** `pcctl rotate finish` found nodes still on the old cluster CA, usually because they were off during the rotation. Moved and unmoved nodes can't reach each other until they are all moved. **Fix.** Start the nodes (a rebooted node still unlocks with the old keys, and `pcctl` says "unlocked with the previous cluster keys"), run `pcctl --repo LOC,... rotate` again to move them, then `rotate finish`. `rotate finish --force` gives up on the rest, which must then be removed and enrolled again. Provisioned nodes that never joined hold the old keys in their join bundle: remove them and provision them again. See [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md). ### Booting and the state disk #### Nothing boots, or the firmware says "Access Denied" **Cause.** The machine isn't booting with UEFI (legacy BIOS or CSM mode), or Secure Boot is on and the image isn't signed with a key the firmware trusts. Release images aren't signed for Secure Boot. **Fix.** Boot in UEFI mode. Either turn Secure Boot off, or build images signed with your own key and enroll it. See [Install a node](https://powercloud.unidatum.com/docs/install.md). #### "state disk ... holds data that is not a PowerCloud volume; refusing to format it" **Cause.** At enrollment, the node found something other than zeros in the first MiB of its state partition, and won't format a disk that might hold data. The usual reason is writing the image with `dd conv=sparse`, which skips zero blocks and leaves the disk's old contents in place. **Fix.** Write the image again without `conv=sparse` (with `conv=fsync`, as in [Install a node](https://powercloud.unidatum.com/docs/install.md)), or erase the disk first with `blkdiscard`. The `powercloud.state.wipe=1` boot option the message mentions can't be given to a released image, whose kernel command line is fixed. #### "several PowerCloud state partitions" **Cause.** More than one disk in the machine carries a PowerCloud state partition, for example an old PowerCloud disk left in a second bay, and the node won't guess which is its own. **Fix.** Remove or erase the disk you don't want, then boot again. #### "no state disk (powercloud.state=); node state is kept in RAM and lost at reboot" **Cause.** The node runs without an encrypted state disk: a kernel booted directly rather than from the disk image, or `pcd` running outside PowerCloud OS. `pcctl status` shows "state is in RAM, lost at reboot". **Fix.** Boot the disk image (see [Install a node](https://powercloud.unidatum.com/docs/install.md)). A node in RAM loses its identity at every reboot. #### The node is locked after a reboot **Cause.** This is normal. A node's state volume is encrypted, and after every reboot it shows an unlock code and waits, unless TPM unlock or peer unlock is on: ```text PowerCloud node NAME (ID) is locked: its state volume is encrypted. On your admin machine run: pcctl --repo ... node unlock NAME unlock code: ... ``` **Fix.** `pcctl --repo LOC,... node unlock NAME CODE`. If the node came back on a different address, add it: `node unlock NAME CODE NEW_HOST`, then `node addr NAME NEW_HOST`. "node key does not match the unlock code" means a mistyped code or something intercepting the connection; read the code again. To have nodes unlock themselves, see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). #### "TPM unlock failed; waiting for an admin to unlock" **Cause.** TPM unlock is sealed to the machine's Secure Boot state. Secure Boot was turned off, its keys changed, or the disk moved to another machine. This is TPM unlock refusing as designed. **Fix.** Unlock with `pcctl --repo LOC,... node unlock NAME CODE`. If the change was intended (new keys), turn TPM unlock on again with `pcctl --repo LOC,... node tpm NAME on`. #### "peer unlock: not yet; an admin can still unlock" **Cause.** The node is asking its peers for key shares and fewer than the threshold have answered: peers are down or locked themselves, or their addresses changed since peer unlock was turned on. "the shares don't open the volume (holders with old shares?)" means the shares are from before a change in the node set or a key rotation. **Fix.** Bring the peers up, or unlock the node yourself. After adding or moving nodes or rotating keys, turn peer unlock on again with `pcctl --repo LOC,... node peer-unlock NAME on`. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). ### Enrolling and joining #### "node key does not match the enrollment code (wrong code or interception)" **Cause.** The code was mistyped, it's from an earlier boot (an unenrolled node may have a new key and code after a reboot), or something between you and the node is posing as it. **Fix.** Read the code from the node's current console output and try again. If it still fails, check that the address really reaches that machine. #### "node is already enrolled in a cluster" **Cause.** The node already has an identity: you enrolled it before, it joined from a join bundle, or someone else enrolled it first. On a cloud VM, whoever reads the console first (including the provider) can enroll it. **Fix.** If it's in your cluster, use it (`pcctl node ls`). If it isn't, don't trust the machine: create a fresh one. A node of yours that you want to start over with can be wiped with `pcctl --repo LOC,... node rm --erase NAME`. #### "a node named "NAME" is already enrolled" **Cause.** Names are unique in the cluster's registry. **Fix.** Pick another name, or remove the old node first with `node rm`. #### "metadata service not answering" and then "could not read user data; waiting for console enrollment" **Cause.** The node is on a cloud (AWS, Google Cloud or Azure) and couldn't read its user data after five tries. On AWS this happens when the instance metadata service is turned off. **Fix.** Turn the metadata service on (on AWS, `aws ec2 modify-instance-metadata-options --instance-id ID --http-endpoint enabled`; the node uses IMDSv2, so `HttpTokens=required` is fine), then reboot: a node with no identity looks for a join bundle again at every boot. Or enroll it from the console. #### "invalid join bundle; waiting for console enrollment" or "join bundle not installed" **Cause.** The user data or fw_cfg file looks like a bundle but is damaged (cut short, or edited), or it doesn't check out against its own CA, or the node couldn't create its state volume (see the state disk entries above). Bundles from before a key rotation are stale too. **Fix.** Remove the node (`pcctl --repo LOC,... node rm NAME`), provision it again, and start a fresh machine with the new bundle, passed as a file (`file://` on AWS) rather than pasted. #### "node enrolled, but its mesh key is unknown (run 'pcctl mesh sync' later)" **Cause.** The node enrolled, but `pcctl` couldn't read its WireGuard key straight away. **Fix.** `pcctl --repo LOC,... mesh sync` once the node answers; it re-reads every node's key and publishes the membership list again. ### Reaching nodes and the mesh #### A node is "unreachable" in pcctl status, or commands time out **Cause.** `pcctl` can't reach the node's API on TCP 7443. Common reasons: a firewall or security group doesn't allow TCP 7443 from your machine, the node is off or still booting, or it came back with a new IP address (a cloud VM after a stop and start). **Fix.** Check the node's console. Allow TCP 7443 from your admin machines. If its address changed, run `pcctl --repo LOC,... node addr NAME NEW_IP`, which also moves its mesh endpoint and tells the other nodes. An Elastic IP or other reserved address avoids this on clouds. #### "NAME: reaches N of M mesh peers" **Cause.** WireGuard between nodes isn't getting through. Common reasons: - UDP 51820 isn't open on a public node, or UDP 51821-51920 on a node that relays for others. - A node's endpoint is wrong, for example a private LAN address that nodes elsewhere can't use, or an address that changed. - Two nodes changed IP addresses at once and nobody ran `node addr`, so they can't find each other. - Two nodes behind NAT have no third node that both can reach. **Fix.** Open the ports, then check endpoints in `pcctl node info NAME` (each peer's endpoint and handshake age). Fix one with `pcctl --repo LOC,... mesh endpoint NAME IP:PORT`, or clear it with `mesh endpoint NAME ''` for a node behind NAT. Keep at least one node with a stable public address. Nodes behind symmetric NATs connect through a relay automatically, which `node info` shows as `(relay NAME)`. #### "node NAME didn't come back on the new image within ..." **Cause.** After `pcctl node upgrade`, the node didn't answer on the new image in time. It may be waiting to be unlocked, or the image may not have booted. **Fix.** If it's locked, unlock it and run `pcctl node upgrade --commit NAME`. Otherwise do nothing: an image that isn't committed reverts at the next boot, and `pcctl node info` then shows `last upgrade ROLLED BACK`. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). #### Replica writes fail with "this node's state volume is full" **Cause.** A node holding a replica (`node://NAME`) refuses writes that would leave less than 256 MiB free on its state volume, so the replica never starves the node's own state. **Fix.** Give the node a bigger disk (it grows its state volume to fill a resized disk at the next boot), or move the replica to another location. ### pcctl on macOS #### macOS won't open pcctl, or says it can't check it for malicious software **Cause.** Gatekeeper holds back programs downloaded from the internet that Apple hasn't notarized. The macOS `pcctl` in a PowerCloud release is signed and notarized, and the release notes say so; a build without notarization, or one you built yourself and copied over, is held back. **Fix.** Check the file against the release's `SHA256SUMS`, make it executable with `chmod +x`, and if the release notes say it isn't notarized, remove the quarantine flag: `xattr -d com.apple.quarantine pcctl-VERSION-darwin-ARCH`. There is no Windows build of `pcctl`. ### Still stuck - `pcctl node info NAME` shows a node's state, certificate, Secure Boot, TPM and mesh status in one place. - The node's console log shows what `pcd` is doing, with every warning and error. - [Concepts](https://powercloud.unidatum.com/docs/overview.md) explains the terms used here, and [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md) lists every command. --- ## pcctl command reference URL: https://powercloud.unidatum.com/docs/pcctl > Every pcctl command, flag, default, environment variable and local file, with what each command needs (the admin certificate, --repo, or --repo or --via) and an example. `pcctl` is the only way to manage a PowerCloud cluster: there is no shell or SSH on the nodes. This page lists every command `pcctl` has, with its exact synopsis, flags and defaults, what it needs to run, and an example. It includes commands that `pcctl`'s own usage text leaves out. For task-oriented guides, see [Quickstart](https://powercloud.unidatum.com/docs/quickstart.md), [Storage](https://powercloud.unidatum.com/docs/storage.md) and [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md). Terms such as cluster, repository, location and node are defined in [Concepts](https://powercloud.unidatum.com/docs/overview.md). `pcctl` runs on Linux and macOS, on amd64 and arm64 (see [Install a node](https://powercloud.unidatum.com/docs/install.md)); there is no Windows build. ### Conventions #### Command line shape ```sh pcctl [GLOBAL OPTIONS] COMMAND [SUBCOMMAND] [FLAGS] [ARGUMENTS] ``` - **Global options come before the command.** `pcctl --repo LOC put a b` works; `pcctl put a b --repo LOC` does not. - **A command's flags come before its positional arguments.** `pcctl app rm --keep-data home-1 web` works; flags after the first positional argument are taken as arguments. The exceptions are noted in each entry (`node peer-unlock`, `node rm --erase`, `node upgrade --commit`, `rotate finish --force`). - **`-flag` and `--flag` are the same**, and a value can be given as `--flag value` or `--flag=value`. Boolean flags take no value (`--dry-run`), or `--flag=false`. - **Repeatable flags** (marked "repeatable") are given once per value: `--env A=1 --env B=2`. - `pcctl -h` prints the usage summary and global flags. Running `pcctl` with no command prints it and exits with status 2. Any other failure prints `pcctl: ` and the error on stderr and exits with status 1. #### What a command needs Each entry says what the command needs: | Needs | Meaning | | --- | --- | | **certificate** | Only the admin certificate saved by `login`, and the node addresses in the local `nodes.json`. No passphrase, no storage access. `--via` is ignored. So is `--repo`, except by `console` (which then holds the keys) and the `s3` subcommands (which then ask for the passphrase although only `s3 enable` and `s3 key add` use it). | | **--repo** | The storage locations themselves, with their credentials, and the passphrase. `--via` is refused. These commands work on each location separately. | | **--repo or --via** | The passphrase, and the storage either directly (`--repo`) or through a node that serves it (`--via NAME`, which also needs the admin certificate). | Commands that need the passphrase unlock the cluster's keys on your machine. With `--via`, the node only relays encrypted blobs and never sees a key. ### Global options | Option | Environment | Default | What it does | | --- | --- | --- | --- | | `--repo LOC[,LOC...]` | `POWERCLOUD_REPO` | none | The cluster's storage locations, comma-separated. Every write is replicated to all of them. See [Location syntax](#location-syntax). | | `--via NODE` | `POWERCLOUD_VIA` | none | Reach the storage through this node instead of `--repo` (the node must serve it; see `node storage`). A flag beats the other's environment variable: `--via` with `POWERCLOUD_REPO` set goes through the node, and `--repo` with `POWERCLOUD_VIA` set uses the locations. If both flags are given, `--repo` wins. | | `--quorum N` | | majority | How many locations must accept a write for it to succeed. `node storage` passes it to the node. | The flag overrides the environment variable. For example: ```sh export POWERCLOUD_REPO=/srv/pc,s3://my-bucket/pc pcctl ls photos/ pcctl --via home-1 ls photos/ ``` #### Location syntax A location in `--repo` (and in `node storage`) is one of: | Form | What it is | | --- | --- | | `/path/to/dir` or `./dir` | A local directory. | | `s3://BUCKET[/PREFIX][?OPTIONS]` | An S3-compatible bucket, optionally under a prefix. | | `node://NAME` | A replica on an enrolled node's encrypted disk, reached with the admin certificate. | | `ec:K:LOC;LOC;...` | Erasure coding over the listed locations: each blob is split into `K` data shards plus parity shards, one per location (`1 <= K <` number of locations `<= 255`). A write needs `K+1` locations (or all, if fewer). The inner locations can be any of the forms above, but not another `ec:`. | S3 options, joined with `&`: | Option | Default | Meaning | | --- | --- | --- | | `endpoint=URL` | AWS (`https://s3.REGION.amazonaws.com`) | The provider's endpoint, `http(s)://HOST[:PORT]` with no path. Setting it turns on path-style requests. | | `region=R` | `us-east-1` | The region to sign requests for (`auto` for Cloudflare R2). | | `creds=NAME` | `AWS` | Which credentials to read from the environment: `NAME_ACCESS_KEY_ID`, `NAME_SECRET_ACCESS_KEY` and, optionally, `NAME_SESSION_TOKEN`, with `NAME` upper-cased. | | `path_style=BOOL` | `false` without `endpoint`, `true` with | Path-style requests. A bucket name with dots needs `path_style=true` over HTTPS. | Quote locations that contain `&` or `;`: ```sh export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... pcctl --repo '/srv/pc,s3://pc?endpoint=https://ACCOUNT.r2.cloudflarestorage.com®ion=auto&creds=r2' ls pcctl --repo 'ec:4:s3://a/pc;s3://b/pc;s3://c/pc;s3://d/pc;s3://e/pc;s3://f/pc' ls ``` #### The passphrase Commands that need the cluster's keys read the passphrase from the terminal without echoing it, or from an environment variable for automation: - `POWERCLOUD_PASSPHRASE`: the cluster passphrase. With it, `init` also skips the "Repeat passphrase" prompt. - `POWERCLOUD_RECOVERY_KEY`: unlock with the recovery key that `init` (or `rotate`) printed, instead of the passphrase. When it is set, no passphrase is asked for. - `POWERCLOUD_NEW_PASSPHRASE`: the new passphrase for `rotate`. Without it, `rotate` asks twice on the terminal. Without a terminal and without these variables, the command fails with `no terminal for passphrase`. ### Environment variables Every variable `pcctl` reads: | Variable | Used by | Meaning | | --- | --- | --- | | `POWERCLOUD_REPO` | all `--repo` commands | Default for `--repo`. | | `POWERCLOUD_VIA` | all `--via` commands | Default for `--via`. | | `POWERCLOUD_PASSPHRASE` | commands that unlock the cluster | The cluster passphrase (new passphrase for `init`). | | `POWERCLOUD_RECOVERY_KEY` | commands that unlock the cluster | Unlock with the recovery key instead. | | `POWERCLOUD_NEW_PASSPHRASE` | `rotate` | The new passphrase. | | `POWERCLOUD_CONFIG` | everything | The local configuration directory (see [Local files](#local-files)). | | `POWERCLOUD_UPGRADE_WAIT` | `node upgrade` | How long to wait for the node to come back on the new image, as a duration such as `20m`. Default `10m`. | | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` | S3 locations without `creds=` | S3 credentials. | | `NAME_ACCESS_KEY_ID`, `NAME_SECRET_ACCESS_KEY`, `NAME_SESSION_TOKEN` | S3 locations with `creds=name` | S3 credentials for that location. `node storage` sends the ones its locations name to the node. | | `DOCKER_CONFIG` | `app deploy --image`, `cluster app deploy --image` | Directory holding the registry logins (`config.json`, as `docker login` writes it). Default `~/.docker`. | | `HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY` | S3 locations, image pulls | The usual proxy settings. Connections to nodes never use a proxy. | | `POWERCLOUD_TEST_NODE_CERT_LIFETIME` | `node enroll`, `node provision`, `node renew` | A test hook that shortens node certificates (a duration). Don't set it on a real cluster: nodes drop out of the mesh when their certificates expire. | ### Local files `pcctl` keeps its state in `$POWERCLOUD_CONFIG`, or else in the user configuration directory plus `powercloud`: `~/.config/powercloud` (or `$XDG_CONFIG_HOME/powercloud`) on Linux and `~/Library/Application Support/powercloud` on macOS. The directory is mode 0700 and the files 0600. | File | Written by | Contents | | --- | --- | --- | | `admin.key`, `admin.crt` | `login`, renewed by any command that unlocks the cluster | The admin certificate (30 days) and its key. | | `ca.crt` | the same | The cluster CA certificate. | | `nodes.json` | `node enroll`, `node provision --addr`, `node addr`; `node rm` removes entries | Each known node's name, node ID and API address. Certificate-only commands find nodes here. | | `amd-kds/` | attestation checks | Cached AMD certificates for SEV-SNP verification. | ### Cluster storage #### init ```sh pcctl --repo LOC[,LOC...] init ``` Creates a new cluster in the given locations: a new root key, protected by the passphrase you choose (asked twice) and by a recovery key, which it prints once. Write the recovery key down and store it offline. It refuses if a cluster already exists there, and also if any location can't be reached (it might hold an existing cluster). Needs: **--repo**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc init ``` ```text Initialized cluster 92283421c94b924005629e1405c47c54 RECOVERY KEY (write it down and store it offline — it is shown only once): PPXR-AHNU-ZWUQ-WODE-USI2-C2TZ-QI4N-7UFI Without your passphrase or this recovery key, no one — including you — can read this data. ``` #### put ```sh pcctl --repo LOC,... put NAME FILE ``` Encrypts `FILE` and stores it as object `NAME`, replicated to every location. Prints the size and chunk count. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc put photos/cat.jpg ./cat.jpg ``` ```text stored photos/cat.jpg (48213 bytes, 1 chunks) ``` #### get ```sh pcctl --repo LOC,... get NAME [FILE] ``` Reads and decrypts object `NAME` into `FILE` (created with mode 0600, replaced if it exists), or to standard output without `FILE`. Every chunk is authenticated; a tampered copy is an error, not altered data. Needs: **--repo or --via**. ```sh pcctl --via home-1 get photos/cat.jpg ./out.jpg ``` #### ls ```sh pcctl --repo LOC,... ls [PREFIX] ``` Lists objects whose names start with `PREFIX` (all without it): size in bytes, modification time in local time, name. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc ls photos/ ``` ```text 48213 2026-10-28 14:02 photos/cat.jpg ``` #### rm ```sh pcctl --repo LOC,... rm NAME ``` Deletes object `NAME`. Its chunks stay until `gc` reclaims them. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc rm photos/cat.jpg ``` #### check ```sh pcctl --repo LOC,... check [--verify] ``` Compares every location and reports, per location, its blobs and how many are missing, stale or corrupt. Changes nothing. Exits with an error if the locations differ (`run 'pcctl repair'`) or if some object has data missing from every reachable location. | Flag | Default | Meaning | | --- | --- | --- | | `--verify` | off | Download and authenticate every copy of every chunk, instead of comparing listings. | Needs: **--repo**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc check --verify ``` ```text 1 objects, 0 deleted, 1 chunks in use, 0 unreferenced (reclaimed by gc) LOCATION BLOBS MISSING STALE CORRUPT REPAIRED STATUS /srv/pc 3 0 0 0 0 ok s3://my-bucket/pc 3 0 0 0 0 ok ``` #### repair ```sh pcctl --repo LOC,... repair [--verify] ``` Brings every location up to date: copies missing blobs, and replaces stale or corrupt copies with a good one. Erasure-coded locations first rebuild missing shards from their own other shards. Use it after a provider was down, or after adding a location. Same flag as `check`. Needs: **--repo**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc,node://home-1 repair ``` #### gc ```sh pcctl --repo LOC,... gc [--grace 24h] [--dry-run] ``` Deletes chunks that no object references any more, from every location, once they have been unreferenced for the grace period. | Flag | Default | Meaning | | --- | --- | --- | | `--grace DURATION` | `24h` | How long a chunk must stay unreferenced before it is deleted. | | `--dry-run` | off | Report what would be deleted, delete nothing. | Needs: **--repo**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc gc --dry-run ``` ```text 1 chunks in use, would delete 0, 0 unreferenced waiting out the 24h0m0s grace period ``` ### Nodes See [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md), [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md), [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md) and [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md) for the tasks behind these commands. #### status ```sh pcctl status ``` Asks every node in `nodes.json` about itself, in parallel (10 seconds each), and prints one line per node (state, architecture, version, uptime, CPU, memory, mesh peers reached, apps running, storage locations served, days left on its certificate), then a `needs attention` list or `all nodes healthy`. Needs: **certificate**. ```sh pcctl status ``` #### console ```sh pcctl console [--port N] [--listen ADDR [--host NAME]...] [--no-open] pcctl --repo LOC,... console [flags] ``` Serves the web console on this machine and opens a browser at its one-time link: live status, node details, app logs, and deploying, restarting and removing apps. Started with `--repo`, it holds the cluster's keys and can also publish and remove cluster apps. Runs until Ctrl-C. | Flag | Default | Meaning | | --- | --- | --- | | `--port N` | any free port | Port to listen on. | | `--listen ADDR` | `127.0.0.1` | IP address to listen on. Another than loopback lets other machines in, over plain HTTP; `0.0.0.0` is every address. | | `--host NAME` | with `--listen`: this machine's name and addresses | A name or address other machines reach the console by. Repeatable. | | `--no-open` | off | Don't open a browser; just print the link. | Needs: **certificate**; with `--repo`, also the passphrase. Given `--via` (and no `--repo`), it runs without the keys, as if neither were given. ```sh pcctl console --port 8080 --no-open ``` #### node ls ```sh pcctl node ls ``` Lists the nodes in this machine's `nodes.json` (name, node ID, API address) without contacting them. Needs: **certificate** (reads only the local file). ```sh pcctl node ls ``` #### node info ```sh pcctl node info NAME ``` Shows one node's details: IDs, architecture, kernel, boot and lockdown, state volume, certificate expiry, cluster CA, confidential computing, image and release, storage served and last sync, TPM and peer unlock, membership and revocation list serials, and its mesh peers with endpoint, handshake age and API round-trip time. Needs: **certificate**. ```sh pcctl node info home-1 ``` #### node enroll ```sh pcctl --repo LOC,... node enroll HOST[:PORT] CODE NAME ``` Joins a new node to the cluster with the enrollment code shown on its console, gives it the key for its encrypted state volume, records it in the cluster registry and in `nodes.json`, and publishes a new membership list to every node. `PORT` defaults to 7443. The node's mesh endpoint is set to the same host on port 51820 (unless the host is `localhost`). Writes an admin certificate if none is saved. Fails if a node named `NAME` is already enrolled. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node enroll 192.0.2.10 XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX home-1 ``` #### node provision ```sh pcctl --repo LOC,... node provision [--addr HOST[:PORT]] [--endpoint IP:PORT] NAME FILE ``` Creates a node's whole identity ahead of time and writes it to `FILE` as a join bundle, for a machine that joins at first boot with nobody at its console (cloud user data, or a QEMU fw_cfg file). Registers the node and publishes a membership list that includes it. `FILE` must not exist. The bundle holds the node's private keys and disk key: delete it once the node is up. Prints how to pass it to AWS, Google Cloud, Azure and QEMU. | Flag | Default | Meaning | | --- | --- | --- | | `--addr HOST[:PORT]` | none | Where `pcctl` reaches the node's API, if known. Without it, run `node addr` once the node is up. | | `--endpoint IP:PORT` | none | Where peers reach the node's WireGuard port, if it is public. `HOST:PORT` is accepted too. | Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node provision web-1 web-1.join ``` #### node addr ```sh pcctl --repo LOC,... node addr NAME HOST[:PORT] ``` Records where `pcctl` reaches a node's API (`PORT` defaults to 7443), in the registry and in `nodes.json`. If the node's mesh endpoint was empty or derived from its old address, it is set to the new host on port 51820 and a new membership list is published. Also the way to add an existing node to `nodes.json` on another admin machine. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node addr cloud-1 203.0.113.50 ``` #### node unlock ```sh pcctl --repo LOC,... node unlock NAME CODE [HOST[:PORT]] ``` Unlocks a rebooted node's encrypted state volume with the unlock code shown on its console. `pcctl` pins the node's temporary key from the code, checks the node and volume IDs against the registry, verifies a SEV-SNP attestation if the node offers one (and requires one if the node was attested before), and only then sends the key. `HOST[:PORT]` reaches the node at another address for this unlock only. During a key rotation, a node not yet moved is unlocked with the previous keys. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node unlock home-1 XXXX-XXXX-XXXX-XXXX # the code from the node's console ``` #### node tpm ```sh pcctl --repo LOC,... node tpm NAME on|off ``` Turns unattended unlock with the node's TPM on or off. With Secure Boot and a TPM 2.0, the node then unlocks its state at boot by itself, only on the same machine with the same Secure Boot keys. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node tpm home-1 on ``` #### node peer-unlock Not in `pcctl`'s usage text. ```sh pcctl --repo LOC,... node peer-unlock NAME on [--threshold K] pcctl --repo LOC,... node peer-unlock NAME off ``` Turns peer unlock on or off: the node splits a key to its state volume into shares held by its peers, and after a reboot unlocks itself while `K` of them answer. `--threshold` comes after `on`. All peers must be reachable over the mesh to turn it on. Turn it on again after adding or moving nodes or rotating the keys. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). | Flag | Default | Meaning | | --- | --- | --- | | `--threshold K` | a majority of the peers | How many peers must answer. | Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node peer-unlock home-1 on --threshold 2 ``` #### node attest ```sh pcctl node attest NAME pcctl --repo LOC,... node attest NAME sev-snp|off ``` With only `NAME`: asks the running node for a fresh AMD SEV-SNP attestation, bound to a random nonce and the node's key, and verifies it against AMD's certificate chain. Needs: **certificate**. With `sev-snp` or `off` (not in `pcctl`'s usage text): sets or clears the registry's requirement that the node present an SEV-SNP attestation at every unlock. `node unlock` sets it by itself the first time a node attests; use `off` when a node has moved to a machine without SEV-SNP on purpose. Needs: **--repo or --via**. ```sh pcctl node attest cvm-1 ``` ```text node cvm-1: AMD SEV-SNP verified (...); launch measurement ... ``` #### node rm ```sh pcctl --repo LOC,... node rm [--erase] NAME ``` Removes a node: revokes its certificate on every node with a CA-signed revocation list, drops it from the mesh and from the registry, and removes it from `nodes.json`. It can no longer be unlocked. `--erase` must come right after `rm`. | Flag | Default | Meaning | | --- | --- | --- | | `--erase` | off | First have the node crypto-erase its state volume and power off. If the node can't be reached, nothing is removed; run without `--erase` to remove it anyway. | Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc,s3://my-bucket/pc node rm --erase home-1 ``` #### node renew ```sh pcctl --repo LOC,... node renew [NAME...] ``` Renews node certificates (they last a year). Without names, renews every registered node whose certificate expires within 30 days or has expired; named nodes are renewed regardless. Works on expired certificates. Prints one line per node. Needs: **--repo or --via**. ```sh pcctl --via home-1 node renew ``` #### node storage ```sh pcctl --repo LOC,... node storage [--sync 1h|off] NAME [LOC,...] ``` Lets node `NAME` serve the cluster's storage, so admins can use `--via NAME`. Sends the node the locations (`--repo`'s, or the ones given, for a node that reaches them by other addresses), the write quorum (`--quorum`) and the S3 credentials those locations name, taken from your environment. The node keeps them on its encrypted disk and refuses locations it can't reach or that belong to another cluster. Only `s3://` and `node://` locations (and `ec:` over them) can be served. | Flag | Default | Meaning | | --- | --- | --- | | `--sync DURATION` | `1h` | How often the node copies blobs some locations lack from one that has them; `off` turns it off. | Needs: **--repo** (refused with `--via`). ```sh pcctl --repo 's3://a/pc?creds=r2,s3://b/pc?creds=b2' node storage --sync 30m home-1 ``` #### node sync ```sh pcctl node sync NAME ``` Has a node that serves storage copy missing blobs between its locations now, and prints the result. Exits with an error (`sync incomplete`) if any copy failed. Needs: **certificate**. ```sh pcctl node sync home-1 ``` #### node upgrade ```sh pcctl node upgrade NAME FILE.efi pcctl node upgrade --commit NAME ``` Installs a PowerCloud node image (`powercloud-VERSION-ARCH.efi` from a PowerCloud release, or `out/powercloud-ARCH.efi` built from source) on one node, A/B: the node writes it beside the running image and reboots into it once. When `pcctl` reaches the node on the new image, it commits it as the default. A node that doesn't come back on it reverts at its next boot. `pcctl` waits up to 10 minutes (`POWERCLOUD_UPGRADE_WAIT`). `--commit` (in that position) commits an image on trial, for example after you had to unlock the node first. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). Needs: **certificate**. ```sh pcctl node upgrade home-1 powercloud-vYYYY.MM.DD-amd64.efi ``` #### node poweroff, node reboot ```sh pcctl node poweroff NAME pcctl node reboot NAME ``` Powers off or reboots a node. After a reboot the node is locked until unlocked (by `node unlock`, its TPM or its peers). Needs: **certificate**. ```sh pcctl node reboot home-1 ``` ### Mesh #### mesh endpoint ```sh pcctl --repo LOC,... mesh endpoint NAME IP:PORT ``` Sets where peers reach node `NAME`'s WireGuard port, and publishes a new membership list. `HOST:PORT` (a DNS name, resolved by the nodes every minute) is accepted, and `''` clears the endpoint, for a node behind NAT. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc mesh endpoint home-1 home.example.net:51820 ``` #### mesh sync ```sh pcctl --repo LOC,... mesh sync ``` Asks every reachable node for its WireGuard key and records any that changed, then publishes the revocation list (if anything was ever revoked) and the membership list to every node. For a node whose key changed or that missed an update. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc mesh sync ``` #### mesh api Not in `pcctl`'s usage text. ```sh pcctl --repo LOC,... mesh api NAME HOST:PORT ``` Sets where peers reach node `NAME`'s API outside the mesh, which locked nodes use for peer unlock. By default that is the address `pcctl` uses (unless it is on this machine only), else the mesh endpoint's host. `''` returns to the default. Publishes a new membership list. Turn peer unlock on again afterwards, as holders' addresses are copied when it is turned on. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc mesh api home-2 192.0.2.11:7443 ``` ### Apps Apps are static Linux binaries or container images that a node runs as sandboxed services. All `app` commands talk to one node and need only the **certificate**. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). #### app deploy ```sh pcctl app deploy [OPTIONS] NODE NAME FILE [-- ARGS...] pcctl app deploy --image NAME[:TAG][@DIGEST] [OPTIONS] NODE NAME [-- COMMAND...] ``` Deploys (or replaces) app `NAME` on node `NODE`. For a binary, everything after `FILE` is passed to it as arguments, with or without `--`. For an image, `pcctl` pulls it on your machine for the node's architecture and sends the flattened files; arguments after `--` replace the image's command. Prints the node's apps afterwards. | Flag | Default | Meaning | | --- | --- | --- | | `--mem SIZE` | `256M` | Memory limit, with `K`, `M` or `G`. | | `--cpu N` | none | CPU limit in CPUs, such as `0.5`. A microVM gets one vCPU per CPU, rounded up. | | `--procs N` | 256 | Process and thread limit. | | `--env K=V` | | Environment variable. Repeatable. | | `--port NODE:APP` | | Publish the app's port `APP` on the node's port `NODE`; `NODE:APP/udp` for UDP. Repeatable. | | `--net host` | the app's own network | Share the node's network instead. | | `--image REF` | | Deploy a container image instead of a binary. | | `--link NAME:PORT` | | Let the app reach app `NAME`'s port `PORT` by its name; `/udp` for UDP. Repeatable. | | `--egress` | off | Let the app connect out, through the node (not to the node itself or other apps). | | `--vm` | off | Run the app in a microVM, under a kernel of its own (nodes with KVM). | ```sh pcctl app deploy --image nginx:alpine --port 80:80 home-1 site ``` #### app ls ```sh pcctl app ls NODE ``` Lists a node's apps: state, uptime, restarts, OOM kills, memory used and limit, CPU used and cap, seccomp, network and ports, last exit. ```sh pcctl app ls home-1 ``` #### app logs ```sh pcctl app logs NODE NAME [BYTES] ``` Prints the end of an app's output, up to `BYTES` (default 65536). ```sh pcctl app logs home-1 site 4096 ``` #### app restart ```sh pcctl app restart NODE NAME ``` Restarts an app. Not in `pcctl`'s usage summary. Prints nothing on success. ```sh pcctl app restart home-1 site ``` #### app rm ```sh pcctl app rm [--keep-data] NODE NAME ``` Removes an app from a node. | Flag | Default | Meaning | | --- | --- | --- | | `--keep-data` | off | Keep the app's data directory. | ```sh pcctl app rm --keep-data home-1 site ``` ### Cluster apps Cluster apps are apps the nodes place among themselves and move when a node goes away. All need **--repo or --via**. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). #### cluster app deploy ```sh pcctl --repo LOC,... cluster app deploy [--replicas N | --exclusive] [--nodes a,b] [--storage] [--volume] [APP OPTIONS] NAME FILE [-- ARGS...] pcctl --repo LOC,... cluster app deploy [--replicas N] [--nodes a,b] [APP OPTIONS] --image REF NAME [-- COMMAND...] ``` Publishes (or replaces) cluster app `NAME`, signed by the cluster CA, and sends it to every node it reaches; others get it from their peers. A binary must be a static ELF for amd64 or arm64. An image is pulled once for each architecture among the nodes it may run on (amd64 if none is reached). Takes every `app deploy` option, plus: | Flag | Default | Meaning | | --- | --- | --- | | `--replicas N` | 1 | How many nodes run the app. | | `--nodes a,b` | any node | The nodes that may run it, comma-separated. | | `--exclusive` | off | Run on at most one node at any time, through the nodes' consensus (needs `consensus init`). Only one replica. | | `--storage` | off | Give the app an S3 endpoint at `127.0.0.1:9000` backed by the cluster's storage. | | `--volume` | off | Keep the app's data directory in the cluster's storage, so it follows the app. Needs `--exclusive`. | ```sh pcctl --repo /srv/pc cluster app deploy --replicas 2 --image nginx:alpine --port 80:80 site ``` #### cluster app ls ```sh pcctl --repo LOC,... cluster app ls ``` Lists the cluster apps: replicas, allowed nodes, where each runs now (and its state), and the source (binary or image, with `+storage`, `+volume`, `+exclusive`). ```sh pcctl --via home-1 cluster app ls ``` #### cluster app rm ```sh pcctl --repo LOC,... cluster app rm NAME ``` Removes a cluster app from every node. ```sh pcctl --repo /srv/pc cluster app rm site ``` ### Consensus and key-value state The nodes can form a Raft group that keeps a key-value state they agree on, while a majority of voters is up. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md) for what uses it. #### consensus init ```sh pcctl --repo LOC,... consensus init [--nodes a,b,c] ``` Founds the group with a CA-signed document naming the first voters, and sends it to every node it reaches. Runs once per cluster; nodes enrolled later become voters by themselves. Warns that two voters tolerate no loss. | Flag | Default | Meaning | | --- | --- | --- | | `--nodes a,b,c` | every node | The first voters, comma-separated. | Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc consensus init ``` #### consensus status ```sh pcctl consensus status ``` Shows each node's view of the group: role, term, leader, applied index, number of keys, voters. Nodes without it show `not started`. Needs: **certificate**. ```sh pcctl consensus status ``` #### kv ```sh pcctl kv [--node NAME] [--stale] put KEY VALUE pcctl kv [--node NAME] [--stale] get KEY pcctl kv [--node NAME] [--stale] rm KEY pcctl kv [--node NAME] [--stale] ls [PREFIX] ``` Reads and changes the consensus state through a node. Changes and (by default) reads need a majority of voters and fail after about 10 seconds without one. `get` prints the value; `ls` prints `KEY=VALUE` lines. The flags come before the subcommand. | Flag | Default | Meaning | | --- | --- | --- | | `--node NAME` | the first node (by name) that answers and has the consensus state | The node to ask. | | `--stale` | off | Read the node's own copy, without a majority (may miss recent changes). | Needs: **certificate**. ```sh pcctl kv put greeting hello pcctl kv --stale get greeting ``` ### S3 gateway The S3 gateway lets S3 clients use the cluster's storage through nodes. See [S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md). #### s3 enable ```sh pcctl --repo LOC,... s3 enable [--port 9443] [--name DNS]... [--ip ADDR]... NODE... ``` Starts the gateway on the nodes, with a certificate from the cluster CA (valid a year) for the node's own addresses plus the names and addresses given. Running it again renews the certificate. Warns if a node serves no storage (`node storage`). | Flag | Default | Meaning | | --- | --- | --- | | `--port N` | 9443 | The port the gateway listens on. | | `--name DNS` | | A DNS name clients reach the gateway by. Repeatable. | | `--ip ADDR` | | An address clients reach the gateway at, besides the node's own. Repeatable. | Needs: **--repo** (refused with `--via`). ```sh pcctl --repo /srv/pc s3 enable --name s3.example.net home-1 cloud-1 ``` #### s3 disable ```sh pcctl s3 disable NODE... ``` Stops the gateway on the nodes. Needs: **certificate**. ```sh pcctl s3 disable home-1 ``` #### s3 key add ```sh pcctl --repo LOC,... s3 key add [--bucket B]... [--read-only] NAME ``` Creates an access key named `NAME` and prints `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` once. Keys are kept in the nodes' consensus state, so `consensus init` must have been run. | Flag | Default | Meaning | | --- | --- | --- | | `--bucket B` | all buckets | A bucket the key may use. Repeatable. | | `--read-only` | off | The key may only read. | Needs: **--repo** (refused with `--via`). ```sh pcctl --repo /srv/pc s3 key add --bucket backups --read-only restic ``` #### s3 key ls, s3 key rm ```sh pcctl s3 key ls pcctl s3 key rm NAME ``` `key ls` lists access keys: name, access key ID, buckets, read-only or read-write, creation date. `key rm` removes a key by name or access key ID; gateways refuse it as soon as their consensus state has the change. Needs: **certificate**. ```sh pcctl s3 key rm restic ``` #### s3 status ```sh pcctl s3 status ``` Shows which nodes run the gateway: port, names, number of access keys, days left on its certificate, or the error it failed with. Needs: **certificate**. #### s3 ca ```sh pcctl s3 ca ``` Prints the cluster CA certificate (PEM), for S3 clients to trust. Needs: **certificate**. ```sh pcctl s3 ca > ca.pem export AWS_CA_BUNDLE=ca.pem ``` ### Releases #### release publish ```sh pcctl --repo LOC,... release publish [--version V] IMAGE.efi... ``` Signs a release of the given node images (at most one per architecture, read from each image's header) and sends it to every node it reaches, with the image for that node's architecture. Nodes that are down get it from their peers. The nodes then upgrade themselves one at a time, and stop the rollout at the first node it fails on. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). | Flag | Default | Meaning | | --- | --- | --- | | `--version V` | the current UTC date and time, such as `2026-10-28T14:02Z` | The release name shown on nodes. | Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc release publish --version vYYYY.MM.DD powercloud-vYYYY.MM.DD-amd64.efi powercloud-vYYYY.MM.DD-arm64.efi ``` #### release status ```sh pcctl release status ``` Shows each node's architecture, running image, the release it has (serial and name) and its state on it, including the node a halted rollout failed on. Needs: **certificate**. ### Keys and access See [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md). #### login ```sh pcctl --repo LOC,... login ``` Issues a new admin certificate (valid 30 days) and writes it, with the CA certificate, to the configuration directory. Needs: **--repo or --via** (with `--via`, you must already hold a working admin certificate to reach the node). ```sh pcctl --repo /srv/pc,s3://my-bucket/pc login ``` #### revoke-admins ```sh pcctl --repo LOC,... revoke-admins ``` Revokes every admin certificate issued so far (for example one on a lost laptop), pushes the revocation list to every node, and issues you a new one. Other admins must run `login` again. Needs: **--repo or --via**. ```sh pcctl --repo /srv/pc revoke-admins ``` #### rotate ```sh pcctl --repo LOC,... rotate pcctl --repo LOC,... rotate finish [--force] ``` Replaces the cluster root key, after a passphrase or recovery key may have leaked. `rotate` asks for a new passphrase (or reads `POWERCLOUD_NEW_PASSPHRASE`), which must differ from the old one, prints a new recovery key, re-encrypts the store, deletes the data under the old keys from every location, and moves every reachable node to a new CA with its volume key rewrapped. Every location must be reachable. Run `rotate` again, with the new passphrase, to move nodes that were offline or to finish an interrupted rotation. `rotate finish` then stops nodes accepting the old CA and drops the old root key; it refuses while nodes are still on the old keys, unless `--force` (in that position) gives up on them, after which they must be removed and enrolled again. Moved nodes stop accepting the old CA by themselves 7 days after the rotation started. Needs: **--repo** (refused with `--via`). ```sh pcctl --repo /srv/pc,s3://my-bucket/pc rotate pcctl --repo /srv/pc,s3://my-bucket/pc rotate finish ``` ### Other #### version ```sh pcctl version ``` Prints the version `pcctl` was built as: the release name (`vYYYY.MM.DD`) for a PowerCloud release, or `dev` when built from source with `go build ./cmd/pcctl`. Needs nothing. ### Commands missing from the usage text `pcctl -h` summarises most commands. These exist in `pcctl` but are not in that summary, or appear there with fewer options: | Command | Where it is described | | --- | --- | | `node peer-unlock NAME on [--threshold K]`, `node peer-unlock NAME off` | [node peer-unlock](#node-peer-unlock) | | `node attest NAME sev-snp\|off` | [node attest](#node-attest) | | `mesh api NAME HOST:PORT` | [mesh api](#mesh-api) | | `app restart NODE NAME` | [app restart](#app-restart) | | `app rm --keep-data`, `app logs NODE NAME BYTES` | [app rm](#app-rm), [app logs](#app-logs) | | `app deploy` options `--procs`, `--port`, `--net`, `--image`, `--link`, `--egress`, `--vm` | [app deploy](#app-deploy) | | `cluster app deploy` options `--exclusive`, `--storage`, `--volume` | [cluster app deploy](#cluster-app-deploy) | ### Quick lookup | Command | Needs | | --- | --- | | `init`, `check`, `repair`, `gc`, `rotate`, `rotate finish`, `node storage`, `s3 enable`, `s3 key add` | --repo | | `put`, `get`, `ls`, `rm`, `login`, `revoke-admins`, `node enroll`, `node provision`, `node addr`, `node unlock`, `node tpm`, `node peer-unlock`, `node attest NAME sev-snp\|off`, `node rm`, `node renew`, `mesh endpoint`, `mesh sync`, `mesh api`, `cluster app deploy\|ls\|rm`, `consensus init`, `release publish` | --repo or --via | | `status`, `console`, `node ls`, `node info`, `node attest NAME`, `node sync`, `node upgrade`, `node poweroff`, `node reboot`, `app ...`, `consensus status`, `kv ...`, `s3 disable`, `s3 key ls`, `s3 key rm`, `s3 status`, `s3 ca`, `release status` | certificate | | `version` | nothing | --- ## Using PowerCloud with an AI assistant URL: https://powercloud.unidatum.com/docs/ai-agents > How an AI coding assistant such as Claude Code should read these docs, run pcctl safely, handle secrets, and which commands need a person's go-ahead. PowerCloud has no login and no shell on its nodes: everything happens through `pcctl`, a single command-line program. That makes it a good fit for an AI assistant working in your terminal, such as Claude Code. The assistant runs the same `pcctl` commands you would, reads their output and explains what it sees. This page is for both of you: the person setting the assistant up, and the assistant itself. ### Point the assistant at the docs Every page of this documentation is also published as plain Markdown, written to be read by machines as well as people: | URL | What it holds | |---|---| | `https://powercloud.unidatum.com/llms.txt` | An index of every page, with a one-line summary each | | `https://powercloud.unidatum.com/llms-full.txt` | Every page in one file, for loading the whole manual at once | | `https://powercloud.unidatum.com/docs/SLUG.md` | One page as Markdown, for example `/docs/pcctl.md` | The [pcctl reference](https://powercloud.unidatum.com/docs/pcctl.md) lists every command and flag. When the assistant is unsure about a command, it should read that page rather than guess flags. `pcctl` itself prints a usage summary when it is run with no arguments. ### Add PowerCloud to your project's instructions Assistants read a project instructions file at the start of each session: `CLAUDE.md` for Claude Code, `AGENTS.md` for many others. Paste this into it, and adjust the first two lines to your cluster: ```text ### PowerCloud - This project's cluster is managed with pcctl. Nodes: node-1, node-2 (see `pcctl node ls`). - Storage locations: set in POWERCLOUD_REPO by the person, never written in this file. - Docs: https://powercloud.unidatum.com/llms.txt (index), /docs/pcctl.md (command reference). - Start with `pcctl status` to see the cluster before changing anything. - Never ask for, print, store or commit the cluster passphrase, the recovery key, enrollment codes, join bundles or anything in the pcctl config directory. - Commands that need the passphrase: ask the person to run them, or to set POWERCLOUD_PASSPHRASE in their own shell first. - Ask before: node rm, gc, rm, rotate, revoke-admins, release publish, node upgrade, node poweroff/reboot, app rm, cluster app rm, s3 key rm, s3 disable, node tpm off. ``` ### What the assistant can do without any secret Once a person has enrolled a node or run `pcctl --repo ... login`, `pcctl` holds an admin certificate in its config directory, and these commands work with no passphrase. They only read, so the assistant can run them freely to find out what's going on: ```sh pcctl status # every node, and what needs attention pcctl node ls pcctl node info NAME pcctl app ls NODE pcctl app logs NODE NAME pcctl release status pcctl consensus status pcctl kv get KEY # and: pcctl kv ls [PREFIX] pcctl s3 status pcctl s3 key ls pcctl version ``` A good first step in any session is `pcctl status`, then `pcctl node info NAME` for any node it flags. ### Commands that need the passphrase Commands that open the cluster's storage need the cluster passphrase. That covers everything given `--repo` or `--via`: `put`, `get`, `ls`, `rm`, `check`, `repair`, `gc`, `login`, enrolling, provisioning and removing nodes, `node unlock`, `node tpm`, `node peer-unlock`, `node attest NAME sev-snp|off`, `node renew`, `node storage`, `mesh`, `rotate`, `release publish`, `cluster app`, `consensus init`, `s3 enable` and `s3 key add`. `pcctl` asks for the passphrase on the terminal. An assistant usually runs commands without a terminal, so `pcctl` stops with: ```text pcctl: no terminal for passphrase; set POWERCLOUD_PASSPHRASE (and POWERCLOUD_NEW_PASSPHRASE for rotate) ``` When the assistant sees this, it should not ask for the passphrase in the chat, because whatever is typed there is stored in the conversation. Do one of these instead: - **The person runs the command.** The assistant shows the exact command and the person runs it in their own terminal, then shares the output if it's useful. - **The person sets the variable in the shell the assistant uses.** Do this only on a machine you trust, and for the session only: `export POWERCLOUD_PASSPHRASE=...` typed by you, not by the assistant, and never written to a file in the project. `POWERCLOUD_RECOVERY_KEY` unlocks the cluster in place of the passphrase. It is meant for emergencies only, and an assistant should never be given it. ### Secrets the assistant must never handle | Secret | Where it appears | Why it matters | |---|---|---| | Cluster passphrase | Typed at `pcctl` prompts, or `POWERCLOUD_PASSPHRASE` | Opens every byte of the cluster's data | | Recovery key | Printed once by `pcctl init` | Same power as the passphrase, and it can't be changed except by `pcctl rotate` | | Enrollment code | A new node's console | Lets whoever has it enroll the node into their own cluster | | Join bundle (`NAME.join`) | Written by `pcctl node provision` | Holds the node's private keys and its disk key | | pcctl config directory | `~/.config/powercloud` on Linux, `~/Library/Application Support/powercloud` on macOS, or `POWERCLOUD_CONFIG` | Admin certificate and key: full control of the nodes | | S3 secret keys | Printed once by `pcctl s3 key add` | Read, and possibly write, access to the gateway | | Storage credentials | `NAME_ACCESS_KEY_ID`, `NAME_SECRET_ACCESS_KEY` in the environment | Access to the encrypted data at the provider | The assistant shouldn't `cat` these files, echo these variables, put them in commit messages or code, or copy them into other files. Keep `*.join` files and the config directory out of version control, with a `.gitignore` entry, for example. ### Commands to confirm with a person first `pcctl` doesn't ask "are you sure?". It does what it's told straight away. An assistant should describe what it is about to do and wait for a yes before running any of these: | Command | Effect | |---|---| | `node rm [--erase] NAME` | Revokes the node for good, and with `--erase` wipes its disk | | `rm NAME`, `gc` | Delete data (`gc --dry-run` is safe and shows what would go) | | `rotate`, `rotate finish` | Change the cluster's keys and passphrase | | `revoke-admins` | Invalidates every admin certificate, including other people's | | `release publish` | Rolls a new image out to every node, one at a time | | `node upgrade NAME FILE.efi` | Reboots the node into a new image | | `node poweroff NAME`, `node reboot NAME` | Takes the node and its apps down | | `node tpm NAME off`, `node peer-unlock NAME off` | The node then needs a person to unlock it after each reboot | | `app rm`, `cluster app rm` | Stop and remove apps | | `s3 key rm`, `s3 disable` | Cut off S3 clients | | `init` | Creates a new cluster in the given locations | Read-only commands and `--dry-run` runs never need confirmation. ### Reading pcctl's output - Output is plain text meant for people, not JSON. Parse it loosely, and prefer asking `pcctl` again over guessing from partial output. - The exit status is 0 on success, 1 on an error, and 2 on a usage mistake. Errors are printed as one line starting `pcctl:`, and many end with the command that fixes them in brackets. For example, an expired certificate suggests `pcctl --repo ... node renew NAME`. - Long operations such as `node upgrade` and `release publish` wait and report progress as they go. Let them finish. Interrupting one doesn't undo it, and a node mid-upgrade rolls back by itself if the new image fails. - [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md) lists the common errors and their fixes. ### A worked example You ask: "Why is the photos app down?" A careful assistant: 1. Runs `pcctl status` and sees that node-2 needs attention because it is locked after a reboot. 2. Runs `pcctl node info node-2` to confirm, and `pcctl app ls node-2` to see which apps were there. 3. Explains that node-2 rebooted and is waiting to be unlocked, and that unlocking needs the passphrase. It shows you `pcctl --repo ... node unlock node-2 CODE` to run yourself, or suggests `pcctl --repo ... node tpm node-2 on` for unattended unlock in future, after reading [Unlocking nodes](https://powercloud.unidatum.com/docs/unlock.md). 4. After you've unlocked it, runs `pcctl app logs node-2 photos` to check that the app is back. At no step does it need, see or store the passphrase.