# Enroll a node > How to bring a freshly booted node into a PowerCloud cluster - from its console with an enrollment code, or unattended with a join bundle given as cloud user data or a QEMU fw_cfg file - and what the node does on its first boot and when it joins the mesh. A freshly booted PowerCloud node belongs to no cluster until you enroll it. There are two ways: read the one-time enrollment code from its console and run `pcctl node enroll`, or issue the node's identity ahead of time with `pcctl node provision` and hand the machine the resulting join bundle, so it joins at first boot with nobody at its console. Either way, the node ends up with a certificate from your cluster CA, an encrypted state volume on its disk, and a place in the mesh. To get a machine booted in the first place, see [Install a node](https://powercloud.unidatum.com/docs/install.md). Both ways need the cluster keys, so the commands below take `--repo` (or `--via NODE`) and ask for your passphrase. ## Enroll from the console When a new node boots, `pcd` prints on its serial console and monitor: ```text PowerCloud node 262c2fdc861a9cf2 is waiting to be enrolled. On your admin machine run: pcctl node enroll enrollment code: EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO ``` On a cloud VM, the console is the provider's console log, for example `aws ec2 get-console-output --instance-id ID --latest` on AWS. From your admin machine, enroll it at an address where you can reach its API (TCP 7443 by default), under a name you choose: ```sh pcctl --repo LOC,... node enroll 192.0.2.10 EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO home-1 ``` ```text enrolled node home-1 (262c2fdc861a9cf2) at 192.0.2.10:7443 mesh membership: 2 nodes (serial 5) NODE MEMBERSHIP colo-1 serial 5 home-1 serial 5 ``` The address may be `HOST` or `HOST:PORT`; without a port, 7443 is used. Node names must be unique in the cluster. ### What enrollment does ```mermaid %% alt: The node shows its enrollment code on its console; pcctl checks the node's key against the code, then sends it a certificate and its disk key under a MAC only the code's secret can produce. sequenceDiagram actor A as You participant C as Console participant N as New node participant P as pcctl N->>C: Code: key fingerprint + secret A->>C: Read the code A->>P: node enroll ADDR CODE NAME P->>N: Fetch node's key over TLS P->>P: Key matches code? P->>N: Certificates, disk key, MAC N->>N: Check MAC, create volume N-->>P: Enrolled P->>P: Record in registry P->>N: Membership list (to every node) ``` 1. **Checks it's the right machine.** The code holds an 80-bit fingerprint of the node's key. `pcctl` connects over TLS and aborts unless the key matches, so nobody in between can pose as the node. 2. **Proves you read the console.** The code also holds an 80-bit secret. `pcctl` sends the node its certificates with a MAC under that secret, and the node accepts them only if the MAC checks out. 3. **Issues the node certificate** from the cluster CA (valid one year), and sends a new volume ID and the key for the node's encrypted state volume. 4. **Records the node** in the cluster's registry (in the encrypted store, so every admin sees it) and in this machine's list of nodes, and writes you an admin certificate if you don't have a current one. 5. **Joins it to the mesh.** `pcctl` reads the node's WireGuard key, sends it the current revocation list, and publishes a new membership list to every node it can reach. From then on, the node accepts only clients with a certificate from your cluster. ## What happens on first boot A node booted from the disk image has a state partition, empty until enrollment. When the node is enrolled (from the console or a join bundle), it: - **Creates its encrypted state volume** on that partition: dm-crypt with AES-256-XTS, and an HMAC-SHA256 tag on every sector (dm-integrity), so a disk changed while the node is off reads as an error, not as altered data. The volume key is wrapped by a key only someone who can unlock the cluster can derive. - **Formats only a blank disk.** `pcd` creates the volume only if the first MiB of the state partition is all zeros, so it never formats a disk that holds something else. - **Moves its identity onto the volume**, then keeps its certificates, mesh configuration, apps and any storage there. - **Grows into the disk** in the background: it writes zeros over the space beyond the initial filesystem (an authenticated volume can't read blocks never written), then grows the filesystem. `pcctl node info NAME` then shows, for example: ```text state encrypted volume, 7811 MiB (7690 MiB free) volume authenticated: AES-256-XTS with HMAC-SHA256 per sector (dm-integrity) ``` After every later reboot, the volume is locked: the node shows an **unlock code** on its console and waits for `pcctl --repo LOC,... node unlock NAME CODE`, unless TPM unlock or peer unlock is on. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). ## Join unattended with a join bundle A cloud VM, or a machine started by automation, has nobody to read its console. Only admins hold the CA key, so you issue the node's whole identity ahead of time: ```sh pcctl --repo LOC,... node provision web-1 web-1.join ``` ```text provisioned node web-1 (92bb5bdf55e3932f) mesh membership: 2 nodes (serial 2) NODE MEMBERSHIP my-node serial 2 web-1 no address yet (it will get it from a mesh peer) join bundle written to web-1.join. Give it to the machine as user data: AWS: aws ec2 run-instances --user-data file://web-1.join ... GCP: gcloud compute instances create --metadata-from-file user-data=web-1.join ... Azure: az vm create --user-data web-1.join ... QEMU: -fw_cfg name=opt/powercloud/join,file=web-1.join It holds the node's private keys and disk key: delete the file once the node is up, and the user data too where the provider allows it. Once the node is up, tell pcctl where it is: pcctl --repo ... node addr web-1 HOST[:PORT] ``` Flags, given before the name: | Flag | Meaning | |---|---| | `--addr HOST[:PORT]` | Where `pcctl` reaches the node's API, if you already know it (for example a reserved IP). Otherwise set it later with `node addr`. | | `--endpoint IP:PORT` | Where peers reach the node's WireGuard port, if it is public. Leave it out for a node behind NAT. | The bundle is a text file: a `#powercloud-join v1` line followed by base64-encoded JSON. It holds the node's key and certificate, the CA certificate, a WireGuard key, the state volume's ID and key, and the signed membership and revocation lists. Provisioning also adds the node to the registry and publishes a membership list that includes it, so its peers accept it the moment it comes up. `pcctl` refuses to overwrite an existing file. ### Giving the bundle to the machine | Where | How | |---|---| | AWS | `aws ec2 run-instances ... --metadata-options HttpTokens=required --user-data file://web-1.join` | | Google Cloud | `gcloud compute instances create ... --metadata-from-file user-data=web-1.join` | | Azure | `az vm create ... --user-data web-1.join` | | QEMU | `-fw_cfg name=opt/powercloud/join,file=web-1.join` | | libvirt, Proxmox | Pass the same fw_cfg file (a libvirt `fwcfg` sysinfo entry, or Proxmox's `args` option) | On first boot, a node with no identity yet looks for a bundle: first a QEMU fw_cfg file, then the user data of the cloud its firmware's SMBIOS strings name (AWS through IMDSv2, Google Cloud's `user-data` attribute, or Azure). A machine anywhere else never waits on a metadata service. A just-started instance's metadata service may need a moment, so the node tries five times before giving up. ```mermaid %% alt: You provision the node ahead of time, so the cluster already expects it; the new machine reads the join bundle from fw_cfg or cloud user data at first boot, checks it and joins the mesh with nobody at its console. sequenceDiagram actor A as You participant P as pcctl participant M as Existing nodes participant U as User data
or fw_cfg participant N as New node A->>P: node provision web-1 FILE P->>P: Issue keys, certificate P->>M: Membership list with web-1 A->>U: Start machine with bundle N->>U: First boot: any bundle? U-->>N: Join bundle N->>N: Verify, create volume N->>M: WireGuard, mutual TLS M-->>N: Accepted, a member ``` The node then checks the bundle: it must hold a node certificate from the bundle's CA for the bundle's key, and membership and revocation lists signed by that CA that name the node. If all is well, it creates its encrypted state volume as at enrollment, installs its identity and starts enrolled, logging `joined the cluster with a join bundle`. It never offers console enrollment. If anything is wrong, it logs why and falls back to waiting for console enrollment. - **User data that isn't a bundle** (a cloud-init script, say) is ignored. - **It joins once.** A node that already has an identity ignores the bundle, which cloud user data keeps serving. After a reboot it is locked like any enrolled node. - **Tested where.** AWS user data and the QEMU fw_cfg path run end to end in automated tests and on real AWS instances. The Google Cloud and Azure metadata readers are tested only against stand-in metadata servers, and the images have not been booted on those clouds yet. ### Tell pcctl where the node is Unless you gave `--addr`, `pcctl` doesn't know the new node's address. Once the machine is up, record it: ```sh pcctl --repo LOC,... node addr web-1 203.0.113.7 ``` ```text node web-1 is at 203.0.113.7:7443 mesh endpoint of web-1 set to 203.0.113.7:51820 mesh membership: 2 nodes (serial 3) NODE MEMBERSHIP my-node serial 3 web-1 serial 3 ``` `node addr` also derives the node's mesh endpoint from the address (port 51820) if none was set, or if the old one was derived from the old address, and tells every node. Run it again whenever the node's address changes, such as a cloud VM that comes back with a new public IP after a stop and start. `node addr` is also how a second admin machine learns about a node it didn't enroll itself. ### Keep the bundle secret The bundle holds the node's private keys and its disk key in the clear: whoever reads it can pose as that node and decrypt that node's disk. A cloud provider can read user data, just as it can read the console used for enrollment, so this adds no exposure on an untrusted node, but user data stays readable for as long as the instance keeps it. - Delete the bundle file once the node is up, and the user data too where the provider allows it. - If a bundle leaks, `pcctl --repo LOC,... node rm NAME` revokes that node. - For a confidential VM, prefer console enrollment: the provider can read user data but not the VM's memory. - Bundles made before a key rotation (`pcctl rotate`) carry the old keys. Remove such a node and provision it again. ## Joining the mesh Enrolled nodes connect to each other over WireGuard (UDP 51820). A node's **endpoint** is where its peers send to: - **At console enrollment**, the endpoint defaults to the enrollment address's host on port 51820. - **With a join bundle**, it is what `--endpoint` gave, or what `node addr` derives later. - **To set it yourself**: `pcctl --repo LOC,... mesh endpoint NAME IP:PORT` (or `HOST:PORT`, which nodes resolve again every minute, for dynamic DNS). For a node behind NAT that nobody can reach first, clear it with `pcctl --repo LOC,... mesh endpoint NAME ''`; it then connects outwards and its peers learn its public address. Each change publishes a new CA-signed membership list. Nodes pass newer lists to each other, so a node that was offline picks up new members from any peer. Two nodes behind NAT find each other through a node both can reach, or relay through a public node when their NATs prevent a direct connection. Public nodes need UDP 51820 open, and UDP 51821-51920 for relaying. Check the mesh with `pcctl node info NAME`, which lists each peer, its endpoint, the WireGuard handshake age and whether its API answers over the mesh: ```text mesh fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820) PEER MESH ADDRESS ENDPOINT HANDSHAKE API OVER MESH colo-1 fdd5:f552:ce70::4c1e:92b0:7d3a:15e2 198.51.100.4:51820 12s ago ok, 3.1 ms ``` `pcctl status` flags any node that reaches fewer peers than it has. If enrollment reports that the node's mesh key is unknown, run `pcctl --repo LOC,... mesh sync` later; it re-reads every node's key and publishes the list again. ## Good to know - **Whoever reads the console first can enroll the node.** On a cloud VM that includes the provider. If someone races you, your own enrollment fails with "already enrolled", so the attack is visible; nodes on plain cloud VMs are untrusted anyway (see [Concepts](https://powercloud.unidatum.com/docs/overview.md)). - **Node certificates last a year.** Renew them with `pcctl --repo LOC,... node renew` (see [Managing nodes](https://powercloud.unidatum.com/docs/nodes.md)). - **Removing a node** (`pcctl --repo LOC,... node rm [--erase] NAME`) revokes its certificate on every node; `--erase` also has it crypto-erase its disk, after which it boots as a blank machine ready to enroll again. - Problems while enrolling are collected in [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md).