Enroll a node

A freshly booted PowerCloud node belongs to no cluster until you enroll it. There are two ways: read the one-time enrollment code from its console and run pcctl node enroll, or issue the node's identity ahead of time with pcctl node provision and hand the machine the resulting join bundle, so it joins at first boot with nobody at its console. Either way, the node ends up with a certificate from your cluster CA, an encrypted state volume on its disk, and a place in the mesh.

To get a machine booted in the first place, see Install a node. Both ways need the cluster keys, so the commands below take --repo (or --via NODE) and ask for your passphrase.

Enroll from the console

When a new node boots, pcd prints on its serial console and monitor:

  PowerCloud node 262c2fdc861a9cf2 is waiting to be enrolled.
  On your admin machine run:
    pcctl node enroll <this-node-address> <code> <name>
  enrollment code: EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO

On a cloud VM, the console is the provider's console log, for example aws ec2 get-console-output --instance-id ID --latest on AWS.

From your admin machine, enroll it at an address where you can reach its API (TCP 7443 by default), under a name you choose:

pcctl --repo LOC,... node enroll 192.0.2.10 EYWC-7XEG-DKOP-F2IR-23UA-GTJQ-5G3M-ZXTO home-1

enrolled node home-1 (262c2fdc861a9cf2) at 192.0.2.10:7443
mesh membership: 2 nodes (serial 5)
  NODE     MEMBERSHIP
  colo-1   serial 5
  home-1   serial 5

The address may be HOST or HOST:PORT; without a port, 7443 is used. Node names must be unique in the cluster.

What enrollment does

1. Checks it's the right machine. The code holds an 80-bit fingerprint of the node's key. pcctl connects over TLS and aborts unless the key matches, so nobody in between can pose as the node.
2. Proves you read the console. The code also holds an 80-bit secret. pcctl sends the node its certificates with a MAC under that secret, and the node accepts them only if the MAC checks out.
3. Issues the node certificate from the cluster CA (valid one year), and sends a new volume ID and the key for the node's encrypted state volume.
4. Records the node in the cluster's registry (in the encrypted store, so every admin sees it) and in this machine's list of nodes, and writes you an admin certificate if you don't have a current one.
5. Joins it to the mesh. pcctl reads the node's WireGuard key, sends it the current revocation list, and publishes a new membership list to every node it can reach.

From then on, the node accepts only clients with a certificate from your cluster.

What happens on first boot

A node booted from the disk image has a state partition, empty until enrollment. When the node is enrolled (from the console or a join bundle), it:

• Creates its encrypted state volume on that partition: dm-crypt with AES-256-XTS, and an HMAC-SHA256 tag on every sector (dm-integrity), so a disk changed while the node is off reads as an error, not as altered data. The volume key is wrapped by a key only someone who can unlock the cluster can derive.
• Formats only a blank disk. pcd creates the volume only if the first MiB of the state partition is all zeros, so it never formats a disk that holds something else.
• Moves its identity onto the volume, then keeps its certificates, mesh configuration, apps and any storage there.
• Grows into the disk in the background: it writes zeros over the space beyond the initial filesystem (an authenticated volume can't read blocks never written), then grows the filesystem.

pcctl node info NAME then shows, for example:

state        encrypted volume, 7811 MiB (7690 MiB free)
volume       authenticated: AES-256-XTS with HMAC-SHA256 per sector (dm-integrity)

After every later reboot, the volume is locked: the node shows an unlock code on its console and waits for pcctl --repo LOC,... node unlock NAME CODE, unless TPM unlock or peer unlock is on. See Unlocking nodes after a reboot.

Join unattended with a join bundle

A cloud VM, or a machine started by automation, has nobody to read its console. Only admins hold the CA key, so you issue the node's whole identity ahead of time:

pcctl --repo LOC,... node provision web-1 web-1.join

provisioned node web-1 (92bb5bdf55e3932f)
mesh membership: 2 nodes (serial 2)
  NODE     MEMBERSHIP
  my-node  serial 2
  web-1    no address yet (it will get it from a mesh peer)
join bundle written to web-1.join. Give it to the machine as user data:
  AWS:     aws ec2 run-instances --user-data file://web-1.join ...
  GCP:     gcloud compute instances create --metadata-from-file user-data=web-1.join ...
  Azure:   az vm create --user-data web-1.join ...
  QEMU:    -fw_cfg name=opt/powercloud/join,file=web-1.join
It holds the node's private keys and disk key: delete the file once the
node is up, and the user data too where the provider allows it.
Once the node is up, tell pcctl where it is: pcctl --repo ... node addr web-1 HOST[:PORT]

Flags, given before the name:

The bundle is a text file: a #powercloud-join v1 line followed by base64-encoded JSON. It holds the node's key and certificate, the CA certificate, a WireGuard key, the state volume's ID and key, and the signed membership and revocation lists. Provisioning also adds the node to the registry and publishes a membership list that includes it, so its peers accept it the moment it comes up. pcctl refuses to overwrite an existing file.

Giving the bundle to the machine

On first boot, a node with no identity yet looks for a bundle: first a QEMU fw_cfg file, then the user data of the cloud its firmware's SMBIOS strings name (AWS through IMDSv2, Google Cloud's user-data attribute, or Azure). A machine anywhere else never waits on a metadata service. A just-started instance's metadata service may need a moment, so the node tries five times before giving up.

The node then checks the bundle: it must hold a node certificate from the bundle's CA for the bundle's key, and membership and revocation lists signed by that CA that name the node. If all is well, it creates its encrypted state volume as at enrollment, installs its identity and starts enrolled, logging joined the cluster with a join bundle. It never offers console enrollment. If anything is wrong, it logs why and falls back to waiting for console enrollment.

• User data that isn't a bundle (a cloud-init script, say) is ignored.
• It joins once. A node that already has an identity ignores the bundle, which cloud user data keeps serving. After a reboot it is locked like any enrolled node.
• Tested where. AWS user data and the QEMU fw_cfg path run end to end in automated tests and on real AWS instances. The Google Cloud and Azure metadata readers are tested only against stand-in metadata servers, and the images have not been booted on those clouds yet.

Tell pcctl where the node is

Unless you gave --addr, pcctl doesn't know the new node's address. Once the machine is up, record it:

pcctl --repo LOC,... node addr web-1 203.0.113.7

node web-1 is at 203.0.113.7:7443
mesh endpoint of web-1 set to 203.0.113.7:51820
mesh membership: 2 nodes (serial 3)
  NODE     MEMBERSHIP
  my-node  serial 3
  web-1    serial 3

node addr also derives the node's mesh endpoint from the address (port 51820) if none was set, or if the old one was derived from the old address, and tells every node. Run it again whenever the node's address changes, such as a cloud VM that comes back with a new public IP after a stop and start. node addr is also how a second admin machine learns about a node it didn't enroll itself.

Keep the bundle secret

The bundle holds the node's private keys and its disk key in the clear: whoever reads it can pose as that node and decrypt that node's disk. A cloud provider can read user data, just as it can read the console used for enrollment, so this adds no exposure on an untrusted node, but user data stays readable for as long as the instance keeps it.

• Delete the bundle file once the node is up, and the user data too where the provider allows it.
• If a bundle leaks, pcctl --repo LOC,... node rm NAME revokes that node.
• For a confidential VM, prefer console enrollment: the provider can read user data but not the VM's memory.
• Bundles made before a key rotation (pcctl rotate) carry the old keys. Remove such a node and provision it again.

Joining the mesh

Enrolled nodes connect to each other over WireGuard (UDP 51820). A node's endpoint is where its peers send to:

• At console enrollment, the endpoint defaults to the enrollment address's host on port 51820.
• With a join bundle, it is what --endpoint gave, or what node addr derives later.
• To set it yourself: pcctl --repo LOC,... mesh endpoint NAME IP:PORT (or HOST:PORT, which nodes resolve again every minute, for dynamic DNS). For a node behind NAT that nobody can reach first, clear it with pcctl --repo LOC,... mesh endpoint NAME ''; it then connects outwards and its peers learn its public address.

Each change publishes a new CA-signed membership list. Nodes pass newer lists to each other, so a node that was offline picks up new members from any peer. Two nodes behind NAT find each other through a node both can reach, or relay through a public node when their NATs prevent a direct connection. Public nodes need UDP 51820 open, and UDP 51821-51920 for relaying.

Check the mesh with pcctl node info NAME, which lists each peer, its endpoint, the WireGuard handshake age and whether its API answers over the mesh:

mesh      fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820)
  PEER    MESH ADDRESS                          ENDPOINT            HANDSHAKE  API OVER MESH
  colo-1  fdd5:f552:ce70::4c1e:92b0:7d3a:15e2   198.51.100.4:51820  12s ago    ok, 3.1 ms

pcctl status flags any node that reaches fewer peers than it has. If enrollment reports that the node's mesh key is unknown, run pcctl --repo LOC,... mesh sync later; it re-reads every node's key and publishes the list again.

Good to know

• Whoever reads the console first can enroll the node. On a cloud VM that includes the provider. If someone races you, your own enrollment fails with "already enrolled", so the attack is visible; nodes on plain cloud VMs are untrusted anyway (see Concepts).
• Node certificates last a year. Renew them with pcctl --repo LOC,... node renew (see Managing nodes).
• Removing a node (pcctl --repo LOC,... node rm [--erase] NAME) revokes its certificate on every node; --erase also has it crypto-erase its disk, after which it boots as a blank machine ready to enroll again.
• Problems while enrolling are collected in Troubleshooting.