Upgrades and releases

A PowerCloud node runs one file, an EFI image holding the kernel, the init system and every service, so upgrading a node means booting a new image. Every upgrade is an A/B upgrade: the node boots the new image once, on trial, and keeps it only once it proves it works, otherwise it falls back to the image it ran before. You can roll a release out to the whole cluster with pcctl release publish, which lets the nodes upgrade themselves one at a time, or upgrade one node by hand with pcctl node upgrade.

What a release holds

A PowerCloud release is named for the day it was built, vYYYY.MM.DD, with .1, .2 and so on added for a later release on the same day. It holds these files, where VERSION is the release name and ARCH is amd64 or arm64:

The Google Cloud and Azure files are built to each cloud's documented import format and have upload scripts, but they have not been booted on those clouds yet.

To upgrade nodes that are already running, you need only the .efi files. The disk images are for adding new machines.

Check the files

Check every file you download against SHA256SUMS before you use it. In the directory holding the downloads:

sha256sum -c --ignore-missing SHA256SUMS            # Linux
shasum -a 256 -c --ignore-missing SHA256SUMS        # macOS

Each file you have should report OK. Don't use a file that doesn't.

Building from source instead

You can also build the same files yourself. make image writes the disk images out/powercloud-amd64.img and out/powercloud-arm64.img and, beside each, the node image out/powercloud-amd64.efi and out/powercloud-arm64.efi. go build -o pcctl ./cmd/pcctl builds pcctl. A pcctl built this way reports its version as dev.

If your machines boot with Secure Boot, build with your signing key (see "Upgrades and Secure Boot" below).

See what your nodes run

pcctl version            # the release this pcctl was built for
pcctl node info home-1   # one node in detail
pcctl status             # every node, and what needs attention

pcctl node info NAME shows, among other lines:

• version: the build the node's software reports.
• image: the first 16 hex digits of the SHA-256 of the image it runs, with (upgrade on trial: ...) while an upgrade hasn't been committed.
• last upgrade: ROLLED BACK: image ... didn't boot or wasn't committed, if the last upgrade failed.
• release: the release serial, its version name, and the node's state on it (see "Follow the rollout" below).

pcctl status lists under "needs attention" any node that runs an upgrade on trial, and any node that rejected or halted on a release.

These commands use only your admin certificate, so they need no --repo. See Admins, keys and revocation for admin certificates.

Roll out a release to every node

Publishing a release is the normal way to upgrade a cluster. Give pcctl release publish one .efi image per architecture you run; pcctl reads each image's architecture from the file itself.

pcctl --repo LOC,... release publish --version vYYYY.MM.DD \
  powercloud-vYYYY.MM.DD-amd64.efi powercloud-vYYYY.MM.DD-arm64.efi

LOC,... is your cluster's storage locations; you can set them once in the POWERCLOUD_REPO environment variable instead of passing --repo, or reach the storage through a node with --via NODE (see Storage). The command asks for the cluster passphrase, because it signs the release with the cluster's keys.

--version is the name nodes show for the release. Use the release's own name. Without it, the name is the current date and time in UTC.

release 4 (vYYYY.MM.DD) published
  home-1: release and amd64 image sent
  colo-1: release and arm64 image sent
  aws-1: not reached (...); it gets the release from its peers
nodes upgrade one at a time; follow with 'pcctl release status'

The number after release is the release's serial, which goes up by one with every publish.

What happens next

1. The release is signed and stored. pcctl signs a release document with the cluster's certificate authority (CA): its serial, its name, and each image's SHA-256 and size. Nodes accept only a release signed by your cluster.
2. Every node gets it. pcctl sends the document and the image for each node's architecture to every node it reaches. A node that was off or unreachable fetches both from its peers over the encrypted mesh once it is back. Every copy is checked against the signed hash and size.
3. Nodes upgrade one at a time, in node ID order. A node installs the release only when every reachable peer before it already runs it, and no peer is installing or trying it. A peer that went quiet in the middle of its upgrade (rebooting, or waiting to be unlocked) holds the rollout for up to 30 minutes.
4. Each node commits itself. On its trial boot, a node keeps the new image once it is healthy: it reaches a peer on the mesh again (or, if it had no peers, it has run for 30 seconds). If it isn't healthy within 5 minutes, it reboots to the previous image.
5. A bad release stops at the first node. A release whose image doesn't boot, doesn't become healthy, or can't be installed is rejected on that node and never retried there. Every node that sees a peer reject or halt on it halts too, so the rest of the cluster keeps running the old image.

To recover from a bad release, publish a new one. Publishing clears the halt.

Follow the rollout

pcctl release status

NAME    ARCH   IMAGE             RELEASE           STATE
aws-1   arm64  1f0c9e2a7b3d4c5e  4 (vYYYY.MM.DD)   pending
colo-1  arm64  9a8b7c6d5e4f3a2b  4 (vYYYY.MM.DD)   current
home-1  amd64  3c4d5e6f7a8b9c0d  4 (vYYYY.MM.DD)   trial

Upgrade one node by hand

pcctl node upgrade upgrades a single node, and waits while you watch:

pcctl node upgrade home-1 powercloud-vYYYY.MM.DD-amd64.efi

uploading powercloud-vYYYY.MM.DD-amd64.efi (24 MiB, sha256 3c4d5e6f7a8b9c0d) to home-1
node home-1 is rebooting into the new image; waiting for it (up to 10m0s)
node home-1 runs 3c4d5e6f7a8b9c0d, committed: it is now the default image

What happens:

1. The node checks the image's hash and that it is an EFI image for its architecture, and writes it next to the running image on its EFI system partition.
2. It asks the firmware to boot the new image on the next boot only, and reboots. No boot loader is involved: the firmware's one-time boot setting does the counting.
3. pcctl waits for the node to answer on the new image, then commits it. The old image is kept as the previous one, and the new one becomes the default.

If the node already runs that image, pcctl says node NAME already runs ... and does nothing.

When the new image fails

Anything short of a commit reverts to the old image: an image the firmware won't load, a kernel that panics, or a trial nobody commits. The next boot is the old image, the node reports the rollback, and pcctl node upgrade fails with:

pcctl: the new image didn't boot on home-1; it is back on 9a8b7c6d5e4f3a2b

A node that must be unlocked

A node without unattended unlock waits at every boot, the trial boot included, until its encrypted state volume is unlocked (see Unlocking nodes after a reboot). If that takes longer than pcctl waits, pcctl stops with a message saying so. Unlock the node, then commit the trial by hand:

pcctl --repo LOC,... node unlock home-1 CODE
pcctl node upgrade --commit home-1

If you don't commit it, the node goes back to the old image at its next boot. pcctl waits 10 minutes by default; set POWERCLOUD_UPGRADE_WAIT (a duration such as 20m) to wait longer.

A trial started with pcctl node upgrade is never committed by the node itself; only pcctl commits it.

Upgrades and Secure Boot

With Secure Boot on, the firmware runs only images signed with your key, and that includes upgrades. An unsigned image, or one signed with another key, is refused by the firmware, which shows up as a rollback (and, during a release, as rejected on the first node, which halts the rest).

The images in a PowerCloud release are not signed with your key. If your nodes boot with Secure Boot, build your images signed with your key, as described in Unlocking nodes after a reboot:

POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image

Then publish out/powercloud-amd64.efi and out/powercloud-arm64.efi. Images signed with the same key keep TPM unlock working across the upgrade.

Keep pcctl current

pcctl has no self-update. To update it, download the pcctl-VERSION-OS-ARCH file for your system from a PowerCloud release, check it against SHA256SUMS, make it executable and put it in place of the old one:

sha256sum -c --ignore-missing SHA256SUMS
chmod +x pcctl-vYYYY.MM.DD-linux-amd64
mv pcctl-vYYYY.MM.DD-linux-amd64 /usr/local/bin/pcctl
pcctl version

Your admin certificate, node list and other settings live in pcctl's configuration directory, not in the binary, so they carry over. It is simplest to keep pcctl at the same release as your nodes. pcctl for Windows doesn't exist yet.

Limits

• A hang isn't a failure. A trial image that hangs instead of panicking stays hung until someone resets the machine (a cloud's reboot button, a power cycle); then it reverts. There is no watchdog.
• Health means "back on the mesh". A release that boots and reaches its peers, but breaks something else, is committed.
• Whoever can unlock the cluster can publish a release. Releases are signed with the cluster CA; there is no separate release-signing key yet. Secure Boot, if on, still requires images signed with your own key.
• A locked node needs its admin. Without TPM or peer unlock, every trial boot waits to be unlocked. If that takes more than 30 minutes, the next node in the rollout goes ahead.
• Only nodes booted from a disk image can be upgraded. A kernel booted directly by a hypervisor has no EFI system partition.
• Room on the EFI system partition. It holds three images during an upgrade (default, previous and next). Current disk images have plenty of room; machines installed from much older images have a smaller partition, and there the node drops the previous image if the new one wouldn't fit.
• Where it has run. A/B upgrades are checked in automated tests on the standard virtual-machine UEFI firmware for both architectures, and have run on AWS's firmware on x86. They have not yet run on Graviton, Google Cloud, Azure or physical machines. Firmware that ignores the one-time boot setting never boots the trial, which shows as a rollback.