# Upgrades and releases > How to keep PowerCloud nodes and pcctl current, from checking a release's files to rolling it out to every node one at a time, upgrading a single node by hand, and what happens when an image fails to boot. A PowerCloud node runs one file, an EFI image holding the kernel, the init system and every service, so upgrading a node means booting a new image. Every upgrade is an A/B upgrade: the node boots the new image once, on trial, and keeps it only once it proves it works, otherwise it falls back to the image it ran before. You can roll a release out to the whole cluster with `pcctl release publish`, which lets the nodes upgrade themselves one at a time, or upgrade one node by hand with `pcctl node upgrade`. ```mermaid %% alt: A node boots a new image once on trial; it becomes the default only when committed, and anything else sends the node back to the image it ran before. stateDiagram-v2 direction LR Running: Old image Trial: New image on trial Committed: New image is default RolledBack: Back on old image [*] --> Running Running --> Trial: install, reboot Trial --> Committed: healthy, commit Trial --> RolledBack: fails, no commit ``` ## What a release holds A PowerCloud release is named for the day it was built, `vYYYY.MM.DD`, with `.1`, `.2` and so on added for a later release on the same day. It holds these files, where `VERSION` is the release name and `ARCH` is `amd64` or `arm64`: | File | What it is for | | --- | --- | | `powercloud-VERSION-ARCH.efi` | The node image. This is what `pcctl release publish` and `pcctl node upgrade` send to running nodes. | | `powercloud-VERSION-ARCH.img.xz` | A bootable disk image for new machines (decompress it, then write it to a disk or upload it to AWS). See [Install a node](https://powercloud.unidatum.com/docs/install.md). | | `powercloud-VERSION-ARCH.gce.tar.gz` | The disk image packed for Google Cloud. | | `powercloud-VERSION-ARCH.vhd.xz` | The disk image packed for Azure, as a fixed VHD. | | `pcctl-VERSION-OS-ARCH` | The `pcctl` command-line tool for `linux` and `darwin` (macOS), each for `amd64` and `arm64`. The macOS builds are signed and notarized. | | `SHA256SUMS` | The SHA-256 checksum of every other file. | The Google Cloud and Azure files are built to each cloud's documented import format and have upload scripts, but they have not been booted on those clouds yet. To upgrade nodes that are already running, you need only the `.efi` files. The disk images are for adding new machines. ### Check the files Check every file you download against `SHA256SUMS` before you use it. In the directory holding the downloads: ```sh sha256sum -c --ignore-missing SHA256SUMS # Linux shasum -a 256 -c --ignore-missing SHA256SUMS # macOS ``` Each file you have should report `OK`. Don't use a file that doesn't. ### Building from source instead You can also build the same files yourself. `make image` writes the disk images `out/powercloud-amd64.img` and `out/powercloud-arm64.img` and, beside each, the node image `out/powercloud-amd64.efi` and `out/powercloud-arm64.efi`. `go build -o pcctl ./cmd/pcctl` builds `pcctl`. A `pcctl` built this way reports its version as `dev`. If your machines boot with Secure Boot, build with your signing key (see "Upgrades and Secure Boot" below). ## See what your nodes run ```sh pcctl version # the release this pcctl was built for pcctl node info home-1 # one node in detail pcctl status # every node, and what needs attention ``` `pcctl node info NAME` shows, among other lines: - `version`: the build the node's software reports. - `image`: the first 16 hex digits of the SHA-256 of the image it runs, with `(upgrade on trial: ...)` while an upgrade hasn't been committed. - `last upgrade`: `ROLLED BACK: image ... didn't boot or wasn't committed`, if the last upgrade failed. - `release`: the release serial, its version name, and the node's state on it (see "Follow the rollout" below). `pcctl status` lists under "needs attention" any node that runs an upgrade on trial, and any node that rejected or halted on a release. These commands use only your admin certificate, so they need no `--repo`. See [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md) for admin certificates. ## Roll out a release to every node Publishing a release is the normal way to upgrade a cluster. Give `pcctl release publish` one `.efi` image per architecture you run; pcctl reads each image's architecture from the file itself. ```sh pcctl --repo LOC,... release publish --version vYYYY.MM.DD \ powercloud-vYYYY.MM.DD-amd64.efi powercloud-vYYYY.MM.DD-arm64.efi ``` `LOC,...` is your cluster's storage locations; you can set them once in the `POWERCLOUD_REPO` environment variable instead of passing `--repo`, or reach the storage through a node with `--via NODE` (see [Storage](https://powercloud.unidatum.com/docs/storage.md)). The command asks for the cluster passphrase, because it signs the release with the cluster's keys. `--version` is the name nodes show for the release. Use the release's own name. Without it, the name is the current date and time in UTC. ```text release 4 (vYYYY.MM.DD) published home-1: release and amd64 image sent colo-1: release and arm64 image sent aws-1: not reached (...); it gets the release from its peers nodes upgrade one at a time; follow with 'pcctl release status' ``` The number after `release` is the release's serial, which goes up by one with every publish. ### What happens next 1. **The release is signed and stored.** pcctl signs a release document with the cluster's certificate authority (CA): its serial, its name, and each image's SHA-256 and size. Nodes accept only a release signed by your cluster. 2. **Every node gets it.** pcctl sends the document and the image for each node's architecture to every node it reaches. A node that was off or unreachable fetches both from its peers over the encrypted mesh once it is back. Every copy is checked against the signed hash and size. 3. **Nodes upgrade one at a time, in node ID order.** A node installs the release only when every reachable peer before it already runs it, and no peer is installing or trying it. A peer that went quiet in the middle of its upgrade (rebooting, or waiting to be unlocked) holds the rollout for up to 30 minutes. 4. **Each node commits itself.** On its trial boot, a node keeps the new image once it is healthy: it reaches a peer on the mesh again (or, if it had no peers, it has run for 30 seconds). If it isn't healthy within 5 minutes, it reboots to the previous image. 5. **A bad release stops at the first node.** A release whose image doesn't boot, doesn't become healthy, or can't be installed is rejected on that node and never retried there. Every node that sees a peer reject or halt on it halts too, so the rest of the cluster keeps running the old image. For example, a release that fails on the second of three nodes: ```mermaid %% alt: pcctl sends a signed release to every node; the nodes upgrade one at a time in node ID order, and when one rejects the release the next node halts and keeps the old image. sequenceDiagram participant P as pcctl participant A as node 1 participant B as node 2 participant C as node 3 P->>A: signed release + image P->>B: signed release + image P->>C: signed release + image Note over A: trial: commits A-->>B: current Note over B: trial: rolls back B-->>C: rejected Note over C: halts ``` To recover from a bad release, publish a new one. Publishing clears the halt. ### Follow the rollout ```sh pcctl release status ``` ```text NAME ARCH IMAGE RELEASE STATE aws-1 arm64 1f0c9e2a7b3d4c5e 4 (vYYYY.MM.DD) pending colo-1 arm64 9a8b7c6d5e4f3a2b 4 (vYYYY.MM.DD) current home-1 amd64 3c4d5e6f7a8b9c0d 4 (vYYYY.MM.DD) trial ``` | State | Meaning | | --- | --- | | `pending` | Waiting for the image or for its turn. | | `installing` | The image is installed; the node is about to reboot to try it. | | `trial` | Running the new image on trial; it commits itself once healthy. | | `current` | Running the release's image. | | `rejected` | It tried the release, and it failed on this node. | | `halted: failed on NAME` | Not trying it, because it failed on the node named. | | `not upgradable` | The node didn't boot from a PowerCloud disk image (for example, a kernel booted directly by a hypervisor), so it can't be upgraded this way. | | `unreachable` | pcctl couldn't reach the node. | ## Upgrade one node by hand `pcctl node upgrade` upgrades a single node, and waits while you watch: ```sh pcctl node upgrade home-1 powercloud-vYYYY.MM.DD-amd64.efi ``` ```text uploading powercloud-vYYYY.MM.DD-amd64.efi (24 MiB, sha256 3c4d5e6f7a8b9c0d) to home-1 node home-1 is rebooting into the new image; waiting for it (up to 10m0s) node home-1 runs 3c4d5e6f7a8b9c0d, committed: it is now the default image ``` What happens: 1. The node checks the image's hash and that it is an EFI image for its architecture, and writes it next to the running image on its EFI system partition. 2. It asks the firmware to boot the new image on the next boot only, and reboots. No boot loader is involved: the firmware's one-time boot setting does the counting. 3. pcctl waits for the node to answer on the new image, then commits it. The old image is kept as the previous one, and the new one becomes the default. If the node already runs that image, pcctl says `node NAME already runs ...` and does nothing. ### When the new image fails Anything short of a commit reverts to the old image: an image the firmware won't load, a kernel that panics, or a trial nobody commits. The next boot is the old image, the node reports the rollback, and `pcctl node upgrade` fails with: ```text pcctl: the new image didn't boot on home-1; it is back on 9a8b7c6d5e4f3a2b ``` ### A node that must be unlocked A node without unattended unlock waits at every boot, the trial boot included, until its encrypted state volume is unlocked (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). If that takes longer than pcctl waits, pcctl stops with a message saying so. Unlock the node, then commit the trial by hand: ```sh pcctl --repo LOC,... node unlock home-1 CODE pcctl node upgrade --commit home-1 ``` If you don't commit it, the node goes back to the old image at its next boot. pcctl waits 10 minutes by default; set `POWERCLOUD_UPGRADE_WAIT` (a duration such as `20m`) to wait longer. A trial started with `pcctl node upgrade` is never committed by the node itself; only pcctl commits it. ## Upgrades and Secure Boot With Secure Boot on, the firmware runs only images signed with your key, and that includes upgrades. An unsigned image, or one signed with another key, is refused by the firmware, which shows up as a rollback (and, during a release, as `rejected` on the first node, which halts the rest). The images in a PowerCloud release are not signed with your key. If your nodes boot with Secure Boot, build your images signed with your key, as described in [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md): ```sh POWERCLOUD_SB_KEY=~/powercloud-sb/db.key POWERCLOUD_SB_CERT=~/powercloud-sb/db.crt make image ``` Then publish `out/powercloud-amd64.efi` and `out/powercloud-arm64.efi`. Images signed with the same key keep TPM unlock working across the upgrade. ## Keep pcctl current `pcctl` has no self-update. To update it, download the `pcctl-VERSION-OS-ARCH` file for your system from a PowerCloud release, check it against `SHA256SUMS`, make it executable and put it in place of the old one: ```sh sha256sum -c --ignore-missing SHA256SUMS chmod +x pcctl-vYYYY.MM.DD-linux-amd64 mv pcctl-vYYYY.MM.DD-linux-amd64 /usr/local/bin/pcctl pcctl version ``` Your admin certificate, node list and other settings live in pcctl's configuration directory, not in the binary, so they carry over. It is simplest to keep pcctl at the same release as your nodes. pcctl for Windows doesn't exist yet. ## Limits - **A hang isn't a failure.** A trial image that hangs instead of panicking stays hung until someone resets the machine (a cloud's reboot button, a power cycle); then it reverts. There is no watchdog. - **Health means "back on the mesh".** A release that boots and reaches its peers, but breaks something else, is committed. - **Whoever can unlock the cluster can publish a release.** Releases are signed with the cluster CA; there is no separate release-signing key yet. Secure Boot, if on, still requires images signed with your own key. - **A locked node needs its admin.** Without TPM or peer unlock, every trial boot waits to be unlocked. If that takes more than 30 minutes, the next node in the rollout goes ahead. - **Only nodes booted from a disk image can be upgraded.** A kernel booted directly by a hypervisor has no EFI system partition. - **Room on the EFI system partition.** It holds three images during an upgrade (default, previous and next). Current disk images have plenty of room; machines installed from much older images have a smaller partition, and there the node drops the previous image if the new one wouldn't fit. - **Where it has run.** A/B upgrades are checked in automated tests on the standard virtual-machine UEFI firmware for both architectures, and have run on AWS's firmware on x86. They have not yet run on Graviton, Google Cloud, Azure or physical machines. Firmware that ignores the one-time boot setting never boots the trial, which shows as a rollback.