# Managing nodes
> Day-to-day operation of PowerCloud nodes with pcctl, including status, node info, the web console, power, storage sync, changing addresses, the mesh, NAT traversal, relays and how pcctl finds and authenticates to nodes.
This page covers running PowerCloud nodes once they are enrolled: seeing their health, opening the web console, rebooting them, and keeping the mesh working when addresses change. Most of these commands need only your admin certificate, not the cluster passphrase. The ones that change the cluster's records (addresses, endpoints) unlock the cluster with `--repo` or `--via`.
A node is a machine running PowerCloud OS that has joined your cluster (see [Enroll a node](https://powercloud.unidatum.com/docs/enroll.md)). Terms such as cluster, repository and admin certificate are defined in [Concepts](https://powercloud.unidatum.com/docs/overview.md). Every command and flag is listed in [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md).
## How pcctl finds and authenticates to nodes
`pcctl` keeps its credentials and its list of nodes in a local configuration directory:
- `$POWERCLOUD_CONFIG` if that variable is set.
- Otherwise the operating system's user configuration directory plus `powercloud`: `~/.config/powercloud` on Linux (or `$XDG_CONFIG_HOME/powercloud`), and `~/Library/Application Support/powercloud` on macOS.
The directory is created with mode 0700 and each file with mode 0600. It holds:
| File | What it is |
| --- | --- |
| `admin.key`, `admin.crt` | Your admin client certificate and its private key, issued by the cluster CA. Valid for 30 days. |
| `ca.crt` | The cluster CA certificate, used to check that a node is really one of yours. |
| `nodes.json` | The nodes this machine knows: each node's name, node ID and API address (`HOST:PORT`). |
| `amd-kds/` | A cache of AMD's certificates, used when verifying SEV-SNP attestation. |
Every connection to a node is mutual TLS. `pcctl` presents the admin certificate, and checks that the node's certificate comes from your cluster CA and names the node ID recorded for it in `nodes.json`. A node accepts only certificates from its own cluster CA, and checks the role (`admin` or `node`) on every request. Nodes have no SSH and no login: the node API on TCP port 7443 is the only way in.
```mermaid
%% alt: pcctl on your computer reaches each node's API directly over mutual TLS, while the nodes talk to each other over their own WireGuard mesh.
flowchart TB
subgraph you ["Your computer"]
direction LR
cfg[("Config directory")] -->|"admin cert,
nodes.json"| pcctl["pcctl"]
end
subgraph cluster ["Your nodes"]
direction LR
a["cloud-1"] <-->|WireGuard mesh| b["home-1"]
end
you -->|"mutual TLS to each
node's API, TCP 7443"| cluster
```
### Getting and renewing the admin certificate
`pcctl --repo LOC,... login` unlocks the cluster with the passphrase and writes a fresh admin certificate:
```sh
pcctl --repo /srv/pc,s3://my-bucket/pc login
```
```text
admin certificate for cluster 92283421c94b924005629e1405c47c54 written to /home/alex/.config/powercloud (valid 30 days)
```
You rarely need to run it by hand. Any command that unlocks the cluster (with `--repo` or `--via`) reissues the saved certificate when it has a week or less left, and prints a note on stderr. `node enroll` and `node provision` write one if none is saved. If the certificate has expired, commands that use it say so and ask you to run `login`. To revoke every admin certificate, for example one on a lost laptop, see [Admins, keys and revocation](https://powercloud.unidatum.com/docs/access.md).
To save typing, set the storage locations once in the environment. `POWERCLOUD_REPO` stands in for `--repo` on every command:
```sh
export POWERCLOUD_REPO=/srv/pc,s3://my-bucket/pc
```
### Which nodes pcctl knows
`nodes.json` is filled in by this machine's own commands: `node enroll`, `node provision --addr`, and `node addr` add or update a node, and `node rm` removes it. The cluster's shared record of nodes (the registry, kept in the encrypted storage) is what `--repo` and `--via` commands use, so every admin sees the same nodes there.
On a second admin machine, `login` writes the certificate but no node list. Record each node's address on that machine with `node addr` (it needs the cluster passphrase), after which `status`, `node info` and the other certificate-only commands work there too:
```sh
pcctl --repo LOC,... login
pcctl --repo LOC,... node addr home-1 192.0.2.10
```
## Cluster status
`pcctl status` asks every known node about itself, all at once, and prints one line per node followed by what needs attention. It uses only the admin certificate.
```sh
pcctl status
```
```text
NODE STATE ARCH VERSION UPTIME CPU MEMORY MESH APPS STORAGE CERT
cloud-1 ok arm64 vYYYY.MM.DD 72h4m10s 3% of 2 412/3891 MiB 2/2 peers 1/1 2 loc 341d
home-1 ok amd64 vYYYY.MM.DD 5h12m3s 12% of 4 1204/7820 MiB 1/2 peers 3/3 - 200d
vps-1 unreachable - - - - - - - - -
needs attention:
home-1: reaches 1 of 2 mesh peers
vps-1: unreachable (dial tcp 198.51.100.4:7443: i/o timeout)
```
The columns:
| Column | Meaning |
| --- | --- |
| `STATE` | `ok`, plus `AMD SEV-SNP` when the node runs on a confidential VM; `unreachable` when it doesn't answer within 10 seconds, or answers with an error. |
| `ARCH`, `VERSION`, `UPTIME` | The node's architecture, the PowerCloud version it runs and how long since it booted. |
| `CPU` | How busy its CPUs are, of how many. |
| `MEMORY` | Memory in use of the total, in MiB. |
| `MESH` | How many of its mesh peers it can reach over the mesh. |
| `APPS` | Apps running, of apps deployed. |
| `STORAGE` | How many storage locations it serves (`-` if it serves none; see [Storage](https://powercloud.unidatum.com/docs/storage.md)). |
| `CERT` | Days until its node certificate expires. |
### Needs attention
When nothing is wrong, the last line is `all nodes healthy` (it also says so when `nodes.json` lists no nodes at all). Otherwise each line names a node and a problem:
| Message | What to do |
| --- | --- |
| `unreachable (...)` | The node is down, locked after a reboot, or its address changed. Check it with `pcctl node info NAME`; unlock it ([Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)) or update its address with `node addr`. |
| `reaches N of M mesh peers` | Some peers can't be reached over the mesh. See [the mesh](#the-mesh) below. |
| `app NAME is STATE (...)` | An app isn't running; the last exit is shown. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). |
| `last storage sync incomplete` | The node's last copy between its storage locations failed for some blobs, or a location couldn't be listed. Run `pcctl node sync NAME` once the location is back. |
| `certificate expires in N days` | Fewer than 7 days left. Run `pcctl --repo LOC,... node renew`. |
| `release V rejected` or `halted` | A published release failed on this node. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). |
| `runs an upgrade on trial` | The node booted a new image that isn't committed yet. See [Upgrades and releases](https://powercloud.unidatum.com/docs/upgrades.md). |
| `cluster app NAME: no node has the memory or CPU for it` | A cluster app runs on fewer nodes than it should. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). |
| `state is in RAM, lost at reboot` | The node has no encrypted state disk, so its identity and apps don't survive a reboot. |
## Listing nodes and node details
`pcctl node ls` prints the nodes in this machine's `nodes.json`, without contacting them:
```sh
pcctl node ls
```
```text
NAME ID ADDRESS
cloud-1 3c91e07d5b2a8f14 203.0.113.7:7443
home-1 fa8adc53910a6a79 192.0.2.10:7443
```
`pcctl node info NAME` asks one node for its details:
```sh
pcctl node info home-1
```
It prints these fields, the later ones only when they apply:
| Field | Meaning |
| --- | --- |
| `name`, `id`, `cluster` | The node's name, node ID and cluster ID. |
| `arch`, `kernel`, `version`, `uptime` | Architecture, kernel release, PowerCloud version, time since boot. |
| `boot` | How it booted, including whether Secure Boot is on. |
| `lockdown` | The kernel lockdown mode. |
| `state` | `encrypted volume, N MiB (N MiB free)`, or `RAM (lost at reboot)`. |
| `certificate` | When the node certificate expires, with a reminder to run `node renew` from 30 days before. |
| `cluster CA` | The CA fingerprint; during a key rotation, also the old CA it still accepts and until when. |
| `confidential` | The confidential computing kind, with a hint to verify it with `pcctl node attest NAME`. |
| `image` | The running image's sha256, and whether an upgrade is on trial; `last upgrade` reports a rollback. |
| `release` | The published release's serial, name and this node's state on it. |
| `storage`, `last sync` | Locations served, write quorum, sync interval, and the last sync's result. |
| `volume`, `tpm unlock`, `peer unlock` | How the state volume is encrypted and how it unlocks at boot ([Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). |
| `membership`, `revocations` | The serials of the signed membership and revocation lists the node holds. |
| `mesh` | Its mesh address and WireGuard port, then a table of peers. |
The peer table shows, for each peer, its mesh address, the endpoint WireGuard sends to (with `(relay NAME)` when the pair is relayed), the age of the last WireGuard handshake, and whether the peer's API answers over the mesh, with the round-trip time:
```text
mesh fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820)
PEER MESH ADDRESS ENDPOINT HANDSHAKE API OVER MESH
cloud-1 fdd5:f552:ce70::3c91:e07d:5b2a:8f14 203.0.113.7:51820 12s ago ok, 18.4 ms
home-2 fdd5:f552:ce70::77e2:1ab:9c3d:4e55 203.0.113.7:51822 (relay cloud-1) 31s ago ok, 41.0 ms
```
If the node refuses the admin certificate, `pcctl` says so and suggests `login` (admin certificates may have been revoked). If the node's own certificate has expired, it suggests `node renew NAME`.
## The web console
`pcctl console` serves a web page on your own machine that shows the cluster as `pcctl status` sees it, live, and lets you read app logs and deploy, restart and remove apps. The nodes serve no web page: the console runs inside `pcctl` and reaches the nodes with your admin certificate, like every other command, so no key leaves your machine.
```sh
pcctl console
```
```text
PowerCloud console at http://127.0.0.1:41377/?t=5b0c...
(for one browser, the first to open it; Ctrl-C to stop)
```
It opens your browser at that link and runs until you press Ctrl-C.
| Flag | Default | What it does |
| --- | --- | --- |
| `--port N` | any free port | The port to listen on. |
| `--listen ADDR` | `127.0.0.1` | The IP address to listen on. Any address other than loopback lets other machines in; `0.0.0.0` listens on every address. Must be an IP address, not a name. |
| `--host NAME` | this machine's name and addresses | A name or address other machines use to reach the console. Repeatable. Only used with `--listen`. |
| `--no-open` | off | Don't open a browser; just print the link. |
### What it shows
- **A summary bar**: nodes up, apps running, CPU busy across all nodes, memory in use, how much memory apps' limits promise of what the nodes set aside for apps, and the number of issues.
- **Needs attention**: the same list as `pcctl status`.
- **Nodes**: one row per node with state, CPU, memory, apps, architecture, version, uptime, mesh peers reached, storage locations served, consensus role and leader, release state, and days until the certificate expires.
- **A node's detail** (select its row): bars for CPU, memory, app memory and app CPU promised and in use, and the state disk; its addresses (where `pcctl` reaches it, the node's own addresses, and its mesh address with each peer's round-trip time or relay); and facts such as node ID, kernel, boot, state encryption, how it unlocks, what apps get, the image and lockdown.
- **The node's apps**: state, uptime, restarts, memory, CPU, network, where each published port is reached (as a link for TCP ports), whether it is a cluster app, and the last exit.
- **Logs**: the last 64 KiB of an app's output, refreshed every 2 seconds while Follow is ticked.
The page refreshes every 5 seconds and shows how old its data is. A short guided tour starts on a browser's first visit, and again from the Guide button.
### What you can do from it
- **Restart** an app, or **Remove** it (with an option to keep its data directory). Removing a cluster app removes it from the whole cluster.
- **Deploy an app** to the selected node, from a container image (pulled on your machine and sent to the node) or a static Linux binary you upload (up to 512 MiB), with memory, CPU cap, published ports, environment, links, microVM and egress options. See [Apps on a node](https://powercloud.unidatum.com/docs/apps.md) for what these mean.
- **Deploy a cluster app**, which the nodes place among themselves, with replicas, allowed nodes, exclusive, storage and volume options. This needs the cluster's keys, so it is offered only when you start the console with the repository: `pcctl --repo LOC,... console`. A console started with `--via` or with neither flag can't publish cluster apps. See [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md).
### Who can open it
- The link carries a random token. The first browser to open it trades the token for a cookie (HttpOnly, SameSite=Strict), and the link lets no other browser in after that. To open it in another browser, stop and restart `pcctl console` for a new link.
- Requests must use one of the console's own host names, which blocks DNS rebinding, and changes must come from the console's own page.
- The page loads nothing from elsewhere (a strict Content-Security-Policy).
### Reaching it from another machine
By default the console listens on loopback only. The safest way to use it from another machine is an SSH tunnel to a loopback console. On the machine that has `pcctl` and your admin certificate (here `admin-box`), start the console on a fixed port without a browser:
```sh
pcctl console --port 8080 --no-open
```
On the machine with the browser, forward the port, then open the link the console printed (it uses `127.0.0.1:8080`, which the tunnel carries):
```sh
ssh -L 8080:127.0.0.1:8080 admin-box
```
`--listen` puts it on the network over plain HTTP, and `pcctl` warns that anyone who can see the traffic can take the link or the session. Use it only on a network you trust:
```sh
pcctl console --listen 0.0.0.0 --port 8080 --host admin-box.lan --no-open
```
Pass `--no-open` whenever the browser that should use the console is on another machine. Without it, `--listen 0.0.0.0` opens a browser on this machine, which then takes the one-time link.
## Power and reboots
```sh
pcctl node reboot home-1
pcctl node poweroff home-1
```
```text
node home-1 is rebooting
```
Both need only the admin certificate. After a reboot a node's encrypted state stays locked until it is unlocked, by you with `pcctl --repo LOC,... node unlock NAME CODE`, or by itself with TPM or peer unlock. See [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md). Until then `pcctl status` reports it as unreachable. A powered-off node has to be started again at the machine or at its provider.
## Storage sync on a node
A node can serve the cluster's storage locations so that admins can use `--via NAME` instead of `--repo` (set up with `node storage`, described in [Storage](https://powercloud.unidatum.com/docs/storage.md)). Such a node copies any blob that some of its locations lack from one that has it, one minute after it starts and then every hour by default. `pcctl node sync NAME` runs that copy now:
```sh
pcctl node sync home-1
```
```text
node home-1 synced its storage: 2026-10-28 14:02 UTC: 18234 blobs, 12 copied
```
It exits with an error (`sync incomplete`) if any copy failed or a location couldn't be listed. Sync only fills gaps and needs no keys; replacing stale or corrupt copies is `pcctl --repo LOC,... repair`.
## When an address changes
There are two addresses per node, and they can change separately:
- **The API address** (`HOST:PORT`, port 7443 by default) is where `pcctl` reaches the node.
- **The mesh endpoint** (`IP:PORT` or `HOST:PORT`, port 51820 by default) is where the other nodes send the node's WireGuard traffic.
### A node's API address: node addr
When a node comes back with a new IP (a cloud VM restarted without a fixed IP, a home router handing out a new lease), tell `pcctl` where it is:
```sh
pcctl --repo LOC,... node addr cloud-1 203.0.113.50
```
```text
node cloud-1 is at 203.0.113.50:7443
mesh endpoint of cloud-1 set to 203.0.113.50:51820
```
This updates the cluster registry and this machine's `nodes.json`. If the node's mesh endpoint was empty or had been derived from its old address, `node addr` also sets it to the new address on port 51820 and publishes a new membership list to every node, so the other nodes find it again. An endpoint you set by hand with `mesh endpoint` is left alone. A DNS name works too (`node addr home-1 home.example.net`), for a node behind dynamic DNS.
### A node's mesh endpoint: mesh endpoint
Set the endpoint explicitly when the other nodes must reach the node at a different address from the one `pcctl` uses, for example a public address with a port forwarded on the router while `pcctl` uses the LAN address:
```sh
pcctl --repo LOC,... mesh endpoint home-1 198.51.100.7:51820
pcctl --repo LOC,... mesh endpoint home-1 home.example.net:51820
pcctl --repo LOC,... mesh endpoint home-1 ''
```
The last form clears the endpoint, for a node behind NAT that nobody can send to first. Every change publishes a new signed membership list to the nodes. With a DNS name, the nodes resolve it themselves, again every minute, and apply a new address only to a peer with no handshake in the last three minutes.
### Re-reading keys: mesh sync
Enrolling, removing and changing endpoints publish the membership list on their own. `pcctl --repo LOC,... mesh sync` asks every reachable node for its WireGuard key, records any that changed, and publishes the membership and revocation lists to every node again. Use it when a node was missed or its key changed:
```sh
pcctl --repo LOC,... mesh sync
```
Nodes that can't be reached keep the key on record (with a warning), and get the new lists from their mesh peers later.
## The mesh
Nodes talk to each other over an encrypted WireGuard mesh. Each node gets a private IPv6 address derived from the cluster ID and its node ID, so there are no addresses to manage. The peer list is the membership list, signed by the cluster CA, which `pcctl` pushes to every node it can reach whenever it changes; nodes that were offline get it from any peer. Inside the tunnel, nodes authenticate each other again with their node certificates. Every node checks each peer's API through the mesh every 5 seconds, which is what `pcctl node info` and the `MESH` column of `pcctl status` report.
### NAT traversal: what's automatic
A node behind NAT (a home router, a cloud VM without a public IP) needs no configuration beyond a clear or unreachable endpoint:
- It connects outwards to the nodes that have a public endpoint, which learn its NAT's public address from its packets.
- Nodes share what they see. A node with a peer it has no session with asks the peers it can reach every 5 seconds, and tries the address they report. Two nodes behind NAT do this at the same time, which opens both NATs, and they get a direct session. Traffic between them doesn't pass through the public node.
- When two nodes still can't connect after 20 seconds (symmetric NAT, as with carrier-grade NAT and many corporate firewalls), they relay through a public node. Both pick the same relay; if it doesn't connect them within another 20 seconds, they move on to the next. The relay forwards WireGuard packets it can't read or change: the session is end to end between the two nodes.
- Relayed pairs try a direct path again every 10 minutes, backing off up to about every 10 hours after repeated failures, in case the network changed.
`pcctl node info` shows a relayed peer as `(relay NAME)`. These paths are checked in automated tests with emulated NATs.
The three kinds of path, in a cluster with one public node:
```mermaid
%% alt: Nodes behind NAT connect out to a public node, learn each other's addresses through it and then talk directly; a node behind symmetric NAT is relayed through the public node instead.
flowchart TB
pub["cloud-1, public endpoint"]
subgraph nat ["Behind ordinary NATs"]
direction LR
a["home-1"] <-->|"direct session"| b["office-1"]
end
subgraph sym ["Behind symmetric NAT"]
c["home-2"]
end
pub <-->|"each connects out;
addresses shared"| nat
pub <-.->|"relays home-2's
WireGuard packets"| sym
```
### What you configure
- **At least one node with a public endpoint.** Two nodes behind NAT find each other only through a third node both can reach. Relays are the nodes with an endpoint in the membership list.
- **Firewall ports** on public nodes:
| Port | Protocol | Who connects | Needed on |
| --- | --- | --- | --- |
| 7443 | TCP | `pcctl` (and, for peer unlock, locked peers) | Every node `pcctl` manages |
| 51820 | UDP | Other nodes' WireGuard | Every node with a public endpoint |
| 51821 to 51920 | UDP | Relayed WireGuard traffic | Public nodes that may act as relays |
| 9443 | TCP | S3 clients | Nodes running the S3 gateway ([S3 gateway](https://powercloud.unidatum.com/docs/s3-gateway.md)) |
- **The right endpoint.** Enrolling a node sets its endpoint from the address you enrolled it at, on port 51820. If that is a LAN address the other nodes can't reach, set the public one with `mesh endpoint` (with a port forward on the router), or clear it with `mesh endpoint NAME ''` so the node is treated as behind NAT. Note that `node addr` sets an empty endpoint again from the new address; clear it afterwards if the node is behind NAT.
### Limits
- Mesh changes start from an admin: nodes only pass on membership lists signed by the CA.
- Relaying costs the relay's bandwidth and a detour. Pairs behind truly symmetric NATs keep relaying.
- When the first peer to report a missing node's address is on the same LAN as it, it reports the LAN address, which a node elsewhere can't use; the two connect once that peer loses its session.
- The mesh carries traffic between the nodes' APIs (health checks, consensus, replicas on other nodes). Your admin machine is not on the mesh: `pcctl` reaches each node's API directly, on its API address.
## Node storage
A node's encrypted disk can hold a replica of the cluster's storage (`node://NAME` locations), and a node can serve the storage locations to admins so they use `--via NAME` with only the passphrase and admin certificate. Both are covered in [Storage](https://powercloud.unidatum.com/docs/storage.md).
## Good to know
- Commands that only read or act on one node (`status`, `node ls`, `node info`, `node reboot`, `node poweroff`, `node sync`, `node upgrade`, `node attest NAME`, `app ...`, `console`) need only the admin certificate.
- Commands that change the cluster's records (`node addr`, `mesh endpoint`, `mesh sync`, `node renew`, `node rm`) unlock the cluster with the passphrase, through `--repo` or `--via`.
- `pcctl status` and the console ask each node for at most 10 seconds, so one slow node doesn't hold up the rest.
- For problems not covered here, see [Troubleshooting](https://powercloud.unidatum.com/docs/troubleshooting.md).