Managing nodes

This page covers running PowerCloud nodes once they are enrolled: seeing their health, opening the web console, rebooting them, and keeping the mesh working when addresses change. Most of these commands need only your admin certificate, not the cluster passphrase. The ones that change the cluster's records (addresses, endpoints) unlock the cluster with --repo or --via.

A node is a machine running PowerCloud OS that has joined your cluster (see Enroll a node). Terms such as cluster, repository and admin certificate are defined in Concepts. Every command and flag is listed in pcctl command reference.

How pcctl finds and authenticates to nodes

pcctl keeps its credentials and its list of nodes in a local configuration directory:

• $POWERCLOUD_CONFIG if that variable is set.
• Otherwise the operating system's user configuration directory plus powercloud: ~/.config/powercloud on Linux (or $XDG_CONFIG_HOME/powercloud), and ~/Library/Application Support/powercloud on macOS.

The directory is created with mode 0700 and each file with mode 0600. It holds:

Every connection to a node is mutual TLS. pcctl presents the admin certificate, and checks that the node's certificate comes from your cluster CA and names the node ID recorded for it in nodes.json. A node accepts only certificates from its own cluster CA, and checks the role (admin or node) on every request. Nodes have no SSH and no login: the node API on TCP port 7443 is the only way in.

Getting and renewing the admin certificate

pcctl --repo LOC,... login unlocks the cluster with the passphrase and writes a fresh admin certificate:

pcctl --repo /srv/pc,s3://my-bucket/pc login

admin certificate for cluster 92283421c94b924005629e1405c47c54 written to /home/alex/.config/powercloud (valid 30 days)

You rarely need to run it by hand. Any command that unlocks the cluster (with --repo or --via) reissues the saved certificate when it has a week or less left, and prints a note on stderr. node enroll and node provision write one if none is saved. If the certificate has expired, commands that use it say so and ask you to run login. To revoke every admin certificate, for example one on a lost laptop, see Admins, keys and revocation.

To save typing, set the storage locations once in the environment. POWERCLOUD_REPO stands in for --repo on every command:

export POWERCLOUD_REPO=/srv/pc,s3://my-bucket/pc

Which nodes pcctl knows

nodes.json is filled in by this machine's own commands: node enroll, node provision --addr, and node addr add or update a node, and node rm removes it. The cluster's shared record of nodes (the registry, kept in the encrypted storage) is what --repo and --via commands use, so every admin sees the same nodes there.

On a second admin machine, login writes the certificate but no node list. Record each node's address on that machine with node addr (it needs the cluster passphrase), after which status, node info and the other certificate-only commands work there too:

pcctl --repo LOC,... login
pcctl --repo LOC,... node addr home-1 192.0.2.10

Cluster status

pcctl status asks every known node about itself, all at once, and prints one line per node followed by what needs attention. It uses only the admin certificate.

pcctl status

NODE     STATE        ARCH   VERSION      UPTIME    CPU       MEMORY         MESH       APPS  STORAGE  CERT
cloud-1  ok           arm64  vYYYY.MM.DD  72h4m10s  3% of 2   412/3891 MiB   2/2 peers  1/1   2 loc    341d
home-1   ok           amd64  vYYYY.MM.DD  5h12m3s   12% of 4  1204/7820 MiB  1/2 peers  3/3   -        200d
vps-1    unreachable  -      -            -         -         -              -          -     -        -

needs attention:
  home-1: reaches 1 of 2 mesh peers
  vps-1: unreachable (dial tcp 198.51.100.4:7443: i/o timeout)

The columns:

Needs attention

When nothing is wrong, the last line is all nodes healthy (it also says so when nodes.json lists no nodes at all). Otherwise each line names a node and a problem:

Listing nodes and node details

pcctl node ls prints the nodes in this machine's nodes.json, without contacting them:

pcctl node ls

NAME     ID                ADDRESS
cloud-1  3c91e07d5b2a8f14  203.0.113.7:7443
home-1   fa8adc53910a6a79  192.0.2.10:7443

pcctl node info NAME asks one node for its details:

pcctl node info home-1

It prints these fields, the later ones only when they apply:

The peer table shows, for each peer, its mesh address, the endpoint WireGuard sends to (with (relay NAME) when the pair is relayed), the age of the last WireGuard handshake, and whether the peer's API answers over the mesh, with the round-trip time:

mesh      fdd5:f552:ce70::fa8a:dc53:910a:6a79 (WireGuard port 51820)
  PEER     MESH ADDRESS                        ENDPOINT                            HANDSHAKE  API OVER MESH
  cloud-1  fdd5:f552:ce70::3c91:e07d:5b2a:8f14  203.0.113.7:51820                   12s ago    ok, 18.4 ms
  home-2   fdd5:f552:ce70::77e2:1ab:9c3d:4e55   203.0.113.7:51822 (relay cloud-1)   31s ago    ok, 41.0 ms

If the node refuses the admin certificate, pcctl says so and suggests login (admin certificates may have been revoked). If the node's own certificate has expired, it suggests node renew NAME.

The web console

pcctl console serves a web page on your own machine that shows the cluster as pcctl status sees it, live, and lets you read app logs and deploy, restart and remove apps. The nodes serve no web page: the console runs inside pcctl and reaches the nodes with your admin certificate, like every other command, so no key leaves your machine.

pcctl console

PowerCloud console at http://127.0.0.1:41377/?t=5b0c...
(for one browser, the first to open it; Ctrl-C to stop)

It opens your browser at that link and runs until you press Ctrl-C.

What it shows

• A summary bar: nodes up, apps running, CPU busy across all nodes, memory in use, how much memory apps' limits promise of what the nodes set aside for apps, and the number of issues.
• Needs attention: the same list as pcctl status.
• Nodes: one row per node with state, CPU, memory, apps, architecture, version, uptime, mesh peers reached, storage locations served, consensus role and leader, release state, and days until the certificate expires.
• A node's detail (select its row): bars for CPU, memory, app memory and app CPU promised and in use, and the state disk; its addresses (where pcctl reaches it, the node's own addresses, and its mesh address with each peer's round-trip time or relay); and facts such as node ID, kernel, boot, state encryption, how it unlocks, what apps get, the image and lockdown.
• The node's apps: state, uptime, restarts, memory, CPU, network, where each published port is reached (as a link for TCP ports), whether it is a cluster app, and the last exit.
• Logs: the last 64 KiB of an app's output, refreshed every 2 seconds while Follow is ticked.

The page refreshes every 5 seconds and shows how old its data is. A short guided tour starts on a browser's first visit, and again from the Guide button.

What you can do from it

• Restart an app, or Remove it (with an option to keep its data directory). Removing a cluster app removes it from the whole cluster.
• Deploy an app to the selected node, from a container image (pulled on your machine and sent to the node) or a static Linux binary you upload (up to 512 MiB), with memory, CPU cap, published ports, environment, links, microVM and egress options. See Apps on a node for what these mean.
• Deploy a cluster app, which the nodes place among themselves, with replicas, allowed nodes, exclusive, storage and volume options. This needs the cluster's keys, so it is offered only when you start the console with the repository: pcctl --repo LOC,... console. A console started with --via or with neither flag can't publish cluster apps. See Cluster apps.

Who can open it

• The link carries a random token. The first browser to open it trades the token for a cookie (HttpOnly, SameSite=Strict), and the link lets no other browser in after that. To open it in another browser, stop and restart pcctl console for a new link.
• Requests must use one of the console's own host names, which blocks DNS rebinding, and changes must come from the console's own page.
• The page loads nothing from elsewhere (a strict Content-Security-Policy).

Reaching it from another machine

By default the console listens on loopback only. The safest way to use it from another machine is an SSH tunnel to a loopback console. On the machine that has pcctl and your admin certificate (here admin-box), start the console on a fixed port without a browser:

pcctl console --port 8080 --no-open

On the machine with the browser, forward the port, then open the link the console printed (it uses 127.0.0.1:8080, which the tunnel carries):

ssh -L 8080:127.0.0.1:8080 admin-box

--listen puts it on the network over plain HTTP, and pcctl warns that anyone who can see the traffic can take the link or the session. Use it only on a network you trust:

pcctl console --listen 0.0.0.0 --port 8080 --host admin-box.lan --no-open

Pass --no-open whenever the browser that should use the console is on another machine. Without it, --listen 0.0.0.0 opens a browser on this machine, which then takes the one-time link.

Power and reboots

pcctl node reboot home-1
pcctl node poweroff home-1

node home-1 is rebooting

Both need only the admin certificate. After a reboot a node's encrypted state stays locked until it is unlocked, by you with pcctl --repo LOC,... node unlock NAME CODE, or by itself with TPM or peer unlock. See Unlocking nodes after a reboot. Until then pcctl status reports it as unreachable. A powered-off node has to be started again at the machine or at its provider.

Storage sync on a node

A node can serve the cluster's storage locations so that admins can use --via NAME instead of --repo (set up with node storage, described in Storage). Such a node copies any blob that some of its locations lack from one that has it, one minute after it starts and then every hour by default. pcctl node sync NAME runs that copy now:

pcctl node sync home-1

node home-1 synced its storage: 2026-10-28 14:02 UTC: 18234 blobs, 12 copied

It exits with an error (sync incomplete) if any copy failed or a location couldn't be listed. Sync only fills gaps and needs no keys; replacing stale or corrupt copies is pcctl --repo LOC,... repair.

When an address changes

There are two addresses per node, and they can change separately:

• The API address (HOST:PORT, port 7443 by default) is where pcctl reaches the node.
• The mesh endpoint (IP:PORT or HOST:PORT, port 51820 by default) is where the other nodes send the node's WireGuard traffic.

A node's API address: node addr

When a node comes back with a new IP (a cloud VM restarted without a fixed IP, a home router handing out a new lease), tell pcctl where it is:

pcctl --repo LOC,... node addr cloud-1 203.0.113.50

node cloud-1 is at 203.0.113.50:7443
mesh endpoint of cloud-1 set to 203.0.113.50:51820

This updates the cluster registry and this machine's nodes.json. If the node's mesh endpoint was empty or had been derived from its old address, node addr also sets it to the new address on port 51820 and publishes a new membership list to every node, so the other nodes find it again. An endpoint you set by hand with mesh endpoint is left alone. A DNS name works too (node addr home-1 home.example.net), for a node behind dynamic DNS.

A node's mesh endpoint: mesh endpoint

Set the endpoint explicitly when the other nodes must reach the node at a different address from the one pcctl uses, for example a public address with a port forwarded on the router while pcctl uses the LAN address:

pcctl --repo LOC,... mesh endpoint home-1 198.51.100.7:51820
pcctl --repo LOC,... mesh endpoint home-1 home.example.net:51820
pcctl --repo LOC,... mesh endpoint home-1 ''

The last form clears the endpoint, for a node behind NAT that nobody can send to first. Every change publishes a new signed membership list to the nodes. With a DNS name, the nodes resolve it themselves, again every minute, and apply a new address only to a peer with no handshake in the last three minutes.

Re-reading keys: mesh sync

Enrolling, removing and changing endpoints publish the membership list on their own. pcctl --repo LOC,... mesh sync asks every reachable node for its WireGuard key, records any that changed, and publishes the membership and revocation lists to every node again. Use it when a node was missed or its key changed:

pcctl --repo LOC,... mesh sync

Nodes that can't be reached keep the key on record (with a warning), and get the new lists from their mesh peers later.

The mesh

Nodes talk to each other over an encrypted WireGuard mesh. Each node gets a private IPv6 address derived from the cluster ID and its node ID, so there are no addresses to manage. The peer list is the membership list, signed by the cluster CA, which pcctl pushes to every node it can reach whenever it changes; nodes that were offline get it from any peer. Inside the tunnel, nodes authenticate each other again with their node certificates. Every node checks each peer's API through the mesh every 5 seconds, which is what pcctl node info and the MESH column of pcctl status report.

NAT traversal: what's automatic

A node behind NAT (a home router, a cloud VM without a public IP) needs no configuration beyond a clear or unreachable endpoint:

• It connects outwards to the nodes that have a public endpoint, which learn its NAT's public address from its packets.
• Nodes share what they see. A node with a peer it has no session with asks the peers it can reach every 5 seconds, and tries the address they report. Two nodes behind NAT do this at the same time, which opens both NATs, and they get a direct session. Traffic between them doesn't pass through the public node.
• When two nodes still can't connect after 20 seconds (symmetric NAT, as with carrier-grade NAT and many corporate firewalls), they relay through a public node. Both pick the same relay; if it doesn't connect them within another 20 seconds, they move on to the next. The relay forwards WireGuard packets it can't read or change: the session is end to end between the two nodes.
• Relayed pairs try a direct path again every 10 minutes, backing off up to about every 10 hours after repeated failures, in case the network changed.

pcctl node info shows a relayed peer as (relay NAME). These paths are checked in automated tests with emulated NATs.

What you configure

• At least one node with a public endpoint. Two nodes behind NAT find each other only through a third node both can reach. Relays are the nodes with an endpoint in the membership list.
• Firewall ports on public nodes:

• The right endpoint. Enrolling a node sets its endpoint from the address you enrolled it at, on port 51820. If that is a LAN address the other nodes can't reach, set the public one with mesh endpoint (with a port forward on the router), or clear it with mesh endpoint NAME '' so the node is treated as behind NAT. Note that node addr sets an empty endpoint again from the new address; clear it afterwards if the node is behind NAT.

Limits

• Mesh changes start from an admin: nodes only pass on membership lists signed by the CA.
• Relaying costs the relay's bandwidth and a detour. Pairs behind truly symmetric NATs keep relaying.
• When the first peer to report a missing node's address is on the same LAN as it, it reports the LAN address, which a node elsewhere can't use; the two connect once that peer loses its session.
• The mesh carries traffic between the nodes' APIs (health checks, consensus, replicas on other nodes). Your admin machine is not on the mesh: pcctl reaches each node's API directly, on its API address.

Node storage

A node's encrypted disk can hold a replica of the cluster's storage (node://NAME locations), and a node can serve the storage locations to admins so they use --via NAME with only the passphrase and admin certificate. Both are covered in Storage.

Good to know

• Commands that only read or act on one node (status, node ls, node info, node reboot, node poweroff, node sync, node upgrade, node attest NAME, app ..., console) need only the admin certificate.
• Commands that change the cluster's records (node addr, mesh endpoint, mesh sync, node renew, node rm) unlock the cluster with the passphrase, through --repo or --via.
• pcctl status and the console ask each node for at most 10 seconds, so one slow node doesn't hold up the rest.
• For problems not covered here, see Troubleshooting.