# Cluster apps > How to publish apps for the whole PowerCloud cluster, how the nodes place them and fail them over, and how to use exclusive apps, app storage, app volumes, the nodes' consensus and the kv commands. A cluster app is an app you publish for the whole cluster rather than deploy to one node: the nodes decide among themselves where it runs, within each node's memory and CPU, and start it elsewhere when a node goes away. This page covers `pcctl cluster app`, exclusive apps, app storage and volumes, and the nodes' consensus with its `pcctl kv` commands. What an app is and the sandbox it runs in are described in [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). ## Publish a cluster app Cluster app commands change the cluster's signed state, so they need the cluster's keys: pass the storage locations with `--repo` (or set `POWERCLOUD_REPO` once in your shell), or reach the storage through a node with `--via NODE`. See [Storage](https://powercloud.unidatum.com/docs/storage.md). ```sh export POWERCLOUD_REPO=s3://my-bucket/cluster,/mnt/backup/cluster ``` The command has two forms, one for a static binary and one for a container image: ```sh pcctl --repo LOC,... cluster app deploy [--replicas N | --exclusive] [--nodes a,b] [--storage] [--volume] [app options] NAME FILE [-- ARGS...] pcctl --repo LOC,... cluster app deploy [--replicas N] [--nodes a,b] [app options] --image REF NAME [-- COMMAND...] ``` Two replicas of an image, on any nodes, with node port 80 published: ```sh pcctl --repo LOC,... cluster app deploy --replicas 2 --image nginx:alpine --port 80:80 site ``` A binary that may run only on two named nodes: ```sh pcctl --repo LOC,... cluster app deploy --nodes home-1,colo-1 --port 8080:8080 api ./api ``` ```text cluster apps 4 published (2 apps) home-1: updated colo-1: updated edge-1: not reached; it gets the change from its peers ``` Each publish is numbered. Nodes that `pcctl` cannot reach catch up from their peers over the mesh, so a node that was off picks up the change when it comes back. Publishing again under the same name replaces the app's definition everywhere. ### Options The cluster options: | Option | Default | What it does | | --- | --- | --- | | `--replicas N` | 1 | How many nodes run the app. | | `--nodes a,b` | any | The nodes that may run it, by name, comma-separated. | | `--exclusive` | off | Run the app on at most one node at any time, through the nodes' consensus. Needs `consensus init`; has one replica. | | `--storage` | off | Give the app an S3 endpoint backed by the cluster's encrypted storage. | | `--volume` | off | Keep the app's data directory in the cluster's storage, so it follows the app. Needs `--exclusive`. | | `--image REF` | | Deploy a container image (`NAME[:TAG][@DIGEST]`) instead of a binary. | The app options are the same as for `pcctl app deploy`: `--mem`, `--cpu`, `--procs`, `--env`, `--port`, `--link`, `--egress`, `--net host` and `--vm`. They are described in [Apps on a node](https://powercloud.unidatum.com/docs/apps.md). Each node that runs the app applies them. ### Binaries, images and architectures An app's files are stored as artifacts named by their SHA-256: the static binary, or the image flattened to one tar for each architecture. `pcctl` sends each node it reaches its architecture's artifact; any other node fetches it from a peer and checks the hash. Artifacts nothing uses any more are deleted. - For an image, `pcctl` pulls one copy for each architecture among the nodes it reaches (only the `--nodes` ones, if given), on your machine. - A binary is for one architecture only. In a cluster that mixes x86-64 and arm64 nodes, use `--nodes` to keep a binary app on nodes that can run it, or publish an image. ## See and remove cluster apps ```sh pcctl --repo LOC,... cluster app ls pcctl --repo LOC,... cluster app rm site ``` ```text NAME REPLICAS NODES RUNNING ON SOURCE api 1 home-1,colo-1 colo-1 (running) binary site 2 any home-1 (running), edge-1 (running) registry-1.docker.io/library/nginx@sha256:... ``` `SOURCE` is `binary` or the image by digest, followed by `+storage`, `+volume` and `+exclusive` where they apply. `cluster app rm NAME` takes the app away from every node. On each node, a cluster app also shows in `pcctl app ls NODE`, with its logs in `pcctl app logs NODE NAME`. The node marks it as managed: change it with `pcctl cluster app`, not by hand. Apps deployed by hand with `pcctl app deploy` live alongside cluster apps, and the cluster leaves them alone. `pcctl status` reports cluster apps that are not running where they should, including an app that no node has the memory or CPU for. ## Placement and failover No node gives orders: every node runs the same calculation on the same signed document and comes to the same answer about where each app runs. ```mermaid %% alt: An admin's pcctl signs the cluster's apps document, which spreads to every node over the mesh; each node ranks the eligible, alive nodes by rendezvous hash and runs the app if it is among the first --replicas with room, and a node unreachable for a minute drops out of the ranking. flowchart TB A["Admin: pcctl
cluster app
deploy"] -->|"signed document"| M["Every node
over the mesh"] M --> R["Rank eligible,
alive nodes
by hash"] X["Node unreachable
for a minute"] -.->|"drops out"| R R --> P["First --replicas
with room
run the app"] ``` - **What should run is signed by an admin.** The cluster's apps are one CA-signed document, kept in the cluster's storage and passed between nodes over the mesh. Only admins write it, so nodes need no agreement on what should run. - **Where it runs is computed by every node alike.** For each app, every node ranks the eligible nodes (`--nodes`, or all) it considers alive, by a hash of the app's name and the node's ID (rendezvous hashing), and runs the app if it is among the first `--replicas`. Losing a node moves only that node's apps. - **Within each node's memory and CPU.** Each node offers apps its RAM less a reserve for itself (the larger of 256 MiB and an eighth), and its CPUs less a reserve (the larger of a quarter CPU and an eighth). An app takes its `--mem` and its `--cpu` on every node it runs on; an app without `--cpu` takes no CPU in placement. Exclusive apps are placed first, then the others in name order, each skipping nodes without room. An app that fits nowhere runs nowhere, rather than overcommitting a node, and `pcctl status` says so. - **MicroVM apps** (`--vm`) go only to nodes with KVM, and count at their memory limit plus 96 MiB. - **Failover.** A node that its peers cannot reach for a minute stops counting, and its apps start on the next nodes in line. When it comes back it takes its apps back, and the others stop their copies. - **Local data stays behind.** An app moved away keeps its data directory on the node it left (and its user, so the files stay its own), in case it returns. To have data follow the app, use `--storage` or, for an exclusive app, `--volume`. - **After a reboot**, a node leaves cluster apps stopped until it has checked again where they should run. Every app is told `POWERCLOUD_NODE`, so replicas can tell themselves apart. Links to a cluster app (`--link NAME:PORT` on another app) reach it on whichever node it was placed, over the encrypted mesh, and follow it when it moves. ## Exclusive apps Rendezvous placement can briefly run an app twice when nodes see each other differently, for example during a network split. `--exclusive` runs an app on at most one node at a time, decided through the nodes' consensus: ```sh pcctl --repo LOC,... consensus init pcctl --repo LOC,... cluster app deploy --exclusive --mem 512M --port 5432:5432 --image postgres:alpine db ``` - **The assignment is agreed.** The consensus state records each exclusive app's node, and only the Raft leader changes it. It moves an app off a node it has not reached for a minute, to the eligible node first in rendezvous order, and the app stays there when the old node returns. - **Fencing.** A node runs an exclusive app only while its copy of the agreed state assigns it the app and was confirmed current within the last 15 seconds. A node cut off from the majority stops the app within about 20 seconds, long before the leader moves it. - **Moving off a live node** (when you change `--nodes`) first clears the assignment, so the node stops the app, and assigns it elsewhere 15 seconds later. A failover of an exclusive app with `--volume`, when its node is cut off: ```mermaid %% alt: An exclusive app with a volume fails over: home-1 saves /data every 30 seconds, is cut off and stops the app within about 20 seconds, then after a minute the Raft leader assigns the app to colo-1, which restores /data from storage and starts it; the app stays on colo-1 when home-1 returns. sequenceDiagram participant H as home-1 (holder) participant L as Raft leader participant C as colo-1 participant S as Cluster storage H->>S: save /data every 30 s Note over H,L: home-1 is cut off from the majority H->>H: fenced: stops the app (about 20 s) Note over L: home-1 not reached for a minute L->>C: assign the app to colo-1 C->>S: restore /data if newer C->>C: start the app Note over H: home-1 returns, the app stays on colo-1 ``` Good to know: a node that freezes (a paused VM, a long stall) and wakes does not know how long it was away. It keeps its copy until it hears the new assignment, a few seconds after it wakes, so two copies can overlap that long. Apps that must never overlap need their own fencing on top, such as conditional writes to their storage. ## App storage (--storage) A cluster app deployed with `--storage` gets an S3 endpoint inside its own network, at `127.0.0.1:9000` (at `10.0.2.2:9000` in a microVM). `AWS_ENDPOINT_URL` and dummy credentials are set for it, so AWS SDKs and the AWS CLI find it unchanged. Behind it is the cluster's storage: encrypted, and replicated or erasure-coded across the locations the nodes serve (see [Storage](https://powercloud.unidatum.com/docs/storage.md)). When the app moves, it finds its objects again on the new node. ```sh pcctl --repo LOC,... cluster app deploy --storage --port 8080:8080 counter ./counter ``` - **Its own key and namespace.** Each app has a random 256-bit key, kept in the cluster's encrypted store. Its objects are encrypted under keys derived from it, under IDs of their own, so another app cannot read them and storage providers do not see app names. - **The key goes only where the app may run.** `pcctl` sends it to the eligible nodes it reaches; another eligible node can fetch it from a peer, which gives it only to a node the signed document allows. Eligible nodes fetch the key and the app's files ahead of time, so a failover does not depend on the node that just went away. - **What it speaks.** Path-style buckets and objects (put, get with ranges, head, delete), ListObjectsV2 with prefixes, delimiters and paging, and multipart uploads, with S3's ETags. That is enough for `aws s3 mb`, `cp`, `ls` and `rm` and for SDKs. There is no copy, versioning or ACLs. Only the app's own network reaches the endpoint, so request signatures are not checked. - **Cleanup.** Nodes that serve storage reclaim the chunks of an app's deleted objects during their storage sync (hourly by default), with a 24-hour grace for uploads in progress. A cluster where no node serves storage does not reclaim them. `--storage` cannot be combined with `--net host`. ## App volumes (--volume) An exclusive app deployed with `--volume` keeps its data directory (`/data`) in the cluster's storage, so its ordinary files follow it to another node: ```sh pcctl --repo LOC,... cluster app deploy --exclusive --volume --port 8080:8080 counter ./counter ``` - **Saved as it runs.** Every 30 seconds, the node running the app saves the directory, uploading only files whose size, time or mode changed. A save cut short leaves the previous one whole. - **And as it stops.** A node stopping the app saves once more after it exits, unless another node has taken the app since. Moving an app off a live node loses nothing; a node that goes away loses at most its last 30 seconds. - **Restored before it starts.** A node starting the app downloads the directory if its own copy is older, into a new directory that it swaps in. Files belong to the app's user, with their modes and times. - **Only the holder writes.** Saves come only from the node assigned the app while it may run it, so a node waking from a freeze does not overwrite newer data. - **Encrypted and rollback-protected.** The volume is encrypted under a key derived from the app's storage key, and protected against a provider serving an older version through the nodes' consensus. - **Symlinks are saved as links**, never followed, so an app cannot make the node copy its own files into the volume. Limits of volumes: - **Snapshots, not replication.** A node that goes away loses up to 30 seconds of changes, and files are not saved at one instant together. Apps whose files must stay consistent with each other, such as databases, should keep their state in `--storage`, or checkpoint it. - **Whole files.** A changed file is uploaded again entire (chunks already stored are deduplicated). - Owners, extended attributes, hard links and special files (sockets, FIFOs) are not kept. ## Consensus among the nodes Some state must be agreed by the nodes themselves, in one order, and survive any minority of them going away: exclusive apps' assignments, and the marks that catch storage rolled back to an older version. For that the nodes form a Raft group replicating a key-value state, with each node's log on its encrypted state volume. ```mermaid %% alt: The nodes form one Raft group whose leader replicates every change to the followers; a change is applied once a majority of voters have it, and the group holds pcctl kv keys, exclusive apps' assignments, rollback marks and S3 access keys. flowchart TB K["pcctl kv,
exclusive apps,
rollback marks,
S3 access keys"] -->|"change through
any node"| L subgraph G["Raft group: applied once a majority has it"] L["home-1
leader"] F1["colo-1
follower"] F2["edge-1
follower"] L -->|"log"| F1 L -->|"log"| F2 end ``` ### Found the group An admin founds the group once: ```sh pcctl --repo LOC,... consensus init pcctl --repo LOC,... consensus init --nodes home-1,colo-1,edge-1 ``` ```text consensus founded with 3 voters: colo-1, edge-1, home-1 colo-1: started edge-1: started home-1: started ``` `--nodes` names the first voters; by default every node is one. `consensus init` signs a document naming them, which spreads over the mesh like the other signed documents. It can run only once: afterwards it says the group was founded already. After that, voters follow the membership list by themselves. The leader adds a member it has reached within 30 seconds, and removes voters that are no longer members or were revoked, one change at a time. A node enrolled later becomes a voter by itself and catches up from the leader. ### Check the group ```sh pcctl consensus status ``` ```text NODE ROLE TERM LEADER APPLIED KEYS VOTERS colo-1 Follower 3 home-1 1842 12 colo-1,edge-1,home-1 edge-1 Follower 3 home-1 1842 12 colo-1,edge-1,home-1 home-1 Leader 3 home-1 1842 12 colo-1,edge-1,home-1 ``` Each node shows its role, term, leader, applied index, number of keys and voters, or `unreachable`, or `not started` if it has no group yet. `consensus status` needs only the admin certificate, no `--repo`. ### Read and write keys `pcctl kv` reads and changes the agreed state through a node. Flags go before the subcommand: ```sh pcctl kv put config/mode prod pcctl kv get config/mode pcctl kv ls config/ pcctl kv rm config/mode pcctl kv --node home-1 --stale get config/mode ``` - `put KEY VALUE` returns once the change is applied on the node asked, after a majority of voters have it. Without a majority it fails after 10 seconds. - `get KEY` prints the value; `ls [PREFIX]` prints `KEY=VALUE` lines. Reads are linearizable: they see every change made before they started, through whichever node. Without a majority, a read fails like a change. - `--stale` answers from the node's own copy instead, which works without a majority but may miss recent changes. - `--node NAME` picks the node to ask; by default it is the first that answers with a group. Keys and values are plain strings in one flat map. The nodes themselves keep keys there too (exclusive apps' assignments under `assign/`), so do not change keys you did not write. ### Limits of consensus - **Two voters tolerate no loss.** A majority of two is both, so a two-node cluster stops taking changes, and exclusive apps stop, when either node is down. Three or more voters tolerate a loss. `consensus init` warns about two. - **The group cannot be refounded** with `pcctl` once founded. Losing a majority of voters for good needs a manual recovery. ## Limits - **No consensus on liveness, unless exclusive.** For ordinary cluster apps, each node judges liveness from its own view. A node cut off from the others, but not from users, may run an app another node also runs. That is fine for stateless apps; use `--exclusive` for the others. - **No locking across nodes.** Two copies of an app (during a split, or with several replicas) see the same `--storage` objects, last writer wins, as with S3 itself. - **Limits, not use.** Placement counts apps' memory limits and CPU caps, not what they use. It ignores apps deployed by hand with `pcctl app`, and apps without a CPU cap share the CPU left over. - **Local data moves only with a volume.** Without `--volume`, an app's `/data` stays on the node it ran on.