Cluster apps

A cluster app is an app you publish for the whole cluster rather than deploy to one node: the nodes decide among themselves where it runs, within each node's memory and CPU, and start it elsewhere when a node goes away. This page covers pcctl cluster app, exclusive apps, app storage and volumes, and the nodes' consensus with its pcctl kv commands. What an app is and the sandbox it runs in are described in Apps on a node.

Publish a cluster app

Cluster app commands change the cluster's signed state, so they need the cluster's keys: pass the storage locations with --repo (or set POWERCLOUD_REPO once in your shell), or reach the storage through a node with --via NODE. See Storage.

export POWERCLOUD_REPO=s3://my-bucket/cluster,/mnt/backup/cluster

The command has two forms, one for a static binary and one for a container image:

pcctl --repo LOC,... cluster app deploy [--replicas N | --exclusive] [--nodes a,b] [--storage] [--volume] [app options] NAME FILE [-- ARGS...]
pcctl --repo LOC,... cluster app deploy [--replicas N] [--nodes a,b] [app options] --image REF NAME [-- COMMAND...]

Two replicas of an image, on any nodes, with node port 80 published:

pcctl --repo LOC,... cluster app deploy --replicas 2 --image nginx:alpine --port 80:80 site

A binary that may run only on two named nodes:

pcctl --repo LOC,... cluster app deploy --nodes home-1,colo-1 --port 8080:8080 api ./api

cluster apps 4 published (2 apps)
  home-1: updated
  colo-1: updated
  edge-1: not reached; it gets the change from its peers

Each publish is numbered. Nodes that pcctl cannot reach catch up from their peers over the mesh, so a node that was off picks up the change when it comes back.

Publishing again under the same name replaces the app's definition everywhere.

Options

The cluster options:

The app options are the same as for pcctl app deploy: --mem, --cpu, --procs, --env, --port, --link, --egress, --net host and --vm. They are described in Apps on a node. Each node that runs the app applies them.

Binaries, images and architectures

An app's files are stored as artifacts named by their SHA-256: the static binary, or the image flattened to one tar for each architecture. pcctl sends each node it reaches its architecture's artifact; any other node fetches it from a peer and checks the hash. Artifacts nothing uses any more are deleted.

• For an image, pcctl pulls one copy for each architecture among the nodes it reaches (only the --nodes ones, if given), on your machine.
• A binary is for one architecture only. In a cluster that mixes x86-64 and arm64 nodes, use --nodes to keep a binary app on nodes that can run it, or publish an image.

See and remove cluster apps

pcctl --repo LOC,... cluster app ls
pcctl --repo LOC,... cluster app rm site

NAME  REPLICAS  NODES          RUNNING ON                         SOURCE
api   1         home-1,colo-1  colo-1 (running)                   binary
site  2         any            home-1 (running), edge-1 (running)  registry-1.docker.io/library/nginx@sha256:...

SOURCE is binary or the image by digest, followed by +storage, +volume and +exclusive where they apply. cluster app rm NAME takes the app away from every node.

On each node, a cluster app also shows in pcctl app ls NODE, with its logs in pcctl app logs NODE NAME. The node marks it as managed: change it with pcctl cluster app, not by hand. Apps deployed by hand with pcctl app deploy live alongside cluster apps, and the cluster leaves them alone.

pcctl status reports cluster apps that are not running where they should, including an app that no node has the memory or CPU for.

Placement and failover

• What should run is signed by an admin. The cluster's apps are one CA-signed document, kept in the cluster's storage and passed between nodes over the mesh. Only admins write it, so nodes need no agreement on what should run.
• Where it runs is computed by every node alike. For each app, every node ranks the eligible nodes (--nodes, or all) it considers alive, by a hash of the app's name and the node's ID (rendezvous hashing), and runs the app if it is among the first --replicas. Losing a node moves only that node's apps.
• Within each node's memory and CPU. Each node offers apps its RAM less a reserve for itself (the larger of 256 MiB and an eighth), and its CPUs less a reserve (the larger of a quarter CPU and an eighth). An app takes its --mem and its --cpu on every node it runs on; an app without --cpu takes no CPU in placement. Exclusive apps are placed first, then the others in name order, each skipping nodes without room. An app that fits nowhere runs nowhere, rather than overcommitting a node, and pcctl status says so.
• MicroVM apps (--vm) go only to nodes with KVM, and count at their memory limit plus 96 MiB.
• Failover. A node that its peers cannot reach for a minute stops counting, and its apps start on the next nodes in line. When it comes back it takes its apps back, and the others stop their copies.
• Local data stays behind. An app moved away keeps its data directory on the node it left (and its user, so the files stay its own), in case it returns. To have data follow the app, use --storage or, for an exclusive app, --volume.
• After a reboot, a node leaves cluster apps stopped until it has checked again where they should run.

Every app is told POWERCLOUD_NODE, so replicas can tell themselves apart. Links to a cluster app (--link NAME:PORT on another app) reach it on whichever node it was placed, over the encrypted mesh, and follow it when it moves.

Exclusive apps

Rendezvous placement can briefly run an app twice when nodes see each other differently, for example during a network split. --exclusive runs an app on at most one node at a time, decided through the nodes' consensus:

pcctl --repo LOC,... consensus init
pcctl --repo LOC,... cluster app deploy --exclusive --mem 512M --port 5432:5432 --image postgres:alpine db

• The assignment is agreed. The consensus state records each exclusive app's node, and only the Raft leader changes it. It moves an app off a node it has not reached for a minute, to the eligible node first in rendezvous order, and the app stays there when the old node returns.
• Fencing. A node runs an exclusive app only while its copy of the agreed state assigns it the app and was confirmed current within the last 15 seconds. A node cut off from the majority stops the app within about 20 seconds, long before the leader moves it.
• Moving off a live node (when you change --nodes) first clears the assignment, so the node stops the app, and assigns it elsewhere 15 seconds later.

Good to know: a node that freezes (a paused VM, a long stall) and wakes does not know how long it was away. It keeps its copy until it hears the new assignment, a few seconds after it wakes, so two copies can overlap that long. Apps that must never overlap need their own fencing on top, such as conditional writes to their storage.

App storage (--storage)

A cluster app deployed with --storage gets an S3 endpoint inside its own network, at 127.0.0.1:9000 (at 10.0.2.2:9000 in a microVM). AWS_ENDPOINT_URL and dummy credentials are set for it, so AWS SDKs and the AWS CLI find it unchanged. Behind it is the cluster's storage: encrypted, and replicated or erasure-coded across the locations the nodes serve (see Storage). When the app moves, it finds its objects again on the new node.

pcctl --repo LOC,... cluster app deploy --storage --port 8080:8080 counter ./counter

• Its own key and namespace. Each app has a random 256-bit key, kept in the cluster's encrypted store. Its objects are encrypted under keys derived from it, under IDs of their own, so another app cannot read them and storage providers do not see app names.
• The key goes only where the app may run. pcctl sends it to the eligible nodes it reaches; another eligible node can fetch it from a peer, which gives it only to a node the signed document allows. Eligible nodes fetch the key and the app's files ahead of time, so a failover does not depend on the node that just went away.
• What it speaks. Path-style buckets and objects (put, get with ranges, head, delete), ListObjectsV2 with prefixes, delimiters and paging, and multipart uploads, with S3's ETags. That is enough for aws s3 mb, cp, ls and rm and for SDKs. There is no copy, versioning or ACLs. Only the app's own network reaches the endpoint, so request signatures are not checked.
• Cleanup. Nodes that serve storage reclaim the chunks of an app's deleted objects during their storage sync (hourly by default), with a 24-hour grace for uploads in progress. A cluster where no node serves storage does not reclaim them.

--storage cannot be combined with --net host.

App volumes (--volume)

An exclusive app deployed with --volume keeps its data directory (/data) in the cluster's storage, so its ordinary files follow it to another node:

pcctl --repo LOC,... cluster app deploy --exclusive --volume --port 8080:8080 counter ./counter

• Saved as it runs. Every 30 seconds, the node running the app saves the directory, uploading only files whose size, time or mode changed. A save cut short leaves the previous one whole.
• And as it stops. A node stopping the app saves once more after it exits, unless another node has taken the app since. Moving an app off a live node loses nothing; a node that goes away loses at most its last 30 seconds.
• Restored before it starts. A node starting the app downloads the directory if its own copy is older, into a new directory that it swaps in. Files belong to the app's user, with their modes and times.
• Only the holder writes. Saves come only from the node assigned the app while it may run it, so a node waking from a freeze does not overwrite newer data.
• Encrypted and rollback-protected. The volume is encrypted under a key derived from the app's storage key, and protected against a provider serving an older version through the nodes' consensus.
• Symlinks are saved as links, never followed, so an app cannot make the node copy its own files into the volume.

Limits of volumes:

• Snapshots, not replication. A node that goes away loses up to 30 seconds of changes, and files are not saved at one instant together. Apps whose files must stay consistent with each other, such as databases, should keep their state in --storage, or checkpoint it.
• Whole files. A changed file is uploaded again entire (chunks already stored are deduplicated).
• Owners, extended attributes, hard links and special files (sockets, FIFOs) are not kept.

Consensus among the nodes

Some state must be agreed by the nodes themselves, in one order, and survive any minority of them going away: exclusive apps' assignments, and the marks that catch storage rolled back to an older version. For that the nodes form a Raft group replicating a key-value state, with each node's log on its encrypted state volume.

Found the group

An admin founds the group once:

pcctl --repo LOC,... consensus init
pcctl --repo LOC,... consensus init --nodes home-1,colo-1,edge-1

consensus founded with 3 voters: colo-1, edge-1, home-1
  colo-1: started
  edge-1: started
  home-1: started

--nodes names the first voters; by default every node is one. consensus init signs a document naming them, which spreads over the mesh like the other signed documents. It can run only once: afterwards it says the group was founded already.

After that, voters follow the membership list by themselves. The leader adds a member it has reached within 30 seconds, and removes voters that are no longer members or were revoked, one change at a time. A node enrolled later becomes a voter by itself and catches up from the leader.

Check the group

pcctl consensus status

NODE    ROLE      TERM  LEADER  APPLIED  KEYS  VOTERS
colo-1  Follower  3     home-1  1842     12    colo-1,edge-1,home-1
edge-1  Follower  3     home-1  1842     12    colo-1,edge-1,home-1
home-1  Leader    3     home-1  1842     12    colo-1,edge-1,home-1

Each node shows its role, term, leader, applied index, number of keys and voters, or unreachable, or not started if it has no group yet. consensus status needs only the admin certificate, no --repo.

Read and write keys

pcctl kv reads and changes the agreed state through a node. Flags go before the subcommand:

pcctl kv put config/mode prod
pcctl kv get config/mode
pcctl kv ls config/
pcctl kv rm config/mode
pcctl kv --node home-1 --stale get config/mode

• put KEY VALUE returns once the change is applied on the node asked, after a majority of voters have it. Without a majority it fails after 10 seconds.
• get KEY prints the value; ls [PREFIX] prints KEY=VALUE lines. Reads are linearizable: they see every change made before they started, through whichever node. Without a majority, a read fails like a change.
• --stale answers from the node's own copy instead, which works without a majority but may miss recent changes.
• --node NAME picks the node to ask; by default it is the first that answers with a group.

Keys and values are plain strings in one flat map. The nodes themselves keep keys there too (exclusive apps' assignments under assign/), so do not change keys you did not write.

Limits of consensus

• Two voters tolerate no loss. A majority of two is both, so a two-node cluster stops taking changes, and exclusive apps stop, when either node is down. Three or more voters tolerate a loss. consensus init warns about two.
• The group cannot be refounded with pcctl once founded. Losing a majority of voters for good needs a manual recovery.

Limits

• No consensus on liveness, unless exclusive. For ordinary cluster apps, each node judges liveness from its own view. A node cut off from the others, but not from users, may run an app another node also runs. That is fine for stateless apps; use --exclusive for the others.
• No locking across nodes. Two copies of an app (during a split, or with several replicas) see the same --storage objects, last writer wins, as with S3 itself.
• Limits, not use. Placement counts apps' memory limits and CPU caps, not what they use. It ignores apps deployed by hand with pcctl app, and apps without a CPU cap share the CPU left over.
• Local data moves only with a volume. Without --volume, an app's /data stays on the node it ran on.