Apps on a node

An app is a service you deploy to a PowerCloud node with pcctl app: a static Linux binary or a container image, run in a sandbox and kept running across crashes and reboots. This page covers apps deployed by hand to one node; to let the nodes choose where an app runs and move it when a node goes away, see Cluster apps.

What an app is

An app is one of two things:

• A static Linux binary. The app's root holds only the binary and its data directory, with no libraries or dynamic loader, so the node refuses dynamically linked ELF files and anything that is not ELF. For Go, build with CGO_ENABLED=0. The binary must match the node's architecture (x86-64 or arm64).
• A container image (--image). pcctl pulls the image from its registry on your machine, not on the node. It takes the image for the node's architecture, checks every manifest, config and layer against its sha256 digest, applies the layers in order and sends the node the flattened files. The command is the image's entrypoint and command, the image's environment and working directory apply, and the node records the image by digest.

Public images (Docker Hub, GHCR and others) are pulled with an anonymous token. Private images use the login that docker login saved in ~/.docker/config.json (or $DOCKER_CONFIG). Docker credential helpers are not used.

Each app lives in a directory on the node's encrypted state volume, holding its spec, its files, its data and its log. Only admins can deploy, remove, restart or read the logs of an app: the commands use your admin certificate, like every other change to a node. The pcctl app commands talk to the node directly and need no --repo.

Deploy an app

The command has two forms, one for a binary and one for an image:

pcctl app deploy [options] NODE NAME FILE [-- ARGS...]
pcctl app deploy --image NAME[:TAG][@DIGEST] [options] NODE NAME [-- COMMAND...]

NODE is a node's name as pcctl node ls shows it. NAME is the app's name: 1 to 32 lowercase letters, digits and dashes, starting with a letter or digit. Options go before NODE.

A binary with a memory limit, a CPU cap, an environment variable and its own arguments:

pcctl app deploy --mem 128M --cpu 0.5 --env MODE=prod home-1 web ./web -- --port 8080

app web deployed to home-1 (6815744 bytes)
NAME  STATE    UPTIME  RESTARTS  OOM KILLS  MEMORY     CPU        SECCOMP  NETWORK  LAST EXIT
web   running  0s      0         0          0/128 MiB  0.00/0.50  on       own

An image, with node port 80 published to the app's port 80:

pcctl app deploy --image nginx:alpine --port 80:80 home-1 site

For an image, arguments after -- replace the image's command (the entrypoint stays). For a binary, everything after FILE (with or without --) is passed to the binary as arguments.

Deploying again under the same name replaces the app. It keeps its data directory and its user, so its files stay its own.

Options

Some options cannot be combined: --net host excludes --port, --link, --egress and --vm, since all of those work on the app's own network.

What the app is told

Besides its --env variables, every app gets POWERCLOUD_APP (its name) and POWERCLOUD_NODE (its node's name). The replicas of a cluster app can use these to tell themselves apart. An --env of the same name wins over both.

Its data directory

An app's data directory is /data as the app sees it. What the app writes there is on the node's encrypted state volume and is kept across restarts, redeploys and reboots. Everything outside /data (the binary, or the image's files) is replaced on each deploy.

The sandbox

Every app runs in a sandbox of its own. In brief:

• Filesystem. The app is chrooted into its own root, so it sees only its binary (or its image's files) and its data: no node state, keys or devices. It gets its own /proc (showing only its processes), a /dev with null, zero, full, random and urandom, and /dev/shm and /tmp. The apps directory is mounted nosuid and nodev, and setuid bits, device nodes and FIFOs are dropped from images.
• User. It runs as a user of its own (numbered from 20000, assigned at deploy) with no capabilities and no supplementary groups, so it cannot write its own binary or another app's data. An image's User setting is ignored, so images that expect to run as root must tolerate running as an ordinary user.
• Processes. It gets new PID, mount and UTS namespaces. It is PID 1 of its own process tree, its hostname is its name, and everything it starts dies with it.
• Limits. It is placed in its own cgroup as it starts, which caps its memory, CPU time and number of processes.
• System calls. It runs under a seccomp filter with no_new_privs. The filter refuses calls from another CPU ABI, and returns EPERM for mounts, chroot, pivot_root, unshare, setns and namespace-creating clone; ptrace and process_vm_*; bpf, perf_event_open, keyrings, userfaultfd and io_uring; module loading, kexec, reboot and swap; setting the clock; and open-by-handle. clone3 gets ENOSYS, so C libraries fall back to clone.
• Network. By default it has a network of its own, with only loopback: it cannot reach the node's network, the mesh or the internet, and nothing reaches it but its published ports and the links other apps were given to it. It may bind ports below 1024.

pcctl app ls shows on in the SECCOMP column for an app running under the filter.

Networking

Publish ports

--port NODE:APP makes the node listen on port NODE and pass each TCP connection to port APP on the app's loopback. --port NODE:APP/udp does the same for UDP: each client address gets its own session to the app's port, closed after two minutes idle, with at most 1024 open at once.

pcctl app deploy --image coredns/coredns --port 53:53/udp --port 53:53 home-1 dns

The app has no address of its own. Users reach it at the node's address on the published port.

Connect out with --egress

--egress gives the app a way out while keeping its own network. Its traffic goes out through the node, masqueraded as the node, to anything outside it. It still cannot reach the node itself (its API, its storage, other apps' published ports) or other apps. The app gets the node's name servers in /etc/resolv.conf. Egress is IPv4 only.

Share the node's network with --net host

--net host puts the app on the node's own network, with all of its access. It is the coarse option: there is nothing in between, such as reaching only some hosts. Prefer --port, --egress and --link.

Link apps by name

--link NAME:PORT lets the app reach another app, NAME, on its TCP port PORT, by that name, without the other app publishing anything. Inside the app, NAME resolves (through its /etc/hosts) to an address in the app's own network where the node picks up each connection and passes it to the target's port.

pcctl app deploy --image postgres:alpine --mem 512M home-1 db
pcctl app deploy --link db:5432 --port 8080:8080 home-1 api ./api

The api app then connects to db:5432. Add /udp for UDP (--link dns:53/udp); each of the app's sockets gets a session, closed after two minutes idle, with at most 1024 at once. UDP to a link address's other ports is refused at once, so an app trying QUIC to a peer it has only a TCP link to falls back to TCP.

Good to know about links:

• An app reaches only the apps and ports it was linked to.
• The app may serve the same port itself, so two peer-to-peer apps can be linked to each other on the port they both use.
• A link to an app deployed by hand works only on the same node. A link to a cluster app works wherever the cluster placed it, over the encrypted mesh, and follows it when it moves (see Cluster apps).

See, restart and remove apps

pcctl app ls home-1
pcctl app logs home-1 web
pcctl app logs home-1 web 1048576
pcctl app restart home-1 web
pcctl app rm home-1 web
pcctl app rm --keep-data home-1 web

• app ls NODE lists the node's apps: state, uptime, restarts, memory kills, memory used against the limit, CPU used against the cap, whether seccomp is on, the network (own, host or microvm, with +egress, ->NAME:PORT for links and NODE->APP for published ports) and the last exit.
• app logs NODE NAME [BYTES] prints the end of the app's output, 64 KiB by default. Standard output and standard error both go to the log, which is rotated at 1 MiB. Programs that reopen /dev/stdout or /dev/stderr (nginx, for one) work.
• app restart NODE NAME stops the app and starts it again.
• app rm NODE NAME stops the app and deletes it with its data. With --keep-data, the data directory is kept, and the app gets its user and its files back if deployed again on that node under the same name.

The web console (pcctl console) shows the same per node, and can deploy, restart and remove apps from a form; see pcctl command reference.

Kept running

An app that exits, or is killed (by its memory limit too), starts again after a delay that doubles from 1 second up to a minute. The delay starts again from 1 second once the app has run for more than a minute. RESTARTS and OOM KILLS in pcctl app ls count these, and LAST EXIT says how it last ended.

Apps start again by themselves when the node boots and is unlocked; until then the state volume, and so every app, is locked (see Unlocking nodes after a reboot).

MicroVMs

By default, apps share the node's kernel, so a kernel bug an app can reach is a way out of its sandbox. --vm runs the app under a kernel of its own, in a microVM, with the CPU's hardware virtualization as the boundary. Use it for code you trust less than the node.

pcctl app deploy --vm --cpu 2 --mem 512M --image nginx:alpine --port 80:80 home-1 site

Nothing else about the app changes: it is deployed, listed, logged, restarted and removed like any app, from a binary or an image, and its microVM runs inside the same sandbox (namespaces, chroot, cgroup, its own user, seccomp). A guest escape lands in that sandbox. Inside the guest, the app again runs as its own user with no capabilities, under the same seccomp filter. A guest boots, runs its app and reports its exit in about a second.

What is different:

• It needs KVM. A node runs microVMs where its CPU offers virtualization to it: bare metal (on AWS, x86 and Graviton .metal instances), or a VM with nested virtualization. Most cloud VMs do not. On a node without KVM, the deploy fails with "this node can't run microVM apps (it needs KVM)".
• vCPUs come from --cpu. The guest gets a vCPU for each CPU of its limit, rounded up (--cpu 1.5 gives two), or, with no --cpu, one for each of the node's CPUs, up to 32. The CPU cap applies to the vCPUs together.
• Memory includes its files. The guest's root filesystem is in its memory, so the app's files (but not its data directory) count against --mem, as does the guest kernel's 64 MiB. Images of a few hundred MiB are fine; multi-GiB ones are not (2 GiB at most). Files the app writes outside /data are in that memory too, and are gone when the guest stops.
• Data stays on the node. The data directory is shared with the guest at /data, not copied in, so what the app writes there is on the node and survives the guest.
• Network. The app always has its own network (not --net host). The guest is at 10.0.2.15 on a private link to the node, which is at 10.0.2.2. Published ports, --egress and links work as for any app; linked apps get addresses from 10.0.2.100 onwards.
• app ls shows microvm in the NETWORK column. The guest kernel's serious messages go to the app's log, without its boot messages.

Limits

• One node, by hand. An app runs where you deploy it. For placement, replicas and failover, use Cluster apps.
• No outbound rules. Egress reaches anything outside the node; there is no list of allowed hosts. Egress is IPv4 only.
• A denylist. The seccomp filter refuses known-dangerous calls and allows the rest, rather than allowing only a known-safe set.
• The node's kernel. Apps share it unless deployed with --vm. Images run without user namespaces, as one unprivileged user.
• One user per app. An app runs as its own user, whatever its image asks for.