# Apps on a node > How to deploy, inspect and remove sandboxed apps (static binaries or container images) on one PowerCloud node, with their limits, ports, links, egress and microVMs. An app is a service you deploy to a PowerCloud node with `pcctl app`: a static Linux binary or a container image, run in a sandbox and kept running across crashes and reboots. This page covers apps deployed by hand to one node; to let the nodes choose where an app runs and move it when a node goes away, see [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). ## What an app is An app is one of two things: - **A static Linux binary.** The app's root holds only the binary and its data directory, with no libraries or dynamic loader, so the node refuses dynamically linked ELF files and anything that is not ELF. For Go, build with `CGO_ENABLED=0`. The binary must match the node's architecture (x86-64 or arm64). - **A container image** (`--image`). `pcctl` pulls the image from its registry on your machine, not on the node. It takes the image for the node's architecture, checks every manifest, config and layer against its sha256 digest, applies the layers in order and sends the node the flattened files. The command is the image's entrypoint and command, the image's environment and working directory apply, and the node records the image by digest. Public images (Docker Hub, GHCR and others) are pulled with an anonymous token. Private images use the login that `docker login` saved in `~/.docker/config.json` (or `$DOCKER_CONFIG`). Docker credential helpers are not used. Each app lives in a directory on the node's encrypted state volume, holding its spec, its files, its data and its log. Only admins can deploy, remove, restart or read the logs of an app: the commands use your admin certificate, like every other change to a node. The `pcctl app` commands talk to the node directly and need no `--repo`. ## Deploy an app The command has two forms, one for a binary and one for an image: ```sh pcctl app deploy [options] NODE NAME FILE [-- ARGS...] pcctl app deploy --image NAME[:TAG][@DIGEST] [options] NODE NAME [-- COMMAND...] ``` `NODE` is a node's name as `pcctl node ls` shows it. `NAME` is the app's name: 1 to 32 lowercase letters, digits and dashes, starting with a letter or digit. Options go before `NODE`. A binary with a memory limit, a CPU cap, an environment variable and its own arguments: ```sh pcctl app deploy --mem 128M --cpu 0.5 --env MODE=prod home-1 web ./web -- --port 8080 ``` ```text app web deployed to home-1 (6815744 bytes) NAME STATE UPTIME RESTARTS OOM KILLS MEMORY CPU SECCOMP NETWORK LAST EXIT web running 0s 0 0 0/128 MiB 0.00/0.50 on own ``` An image, with node port 80 published to the app's port 80: ```sh pcctl app deploy --image nginx:alpine --port 80:80 home-1 site ``` For an image, arguments after `--` replace the image's command (the entrypoint stays). For a binary, everything after `FILE` (with or without `--`) is passed to the binary as arguments. Deploying again under the same name replaces the app. It keeps its data directory and its user, so its files stay its own. ### Options | Option | Default | What it does | | --- | --- | --- | | `--mem SIZE` | `256M` | Memory limit, with `K`, `M` or `G`; at least 4 MiB. There is no swap. | | `--cpu N` | none | CPU limit in CPUs, such as `0.5` or `2`. Without it the app may use any CPU left over. | | `--procs N` | 256 | Limit on processes and threads. | | `--env K=V` | | An environment variable. Repeatable. | | `--port NODE:APP` | | Publish the app's TCP port `APP` on the node's port `NODE`. `NODE:APP/udp` publishes a UDP port. Repeatable. | | `--link NAME:PORT` | | Let the app reach app `NAME`'s TCP port `PORT` by its name. `NAME:PORT/udp` for UDP. Repeatable, up to 32. | | `--egress` | off | Let the app connect out through the node. | | `--net host` | own network | Share the node's network instead of having a network of its own. | | `--vm` | off | Run the app in a microVM, under a kernel of its own (nodes with KVM). | | `--image REF` | | Deploy a container image instead of a binary. | Some options cannot be combined: `--net host` excludes `--port`, `--link`, `--egress` and `--vm`, since all of those work on the app's own network. ### What the app is told Besides its `--env` variables, every app gets `POWERCLOUD_APP` (its name) and `POWERCLOUD_NODE` (its node's name). The replicas of a cluster app can use these to tell themselves apart. An `--env` of the same name wins over both. ### Its data directory An app's data directory is `/data` as the app sees it. What the app writes there is on the node's encrypted state volume and is kept across restarts, redeploys and reboots. Everything outside `/data` (the binary, or the image's files) is replaced on each deploy. ## The sandbox Every app runs in a sandbox of its own. In brief: - **Filesystem.** The app is chrooted into its own root, so it sees only its binary (or its image's files) and its data: no node state, keys or devices. It gets its own `/proc` (showing only its processes), a `/dev` with `null`, `zero`, `full`, `random` and `urandom`, and `/dev/shm` and `/tmp`. The apps directory is mounted `nosuid` and `nodev`, and setuid bits, device nodes and FIFOs are dropped from images. - **User.** It runs as a user of its own (numbered from 20000, assigned at deploy) with no capabilities and no supplementary groups, so it cannot write its own binary or another app's data. An image's `User` setting is ignored, so images that expect to run as root must tolerate running as an ordinary user. - **Processes.** It gets new PID, mount and UTS namespaces. It is PID 1 of its own process tree, its hostname is its name, and everything it starts dies with it. - **Limits.** It is placed in its own cgroup as it starts, which caps its memory, CPU time and number of processes. - **System calls.** It runs under a seccomp filter with `no_new_privs`. The filter refuses calls from another CPU ABI, and returns `EPERM` for mounts, `chroot`, `pivot_root`, `unshare`, `setns` and namespace-creating `clone`; `ptrace` and `process_vm_*`; `bpf`, `perf_event_open`, keyrings, `userfaultfd` and `io_uring`; module loading, kexec, `reboot` and swap; setting the clock; and open-by-handle. `clone3` gets `ENOSYS`, so C libraries fall back to `clone`. - **Network.** By default it has a network of its own, with only loopback: it cannot reach the node's network, the mesh or the internet, and nothing reaches it but its published ports and the links other apps were given to it. It may bind ports below 1024. `pcctl app ls` shows `on` in the `SECCOMP` column for an app running under the filter. ## Networking ### Publish ports `--port NODE:APP` makes the node listen on port `NODE` and pass each TCP connection to port `APP` on the app's loopback. `--port NODE:APP/udp` does the same for UDP: each client address gets its own session to the app's port, closed after two minutes idle, with at most 1024 open at once. ```sh pcctl app deploy --image coredns/coredns --port 53:53/udp --port 53:53 home-1 dns ``` The app has no address of its own. Users reach it at the node's address on the published port. ### Connect out with --egress `--egress` gives the app a way out while keeping its own network. Its traffic goes out through the node, masqueraded as the node, to anything outside it. It still cannot reach the node itself (its API, its storage, other apps' published ports) or other apps. The app gets the node's name servers in `/etc/resolv.conf`. Egress is IPv4 only. ### Share the node's network with --net host `--net host` puts the app on the node's own network, with all of its access. It is the coarse option: there is nothing in between, such as reaching only some hosts. Prefer `--port`, `--egress` and `--link`. ### Link apps by name `--link NAME:PORT` lets the app reach another app, `NAME`, on its TCP port `PORT`, by that name, without the other app publishing anything. Inside the app, `NAME` resolves (through its `/etc/hosts`) to an address in the app's own network where the node picks up each connection and passes it to the target's port. ```sh pcctl app deploy --image postgres:alpine --mem 512M home-1 db pcctl app deploy --link db:5432 --port 8080:8080 home-1 api ./api ``` The `api` app then connects to `db:5432`. Add `/udp` for UDP (`--link dns:53/udp`); each of the app's sockets gets a session, closed after two minutes idle, with at most 1024 at once. UDP to a link address's other ports is refused at once, so an app trying QUIC to a peer it has only a TCP link to falls back to TCP. Good to know about links: - An app reaches only the apps and ports it was linked to. - The app may serve the same port itself, so two peer-to-peer apps can be linked to each other on the port they both use. - A link to an app deployed by hand works only on the same node. A link to a cluster app works wherever the cluster placed it, over the encrypted mesh, and follows it when it moves (see [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md)). ## See, restart and remove apps ```sh pcctl app ls home-1 pcctl app logs home-1 web pcctl app logs home-1 web 1048576 pcctl app restart home-1 web pcctl app rm home-1 web pcctl app rm --keep-data home-1 web ``` - `app ls NODE` lists the node's apps: state, uptime, restarts, memory kills, memory used against the limit, CPU used against the cap, whether seccomp is on, the network (`own`, `host` or `microvm`, with `+egress`, `->NAME:PORT` for links and `NODE->APP` for published ports) and the last exit. - `app logs NODE NAME [BYTES]` prints the end of the app's output, 64 KiB by default. Standard output and standard error both go to the log, which is rotated at 1 MiB. Programs that reopen `/dev/stdout` or `/dev/stderr` (nginx, for one) work. - `app restart NODE NAME` stops the app and starts it again. - `app rm NODE NAME` stops the app and deletes it with its data. With `--keep-data`, the data directory is kept, and the app gets its user and its files back if deployed again on that node under the same name. The web console (`pcctl console`) shows the same per node, and can deploy, restart and remove apps from a form; see [pcctl command reference](https://powercloud.unidatum.com/docs/pcctl.md). ## Kept running An app that exits, or is killed (by its memory limit too), starts again after a delay that doubles from 1 second up to a minute. The delay starts again from 1 second once the app has run for more than a minute. `RESTARTS` and `OOM KILLS` in `pcctl app ls` count these, and `LAST EXIT` says how it last ended. Apps start again by themselves when the node boots and is unlocked; until then the state volume, and so every app, is locked (see [Unlocking nodes after a reboot](https://powercloud.unidatum.com/docs/unlock.md)). ## MicroVMs By default, apps share the node's kernel, so a kernel bug an app can reach is a way out of its sandbox. `--vm` runs the app under a kernel of its own, in a microVM, with the CPU's hardware virtualization as the boundary. Use it for code you trust less than the node. ```sh pcctl app deploy --vm --cpu 2 --mem 512M --image nginx:alpine --port 80:80 home-1 site ``` Nothing else about the app changes: it is deployed, listed, logged, restarted and removed like any app, from a binary or an image, and its microVM runs inside the same sandbox (namespaces, chroot, cgroup, its own user, seccomp). A guest escape lands in that sandbox. Inside the guest, the app again runs as its own user with no capabilities, under the same seccomp filter. A guest boots, runs its app and reports its exit in about a second. What is different: - **It needs KVM.** A node runs microVMs where its CPU offers virtualization to it: bare metal (on AWS, x86 and Graviton `.metal` instances), or a VM with nested virtualization. Most cloud VMs do not. On a node without KVM, the deploy fails with "this node can't run microVM apps (it needs KVM)". - **vCPUs come from `--cpu`.** The guest gets a vCPU for each CPU of its limit, rounded up (`--cpu 1.5` gives two), or, with no `--cpu`, one for each of the node's CPUs, up to 32. The CPU cap applies to the vCPUs together. - **Memory includes its files.** The guest's root filesystem is in its memory, so the app's files (but not its data directory) count against `--mem`, as does the guest kernel's 64 MiB. Images of a few hundred MiB are fine; multi-GiB ones are not (2 GiB at most). Files the app writes outside `/data` are in that memory too, and are gone when the guest stops. - **Data stays on the node.** The data directory is shared with the guest at `/data`, not copied in, so what the app writes there is on the node and survives the guest. - **Network.** The app always has its own network (not `--net host`). The guest is at 10.0.2.15 on a private link to the node, which is at 10.0.2.2. Published ports, `--egress` and links work as for any app; linked apps get addresses from 10.0.2.100 onwards. - `app ls` shows `microvm` in the `NETWORK` column. The guest kernel's serious messages go to the app's log, without its boot messages. ## Limits - **One node, by hand.** An app runs where you deploy it. For placement, replicas and failover, use [Cluster apps](https://powercloud.unidatum.com/docs/cluster-apps.md). - **No outbound rules.** Egress reaches anything outside the node; there is no list of allowed hosts. Egress is IPv4 only. - **A denylist.** The seccomp filter refuses known-dangerous calls and allows the rest, rather than allowing only a known-safe set. - **The node's kernel.** Apps share it unless deployed with `--vm`. Images run without user namespaces, as one unprivileged user. - **One user per app.** An app runs as its own user, whatever its image asks for.