Use cases with the Unidatum Data Engine

The Unidatum Data Engine is a sibling Unidatum product: a peer-to-peer data engine with MongoDB-style document collections, distributed SQL over parquet files (a lakehouse), a content-addressed file layer built on BitTorrent v2, and SQL wire-protocol front ends. PowerCloud supplies what the engine leaves to its host: encrypted disks, a sandbox that keeps services running, an encrypted mesh between your machines, and encrypted storage spread across providers. This page says exactly what of that combination works today, then describes use cases as designs: no combined deployment has been tested yet.

The engine's own product site is https://www.unidatum.ie/en. It is licensed separately from PowerCloud. Its executable is called unidatum, and some of its settings are still environment variables named P2PFS_*.

Status

Each product works on its own and has its own tests. Nothing on this page has been run end to end, so treat every command below as a starting point to verify on your own cluster.

What is established today

Should work, but untested

• The engine as a single static binary app with DuckDB linked in: one file, nothing beside it, queries included. This needs the fully static (musl) in-process build; the engine's standard releases don't ship it, so you build it.
• The engine as a container-image app (pcctl app deploy --image), with the companion programs it needs, and either the in-process build or the DuckDB command-line program in the image. There is no published engine image, so you build one (see "Packaging the engine" below).
• The engine's blob tier on PowerCloud app storage, with the engine as a cluster app deployed with --storage, and P2PFS_S3_ENDPOINT=http://127.0.0.1:9000.
• The engine's blob tier on the PowerCloud S3 gateway, from an engine running anywhere, trusting the cluster CA.
• Engine peers on several PowerCloud nodes, each a cluster app pinned to its node with --nodes, linked to each other by name over the encrypted mesh.
• SQL front ends (PostgreSQL, MySQL and SQL Server wire protocols, Arrow Flight SQL) and the MongoDB wire protocol served from an engine app through published ports or links.
• The engine's Node.js edition in an image with Node.js. PowerCloud runs any image whose programs tolerate its sandbox, but the Go executable is the one this page assumes: it needs no Node.js.

What doesn't work yet, and why

• The standard release binary can write but not query as a binary app. The engine reads documents and runs SQL through DuckDB. Its standard Linux releases are built without cgo and run the duckdb program beside them, but a PowerCloud binary app's root holds only its one binary and /data. Such a node can write documents, sync, hold and serve files, but cannot read documents back or run SQL. Use the in-process build (DuckDB linked in) or an image for anything that queries.
• Companion programs need an image. unidatum-sqld (SQL wire serving), unidatum-kafka (the Kafka sink) and unidatum-xfer (fast bulk transfer) are separate programs that must sit beside unidatum, which a binary app can't do.
• No shell in an app. PowerCloud has no command to run a one-off command inside an app. The engine's setup commands (unidatum init, unidatum store attach, unidatum sync PEER) must run from the image's entrypoint script, or as separate deploys of a binary app (each deploy under the same name keeps /data).
• No LAN or internet discovery in an app's own network. An app's network holds only itself, so the engine's mDNS finds nothing, and its DHT needs --egress to reach anyone. Peers must be named explicitly, by link name.
• The engine can't tell clients apart. PowerCloud passes each connection to a published port or link from inside the app's network, so the engine sees it coming from loopback (in a microVM, from the node's side of the guest's link). The engine always admits loopback, so its own address allow-lists and per-address limits (.p2pfs/access.json) don't work behind PowerCloud. Rules by node ID still apply.
• Memory is sized from the node, not the app. The engine sizes DuckDB's memory from the machine's total RAM, which in an app is the node's, not the app's --mem. Set P2PFS_DUCKDB_MEMORY and P2PFS_DUCKDB_RSS_MAX yourself, or the app's memory limit kills large queries.
• A Kafka broker isn't part of the engine. The engine ingests from an existing Kafka cluster (its Kafka sink), and has a Kafka-shaped producer of its own. It doesn't accept Kafka clients as a broker.

How the two fit together

The main gains from running them together:

• Nothing in plaintext on any disk or bucket you don't hold the key to. The engine's metadata, documents and file bytes sit on the node's encrypted state volume. Its bulk data, through the blob tier, lands in PowerCloud storage, where providers see only ciphertext without names.
• No single provider. The bucket behind the blob tier isn't one provider's bucket: PowerCloud replicates or erasure-codes it across the locations your nodes serve (directories, S3-compatible buckets, your own nodes), so losing a provider or moving off one doesn't touch the engine.
• Two layers of failover. The engine replicates between peers, so a peer that goes away is one copy less, not an outage. PowerCloud restarts a crashed engine, and can move an engine app to another node.
• Encrypted links between peers. Links between apps on different nodes ride the WireGuard mesh, so the engine's own first contact with a peer, which it makes in plaintext before switching to its secure wire, is still inside an encrypted tunnel.

Packaging the engine

As a container image (recommended)

An image can hold the engine, its companion programs, DuckDB (linked into the engine, or as the command-line program) and a start-up script. It is pulled by pcctl on your machine from a registry over HTTPS (private registries work with docker login). A sketch with the standard release build, so with unidatum, unidatum-sqld and a duckdb program for the image's architecture in the build directory (with the in-process build, leave out duckdb and P2PFS_DUCKDB):

FROM debian:stable-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
COPY unidatum unidatum-sqld duckdb /usr/local/bin/
COPY start.sh /start.sh
ENV P2PFS_DUCKDB=/usr/local/bin/duckdb HOME=/data
ENTRYPOINT ["/start.sh"]

The script prepares the library on first start, attaches the blob tier once, and runs the node. POWERCLOUD_APP is the app's name, which PowerCloud sets:

#!/bin/sh
set -e
cd /data
[ -f .p2pfs/config.json ] || unidatum init --library "$LIBRARY" --node-name "$POWERCLOUD_APP"
if [ -n "$BLOB_ROOT" ] && [ ! -f .blob-tier-attached ]; then
  unidatum store attach "$BLOB_ROOT" && touch .blob-tier-attached
fi
for peer in $PEERS; do unidatum sync "$peer" || true; done
exec unidatum ui --bind "" --port 47800 --no-mdns --no-dht --rendezvous off \
  --advertise "$POWERCLOUD_APP:47800" "$@"

Attach only once: attaching again makes the engine forget which blobs the bucket holds. A peer that isn't up yet when the script runs isn't registered; restart the app later with pcctl app restart NODE NAME. Arguments after -- in pcctl app deploy replace the image's command, which here are passed on to unidatum ui (such as --sql or --mongo-port 27017).

Good to know about images:

• PowerCloud ignores the image's User: the app runs as its own unprivileged user, which owns the image's files.
• /data is the app's data directory, kept across restarts and redeploys.
• The engine's auto-update (off by default) replaces its own binary; leave it off and update by deploying a new image.
• In a microVM (--vm), the image's files count against the app's memory, and /data is shared into the guest from the node. The engine's metadata database, SQLite in write-ahead-log mode, hasn't been tried on that shared directory.

As a static binary

The engine runs as a binary app with no image. Which build decides what it can do:

• With DuckDB linked in (-tags duckdb_inproc), it does everything a node does, queries included, from one file. A binary app must be fully static, so build DuckDB with the engine's scripts/linux-duckdb.sh for a musl toolchain with PIC=OFF, then build the engine against it. Linking DuckDB in also removes the two DuckDB process starts the standard build pays on every query.
• The standard release build (no cgo) writes, syncs and serves files, but can't read documents back or run SQL, because its duckdb program can't sit beside it.

Each step is a deploy under the same name, which keeps /data:

pcctl app deploy --mem 512M edge-1 ud ./unidatum -- init --library plant --node-name edge-1
pcctl app logs edge-1 ud
pcctl app deploy --mem 512M edge-1 ud ./unidatum -- ui --port 47800 --no-mdns --no-dht --rendezvous off

The first deploy creates the library, exits, and is restarted until the next deploy replaces it; a second init refuses to overwrite the library, so the restarts do no harm. A binary is for one architecture: build or fetch the one that matches the node.

Use cases

Each use case below is a design. The names ud-home, ud-colo and ud-cloud are engine apps; home-1, colo-1 and cloud-1 are nodes.

Document database for your own apps

Setup. One engine image app, and your application as another app linked to it. Your app talks to the engine's HTTP API, or to its MongoDB wire protocol with a MongoDB driver:

pcctl app deploy --image registry.example.com/ud:latest --mem 2G --cpu 2 \
  --env LIBRARY=apps --env P2PFS_DUCKDB_MEMORY=1GB --env P2PFS_DUCKDB_RSS_MAX=1600MB \
  home-1 ud-home -- --mongo-port 27017
pcctl app deploy --link ud-home:7480 --link ud-home:27017 --port 8080:8080 home-1 api ./api

api then reaches http://ud-home:7480 and mongodb://ud-home:27017. The engine's MongoDB users come from P2PFS_MONGO_USERS.

Why together. The engine needs no published port at all, so its API is reachable only by the apps you linked. Its files are on the node's encrypted disk.

Sizing. Writes need little memory; reads and queries run in DuckDB. 1 to 2 GiB is a reasonable start for a small database. Set P2PFS_DUCKDB_MEMORY to about 60% of --mem and P2PFS_DUCKDB_RSS_MAX to about 80%.

Caveats. A link to an app deployed by hand works only on the same node. For a database that survives the node, add peers (below).

Bulk data encrypted across providers

Setup. The engine as a cluster app with --storage, its blob tier pointed at the app's own S3 endpoint:

pcctl --repo LOC,... cluster app deploy --nodes colo-1 --storage --mem 2G \
  --env P2PFS_S3_ENDPOINT=http://127.0.0.1:9000 --env BLOB_ROOT=s3://ud/colo \
  --env P2PFS_BLOB_CACHE=20G --env LIBRARY=archive \
  --image registry.example.com/ud:latest ud-colo

In a microVM the endpoint is http://10.0.2.2:9000. P2PFS_BLOB_CACHE bounds the blobs kept on the node; the rest are fetched back from storage when read. An engine outside PowerCloud (a workstation, a server you already run) can use the S3 gateway instead:

pcctl --repo LOC,... s3 key add --bucket ud workstation
pcctl s3 ca > powercloud-ca.pem
cat /etc/ssl/certs/ca-certificates.crt powercloud-ca.pem > ca-bundle.pem
export SSL_CERT_FILE=$PWD/ca-bundle.pem P2PFS_S3_ENDPOINT=https://s3.example.com:9443
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
unidatum store attach s3://ud/workstation

SSL_CERT_FILE replaces the system's list, hence the combined bundle. On macOS and Windows, Go programs use the system's trust store instead, so the cluster CA would have to be trusted there; that hasn't been tried. See S3 gateway.

Why together. The engine's blobs (every file and every parquet member) are content-addressed and verified on fetch; PowerCloud encrypts them before any provider sees them, spreads them over several, and keeps bucket and object names private. With --storage, the blobs follow the app if it moves to another node.

Sizing. Every engine blob is one object, sent in a single put; PowerCloud splits it into 4 MiB encrypted chunks, each stored once per replica. Provider request counts and bytes therefore multiply by the replication factor. Plan the local cache to hold the working set: a cache smaller than what queries read makes every query fetch again.

Caveats. The blob tier holds blobs only. The engine's metadata (its database of documents' history, manifests and catalog) stays in /data on the node, so an app moved to another node finds its blobs but not its library; peers are what keep the library itself. --volume saves /data every 30 seconds as whole files, which isn't a safe copy of a live database. The blob tier only adds: run unidatum store prune to delete what nothing references.

One library across home, colo and cloud

Setup. One engine cluster app per node, each pinned with --nodes, each linked to the others on the sync port:

pcctl --repo LOC,... cluster app deploy --nodes home-1 --mem 4G --cpu 2 \
  --link ud-colo:47800 --link ud-cloud:47800 \
  --env LIBRARY=fleet --env "PEERS=ud-colo:47800 ud-cloud:47800" \
  --image registry.example.com/ud:latest ud-home -- --sql

Repeat for ud-colo on colo-1 and ud-cloud on cloud-1, each linking the other two. Add --link NAME:47800/udp to carry the engine's QUIC too, or start it with --quic-port -1 to use TCP only. Each engine advertises its app name, which the others resolve through their links.

Why together. Links between nodes run over the mesh, encrypted, using the mesh's own NAT traversal, so no engine port is opened on a router or the internet. Every peer holds the library, so a node that goes away loses nothing; when it comes back, its engine catches up.

Sizing. Each peer holds what its library holds, unless its blobs live in the blob tier. Links carry the engine's traffic through the nodes' own processes and the mesh; expect bulk transfer between peers to be slower than over a direct connection (not measured).

Caveats. Use one app per node rather than --replicas: replicas share a name, so they can't be linked to each other. Registration in the engine goes one way; each script syncs to its peers at start, so start all three and restart any that started before the others were up.

Analytics over private data in several places

Setup. The peer group above, started with --sql so each peer answers its share of a distributed query, plus the SQL front ends in the image. A start script can start unidatum sql-serve --pg-port 5433 --user U --password P beside the node, and the app publishes or links that port.

Why together. Data stays where it was produced (a home server, a colo box, a cloud VM), each copy on an encrypted disk, and queries go to the data. BI tools connect with their own PostgreSQL, MySQL or SQL Server drivers.

Sizing. From the engine's own measurements: about 1 GB per node for a GROUP BY on an integer or a short text key, and about 3 GB plus 5.7 bytes per input row for text keys longer than 12 characters, with 8 DuckDB threads. Thread count changes memory a lot; set P2PFS_DUCKDB_THREADS and measure. DuckDB in the engine doesn't spill to disk: a query fits or runs out of memory. Give PowerCloud's placement the real figure with --mem. See Costs and sizing for the nodes.

Caveats. Querying parquet that sits directly in a bucket (the engine's store path) goes through DuckDB's own S3 support, which needs its httpfs extension and its own endpoint settings; that hasn't been checked against PowerCloud storage.

File sharing and sync

Setup. Files added to the library (through the engine's API, or on any other peer) sync to every peer as content-addressed blobs. Publish the sync port to let devices outside the cluster join the library, or let ordinary BitTorrent clients download from it:

pcctl app deploy --image registry.example.com/ud:latest --port 47800:47800 --env LIBRARY=family home-1 ud-home

Why together. Every copy is on an encrypted disk, and with the blob tier the bulk lives in encrypted storage across providers rather than on any one machine.

Sizing. A peer without the blob tier needs disk for the whole library. The engine's fast bulk transfer (unidatum-xfer) is a separate program that needs the C crypto library and a port of its own; without it, transfers use the engine's slower built-in path.

Caveats. The sync port is the engine's front door: limit who can join by node ID in its access policy, since every connection reaches it from loopback.

Event streams

Setup. The engine's Kafka sink (unidatum kafka-sink, needing unidatum-kafka in the image) consumes topics from a Kafka cluster and lands each batch with its offsets in one atomic commit. With --egress it reaches an external Kafka; a Kafka broker run as another app is reached by a link. Without Kafka, producers write through unidatum produce or the HTTP API.

Why together. Topics become queryable, encrypted tables, and the sink is restarted by PowerCloud if it stops.

Caveats. One sink per topic. The engine isn't a broker, and a Kafka broker as a PowerCloud app hasn't been tried.

Edge nodes

Setup. A small node (an arm64 board or a branch-office box) runs an engine that writes locally and syncs to the peers at home or in a colo. It runs as one static binary: the in-process build if it should answer queries too, or the standard release build if it only produces data and leaves queries to bigger peers. Your collector app links to it on port 7480.

Why together. The edge disk is encrypted, the edge node needs no open port (it reaches its peers through the mesh), and a stolen disk holds nothing readable. How much a stolen machine gives away depends on how it unlocks (see Unlocking nodes after a reboot).

Sizing. A write-only engine is light; leave room for DuckDB only on nodes that query.

Caveats. For the edge node to reach peers on other nodes, both must be cluster apps (links to hand-deployed apps work only on the same node).

Good to know

• Keep the engine's UI private. Reach it through links from your own apps. If you publish port 7480 to use it from a browser, do it only on a node whose network you trust.
• Name peers explicitly. Start the engine with --no-mdns --no-dht --rendezvous off in an app's own network, and give it --advertise NAME:47800 so peers dial the link name.
• Set memory by hand. --mem for PowerCloud, P2PFS_DUCKDB_MEMORY and P2PFS_DUCKDB_RSS_MAX for the engine.
• Consistency. The engine's sync is conflict-free between its peers. PowerCloud storage under it is last-writer-wins per object, which suits the engine's content-addressed blobs: a blob's name is its content.

Related pages: Apps on a node, Cluster apps, S3 gateway, Storage, Costs and sizing, Security model.