Storage

PowerCloud keeps the cluster's data in an encrypted, deduplicating object store spread over several storage locations: local directories, S3-compatible buckets, and the encrypted disks of your own nodes. Everything is encrypted by pcctl on your machine before it leaves, so every location holds only ciphertext, and losing any one location loses nothing because each object is replicated (or erasure-coded) across them.

How the store works

• Objects are split into 4 MiB chunks, each sealed with XChaCha20-Poly1305 under a key derived from the cluster root key. A chunk's ID is an HMAC of its content, so identical chunks are stored once (deduplication) without letting a provider confirm what is inside.
• Names live in encrypted manifests. An object's name, size and chunk list are sealed too, under an ID that is an HMAC of the name. Providers see no names, only opaque blobs.
• Every read is checked. Each chunk is decrypted and its MAC compared with its ID, so a provider that changes or swaps a blob is caught.
• The cluster header is the one unencrypted blob. It holds the keyslots (passphrase and recovery key), which wrap the root key but contain no plaintext key material.

The cluster root key never leaves your machine in the clear. Nodes that relay or hold storage never get it. For the key hierarchy and what a passphrase protects, see Security model and Concepts.

Storage locations

A repository is a comma-separated list of locations, given with --repo or the POWERCLOUD_REPO environment variable:

export POWERCLOUD_REPO='/srv/pc,s3://my-bucket/pc?region=us-east-2,node://home-1'
pcctl ls          # same as pcctl --repo "$POWERCLOUD_REPO" ls

Each location is one of:

Quote --repo values in the shell whenever they contain & or ;.

S3 bucket options

A bucket name containing dots needs path_style=true over HTTPS. Unknown options are refused.

Credentials

Credentials never go in the location string. For creds=NAME, pcctl reads them from the environment:

• NAME_ACCESS_KEY_ID
• NAME_SECRET_ACCESS_KEY
• NAME_SESSION_TOKEN (optional)

NAME is upper-cased, so creds=r2 reads R2_ACCESS_KEY_ID. Without creds=, the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are used. Each location can use its own provider's keys. A location whose credentials are missing fails with a message naming the variables to set.

Use one bucket and one key scoped to that bucket per provider. Stolen storage keys let an attacker delete or corrupt that provider's copy, but not read it, and the other locations cover the loss. Where a provider offers bucket versioning or object lock, it protects that copy as well.

Provider examples

A repository over a local disk, R2 and B2:

export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=...
export B2_ACCESS_KEY_ID=... B2_SECRET_ACCESS_KEY=...
pcctl --repo "/srv/pc,s3://my-r2-bucket/pc?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com&region=auto&creds=r2,s3://my-b2-bucket/pc?endpoint=https://s3.us-west-004.backblazeb2.com&region=us-west-004&creds=b2" init

The S3 client in pcctl needs no SDK: it uses PUT, GET, DELETE and ListObjectsV2, signed with AWS Signature Version 4, and signs the SHA-256 of every request body so a body changed in transit is rejected. Network errors, throttling (429) and server errors (5xx) are retried with backoff, up to 4 attempts. Authentication errors and missing buckets fail at once, and a missing bucket is never mistaken for a missing object. It is checked against real S3 servers (RustFS) in automated tests, and against AWS S3.

Replication and the write quorum

With several locations, every write goes to all of them and succeeds once a write quorum has accepted it. Reads try each location until one serves an authentic copy.

• The default quorum is a majority: 1 of 1, 2 of 2, 2 of 3, 3 of 4 or 5.
• --quorum N changes it for one command, for example to keep writing while a provider is down:

pcctl --repo /srv/pc,s3://a/pc,s3://b/pc --quorum 1 put docs/a.pdf a.pdf

A location that missed writes is brought up to date later with pcctl repair, or by a serving node's scheduled sync (see "Storage through a node").

• Newest version wins. Reads fetch the manifest from every reachable location and use the newest one that authenticates, so a location that missed an overwrite never serves the old version.
• Deletes are tombstones. rm writes an encrypted "deleted" manifest rather than removing the object, so a location that missed the delete can't bring the object back.
• Unreachable is not "not found". An object missing from some locations while others are unreachable is reported as an error, since it may exist on the unreachable ones.

Erasure coding

Full replicas cost n times the data. An erasure-coded location spreads each blob over n providers as k data shards and n−k parity shards (Reed-Solomon). Any k shards rebuild it, so it survives losing n−k providers while storing n/k times the data:

pcctl --repo 'ec:4:s3://a/pc?endpoint=...&creds=r2;s3://b/pc?endpoint=...&creds=b2;s3://c/pc?endpoint=...&creds=wasabi;s3://d/pc?region=us-east-2;node://home-1;node://colo-1' init

That is 4 data and 2 parity shards over six providers: any two can be lost, at 1.5 times the data instead of 3 times for three replicas.

• ec:K:LOC;LOC;... separates its parts with ;. Parts can be directories, S3 buckets or node:// replicas. K must be at least 1 and less than the number of parts. Erasure-coded sets can't be nested.
• The whole set is one location to everything else (replication, quorum, check, repair, gc, node sync), so it can stand alone or sit next to plain locations: --repo 'ec:2:/a;/b;/c,/srv/pc'.
• A write must store k+1 shards (or all n, if fewer), so a fresh write survives losing one more provider; a write that stores fewer fails. A delete must reach every part.
• No single provider is trusted: a reader decodes a version only when k shards agree on it, so a provider forging its shard is outvoted.
• Lost shards rebuild themselves: a read rewrites the shard of any part that answered without a usable one, and pcctl repair and a serving node's sync heal every blob of each erasure-coded location first.

Creating a repository and using it

init

pcctl --repo /srv/pc,s3://a/pc,s3://b/pc init

init asks for a new passphrase twice, writes the cluster header to every location and prints the cluster ID and a one-time recovery key:

Initialized cluster ...

RECOVERY KEY (write it down and store it offline — it is shown only once):

    ...

Store the recovery key offline. Without the passphrase or the recovery key no one, including you, can read the data. init refuses a repository that is already initialised, and refuses to run if any location can't be reached, so it never writes a new header over an existing cluster it can't see.

Every other command asks for the passphrase. For automation, set POWERCLOUD_PASSPHRASE, or POWERCLOUD_RECOVERY_KEY to unlock with the recovery key instead.

put, get, ls, rm

pcctl --repo "$POWERCLOUD_REPO" put docs/a.pdf ./a.pdf     # NAME then FILE
pcctl --repo "$POWERCLOUD_REPO" get docs/a.pdf ./out.pdf   # FILE optional: stdout without it
pcctl --repo "$POWERCLOUD_REPO" ls docs/                   # PREFIX optional
pcctl --repo "$POWERCLOUD_REPO" rm docs/a.pdf

put prints what it stored; ls prints size, modification time and name:

stored docs/a.pdf (482113 bytes, 1 chunks)

482113  2026-10-28 14:02  docs/a.pdf

Writing a name that exists replaces it. When two admins write the same name at the same time, the later one wins.

check, repair and gc

These commands work on each location separately, so they always need --repo (not --via).

check

pcctl --repo /srv/pc,s3://a/pc,s3://b/pc check [--verify]

check lists every location and reports, per location, missing, stale (older than the newest version) and corrupt copies:

12 objects, 3 deleted, 40 chunks in use, 2 unreferenced (reclaimed by gc)

LOCATION      BLOBS  MISSING  STALE  CORRUPT  REPAIRED  STATUS
/srv/pc       55     0        0      0        0         ok
s3://a/pc     51     4        0      0        0         needs repair
s3://b/pc     0      0        0      0        0         unreachable: ...

It exits with an error when locations differ. --verify also downloads and authenticates every copy of every chunk, catching bit rot or tampering at rest; without it, chunks are checked by listing only (manifests are always read and authenticated).

An object reported as DAMAGED has data missing from every reachable location. A WARNING: ... has a different cluster header means a location holds another cluster's header (or one left by an interrupted key rotation); it is reported, never changed.

repair

pcctl --repo /srv/pc,s3://a/pc,s3://b/pc repair [--verify]

repair writes the newest manifest, every chunk a live object uses, and the header wherever they are missing or outdated. Every copy is authenticated before it is written, so repair can't spread corruption. With --verify, corrupt chunk copies are found and replaced too.

Adding a provider is the same operation: add its location to --repo and run repair. Dropping a provider means removing it from --repo (and from any node serving the storage).

pcctl --repo /srv/pc,s3://a/pc,s3://new/pc repair   # copies everything onto s3://new/pc

gc

pcctl --repo /srv/pc,s3://a/pc,s3://b/pc gc [--grace 24h] [--dry-run]

gc deletes chunks no live object uses: from deleted objects, overwritten versions, or a put that failed partway.

• A chunk is deleted only after it has been unreferenced for the grace period (default 24h) across two runs. The first run records it, a later run deletes it if it is still unused. This protects chunks that a put in progress has uploaded but not yet listed.
• --dry-run reports what would be deleted.
• gc refuses to run unless every location is reachable, since a manifest on an unreachable one might use chunks that look unused elsewhere.

40 chunks in use, deleted 2, 1 unreferenced waiting out the 24h0m0s grace period

Use a longer --grace if a single put can take longer than a day.

Rollback protection

Encryption stops a provider from forging or swapping objects, but not from serving an older, authentic version of an object, or hiding it. Because every location is untrusted, PowerCloud records the newest version of each object with your nodes instead.

Once the nodes run their consensus group (pcctl --repo ... consensus init, see Managing nodes):

• After every write, pcctl records the object's version time with the nodes. The record is keyed by the object's encrypted ID, so the nodes learn nothing about names.
• On every read, pcctl compares what storage returned with that record. An object older than its record, or a recorded object that no location has, fails with store: rollback, naming both times. Deletions are covered the same way.
• To recover, write the object again: the new version replaces the rolled-back one.
• If no node with the consensus answers, pcctl warns no rollback protection this time and carries on. If only one node answers without a majority, it warns that the check uses that node's copy, which may miss recent writes.

Chunks are covered through the manifest that lists them. The cluster header and the node registry are read before pcctl can find a node, so they are not covered. Before consensus init there is no rollback protection at all.

Storage through a node

With --repo, every admin machine needs every location and the credentials for each. An enrolled node can hold them instead, so other admins (or you on another machine) need only the passphrase and an admin certificate:

export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=...
pcctl --repo "s3://...&creds=r2,s3://...&creds=b2,node://home-1" node storage home-1   # once
pcctl --via home-1 put docs/a.pdf ./a.pdf                                             # from then on

• Syntax: pcctl --repo LOC,... node storage [--sync 1h|off] NAME [LOC,...]. The node's locations default to --repo's. Give them explicitly when the node reaches them by other addresses, such as a private endpoint or a LAN address. --quorum sets the node's write quorum.
• Only s3://, node:// and ec: sets of them. A node's root filesystem is in RAM, so it does not serve local directories.
• Credentials named by creds= are taken from pcctl's environment and stored on the node's encrypted state volume. They survive a reboot, a crypto-erase destroys them, and pcctl node info shows the locations but never the credentials.
• Checked before use. The node refuses a location it can't reach, or one that holds another cluster's header, so a typo can't point admins at the wrong bucket.
• The node only relays ciphertext. pcctl still unlocks the keys with the passphrase and encrypts and decrypts locally. A node serving storage sees what a storage provider sees. Only admin certificates may use these endpoints, so a compromised node can't read or overwrite storage through another node.
• POWERCLOUD_VIA can stand in for --via.

--via works for every command except init, check, repair, gc, rotate and node storage, which work on each location separately and need --repo.

Scheduled sync

A serving node copies every blob that some of its locations lack from one that has it: one minute after it starts, then every hour by default. A provider that was down during writes catches up without an admin.

pcctl --repo ... node storage --sync 30m home-1   # sync every 30 minutes
pcctl --repo ... node storage --sync off home-1   # no scheduled sync
pcctl node sync home-1                            # sync now
pcctl node info home-1                            # shows the storage and the last sync

Sync needs no keys: it works on opaque IDs and only fills gaps. Replacing a stale or corrupt copy takes decrypting to tell which copy is right, so that remains the job of pcctl repair. A location that can't be listed is left alone.

Nodes as storage locations

An enrolled node's encrypted state volume can hold a replica of the cluster's blobs, so your own home or colo machines are providers alongside the clouds:

pcctl --repo /srv/pc,s3://a/pc,node://home-1,node://colo-1 repair        # copy everything onto the nodes
pcctl --repo node://home-1,node://colo-1,s3://a/pc put docs/a.pdf ./a.pdf
pcctl --repo node://home-1,node://colo-1 node storage home-1             # then: pcctl --via home-1 ...

• node://NAME is a location like any other, reached with your admin certificate, under the same replication, quorum, check, repair and gc rules.
• The blobs are encrypted twice: by the cluster's keys, which the node never has, and by the node's disk encryption. A crypto-erase of the node destroys the replica. A node whose state is only in RAM refuses to hold one.
• A node serving storage reaches other nodes' replicas over the mesh with its own node certificate.
• Nodes may read and write each other's replicas, but only admins may delete. A compromised node could overwrite blobs with garbage, which is caught on read and replaced by repair, but it can't erase another node's replica.
• A write that would leave less than 256 MiB free on the node's state volume is refused, so a full replica never starves the node.
• Connections to a node give up after 10 seconds, so a node that is down costs reads seconds, not minutes.

What a provider can and can't see

Because each blob's size is visible, a provider can estimate the total amount of data stored, and the size of a small file that fits in one chunk.

Limits

• Last writer wins. Two admins writing the same name at once can't both win; the later modification time does.
• Tombstones are kept forever. They are small, and removing them safely would need to know that every provider has seen them.
• gc assumes no put runs longer than the grace period. Raise --grace for very slow uploads.
• Rollback protection needs the nodes' consensus and a reachable node; otherwise pcctl warns and goes on without it. The cluster header and node registry are not covered.
• check, repair, gc, init and rotate need --repo with every location, not --via.
• Responses over 256 MiB are refused from S3 services, as a guard against a hostile service; chunks are far smaller.
• Losing the passphrase and the recovery key loses the data. There is no backdoor.

For an S3 endpoint over this storage that any S3 client can use, see S3 gateway. For troubleshooting unreachable locations and rollback errors, see Troubleshooting.