# What a VolObj hub sees, keeps and forgets

This page is for anyone who points a volume at a hub they do not run — the
public hub, typically — and for whoever operates one. It lists what the hub
receives, what it stores, for how long, what it never stores, and how to leave.
Every statement is tied to the code that makes it true; the file names are in
the last column or in the text. If the code and this page ever disagree, the
code is right and this page is a bug.

A hub is an invalidation relay. It never receives file contents and never talks
to your bucket. Your data, your S3 credentials and your volume key never leave
your members. See [SECURITY.md](SECURITY.md) for the threat model and
[HUB-PUBLIC.md](HUB-PUBLIC.md) for operating a multi-tenant hub.

## The public hub

| | |
|---|---|
| URL | `wss://hub.volobj.quave.one` (members), `https://hub.volobj.quave.one` (tenant API) |
| Operator | Quave, on its own platform Quave One (account `quave-one-oss`) |
| Where | Quave One region us-5: Ashburn, Virginia, USA, on Oracle Cloud Infrastructure |
| Software | `quaveone/volobj-hub:0.5.6`, public mode, anonymous tenants on; configuration in [deploy/hub/README.md](../deploy/hub/README.md#running-on-quave-one) |
| Storage | a Quave One managed Redis (tenant records, revocation flags, rate counters: §2) |
| In front of it | the Quave One ingress, which terminates TLS and passes the client address in `X-Forwarded-For` (the hub runs with `--client-ip-header X-Forwarded-For`). The ingress keeps its own access log under Quave One's configuration; this page covers the hub only. |
| E-mail | activation codes are sent over SMTP through Postmark (`smtp.postmarkapp.com`, port 587, STARTTLS). The address goes to Postmark to deliver the code; the hub keeps only its SHA-256 hash (§2). Both e-mail registration and anonymous tenants are available. |

Retention on the public hub is the default of §3: anonymous tenants unused
for 90 days, or whose credential expired more than 7 days ago, are purged.
As deployed on 2026-09-28 its configuration sets no `--stats-postgres-url`,
so no daily rollups are written to Postgres; if that changes, §2 says what
they hold.

## 1. What the hub receives

### Per connection (a member joining a volume's room)

| Item | Why | Code |
|---|---|---|
| Client IP address (TCP peer, or the first entry of `--client-ip-header` behind a trusted proxy) | per-IP rate limits of the self-service endpoints; country lookup with `--geoip-db` | `internal/hub/clientinfo.go` (`requestIP`), `register.go` |
| `User-Agent: volobj/<version> (<os>/<arch>; <mode>)` | usage statistics by client kind | `internal/core/useragent.go`, `internal/hub/clientinfo.go` |
| Credential (`Authorization: Bearer t1.…` / `t2.…`, or the legacy token) — it names the **tenant id** | authentication, caps, revocation | `internal/hub/hub.go` (`authenticate`), `credential.go` |
| Room id (in the URL: `/v1/rooms/<id>`, `hex(sha256(volumeKey + ":events"))`) | which volume's events to relay | `internal/hub/hub.go` (`serveRoom`) |
| A country header (`--country-header`, e.g. `CF-IPCountry`) if the operator configured one and a proxy sets it | usage statistics | `clientinfo.go` |

The `User-Agent` carries no hostname, user, volume or path: the mode is one of
`mount`, `csi`, `plugin`, `nfs`, `session`, `cli`.

### Per event (every write, delete or rename a member publishes)

| Field | Content |
|---|---|
| `vol` | the volume id (`hex(sha256(key))[:32]`) |
| `op` | `put`, `delete` or `rename` |
| `path`, `oldPath` | **the object path, in clear text** |
| `etag`, `size` | the object's ETag and size |
| `ts` | a Unix timestamp |
| `writer` | the member identity (see §5) |
| `seq` | assigned by the hub |

**Paths transit the hub in clear text, by protocol design.** The hub has to
relay them for the members to invalidate the right cache entries. TLS (`wss://`)
protects them on the wire; nothing protects them from the hub's own memory, and
at-rest encryption (`--encrypt`) encrypts object bodies, not paths. If a path is
itself sensitive, run your own hub (§8) or use polling mode (no hub at all:
members only reconcile by listing the bucket).

With several hub replicas the event also crosses the operator's Redis over
pub/sub (`internal/hub/redis.go`): pub/sub delivers and forgets, it stores
nothing, but whoever can `SUBSCRIBE` on that Redis sees the events pass.

## 2. What the hub stores

### Tenant record (public hubs)

Where these live depends on the hub's tenant store
([HUB-PUBLIC.md](HUB-PUBLIC.md) §6). With a SQL store — Postgres, as on the
public hub, or a SQLite file — the database is the system of record and holds
the same facts as the Redis keys below, digests only:

| Table | Content |
|---|---|
| `volobj_hub_tenants` | `id`, `kind`, `created_at`, `credential_expires_at` (anon), `email_digest` (the address **digest**, email only), `last_seen_at` (minute resolution, written at most every 5 minutes per replica) |
| `volobj_hub_revocations` | `tenant_id`, `revoked_at`, `revoked_until` (empty = forever); it outlives the record it revoked |
| `volobj_hub_email_bindings` | e-mail **digest** → tenant id |
| `volobj_hub_daily_counters` | `("anon-created", day)` → count (the hub-wide daily cap) |

DDL: [`deploy/hub/tenants.sql`](../deploy/hub/tenants.sql). Nothing else about
a tenant is in the database: no address, no IP, no room, no path, no event.
Redis is then a cache of those rows (the keys below, plus
`volobj:tenant:ok:<id>` and `volobj:tenant:norec:<id>`, 10-minute markers
meaning "not revoked" and "no record"). Pending registrations stay in Redis
only. `volobj hub tenants export` writes these rows as JSON lines, with the
digest and never an address. Without a SQL store the keys below are the only
copy (Redis alone), or everything is in the hub's memory and gone at its next
restart (no store configured).

| Key | Content | Kinds |
|---|---|---|
| `volobj:tenant:rec:<id>` | `kind` (`anon` / `email`), `created` (Unix seconds), `exp` (expiry of the current t2 credential, anon only), `email` (the address **digest**, email only) | anon, email |
| `volobj:tenant:seen:<id>` | last seen, rounded down to the minute, written at most once a minute per replica | anon, email, operator-issued t1 |
| `volobj:tenant:revoked:<id>` | the revocation flag | any revoked tenant |
| `volobj:tenant:email:<digest>` → id, `volobj:tenant:id:<id>` → digest | the e-mail binding | email |
| `volobj:tenants:anon`, `volobj:tenants:email` | index sets of tenant ids (for the idle sweep and the `volobj_hub_tenants_known` gauge) | anon, email |
| `volobj:tenant:pending:<digest>` | a pending registration: the code **hash**, when it was sent, attempts | email, 15 minutes |
| `volobj:daily:anon-created:<day>` | a counter (the hub-wide daily cap; Redis-only hubs — with a SQL store it is `volobj_hub_daily_counters`) | 48 hours |

The e-mail address is stored only as `sha256("volobj:email:" + address)`
(`internal/hub/register.go`, `emailDigest`). That digest is **not salted**:
someone with read access to that Redis (or that database, or an export) and a
candidate address can check whether it is registered. The address in clear is never stored; it exists in
memory while the code e-mail is sent. Registration codes are stored only as a
hash.

Rooms: the sequence counter of each room, `volobj:seq:<hash>`, is keyed by a
hash of the tenant and room id and holds a number (`redis.go`, `redisName`).

### Usage statistics (with a SQL store)

During the day, in Redis (every key expires 48 hours after its last write;
`internal/hub/stats_redis.go`; without Redis the same counts in the hub's
memory, with exact sets instead of HyperLogLogs):

- counters per *(tenant kind, client mode, version, os, country)*: connections,
  events, event bytes, join failures, tenants created;
- **HyperLogLogs** of the tenant ids and of hashed room ids seen that day, per
  kind and per tuple. A HyperLogLog keeps hashed registers from which the ids
  cannot be listed or recovered; it answers "about how many distinct".

Every `--stats-flush-interval` (30 minutes by default) one replica writes the
running day, and after 00:05 UTC the final version of the previous day, to the
SQL store (`deploy/hub/stats.sql`, `internal/hub/store_sql.go`): one row per
*(day, tenant_kind, client, version, os, country)* with `tenants_active`,
`tenants_created`, `connections`, `rooms_active`, `events`, `bytes`,
`join_failures`, plus one `*` row per kind with the exact distinct counts.
**There are no per-tenant rows.**

### Prometheus metrics

Labels come from closed or capped sets only (`internal/metrics/metrics.go`,
pinned by `TestHubMetricLabelsAreBounded`): `tenant_kind`, `client`, `version`
(major.minor, at most 32 distinct values per replica, then `other`), `os`
(allowlist), `country` (ISO code, at most 300, then `other`), `reason`,
`direction`. No tenant id, room id, IP address, path or e-mail is ever a label.

### Statistics page (only with `--stats-page`)

`GET /stats` and `GET /stats.json` on the hub's main address
(`internal/hub/statspage.go`) show **aggregates only**: live connections,
rooms and members per room (average and maximum), tenants with a live member
by kind (`anon`, `email`, `legacy`), recent events and bytes per second,
per-day and since-start counts of events, bytes, tenants created by kind, join
failures by reason and authentication failures, the uptime, the version and
the replica's random instance id; with the Redis backplane, sums over the live
replicas; with `--stats-postgres-url`, today's shared counters and the `*` row
per kind of the last 14 daily rollups. **It never shows a tenant id, room id,
IP address, e-mail, path or client tuple.** To add up the replicas, each one
writes its own counts (no identifier but its instance id) to the Redis hash
`volobj:hub:replicas`, which expires a minute after the last replica stops.
With `--stats-user`/`--stats-pass` the page asks for Basic auth; a wrong
attempt is counted in `volobj_hub_auth_failures_total`, and the client IP is
held in memory, in the page's rate-limit bucket, for about a minute (the same
rule as the registration endpoints). Without them the page is public, which is
safe only because it carries nothing but aggregates.

### Landing page (`--landing`; on by default with `--public`)

`GET /` on the hub's main address (`internal/hub/landing.go`) describes the
hub and how to use it, from the running configuration; `GET /llms.txt` and
`GET /privacy` serve copies, embedded in the binary, of the sandbox
instructions for LLMs and of this document. The pages set no cookie, run no
script and load nothing from elsewhere. The only request input they use is
the `Host` header, to write the commands shown on the page, and it is not
kept. **They store and log nothing**: a request to them is not logged,
counted or stored by the hub.

### Logs

The hub logs, at its only level: a member's random connection id, its **tenant
id** and the first 12 characters of the room id on join and leave; the tenant
id when a join is refused or a lookup fails; `tenant <id> activated` on an
e-mail activation. It never logs an event field (path, ETag, size, volume id,
writer; `TestHubLogsNoEventPayload`), a credential, or a client IP address:
the lines `net/http` itself writes (TLS handshake errors name the peer) go
through an address scrubber (`internal/hub/logscrub.go`). One exception, for
tests only: without `--mail-command` or SMTP the public hub writes activation
e-mails, address and code included, to its log, and says so at start.

A reverse proxy or ingress in front of the hub has its own access log; that is
the operator's configuration, not the hub's.

## 3. Retention

| What | How long |
|---|---|
| Client IP address | in memory only, in the per-IP rate-limit bucket of one replica, until the bucket has refilled (`register.go`, `sweepLimiters`, run every minute): **about 1 minute** for register/activate/renew/delete at the default 5/min, **at most 1 hour** for anonymous creation at the default 10/hour. Never in Redis, Postgres, metrics or logs. |
| Event payload | in memory while it is being relayed (milliseconds). Never stored. |
| Tenant record, last seen, e-mail binding (Redis and, with a SQL store, the database) | until the tenant is deleted (`DELETE /v1/tenants/me`), revoked and purged, or — for **anonymous** tenants — purged by the sweep after **90 days without use** (`--anon-idle-purge`) or **7 days after its credential expired** unrenewed (`internal/hub/lifecycle.go`, `SweepTenants`) |
| Revocation flag (Redis key and, with a SQL store, `volobj_hub_revocations` row) | a t2 (anonymous) tenant: until its last credential could no longer be renewed (expiry + 7 days); a t1 tenant: indefinitely (the flag is the tenant id and nothing else) |
| Pending registration | 15 minutes |
| Day counters and HyperLogLogs | 48 hours |
| Daily anonymous counter (SQL store) | indefinitely — one number per day, no identifier |
| SQL rollups (Postgres or SQLite) | indefinitely — they are aggregates |
| Anything in a hub without a store (memory mode) | until the hub restarts |
| Prometheus series | whatever retention the operator's Prometheus is configured with |

## 4. What the hub never stores

- file paths, ETags, sizes, contents — of events or of anything else;
- e-mail addresses in clear;
- IP addresses beyond the rate-limit window above;
- your volume key, S3 credentials or bucket coordinates (the hub never receives them);
- per-tenant usage rows (the rollups are per kind and client tuple only).

## 5. Member identity (`writer`, `createdBy`)

Every event carries the writer's member identity, and the control file records
the member that created it. By default it is now a **random `m-<12 base32>`
drawn once per process** (`cmd/volobj/version.go`, `defaultIdentity`) — it
used to be the hostname. `--identity` / `VOLOBJ_IDENTITY` still override it.
The CSI driver keeps a per-mount identity but hashes it:
`csi-<sha256(nodeID)[:12]>-<sha256(volume id | mount)[:12]>`, so node names and
PV names do not reach the hub (`internal/csi/driver.go`, `mountIdentity`). The
Docker plugin uses `p-<sha256(host / volume name)[:12]>`.

## 6. Anonymous tenants

`POST /v1/tenants/anonymous` (or `volobj hub anon --hub <url> --save`) gives a
tenant with no e-mail at all: the hub draws a random id (`anon-` + 20 base32
characters) and returns a **t2** credential that expires after `--anon-ttl`
(30 days by default). Renew it with `POST /v1/tenants/renew` (`volobj hub
renew --save`) any time before, or up to 7 days after, its expiry. Anonymous
tenants have lower caps (see [HUB-PUBLIC.md](HUB-PUBLIC.md)).

## 7. Leaving: revoke and delete

```bash
volobj hub delete-tenant --hub wss://hub.example.com --yes
# = DELETE /v1/tenants/me with Authorization: Bearer <your credential>
```

This revokes the tenant (every credential of it stops working at the next
join) and deletes its record, last-seen stamp, index entry and, for an e-mail
tenant, the e-mail binding — the address can register again as a new tenant.
What remains: the revocation flag (the tenant id only, §3), and the
aggregates it contributed to (counters, HyperLogLogs, rollup rows), which
cannot be traced back to it. Members already connected keep their connection
until it drops. Tenants issued by the operator (for example Quave's region
credentials) cannot delete themselves; that is the operator's call.

## 8. The alternative: your own hub

A hub is one binary (`volobj hub`) and a Redis if you want replicas. Run your
own and nothing in this page applies to anyone but you:

```bash
volobj hub --addr :8787 --tenant-secret "$(openssl rand -hex 32)"
volobj hub-migrate --to wss://my-hub.example.com   # move a volume, members follow within ~30 s
```

Or use no hub at all: a volume without `hubUrl` works in polling mode.

---

## Resumo em português

- O hub **recebe**: por conexão, o IP, o `User-Agent` (`volobj/<versão> (<so>/<arq>; <modo>)`), o id do tenant (via credencial) e o id da sala; por evento, o caminho do arquivo **em texto claro** (por desenho do protocolo), ETag, tamanho, id do volume e a identidade de quem escreveu. Nunca recebe conteúdo de arquivos, chave do volume ou credenciais S3.
- O hub **guarda**: um registro por tenant (id, tipo, criação, última vez visto por minuto, expiração, hash SHA-256 do e-mail só para tenants por e-mail, flag de revogação) no Postgres (ou SQLite) com o Redis como cache, e estatísticas **agregadas** (métricas Prometheus com labels limitados; contadores diários em Redis por 48 h; linhas diárias no banco, parciais a cada 30 min e finais após a meia-noite UTC, por tipo de tenant/cliente/versão/SO/país, sem linha por tenant).
- O hub **nunca guarda**: caminhos, ETags, conteúdo, e-mail em claro, IPs além da janela de rate limit (≈1 min nos endpoints de registro, até 1 h na criação anônima, só em memória).
- **Retenção**: tenants anônimos somem após 90 dias sem uso ou 7 dias após a credencial expirar; tenants por e-mail até serem apagados; rollups indefinidamente (são agregados); Prometheus conforme o servidor.
- **Identidade do membro**: o padrão agora é aleatório por processo (`m-<12 base32>`), não mais o hostname; no CSI é um hash do nó e do mount.
- **Sair**: `volobj hub delete-tenant --hub <url> --yes` (`DELETE /v1/tenants/me`) revoga e apaga o registro.
- **Hub público** `wss://hub.volobj.quave.one`: operado pela Quave no Quave One (conta `quave-one-oss`), região us-5 (Ashburn, EUA, OCI). O registro por e-mail funciona (código enviado por SMTP via Postmark, credencial `t1` permanente); o tenant anônimo continua disponível.
- **Alternativa**: rodar seu próprio hub (`volobj hub`) e mover volumes com `volobj hub-migrate`, ou usar modo polling sem hub.