235 lines
8.4 KiB
Markdown
235 lines
8.4 KiB
Markdown
# Parrhesia clustering and distributed fanout
|
|
|
|
This document describes:
|
|
|
|
1. the **current** distributed fanout behavior implemented today, and
|
|
2. a practical evolution path to a more production-grade clustered relay.
|
|
|
|
---
|
|
|
|
## 1) Current state (implemented today)
|
|
|
|
### 1.1 What exists right now
|
|
|
|
Parrhesia currently includes a lightweight multi-node live fanout path (untested!):
|
|
|
|
- `Parrhesia.Fanout.MultiNode` (`lib/parrhesia/fanout/multi_node.ex`)
|
|
- GenServer that joins a `:pg` process group.
|
|
- Receives locally-published events and forwards them to other group members.
|
|
- Receives remote events and performs local fanout lookup.
|
|
- `Parrhesia.Web.Connection` (`lib/parrhesia/web/connection.ex`)
|
|
- On successful ingest, after ACK scheduling, it does:
|
|
1. local fanout (`fanout_event/1`), then
|
|
2. cross-node publish (`maybe_publish_multi_node/1`).
|
|
- `Parrhesia.Subscriptions.Supervisor` (`lib/parrhesia/subscriptions/supervisor.ex`)
|
|
- Starts `Parrhesia.Fanout.MultiNode` unconditionally.
|
|
|
|
In other words: **if BEAM nodes are connected, live events are fanned out cross-node**.
|
|
|
|
### 1.2 What is not included yet
|
|
|
|
- No automatic cluster formation/discovery (no `libcluster`, DNS polling, gossip, etc.).
|
|
- No durable inter-node event transport.
|
|
- No replay/recovery of missed cross-node live events.
|
|
- No explicit per-node delivery ACK between relay nodes.
|
|
|
|
---
|
|
|
|
## 2) Current runtime behavior in detail
|
|
|
|
### 2.1 Local ingest flow and publish ordering
|
|
|
|
For an accepted event in `Parrhesia.Web.Connection`:
|
|
|
|
1. validate/policy/persist path runs.
|
|
2. Client receives `OK` reply.
|
|
3. A post-ACK message triggers:
|
|
- local fanout (`Index.candidate_subscription_keys/1` + send `{:fanout_event, ...}`),
|
|
- multi-node publish (`MultiNode.publish/1`).
|
|
|
|
Important semantics:
|
|
|
|
- Regular persisted events: ACK implies DB persistence succeeded.
|
|
- Ephemeral events: ACK implies accepted by policy, but no DB durability.
|
|
- Cross-node fanout happens **after** ACK path is scheduled.
|
|
|
|
### 2.2 Multi-node transport mechanics
|
|
|
|
`Parrhesia.Fanout.MultiNode` uses `:pg` membership:
|
|
|
|
- On init:
|
|
- ensures `:pg` is started,
|
|
- joins group `Parrhesia.Fanout.MultiNode`.
|
|
- On publish:
|
|
- gets all group members,
|
|
- excludes itself,
|
|
- sends `{:remote_fanout_event, event}` to each member pid.
|
|
- On remote receive:
|
|
- runs local subscription candidate narrowing via `Parrhesia.Subscriptions.Index`,
|
|
- forwards matching candidates to local connection owners as `{:fanout_event, sub_id, event}`.
|
|
|
|
No republish on remote receive, so this path does not create fanout loops.
|
|
|
|
### 2.3 Subscription index locality
|
|
|
|
The subscription index is local ETS state per node (`Parrhesia.Subscriptions.Index`).
|
|
|
|
- Each node only tracks subscriptions of its local websocket processes.
|
|
- Each node independently decides which local subscribers match a remote event.
|
|
- There is no global cross-node subscription registry.
|
|
|
|
### 2.4 Delivery model and guarantees (current)
|
|
|
|
Current model is **best-effort live propagation** among connected nodes.
|
|
|
|
- If nodes are connected and healthy, remote live subscribers should receive events quickly.
|
|
- If there is a netsplit or temporary disconnection:
|
|
- remote live subscribers may miss events,
|
|
- persisted events can still be recovered by normal `REQ`/history query,
|
|
- ephemeral events are not recoverable.
|
|
|
|
### 2.5 Cluster preconditions
|
|
|
|
For cross-node fanout to work, operators must provide distributed BEAM connectivity:
|
|
|
|
- consistent Erlang cookie,
|
|
- named nodes (`--name`/`--sname`),
|
|
- network reachability for Erlang distribution ports,
|
|
- explicit node connections (or external discovery tooling).
|
|
|
|
Parrhesia currently does not automate these steps.
|
|
|
|
---
|
|
|
|
## 3) Operational characteristics of current design
|
|
|
|
### 3.1 Performance shape
|
|
|
|
For each accepted event on one node:
|
|
|
|
- one local fanout lookup + local sends,
|
|
- one cluster publish that sends to `N - 1` remote bus members,
|
|
- on each remote node: one local fanout lookup + local sends.
|
|
|
|
So inter-node traffic scales roughly linearly with node count per event (full-cluster broadcast).
|
|
|
|
This is simple and low-latency for small-to-medium clusters, but can become expensive as node count grows.
|
|
|
|
### 3.2 Failure behavior
|
|
|
|
- Remote node down: send attempts to that member stop once membership updates; no replay.
|
|
- Netsplit: live propagation gap during split.
|
|
- Recovery: local clients can catch up via DB-backed queries (except ephemeral kinds).
|
|
|
|
### 3.3 Consistency expectations
|
|
|
|
- No global total-ordering guarantee for live delivery across nodes.
|
|
- Per-connection ordering is preserved by each connection process queue/drain behavior.
|
|
- Duplicate suppression for ingestion uses storage semantics (`duplicate_event`), but transport itself is not exactly-once.
|
|
|
|
### 3.4 Observability today
|
|
|
|
Relevant metrics exist for fanout/queue pressure (see `Parrhesia.Telemetry`), e.g.:
|
|
|
|
- `parrhesia.fanout.duration.ms`
|
|
- `parrhesia.connection.outbound_queue.depth`
|
|
- `parrhesia.connection.outbound_queue.pressure`
|
|
- `parrhesia.connection.outbound_queue.overflow.count`
|
|
|
|
These are useful but do not yet fully separate local-vs-remote fanout pipeline stages.
|
|
|
|
---
|
|
|
|
## 4) Practical extension path to a fully-fledged clustered system
|
|
|
|
A realistic path is incremental. Suggested phases:
|
|
|
|
### Phase A — hardened BEAM cluster control plane
|
|
|
|
1. Add cluster discovery/formation (e.g. `libcluster`) with environment-specific topology:
|
|
- Kubernetes DNS,
|
|
- static nodes,
|
|
- cloud VM discovery.
|
|
2. Add clear node liveness/partition telemetry and alerts.
|
|
3. Provide operator docs for cookie, node naming, and network requirements.
|
|
|
|
Outcome: simpler and safer cluster operations, same data plane semantics.
|
|
|
|
### Phase B — resilient distributed fanout data plane
|
|
|
|
Introduce a durable fanout stream for persisted events.
|
|
|
|
Recommended pattern:
|
|
|
|
1. On successful DB commit of event, append to a monotonic fanout log (or use DB sequence-based stream view).
|
|
2. Each relay node runs a consumer with a stored cursor.
|
|
3. On restart/partition recovery, node resumes from cursor and replays missed events.
|
|
4. Local fanout remains same (subscription index + per-connection queues).
|
|
|
|
Semantics target:
|
|
|
|
- **at-least-once** node-to-node propagation,
|
|
- replay after downtime,
|
|
- idempotent handling keyed by event id.
|
|
|
|
Notes:
|
|
|
|
- Ephemeral events can remain best-effort (or have a separate short-lived transport), since no storage source exists for replay.
|
|
|
|
### Phase C — scale and efficiency improvements
|
|
|
|
As cluster size grows, avoid naive full broadcast where possible:
|
|
|
|
1. Optional node-level subscription summaries (coarse bloom/bitset or keyed summaries) to reduce unnecessary remote sends.
|
|
2. Shard fanout workers for CPU locality and mailbox control.
|
|
3. Batch remote delivery payloads.
|
|
4. Separate traffic classes (e.g. Marmot-heavy streams vs generic) with independent queues.
|
|
|
|
Outcome: higher throughput per node and lower inter-node amplification.
|
|
|
|
### Phase D — stronger observability and SLOs
|
|
|
|
Add explicit distributed pipeline metrics:
|
|
|
|
- publish enqueue/dequeue latency,
|
|
- cross-node delivery lag (commit -> remote fanout enqueue),
|
|
- replay backlog depth,
|
|
- per-node dropped/expired transport messages,
|
|
- partition detection counters.
|
|
|
|
Define cluster SLO examples:
|
|
|
|
- p95 commit->remote-live enqueue under nominal load,
|
|
- max replay catch-up time after node restart,
|
|
- bounded message loss for best-effort channels.
|
|
|
|
---
|
|
|
|
## 5) How a fully-fledged system would behave in practice
|
|
|
|
With Phases A-D implemented, expected behavior:
|
|
|
|
- **Normal operation:**
|
|
- low-latency local fanout,
|
|
- remote nodes receive events via stream consumers quickly,
|
|
- consistent operational visibility of end-to-end lag.
|
|
- **Node restart:**
|
|
- node reconnects and replays from stored cursor,
|
|
- local subscribers begin receiving new + missed persisted events.
|
|
- **Transient partition:**
|
|
- live best-effort path may degrade,
|
|
- persisted events converge after partition heals via replay.
|
|
- **High fanout bursts:**
|
|
- batching + sharding keeps queue pressure bounded,
|
|
- overflow policies remain connection-local and measurable.
|
|
|
|
This approach gives a good trade-off between Nostr relay latency and distributed robustness without requiring strict exactly-once semantics.
|
|
|
|
---
|
|
|
|
## 6) Current status summary
|
|
|
|
Today, Parrhesia already supports **lightweight distributed live fanout** when BEAM nodes are connected.
|
|
|
|
It is intentionally simple and fast for smaller clusters, and provides a solid base for a more durable, observable cluster architecture as relay scale and availability requirements grow.
|