tribes/parrhesia
1
0
Fork 0
Code Issues Pull Requests Actions Releases 1 Activity
Files
101ded43cbd95593eb5bb90282113d6149ae667a
parrhesia/docs/ARCH.md

12 KiB
Raw Blame History

Parrhesia Nostr Relay Architecture

1) Goals

Build a robust, high-performance Nostr relay in Elixir/OTP with PostgreSQL as first adapter, while keeping a strict boundary so storage can be swapped later.

Primary targets:

  • Broad relay feature support (core + modern relay-facing NIPs)
  • Strong correctness around NIP-01 semantics
  • Clear OTP supervision and failure isolation
  • High fanout throughput and bounded resource usage
  • Storage abstraction via behavior-driven ports/adapters
  • Full test suite (unit, integration, conformance, perf, fault-injection)
  • Support for Marmot protocol interoperability (MIP-00..03 mandatory, MIP-04/05 optional)

2) NIP and Marmot support scope

Mandatory baseline

  • NIP-01 (includes behavior moved from NIP-12/NIP-16/NIP-20/NIP-33)
  • NIP-11 (relay info document)

Relay-facing features to include

  • NIP-09 (deletion requests)
  • NIP-13 (PoW gating)
  • NIP-17 + NIP-44 + NIP-59 (private DMs / gift wraps)
  • NIP-40 (expiration)
  • NIP-42 (AUTH)
  • NIP-43 (relay membership requests/metadata)
  • NIP-45 (COUNT, optional HLL)
  • NIP-50 (search)
  • NIP-62 (request to vanish)
  • NIP-66 (relay discovery events; store/serve as normal events)
  • NIP-70 (protected events)
  • NIP-77 (negentropy sync)
  • NIP-86 + NIP-98 (relay management API auth)

Marmot interoperability profile

Source of truth: ~/marmot/README.md and required MIPs.

Mandatory for compatibility:

  • MIP-00 (Credentials & KeyPackages)
  • MIP-01 (Group construction + marmot_group_data extension semantics)
  • MIP-02 (Welcome events)
  • MIP-03 (Group messages)

Optional (feature-flagged):

  • MIP-04 (encrypted media metadata flow)
  • MIP-05 (push notification flow)

Relay-facing Marmot event surface to support:

  • kind 443 KeyPackage events
  • kind 10051 KeyPackage relay list events
  • kind 445 group events
  • wrapped delivery via kind 1059 (NIP-59) for Welcome/private flows

Notes:

  • Legacy NIP-EE is superseded by Marmot MIPs and is not the target compatibility profile.
  • No dedicated “Marmot transition compatibility mode” is planned.

3) System architecture (high level)

Configured WS/HTTP Listeners (Bandit/Plug)
  -> Protocol Decoder/Encoder
  -> Command Router (EVENT/REQ/CLOSE/AUTH/COUNT/NEG-*)
  -> Policy Pipeline (listener baseline, validation, auth, ACL, PoW, NIP-70)
  -> Event Service / Query Service
       -> Storage Port (behavior)
           -> Postgres Adapter (Ecto)
       -> Subscription Index (ETS)
       -> Fanout Dispatcher
  -> Telemetry + Metrics + Tracing

4) OTP supervision design

Parrhesia.Application children (top-level):

  1. Parrhesia.Telemetry – metric definitions/reporters
  2. Parrhesia.Config – runtime config cache (ETS-backed)
  3. Parrhesia.Storage.Supervisor – adapter processes (Repo, pools)
  4. Parrhesia.Subscriptions.Supervisor – subscription index + fanout workers
  5. Parrhesia.Auth.Supervisor – AUTH challenge/session tracking
  6. Parrhesia.Policy.Supervisor – rate limiters / ACL caches
  7. Parrhesia.Web.Endpoint – supervises configured WS + HTTP listeners
  8. Parrhesia.Tasks.Supervisor – background jobs (expiry purge, maintenance)

Failure model:

  • Connection failures are isolated per socket process.
  • Listener failures are isolated per Bandit child and restarted independently.
  • Storage outages degrade with explicit OK/CLOSED error prefixes (error:) per NIP-01.
  • Non-critical workers are :transient; core infra is :permanent.

Ingress model:

  • Ingress is defined through config :parrhesia, :listeners, ....
  • Each listener has its own bind/transport settings, TLS mode, proxy trust, network allowlist, enabled features (nostr, admin, metrics), auth requirements, and baseline read/write ACL.
  • Listeners can therefore expose different security postures, for example a public relay listener and a VPN-only sync-capable listener.
  • TLS-capable listeners support direct server TLS, mutual TLS with optional client pin checks, and proxy-terminated TLS identity on explicitly trusted proxy hops.
  • Certificate reload is currently implemented as admin-triggered listener restart from disk rather than background file watching.

5) Core runtime components

5.1 Connection process

Per websocket connection:

  • Parse frames, enforce max frame/message limits
  • Maintain authenticated pubkeys (NIP-42)
  • Track active subscriptions (sub_id scoped to connection)
  • Handle backpressure (bounded outbound queue + drop/close strategy)

5.2 Command router

Dispatches:

  • EVENT -> ingest pipeline
  • REQ -> initial DB query + live subscription
  • CLOSE -> unsubscribe
  • AUTH -> challenge validation, session update
  • COUNT -> aggregate path
  • NEG-OPEN/NEG-MSG/NEG-CLOSE -> negentropy session engine

5.3 Event ingest pipeline

Ordered stages:

  1. Decode + schema checks
  2. id recomputation and signature verification
  3. NIP semantic checks (timestamps, tag forms, size limits)
  4. Policy checks (banlists, kind allowlists, auth-required, NIP-70, PoW)
  5. Storage write (including ephemeral events with short TTL retention)
  6. Live fanout to matching subscriptions
  7. Return canonical OK response with machine prefix when needed, only after durable DB commit succeeds

5.4 Subscription index + fanout

  • ETS-backed inverted indices (kind, author, single-letter tags)
  • Candidate narrowing before full filter evaluation
  • OR semantics across filters, AND within filter
  • limit only for initial query phase; ignored in live phase (NIP-01)

5.5 Query service

  • Compiles NIP filters into adapter-neutral query AST
  • Pushes AST to storage adapter
  • Deterministic ordering (created_at desc, id lexical tie-break)
  • Emits EOSE exactly once per subscription initial catch-up

6) Storage boundary (swap-friendly by design)

6.1 Port/adapter contract

Define behaviors under Parrhesia.Storage:

  • Parrhesia.Storage.Events
    • put_event/2, get_event/2, query/3, count/3
    • delete_by_request/2, vanish/2, purge_expired/1
  • Parrhesia.Storage.Moderation
    • pubkey/event bans, allowlists, blocked IPs
  • Parrhesia.Storage.Groups
    • NIP-29/NIP-43 membership + role operations
  • Parrhesia.Storage.Admin
    • backing for NIP-86 methods

All domain logic depends only on these behaviors.

6.2 Postgres adapter notes

Initial adapter: Parrhesia.Storage.Adapters.Postgres with Ecto.

Schema outline:

  • events (partitioned by created_at; id, pubkey, sig stored in compact binary form; kind, content, d_tag, deleted_at, expires_at)
  • event_tags (event_id, name, value, idx)
  • moderation tables (banned/allowed pubkeys, banned events, blocked IPs)
  • relay/group membership tables
  • optional count/HLL helper tables

Indexing strategy:

  • (kind, created_at DESC)
  • (pubkey, created_at DESC)
  • (created_at DESC)
  • (name, value, created_at DESC) on event_tags
  • partial/unique indexes and deterministic upsert paths for replaceable (pubkey, kind) and addressable (pubkey, kind, d_tag) semantics
  • targeted partial indexes for high-traffic single-letter tags (e, p, d, h, i first), with additional tag indexes added from production query telemetry

Retention strategy:

  • Mandatory time partitioning for events (monthly default, configurable)
  • Partition-aligned pruning for expired/deleted data where possible
  • Periodic purge job for expired/deleted tombstoned rows

6.3 Postgres operating defaults (locked before implementation)

  • Durability invariant: relay returns OK only after transaction commit for accepted events.
  • Pool separation: independent DB pools/queues for ingest writes, REQ/COUNT reads, and maintenance/admin operations.
  • Server-side guardrails: enforce max_filter_limit, max filters per REQ, max entries for ids/authors/#tag, and bounded since/until windows.
  • Deterministic conflict resolution: tie-break replaceable/addressable collisions by created_at, then lexical id (NIP-01-consistent).
  • Conformance lock-in: treat since <= created_at <= until, newest-first initial query ordering, and single EOSE emission as fixed behavior.

7) Feature-specific implementation notes

7.1 NIP-11

  • Serve on WS URL with Accept: application/nostr+json
  • Include accurate supported_nips and limitation

7.2 NIP-42 + NIP-70

  • Connection-scoped challenge store
  • Protected (["-"]) events rejected by default unless auth+pubkey match

7.3 NIP-17/59 privacy guardrails

  • Relay can enforce recipient-only reads for kind 1059 (AUTH required)
  • Query path validates requester access for wrapped DM fetches

7.4 NIP-45 COUNT

  • Exact count baseline
  • Optional approximate mode and HLL payloads for common queries

7.5 NIP-50 search

  • Use Postgres FTS (tsvector) with ranking
  • Apply limit after ranking

7.6 NIP-77 negentropy

  • Track per-negentropy-session state in dedicated GenServer
  • Use bounded resources + inactivity timeout

7.7 NIP-62 vanish

  • Hard-delete all events by pubkey up to created_at
  • Also delete matching gift wraps where feasible (#p target)
  • Persist minimal audit record if needed for operations/legal trace

7.8 Marmot (MIP-00..03 required)

  • MIP-00 / kind 443 + 10051

    • Accept/store KeyPackage events and relay-list events.
    • Validate required Marmot tags/shape relevant to relay interoperability (encoding=base64, protocol/ciphersuite metadata, relay tags).
    • Support efficient #i tag querying for KeyPackageRef discovery.
    • Preserve replaceable semantics for kind 10051.

  • MIP-01 / group metadata anchoring

    • Relay remains cryptographically MLS-agnostic; it stores and routes events by Nostr fields/tags.
    • Enforce ingress/query constraints that Marmot relies on (h-tag routing, deterministic ordering, bounded filters).

  • MIP-02 / Welcome flow

    • Support NIP-59 wrapped delivery (1059) and recipient-gated reads.
    • Keep strict ACK-after-commit durability semantics so clients can sequence Commit before Welcome as required by spec.

  • MIP-03 / kind 445 group events

    • Accept/store high-volume encrypted group events with #h-centric routing/indexing.
    • Keep relay out of MLS decryption path; relay validates envelope shape only.
    • Apply configurable retention policy for group traffic where operators need bounded storage.

  • Optional MIP-04 / MIP-05

    • Treat media/push metadata events as ordinary Nostr payloads unless explicitly policy-gated.
    • Keep optional behind feature flags.

8) Performance model

  • Bounded mailbox and queue limits on connections
  • ETS-heavy hot path (subscription match, auth/session cache)
  • DB writes batched where safe; reads via prepared plans
  • Avoid global locks; prefer partitioned workers and sharded ETS tables
  • Telemetry-first tuning: p50/p95/p99 for ingest, query, fanout
  • Expose Prometheus-compatible /metrics endpoint for scraping

Targets (initial):

  • p95 EVENT ack < 50ms under nominal load
  • p95 REQ initial response start < 120ms on indexed queries
  • predictable degradation under overload via rate-limit + backpressure

9) Testing strategy (full suite)

  1. Unit tests: parser, filter evaluator, policy predicates, NIP validators
  2. Property tests: filter semantics, replaceable/addressable conflict resolution
  3. Adapter contract tests: shared behavior tests run against Postgres adapter
  4. Integration tests: websocket protocol flows (EVENT/REQ/CLOSE/AUTH/COUNT/NEG-*)
  5. NIP conformance tests: machine-prefix responses, ordering, EOSE behavior
  6. Marmot conformance tests: MIP-00..03 event acceptance, routing, ordering, and policy handling
  7. Performance tests: soak + burst + large fanout profiles
  8. Query-plan regression tests: representative EXPLAIN (ANALYZE, BUFFERS) checks for core REQ/COUNT shapes
  9. Fault-injection tests: DB outage, slow query, connection churn, node restart

10) Implementation principles

  • Keep relay event-kind agnostic by default; special-case only where NIPs require
  • Prefer explicit feature flags for expensive/experimental modules
  • No direct Ecto usage outside Postgres adapter and migration layer
  • Every feature lands with tests + telemetry hooks

Implementation task breakdown is tracked in ./PROGRESS.md and Marmot-specific work in ./PROGRESS_MARMOT.md.

Reference in New Issue View Git Blame Copy Permalink