From b62877051737109ff9dfef3a89abc9706e2e0234 Mon Sep 17 00:00:00 2001 From: Steffen Beyer Date: Mon, 16 Mar 2026 12:58:21 +0100 Subject: [PATCH] docs: Narrow NIP-DBSYNC protocol --- docs/NIP-DBSYNC.md | 572 ++++++++++++++++++--------------------------- docs/NIP-REVIEW.md | 346 --------------------------- 2 files changed, 226 insertions(+), 692 deletions(-) delete mode 100644 docs/NIP-REVIEW.md diff --git a/docs/NIP-DBSYNC.md b/docs/NIP-DBSYNC.md index 9ebab35..a95a6a6 100644 --- a/docs/NIP-DBSYNC.md +++ b/docs/NIP-DBSYNC.md @@ -1,452 +1,325 @@ -# NIP-DBSYNC — Database Replication over Nostr +# NIP-DBSYNC — Minimal Mutation Events over Nostr `draft` `optional` -Defines a set of custom Nostr event kinds for replicating database record state across distributed nodes via Nostr relays. +Defines a minimal event format for publishing immutable application mutation events over Nostr. + +This draft intentionally standardizes only the wire format for mutation transport. It does **not** standardize database replication strategy, conflict resolution, relay retention, or key derivation. --- ## Abstract -This NIP specifies event kinds **5000–5002** for distributing database create, update, and destroy operations as Nostr events. Each participating node maintains a local database as its read/write model and uses Nostr as the replication bus. Events carry the full mutation payload (caller input, computed attributes, metadata) and form per-record causal chains via `e` tags, enabling conflict detection and field-level merge resolution. +This NIP defines one regular event kind, **5000**, for signed mutation events. -A three-tier signing model supports node-level, user-server-level, and user-personal-level event authorship with deterministic key derivation for the custodial tier. +A mutation event identifies: + +- the object namespace being mutated, +- the object identifier within that namespace, +- the mutation operation, +- an optional parent mutation event, +- an application-defined payload. + +The purpose of this NIP is to make signed mutation logs portable across Nostr clients and relays without requiring relays to implement database-specific behavior. --- ## Motivation -Applications backed by a single database instance face a single point of failure. For multi-node deployments, sharing one database requires all nodes to have network access to it and introduces a central bottleneck. By replicating database mutations as Nostr events: +Many applications need a way to distribute signed state changes across multiple publishers, consumers, or services. -- Each node operates independently against its own local database. -- The Nostr relay mesh handles event distribution and persistence. -- Cryptographic signatures provide tamper detection and authorship verification that a database column cannot. -- Nodes can recover from downtime by replaying missed events. -- New nodes can bootstrap by replaying the full event history. +Today this can be done with private event kinds, but private schemas make cross-implementation interoperability harder than necessary. This NIP defines a small shared envelope for mutation events while leaving application-specific state semantics in the payload. -The consistency model is deliberately relaxed — closer to a social network than a central bank. Eventual consistency with per-field last-write-wins conflict resolution. +This NIP is intended for use cases such as: + +- synchronizing object changes between cooperating services, +- publishing auditable mutation logs, +- replaying application events from ordinary Nostr relays, +- bridging non-Nostr systems into a Nostr-based event stream. + +This NIP is **not** a consensus protocol. It does not provide: + +- total ordering, +- transactional guarantees, +- global conflict resolution, +- authorization rules, +- guaranteed relay retention. + +Applications that require those properties MUST define them separately. --- ## Specification -### Event Kinds +### Event Kind -| Kind | Category | Name | Relay Behaviour | -|------|----------|------|-----------------| -| 5000 | Regular | Record Create | Stored permanently, full history retained | -| 5001 | Regular | Record Update | Stored permanently, full history retained | -| 5002 | Regular | Record Destroy | Stored permanently, full history retained | +| Kind | Category | Name | +|------|----------|------| +| 5000 | Regular | Mutation | -All kinds fall in the regular range (1000–9999). Relays MUST store all events of these kinds and MUST NOT treat them as replaceable. Full history is required for replay and recovery. +Kind `5000` is a regular event. Relays that support this NIP MAY store it like any other regular event. -Standard kind 5 (NIP-09 deletion requests) MAY be used to retract erroneous sync events. Receiving implementations SHOULD honour deletion requests from the same pubkey. +This NIP does **not** require relays to: + +- retain all historical events, +- index any specific tag beyond normal NIP-01 behavior, +- deliver events in causal or chronological order, +- detect or resolve conflicts. + +Applications that depend on durable replay or custom indexing MUST choose relays whose policies satisfy those needs. ### Event Structure ```json { - "id": "<32-byte hex, sha256 of serialised event>", - "pubkey": "<32-byte hex, signer's public key>", + "id": "<32-byte lowercase hex>", + "pubkey": "<32-byte lowercase hex>", "created_at": "", "kind": 5000, "tags": [ - ["r", ""], - ["i", ""], - ["act", ""], - ["v", ""], - ["n", ""], - ["e", ""], - ["f", ""], - ["u", ""], - ["seq", ""] + ["r", ""], + ["i", ""], + ["op", ""], + ["e", ""] ], - "content": "", - "sig": "<64-byte hex, Schnorr signature>" + "content": "", + "sig": "<64-byte lowercase hex>" } ``` -### Tags - -All tags use single-letter keys where possible to ensure relay indexing (NIP-01 guarantees single-letter tag indexing). Multi-letter tags are used only where no single-letter tag is appropriate. - -| Tag | Key | Required | Indexed | Description | -|-----|-----|----------|---------|-------------| -| `r` | Resource | Yes | Yes | Qualified name identifying the resource type (table, collection, entity). Format is implementation-defined but MUST be unique within the cluster. Examples: `"accounts.users"`, `"MyApp.Accounts.User"`, `"public.orders"`. | -| `i` | Record ID | Yes | Yes | Primary key value of the affected record, as a string. UUIDs recommended. Composite keys SHOULD be serialised as a deterministic JSON array (e.g., `"[\"tenant_a\",\"123\"]"`). | -| `e` | Parent | No | Yes | Event ID of the most recent prior event for this record from the signer's perspective. Omitted on the first event for a record (kind 5000 create with no prior history). Forms the causal chain. | -| `f` | Fields | No | No | Comma-separated list of column/attribute names changed in this mutation. Example: `"name,email,status"`. Omitted for creates (all fields are new) and destroys. Used for field-level conflict resolution. | -| `v` | Version | Yes | No | Integer schema version as a string. Incremented when the event content structure changes for a given resource/operation combination. Default: `"1"`. | -| `n` | Node | Yes | Yes | Public key (hex) of the originating node. Used for echo suppression — a node discards events where the `n` tag matches its own pubkey. | -| `u` | User | No | Yes | Public key (hex) of the acting user. Present when a node key or user-server key signs on behalf of a user. See [Signing Model](#signing-model). | -| `act` | Operation | Yes | No | Name of the operation that produced this event. Examples: `"create"`, `"update_email"`, `"soft_delete"`. Receiving nodes use this to determine how to apply the mutation. | -| `seq` | Sequence | No | No | Monotonically increasing integer (as string) per record per originating node. Provides a secondary ordering signal and gap detection. | - -#### Tag Ordering Convention - -Tags SHOULD appear in the order listed above. Implementations MUST NOT depend on tag ordering. - -### Content Payload - -The `content` field contains a JSON object with three keys: - -```json -{ - "data": { }, - "computed": { }, - "metadata": { } -} -``` - -| Field | Description | -|-------|-------------| -| `data` | The original mutation input — fields and values as provided by the caller before any defaults, triggers, or computed columns were applied. Keys are column/attribute names as strings. Values are JSON-serialisable representations. | -| `computed` | Fields whose values were set by the system during mutation execution (auto-generated IDs, timestamps, sequences, computed columns, default values). These were NOT in the original caller input. On replay, these MUST be force-applied to reproduce exact state. | -| `metadata` | Freeform object for application-specific context. Examples: `"source"` (API, web, CLI), `"request_id"`, `"ip_address"`, `"tenant"`. Implementations SHOULD NOT use metadata for replay logic. | - -#### Content by Kind - -**Kind 5000 (Create):** -- `data`: all input fields provided by the caller. -- `computed`: all fields set by the system (generated ID, timestamps, defaults, computed columns). -- `f` tag: SHOULD be omitted (all fields are new). - -**Kind 5001 (Update):** -- `data`: only the fields explicitly changed by the caller. -- `computed`: only fields modified by the system as a side effect of this update (e.g., `updated_at`). -- `f` tag: MUST list all field names present in `data` AND `computed`. - -**Kind 5002 (Destroy):** -- `data`: any arguments passed to the destroy operation (e.g., soft-delete reason). -- `computed`: typically empty. May contain final state modifications for soft deletes (e.g., `deleted_at`, `status`). -- `f` tag: SHOULD be omitted. - -#### Sensitive Fields - -Fields marked as sensitive or secret in the application schema (passwords, tokens, PII as required by policy) MUST be excluded from `data` and `computed` unless the implementation provides encryption for the content payload. If sensitive fields are included, the `content` field SHOULD be encrypted (application-level; encryption scheme is out of scope for this NIP). - -#### Serialisation - -All values in `data` and `computed` MUST be JSON-serialisable. Types that have no native JSON representation require explicit serialisation. The following conventions are RECOMMENDED: - -| Type | JSON Representation | -|------|-------------------| -| Timestamps / datetimes | ISO 8601 string (e.g., `"2025-03-15T14:30:00Z"`) | -| Arbitrary-precision decimals | String (to preserve precision) | -| Enumerations / symbols | String | -| Sets | Array | -| Binary data (non-UTF-8) | Base64-encoded string with `{"__binary__": ""}` wrapper | -| Nested/embedded objects | JSON object | -| NULL | JSON `null` | - -Implementations MAY define additional type mappings. Custom mappings SHOULD be documented alongside the schema version. +The `content` field is a JSON-encoded string. Its structure is defined below. --- -### Signing Model +## Tags -Events use a three-tier signing hierarchy. The `pubkey` field always identifies the signer. +| Tag | Required | Description | +|-----|----------|-------------| +| `r` | Yes | Stable resource namespace for the mutated object type. Reverse-DNS style names are RECOMMENDED, for example `com.example.accounts.user`. | +| `i` | Yes | Opaque object identifier, unique within the `r` namespace. Consumers MUST treat this as a string. | +| `op` | Yes | Mutation operation. This NIP defines only `upsert` and `delete`. | +| `e` | No | Parent mutation event id, if the publisher wants to express ancestry. At most one `e` tag SHOULD be included in this version of the protocol. | +| `v` | No | Application payload schema version as a string. RECOMMENDED when the payload format may evolve over time. | -#### Tier 1: Node Key +### Tag Rules -Each node in the cluster holds a secp256k1 keypair. Used for: -- System-initiated operations (background jobs, migrations, automated maintenance) -- Any operation where no user context is available +Publishers: -The node pubkey is registered in the cluster's trusted key set. +- MUST include exactly one `r` tag. +- MUST include exactly one `i` tag. +- MUST include exactly one `op` tag. +- MUST set `op` to either `upsert` or `delete`. +- SHOULD include at most one `e` tag. +- MAY include one `v` tag. -#### Tier 2: User Server Key (Custodial) +Consumers: -Each user has a server-side keypair derived deterministically from a shared cluster secret and the user's stable identifier. Used for: -- User-triggered mutations executed on the server (profile updates, settings changes, etc.) -- Any operation where the user is authenticated but their personal key is not available for signing +- MUST ignore unknown tags. +- MUST NOT assume tag ordering. +- MUST treat the `e` tag as an ancestry hint, not as proof of global ordering. -**Deterministic derivation:** +### Resource Namespaces -``` -user_server_privkey = HMAC-SHA256(cluster_secret, "nostr-dbsync-user-key:" || user_id) -``` +The `r` tag identifies an application-level object type. -The HMAC output (32 bytes) is used directly as the secp256k1 private key. The corresponding public key is derived per standard secp256k1 operations (x-only, as per BIP-340). +This NIP does not define a global registry of resource namespaces. To reduce collisions, publishers SHOULD use a stable namespace they control, such as reverse-DNS notation. -All nodes sharing the same `cluster_secret` and `user_id` independently derive the same keypair. No key distribution is required. +Examples: -**Requirements:** -- `cluster_secret` MUST be at least 32 bytes of cryptographically random data. -- `cluster_secret` MUST be identical across all nodes in the cluster. -- `cluster_secret` MUST be stored securely (environment variable, secrets manager) and MUST NOT appear in event data, logs, or Nostr content. -- `user_id` MUST be a stable, unique identifier for the user (UUID recommended). It MUST NOT change over the user's lifetime. +- `com.example.accounts.user` +- `org.example.inventory.item` +- `net.example.billing.invoice` -When a user-server key signs a sync event, the `u` tag SHOULD also be set to the user's personal pubkey (if known), to enable cross-referencing. - -#### Tier 3: User Personal Key (Non-Custodial) - -The user's own Nostr keypair, held on their device. Used for: -- Signing regular Nostr content (kind 1, etc.) -- Future: signing high-trust database operations via NIP-46 (Nostr Connect) - -User personal keys do NOT sign sync events in the initial implementation. NIP-46 integration is a future extension. - -When a user personal key eventually signs a sync event directly, the `u` tag is omitted (the `pubkey` field IS the user). - -#### Trust Verification - -Receiving nodes MUST verify that the event's `pubkey` is trusted before applying it: - -1. **Node keys:** Check against the configured trusted node pubkey set. -2. **User server keys:** Derive the expected pubkey from `cluster_secret` + user ID (looked up via the `u` tag or record context) and verify it matches. -3. **User personal keys:** (Future) Verify the pubkey corresponds to a known user in the local database. - -Events from untrusted pubkeys MUST be rejected. Implementations SHOULD log rejected events for debugging. +Publishers MUST document the payload schema associated with each resource namespace they use. --- -### Causal Chain +## Content Payload -Each record's mutation history forms a singly-linked list via `e` tags: +The `content` field MUST be a JSON-encoded object. -``` -E1 (create) ←── E2 (update) ←── E3 (update) ←── E4 (destroy) - [no e tag] [e: E1.id] [e: E2.id] [e: E3.id] +```json +{ + "value": {}, + "patch": "merge" +} ``` -The `e` tag points to the most recent event the signer was aware of for this record at the time of mutation. This is NOT necessarily the globally latest event — under concurrent modification, two nodes may each produce events pointing to the same parent: +| Field | Required | Description | +|-------|----------|-------------| +| `value` | Yes | Application-defined mutation payload. For `upsert`, this is the state fragment or full post-mutation state being published. For `delete`, this MAY be an empty object or a small reason object. | +| `patch` | No | How `value` should be interpreted. This NIP defines `merge` and `replace`. If omitted, consumers MUST treat it as application-defined. | -``` - E2 (Node A) [e: E1.id] - / -E1 (create) ── - \ - E3 (Node B) [e: E1.id] -``` +### Payload Rules -This fork is a **conflict**. See [Conflict Resolution](#conflict-resolution). +For `op = upsert`: -#### Sequence Numbers +- `value` MUST be a JSON object. +- Publishers SHOULD publish either: + - a partial object intended to be merged, or + - a full post-mutation object intended to replace prior state. +- If the interpretation is important for interoperability, publishers SHOULD set `patch` to `merge` or `replace`. -The `seq` tag provides a secondary ordering signal per (record, originating node). It is a monotonically increasing integer starting at 1 for the create event. Gaps in sequence numbers from a given node indicate missed events. Implementations MAY use sequence gaps to trigger catch-up queries. +For `op = delete`: + +- `value` MAY be `{}`. +- Consumers MUST treat `delete` as an application-level tombstone signal. +- This NIP does not define whether deletion means hard delete, soft delete, archival, or hiding. Applications MUST define that separately. + +### Serialization + +All payload values MUST be JSON-serializable. + +The following representations are RECOMMENDED: + +| Type | Representation | +|------|----------------| +| Timestamp / datetime | ISO 8601 string | +| Decimal | String | +| Binary | Base64 string | +| Null | JSON `null` | + +Publishers MAY define additional type mappings, but those mappings are application-specific and MUST be documented outside this NIP. --- -### Conflict Resolution +## Ancestry and Replay -Conflict resolution is an application-level concern, not a relay concern. This section defines the RECOMMENDED algorithm for implementations. +The optional `e` tag allows a publisher to indicate which prior mutation event it considered the parent when creating a new mutation. -#### Detection +This supports applications that want ancestry hints for: -A conflict exists when two events for the same record reference the same parent `e` tag (or both omit it, which can only happen if two nodes independently create a record with the same ID — an error condition). +- local conflict detection, +- replay ordering, +- branch inspection, +- audit tooling. -#### Resolution: Per-Field Last-Write-Wins +However: -1. Parse the `f` tag of both conflicting events to determine which fields each changed. -2. **Disjoint fields:** Apply both changes. No data loss. -3. **Overlapping fields:** The event with the higher `created_at` wins. Tie-break: the event with the lexicographically lower `id` wins (consistent with NIP-01 replaceable event semantics). -4. After resolution, the resolving node emits a new update event (kind 5001) with `e` tags referencing BOTH conflicting events. This **merge event** collapses the fork into a single chain head. +- the `e` tag does **not** create a global ordering guarantee, +- relays are not required to deliver parents before children, +- consumers MUST be prepared to receive out-of-order events, +- consumers MAY buffer, defer, ignore, or immediately apply parent-missing events according to local policy. -#### Merge Event Structure +This NIP does not define a merge event format. -A merge event has multiple `e` tags — one for each parent being merged: - -```json -{ - "kind": 5001, - "tags": [ - ["e", ""], - ["e", ""], - ["f", ""], - ["r", "..."], - ["i", "..."], - ... - ] -} -``` - -Implementations receiving a merge event with multiple `e` tags SHOULD treat it as authoritative resolution and update their local state accordingly. - -#### No Conflict (Fast Path) - -If an incoming event's `e` tag matches the receiving node's latest known event for that record, there is no conflict. Apply directly. - -If an incoming event's `e` tag references an event the receiving node has NOT yet seen, the event SHOULD be buffered until the parent arrives (or a timeout triggers a catch-up query). +This NIP does not define conflict resolution. If two valid mutation events for the same `(r, i)` object are concurrent or incompatible, consumers MUST resolve them using application-specific rules. --- -### Relay Behaviour +## Authorization -Relays implementing this NIP: +This NIP does not define who is authorized to publish mutation events for a given resource or object. -- MUST store events of kinds 5000, 5001, 5002 as regular (non-replaceable) events. -- MUST support filtering by `r`, `i`, `e`, `n`, and `u` tags. -- SHOULD support the `since` filter for efficient catch-up queries. -- SHOULD NOT impose aggressive retention limits on these kinds (full history is needed for replay). -- MAY apply rate limits consistent with the expected mutation rate of the cluster. +Authorization is application-specific. + +Consumers MUST NOT assume that a valid Nostr signature alone authorizes a mutation. Consumers MUST apply their own trust policy, which MAY include: + +- explicit pubkey allowlists, +- per-resource ACLs, +- external capability documents, +- relay-level write restrictions, +- application-specific verification. + +This NIP does not define custodial keys, deterministic key derivation, shared cluster secrets, or delegation schemes. --- -### Subscription Filters +## Relay Behavior -#### Live sync (all resource events from other nodes) +A relay implementing only NIP-01 remains compatible with this NIP. + +No new relay messages are required beyond `REQ`, `EVENT`, and `CLOSE`. + +Relays: + +- MAY index the `r` and `i` tags using existing single-letter tag indexing conventions. +- MAY apply normal retention, rate-limit, and access-control policies. +- MAY reject events that are too large or otherwise violate local policy. +- MUST NOT be expected to validate application payload semantics. + +Applications that require stronger guarantees, such as durable retention or strict admission control, MUST obtain those guarantees from relay policy or from a separate protocol profile. + +--- + +## Subscription Filters + +This NIP works with ordinary NIP-01 filters. + +### All mutations for one resource ```json { - "kinds": [5000, 5001, 5002] + "kinds": [5000], + "#r": ["com.example.accounts.user"] } ``` -Post-filter by `n` tag client-side for echo suppression (discard events where `n` matches own node pubkey). - -#### Catch-up after downtime +### Mutation history for one object ```json { - "kinds": [5000, 5001, 5002], - "since": -} -``` - -#### Single resource type - -```json -{ - "kinds": [5000, 5001, 5002], - "#r": ["accounts.users"] -} -``` - -#### Single record history - -```json -{ - "kinds": [5000, 5001, 5002], + "kinds": [5000], + "#r": ["com.example.accounts.user"], "#i": ["550e8400-e29b-41d4-a716-446655440000"] } ``` ---- - -### Recovery - -#### Node restart (catch-up) - -1. Read `last_processed_at` from local cursor storage. -2. Subscribe with `since: last_processed_at`. -3. Process all backlog events (delivered before EOSE). -4. Continue with live subscription. - -#### New node (full replay) - -1. Subscribe with no `since` filter. -2. Relay delivers all stored events (oldest first by `created_at`, tie-break by `id`). -3. Apply each event to local database via the appropriate operation. -4. After EOSE, continue with live subscription. - -Full replay is the only bootstrap mechanism specified. Snapshot-based bootstrap (e.g., via database dumps on object storage) is an implementation optimisation outside the scope of this NIP. - ---- - -### Schema Evolution - -The `v` tag carries the schema version for the event's content structure. When the shape of `data` or `computed` changes for a resource/operation combination: - -1. Increment the version number in the publishing implementation. -2. Receiving implementations MUST support transforming older versions to the current shape. -3. Replay of historical events MUST apply the appropriate version transformation before processing. - -Version `"1"` is the default. Implementations MUST NOT omit the `v` tag. - ---- - -## Example Events - -### Create (Kind 5000) +### Mutations from trusted authors ```json { - "id": "a1b2c3...", - "pubkey": "node_a_pubkey_hex", - "created_at": 1710500000, + "kinds": [5000], + "authors": [ + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + ] +} +``` + +Applications SHOULD prefer narrow subscriptions over broad network-wide firehoses. + +--- + +## Examples + +### Upsert with parent + +```json +{ + "id": "1111111111111111111111111111111111111111111111111111111111111111", + "pubkey": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "created_at": 1710500300, "kind": 5000, "tags": [ - ["r", "accounts.users"], + ["r", "com.example.accounts.user"], ["i", "550e8400-e29b-41d4-a716-446655440000"], - ["act", "create"], - ["v", "1"], - ["n", "node_a_pubkey_hex"], - ["u", "user_personal_pubkey_hex"], - ["seq", "1"] + ["op", "upsert"], + ["e", "0000000000000000000000000000000000000000000000000000000000000000"], + ["v", "1"] ], - "content": "{\"data\":{\"name\":\"Jane Doe\",\"email\":\"jane@example.com\"},\"computed\":{\"id\":\"550e8400-e29b-41d4-a716-446655440000\",\"status\":\"active\",\"slug\":\"jane-doe\",\"created_at\":\"2025-03-15T14:30:00Z\",\"updated_at\":\"2025-03-15T14:30:00Z\"},\"metadata\":{\"source\":\"api\",\"request_id\":\"req-abc123\"}}", - "sig": "..." + "content": "{\"value\":{\"email\":\"jane.doe@newdomain.com\",\"updated_at\":\"2025-03-15T14:35:00Z\"},\"patch\":\"merge\"}", + "sig": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" } ``` -### Update (Kind 5001) +### Delete tombstone ```json { - "id": "d4e5f6...", - "pubkey": "user_server_key_hex", - "created_at": 1710500300, - "kind": 5001, - "tags": [ - ["r", "accounts.users"], - ["i", "550e8400-e29b-41d4-a716-446655440000"], - ["act", "update_email"], - ["v", "1"], - ["n", "node_b_pubkey_hex"], - ["e", "a1b2c3..."], - ["f", "email,updated_at"], - ["u", "user_personal_pubkey_hex"], - ["seq", "2"] - ], - "content": "{\"data\":{\"email\":\"jane.doe@newdomain.com\"},\"computed\":{\"updated_at\":\"2025-03-15T14:35:00Z\"},\"metadata\":{\"source\":\"web\"}}", - "sig": "..." -} -``` - -### Destroy (Kind 5002) - -```json -{ - "id": "g7h8i9...", - "pubkey": "node_a_pubkey_hex", + "id": "2222222222222222222222222222222222222222222222222222222222222222", + "pubkey": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "created_at": 1710500600, - "kind": 5002, + "kind": 5000, "tags": [ - ["r", "accounts.users"], + ["r", "com.example.accounts.user"], ["i", "550e8400-e29b-41d4-a716-446655440000"], - ["act", "deactivate"], - ["v", "1"], - ["n", "node_a_pubkey_hex"], - ["e", "d4e5f6..."], - ["seq", "3"] + ["op", "delete"], + ["e", "1111111111111111111111111111111111111111111111111111111111111111"], + ["v", "1"] ], - "content": "{\"data\":{\"reason\":\"user_requested\"},\"computed\":{\"status\":\"deactivated\",\"deactivated_at\":\"2025-03-15T14:40:00Z\"},\"metadata\":{\"source\":\"admin\"}}", - "sig": "..." -} -``` - -### Merge Event (Kind 5001, conflict resolution) - -```json -{ - "id": "m1n2o3...", - "pubkey": "node_b_pubkey_hex", - "created_at": 1710500350, - "kind": 5001, - "tags": [ - ["r", "accounts.users"], - ["i", "550e8400-e29b-41d4-a716-446655440000"], - ["act", "merge"], - ["v", "1"], - ["n", "node_b_pubkey_hex"], - ["e", "conflict_event_A_id"], - ["e", "conflict_event_B_id"], - ["f", "name,email,updated_at"], - ["seq", "3"] - ], - "content": "{\"data\":{\"name\":\"Alice Smith\",\"email\":\"alice@newdomain.com\"},\"computed\":{\"updated_at\":\"2025-03-15T14:35:50Z\"},\"metadata\":{\"merge_resolution\":true}}", - "sig": "..." + "content": "{\"value\":{\"reason\":\"user_requested\"}}", + "sig": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" } ``` @@ -454,21 +327,28 @@ Version `"1"` is the default. Implementations MUST NOT omit the `v` tag. ## Security Considerations -- **Cluster secret compromise:** If `cluster_secret` is leaked, an attacker can derive all user server keys. Rotate the secret and re-derive keys. Events signed with old keys remain verifiable (the pubkeys don't change retroactively, but the attacker could forge new events). Implementations SHOULD support secret rotation with a grace period. -- **Node key compromise:** Attacker can forge system events and events for users who don't have personal keys. Revoke the node's pubkey from the trusted set. Events previously signed by the compromised key remain in the log and may need manual review. -- **Replay of valid events:** An attacker who can publish to the relay could replay old valid events. Mitigated by event ID uniqueness — relays deduplicate by ID, and receiving nodes track processed event IDs. -- **Clock manipulation:** A compromised node could backdate `created_at` to win LWW conflicts. Mitigated by causal chain verification — even with a favourable timestamp, an event's `e` tag must reference a real parent. Implementations MAY reject events with `created_at` significantly in the past. +- **Unauthorized writes:** A valid signature proves authorship, not authorization. Consumers MUST enforce their own trust policy. +- **Replay:** Old valid events may be redelivered by relays or attackers. Consumers SHOULD deduplicate by event id and apply local replay policy. +- **Reordering:** Events may arrive out of order. Consumers MUST NOT treat `created_at` or `e` as a guaranteed total order. +- **Conflict flooding:** Multiple valid mutations may target the same object. Consumers SHOULD rate-limit, bound buffering, and define local conflict policy. +- **Sensitive data exposure:** Nostr events are typically widely replicable. Publishers SHOULD NOT put secrets or regulated data in mutation payloads unless they provide application-layer encryption. +- **Relay retention variance:** Some relays will prune history. Applications that depend on full replay MUST choose relays accordingly or maintain an external archive. --- -## Reserved Kind Ranges +## Extension Points -Kinds 5003–5099 are reserved for future extensions to this protocol (e.g., schema migration events, snapshot events, cluster membership events). Implementations MUST NOT use these kinds for other purposes. +Future drafts or companion NIPs may define: + +- snapshot events for faster bootstrap, +- object-head or checkpoint events, +- capability or delegation profiles for authorized writers, +- standardized conflict-resolution profiles for specific application classes. + +Such extensions SHOULD remain optional and MUST NOT change the meaning of kind `5000` mutation events defined here. --- ## References - [NIP-01](https://github.com/nostr-protocol/nips/blob/master/01.md) — Basic protocol flow description -- [NIP-09](https://github.com/nostr-protocol/nips/blob/master/09.md) — Event deletion request -- [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki) — Schnorr signatures for secp256k1 diff --git a/docs/NIP-REVIEW.md b/docs/NIP-REVIEW.md deleted file mode 100644 index 7b55a7d..0000000 --- a/docs/NIP-REVIEW.md +++ /dev/null @@ -1,346 +0,0 @@ -# Review of `docs/NIP-DBSYNC.md` - -## 1. Scope and motivation - -### Scope is too broad - -This draft is trying to standardize at least five separate things at once: - -1. a transport format for mutation logs, -2. a conflict-resolution algorithm, -3. a trust and key-management model, -4. relay storage/indexing expectations, -5. schema/versioning and replay semantics. - -That is too much surface area for one NIP, especially for a first draft. The problem statement in the abstract already bundles all of it: “**replicating database record state across distributed nodes via Nostr relays**” and “**field-level merge resolution**” plus “**A three-tier signing model**” (`docs/NIP-DBSYNC.md:11`, `docs/NIP-DBSYNC.md:13`). A NIP should usually standardize the interoperable wire contract, not the entire replication strategy. - -### Motivation is not yet convincing as a NIP motivation - -The motivation section argues from deployment convenience, not from a protocol gap: - -- “**Applications backed by a single database instance face a single point of failure**” (`docs/NIP-DBSYNC.md:19`) -- “**The Nostr relay mesh handles event distribution and persistence**” (`docs/NIP-DBSYNC.md:22`) -- “**New nodes can bootstrap by replaying the full event history**” (`docs/NIP-DBSYNC.md:25`) - -Those are operational goals, but they do not show why a new shared Nostr standard is needed. Today, an application can already publish regular events containing mutation deltas or state snapshots on private kinds. What is still missing after existing Nostr primitives are composed? The draft does not identify a cross-implementation interoperability problem; it mostly describes one cluster’s internal replication design. - -### Existing Nostr composition likely covers the transport layer - -If the real goal is “signed application mutation log over relays”, that can already be built from: - -- regular events for immutable mutations, -- optional addressable events for latest snapshots or heads, -- existing relay ACL / auth choices, -- application-defined content schemas. - -What this draft adds beyond that is mostly database-specific replay semantics and custodial key derivation. Those are the least portable parts and the weakest candidates for standardization. - -### Concrete fix - -Split the work: - -- **Base NIP:** immutable mutation events only. -- **Optional profile:** conflict resolution, snapshots, bootstrapping. -- **Out of scope:** custodial key derivation and app-specific operation execution. - -## 2. Protocol fit - -### This does not feel native to Nostr’s strengths - -Nostr is very good at exchanging signed events. It is not good at providing ordered, gap-free, globally consistent transactional replication. The draft quietly assumes those properties anyway. - -Examples: - -- “**Full history is required for replay and recovery**” (`docs/NIP-DBSYNC.md:41`) -- “**Gaps in sequence numbers from a given node indicate missed events**” (`docs/NIP-DBSYNC.md:227`) -- “**Relay delivers all stored events (oldest first by `created_at`, tie-break by `id`)**” (`docs/NIP-DBSYNC.md:339`) - -NIP-01 does not guarantee oldest-first replay, does not guarantee complete retention, and does not give you transactional gap detection. A NIP can define behavior for implementing relays, but this draft treats cluster-grade log semantics as if they are natural extensions of REQ/EVENT/EOSE. They are not. - -### It introduces coordination assumptions Nostr cannot reliably provide - -The biggest hidden assumptions are: - -- `created_at` is a trustworthy ordering signal, -- parents arrive before children or can be recovered cheaply, -- “latest known event for a record” is coherent across nodes, -- all nodes share the same operation semantics, -- a relay can act as durable replay storage indefinitely. - -Those assumptions break routinely in Nostr environments. - -### Relay behavior section is too strong and too cluster-specific - -“**Relays MUST store all events of these kinds**” and “**MUST support filtering by `r`, `i`, `e`, `n`, and `u` tags**” (`docs/NIP-DBSYNC.md:41`, `docs/NIP-DBSYNC.md:279`) turns a generic relay into a specialized replication service. That may be fine for a private deployment profile, but it is not a good default fit for the wider Nostr relay ecosystem. - -### Deletion semantics are a protocol mismatch - -“**Standard kind 5 (NIP-09 deletion requests) MAY be used to retract erroneous sync events**” (`docs/NIP-DBSYNC.md:43`) is the wrong tool. NIP-09 is advisory deletion/hiding, not a causal rollback mechanism for replicated state. Once downstream events depend on a mutation, “retracting” the earlier event does not define what replicas should do. - -### Concrete fix - -Reframe this as a **private relay profile** unless the draft is narrowed to a minimal event schema that works over ordinary Nostr assumptions. Remove NIP-09-based rollback entirely. - -## 3. Event design - -### Three kinds are unnecessary; one mutation kind is probably enough - -The draft uses kinds 5000/5001/5002 for create/update/destroy, but also has an `act` tag carrying the operation name: - -- “**`act` | Operation | Yes | No | Name of the operation that produced this event**” (`docs/NIP-DBSYNC.md:82`) - -That duplicates semantics. If operation meaning lives in `act`, separate kinds add little value. If operation meaning lives in kind, `act` should not also be required. Right now the design is neither minimal nor crisp. - -### `act` is implementation lock-in disguised as protocol - -“**Receiving nodes use this to determine how to apply the mutation**” (`docs/NIP-DBSYNC.md:82`) is a red flag. A protocol should not depend on every implementation having the same executable operation names like `"update_email"` or `"soft_delete"`. That is application code, not interoperable event semantics. - -A different stack, schema, or business rule engine cannot safely “apply” `update_email` just because the string matches. - -### `content` is over-structured for a supposed generic NIP - -The split into `data`, `computed`, and `metadata` (`docs/NIP-DBSYNC.md:91-105`) is not generic protocol design. It reflects one event-sourced database replay strategy. - -Specific concerns: - -- `computed` says receivers “**MUST be force-applied to reproduce exact state**” (`docs/NIP-DBSYNC.md:104`). That overrides local invariants and assumes homogeneous schemas. -- `metadata` is “**Freeform**” (`docs/NIP-DBSYNC.md:105`), which means two implementations can both “support the NIP” and still be unable to replay each other’s events if behavior leaks into metadata. -- “**Fields marked as sensitive or secret ... MUST be excluded ... unless** ... encrypted” (`docs/NIP-DBSYNC.md:126`) conflicts with the replay story. If important fields are omitted, replay is incomplete. If they are included, encryption stops being optional in practice. - -### Tag choices are awkward and sometimes misleading - -- `r` for resource, `i` for record ID, `u` for acting user, `n` for node are internally consistent, but they create a protocol whose meaning depends on a private application namespace. “**Format is implementation-defined**” for `r` (`docs/NIP-DBSYNC.md:75`) undermines interoperability immediately. -- `f` as “**Comma-separated list of column/attribute names**” (`docs/NIP-DBSYNC.md:78`) is ad-hoc packing. Use repeated tags or multi-value tags, not CSV inside a tag value. -- `i` for composite keys as “**a deterministic JSON array**” string (`docs/NIP-DBSYNC.md:76`) does not specify canonicalization. - -### `seq` semantics are internally inconsistent - -The spec says `seq` is “**per record per originating node**” (`docs/NIP-DBSYNC.md:83`, `docs/NIP-DBSYNC.md:227`), but the examples show a create from node A with `seq: 1` and then an update from node B with `seq: 2` (`docs/NIP-DBSYNC.md:374-376`, `docs/NIP-DBSYNC.md:396-400`). That contradicts the definition. - -### Merge design is underspecified and brittle - -The merge rule says the resolver emits “**a new update event (kind 5001) with `e` tags referencing BOTH conflicting events**” (`docs/NIP-DBSYNC.md:244`). Problems: - -- conflict is defined only for sibling events sharing a parent (`docs/NIP-DBSYNC.md:237`), not longer divergent branches; -- multi-parent merge events are introduced, but later “latest known event for that record” logic still reads like a single-head chain; -- merge semantics are application-specific, yet the draft treats them as protocol-level authority. - -### Concrete fix - -Use one regular kind for immutable mutations and make `content` either: - -- a generic application-defined delta, or -- a fully materialized post-state object. - -Do not standardize both `data` and `computed` in the base NIP. Move merge rules to an implementation profile. - -## 4. Interoperability and ecosystem impact - -### Non-implementing relays and clients degrade badly - -A relay that does not agree to full retention or arbitrary single-letter tag indexing is effectively unusable here. A client that does not implement this NIP sees opaque events, which is acceptable, but a relay that only partially implements it breaks replication correctness. - -That means this is not merely “optional application semantics”; it is a specialized relay/storage contract. - -### The draft has undeclared dependencies on local application knowledge - -Several parts require data that is not actually in the event: - -- trust verification of user-server keys requires `user_id`, but the event only defines `u` as “**acting user pubkey hex**” (`docs/NIP-DBSYNC.md:81`) -- the verifier is told to derive the expected pubkey from “**`cluster_secret` + user ID (looked up via the `u` tag or record context)**” (`docs/NIP-DBSYNC.md:197`) - -That is not interoperable. A remote implementation cannot reconstruct trust from the event alone. - -### It is not implementation-agnostic - -The resource examples already leak implementation details: - -- `"accounts.users"` -- `"MyApp.Accounts.User"` -- `"public.orders"` - -from `docs/NIP-DBSYNC.md:75`. - -The `act` tag also assumes shared executable operation names. The whole design reads like “replicate one service’s ORM mutations” rather than “interoperate across different Nostr apps”. - -### Too domain-specific for the NIP registry in current form - -As drafted, this is a database mutation replication profile for cooperating cluster nodes, not a broadly reusable Nostr primitive. A standard NIP should define something others can adopt without inheriting one stack’s replication model. - -### Concrete fix - -If standardization is desired, retarget the NIP to a more general primitive such as: - -- signed state-delta events, -- signed per-object snapshots, -- signed conflict annotations / ancestry references. - -Leave database table names, ORM operations, and replay internals out. - -## 5. Security and abuse surface - -### Authorization model is incomplete - -“**Receiving nodes MUST verify that the event's `pubkey` is trusted before applying it**” (`docs/NIP-DBSYNC.md:194`) is not enough. Trusted to do what? - -The draft lacks per-resource or per-operation authorization. If a node key is trusted at all, can it mutate every table? Can any derived user-server key mutate any record for that user? What prevents a valid signer from publishing semantically invalid state transitions? - -### Deterministic custodial key derivation is a major liability - -“**Each user has a server-side keypair derived deterministically from a shared cluster secret**” (`docs/NIP-DBSYNC.md:160`) means every node that can replicate can derive every custodial user key. That is an operational shortcut, not a good base protocol. - -More problems: - -- the spec uses HMAC output “**directly as the secp256k1 private key**” (`docs/NIP-DBSYNC.md:170`) without defining invalid-key handling; -- secret rotation is not actually specified, but future verification depends on it; -- compromise of one cluster secret compromises the entire custodial identity layer. - -### Replay and conflict abuse are understated - -“**Replay of valid events**” is said to be mitigated by “**event ID uniqueness**” (`docs/NIP-DBSYNC.md:459`). That only deduplicates exact same events. It does not address: - -- replay across relays and reconnects, -- adversarial reordering, -- conflict flooding with many validly signed branches, -- resource exhaustion from buffering children while waiting for parents. - -### `created_at`-based LWW is easy to game - -The draft resolves overlapping updates by higher `created_at` (`docs/NIP-DBSYNC.md:243`). A compromised or buggy node can simply lie about time. The draft’s mitigation claim — “**causal chain verification**” (`docs/NIP-DBSYNC.md:460`) — does not solve that. Two siblings with the same parent can still use fake timestamps to win. - -### Relay DoS risks are significant - -The relay/storage contract invites unbounded growth: - -- permanent retention, -- full-history replay, -- broad subscriptions with no author filter, -- parent-missing buffering, -- conflict merge amplification. - -The “live sync” example subscribes to all three kinds with no authors, no resources, no trusted-key filter (`docs/NIP-DBSYNC.md:290-294`). That is a gift to event flooding unless the deployment is tightly closed. - -### Concrete fix - -- Remove custodial derivation from the base NIP. -- Require explicit signer allowlists or signed capability documents if auth is in scope. -- Treat conflict resolution as local policy, not network truth. -- Require narrow subscription filters in examples. - -## 6. Long-term evolution - -### Versioning path is weak - -The `v` tag only versions “**the event's content structure**” (`docs/NIP-DBSYNC.md:349`), but many breaking changes would hit tag semantics, trust rules, conflict policy, and replay rules too. A single integer on the event is not enough governance for the full protocol surface being claimed. - -### The draft over-specifies internals and under-specifies critical edges - -It over-specifies: - -- `data` vs `computed`, -- deterministic key derivation, -- field-level merge strategy, -- full-replay expectations. - -It under-specifies: - -- invalid/rotated derived keys, -- authorization scope, -- parent lookup behavior, -- how to recover from missing history, -- how to handle partial relay retention, -- how to migrate resource identifiers. - -That is the worst combination for long-term evolution: too much locked down, too many dangerous gaps. - -### Network-topology assumptions will age badly - -The draft assumes a stable cluster of mutually trusting nodes sharing one secret and one resource namespace. That may work for one deployment, but it will age poorly as soon as: - -- multiple organizations interoperate, -- relays are semi-public or policy-diverse, -- mobile/offline publishers appear, -- snapshots become necessary because full replay is too expensive. - -### Concrete fix - -Define an extension path now: - -- base mutation event; -- optional snapshot/head events; -- optional auth profile; -- optional merge/conflict profile. - -Do not bundle them into a single mandatory behavior set. - -## 7. Micro issues - -- “**Eventual consistency with per-field last-write-wins conflict resolution.**” is a sentence fragment and should be rewritten (`docs/NIP-DBSYNC.md:27`). -- “**NIP-01 guarantees single-letter tag indexing**” is too strong; NIP-01 describes a convention, not a universal guarantee (`docs/NIP-DBSYNC.md:71`). -- “**The `content` field contains a JSON object**” is technically wrong; `content` is a string containing JSON-encoded text (`docs/NIP-DBSYNC.md:91`). -- `r` is said to be “**implementation-defined**” yet also “**MUST be unique within the cluster**”; that scopes uniqueness only to one cluster, not to interoperable consumers (`docs/NIP-DBSYNC.md:75`). -- Composite `i` values use a “**deterministic JSON array**” string but no canonicalization is defined (`docs/NIP-DBSYNC.md:76`). -- `f` as “**Comma-separated**” field names is an unnecessary custom encoding (`docs/NIP-DBSYNC.md:78`). -- “**Used for echo suppression — a node discards events where the `n` tag matches its own pubkey**” assumes `n` is trustworthy before trust evaluation order is stated (`docs/NIP-DBSYNC.md:80`). -- “**Receiving nodes use this to determine how to apply the mutation**” leaves operation semantics undefined and stack-specific (`docs/NIP-DBSYNC.md:82`). -- The binary encoding convention `{"__binary__": ""}` reserves a magic object shape without escape rules or collision handling (`docs/NIP-DBSYNC.md:138`). -- “**initial implementation**” is not NIP language; specs should not speak from one implementation’s release plan (`docs/NIP-DBSYNC.md:188`). -- “**looked up via the `u` tag or record context**” is undefined; `u` does not carry `user_id`, and “record context” is not a protocol concept (`docs/NIP-DBSYNC.md:197`). -- The conflict definition only covers same-parent siblings and misses deeper branch divergence (`docs/NIP-DBSYNC.md:237`). -- The tie-break rule borrows replaceable-event wording — “**consistent with NIP-01 replaceable event semantics**” — but these are regular events, so the analogy is weak and not obviously appropriate (`docs/NIP-DBSYNC.md:243`). -- “**authoritative resolution**” for merge events conflicts with the earlier statement that conflict resolution is application-level (`docs/NIP-DBSYNC.md:233`, `docs/NIP-DBSYNC.md:264`). -- “**Relay delivers all stored events (oldest first by `created_at`, tie-break by `id`)**” specifies behavior not grounded in NIP-01 relay semantics (`docs/NIP-DBSYNC.md:339`). -- The examples are not wire-valid Nostr examples because fields like `id`, `pubkey`, and `sig` use placeholders such as `"a1b2c3..."` and `"node_a_pubkey_hex"` instead of valid lowercase hex (`docs/NIP-DBSYNC.md:365-379`, `docs/NIP-DBSYNC.md:387-403`). -- The destroy example is semantically inconsistent: kind 5002 is “destroy” but the example `act` is `"deactivate"`, which sounds like soft delete/state transition, not deletion (`docs/NIP-DBSYNC.md:407-425`). -- The `seq` example contradicts the stated “per record per originating node” rule (`docs/NIP-DBSYNC.md:227`, `docs/NIP-DBSYNC.md:374-376`, `docs/NIP-DBSYNC.md:396-400`). -- “**Rotate the secret and re-derive keys**” is incomplete because no key epoch/version is carried in events, so verifiers cannot know which secret version to use (`docs/NIP-DBSYNC.md:457`). -- “**the pubkeys don't change retroactively**” is muddled wording; future derived pubkeys do change after rotation unless the derivation input is versioned separately (`docs/NIP-DBSYNC.md:457`). -- “**event ID uniqueness**” is not a sufficient replay defense (`docs/NIP-DBSYNC.md:459`). -- “**Kinds 5003–5099 are reserved**” overreaches; a draft should not unilaterally reserve a broad global kind range unless the community has agreed to it (`docs/NIP-DBSYNC.md:466`). - -### Borrowed conventions that need adaptation to Nostr - -- The per-field LWW merge model reads like database/CRDT literature, but Nostr does not provide the causal metadata needed to make that portable or trustworthy. -- The “operation name” approach feels closer to RPC/event-sourcing inside one service than to Nostr event interoperability. -- The deterministic cluster-wide custodial key derivation looks like an infrastructure shortcut from a single deployment architecture, not a Nostr-native identity model. - -## 8. Verdict and recommendations - -### Top 3 issues to resolve first - -1. **The scope is too broad.** The draft mixes wire format, replay engine, auth model, and conflict semantics. -2. **The trust model is not interoperable.** User-server key verification depends on out-of-band `user_id` and a shared secret. -3. **The protocol assumes relay/storage/order guarantees that ordinary Nostr infrastructure does not provide.** - -### Recommended structural alternatives - -#### Option A: Narrow to a transport primitive - -Define a single regular event kind for immutable application mutations: - -- required tags: object namespace, object id, optional parent event id; -- content: application-defined JSON string; -- no standard merge algorithm; -- no standard key derivation. - -This is the most Nostr-native option. - -#### Option B: Add snapshots, not replay mandates - -If bootstrap matters, add an optional addressable snapshot/head event profile: - -- regular mutation log for history; -- addressable snapshot per object or per resource; -- new nodes bootstrap from snapshot + tail, not full replay. - -This fits Nostr much better than requiring permanent full history everywhere. - -#### Option C: Keep this as an implementation profile, not a NIP - -If the goal is specifically “database replication for cooperating nodes in one cluster”, publish it as implementation documentation or a project-specific profile. In current form it is too opinionated and stack-shaped for a general NIP. - -### Overall readiness - -**Needs major revision**. - -The current draft is not ready for wider Nostr community review as a standards-track NIP. It first needs to be narrowed to a small interoperable core and stripped of implementation-specific replication and key-management assumptions.