tribes-plugin-sender/docs/arch.md

Sender Streaming Plugin Architecture

Sender is planned as a Tribes audio/video streaming plugin. The plugin provides RTMP ingest, HLS/LL-HLS playback, and a browser video player while fitting Tribes' existing cluster architecture.

The central design rule is: sync topology and operational state, not media bytes. AshNostrSync is used for replicated metadata, endpoint source links, endpoint declarations, and bounded stats. RTMP packets, HLS playlists, HLS segments, and LL-HLS parts stay in the media byte plane.

The MVP control-plane assumption is that Legion drives stream administration and media runtime orchestration through Tribes' NIP-98 admin management API. Sender still owns the streaming contract that visitors depend on: player pages, playback metadata, stream/generation state, endpoint and route metadata, rendition metadata, local media backends on Tribes nodes, and stats snapshots. Standalone plugin UI controls can be added later by calling the same internal services, but they are not required for the first useful deployment.

Goals

• Accept RTMP input from tools such as OBS.
• Produce regular HLS first, then LL-HLS once the basic pipeline is stable.
• Provide a JS/TS web player that works from plugin-owned Tribes pages.
• Let any Tribes node answer playback/routing questions from synced state.
• Allow Legion to configure and start a minimal single-node stream first.
• Allow multi-node deployments where external mux, transcode, or HLS fanout nodes participate without running the full Tribes plugin.
• Collect fresh node-local stats plus compact Tribes metrics rollups for load-aware playback routing.

Non-Goals

• Do not replicate media bytes through AshNostrSync.
• Do not add media fields to Tribes' cluster_nodes control-plane resource.
• Do not require every external media node to become a trusted Tribes sync writer.
• Do not solve source failover magically: if OBS sends RTMP to one node and that node dies, another node can route playback but cannot reconstruct the lost live source unless OBS, an external mux, or the ingest layer provides failover.
• Do not make Legion part of the visitor playback path. Legion may plan and provision topology, but browsers should use Sender/Tribes playback APIs.
• Do not require standalone admin/creator controls in Sender's web UI for the MVP. Those controls can be layered on top of Sender services later.

Tribes Fit

Tribes separates cluster state into two broad areas:

• Control plane: node membership, trusted pubkeys, sync transport addresses, TLS material, and orchestrator-managed infrastructure. This is represented by host resources such as cluster_nodes and is intentionally narrow.
• Data plane: application state replicated with AshNostrSync as Nostr kind:5000 full-state upserts.

Sender should live in the data plane. It references Tribes nodes by pubkey through MediaEndpoint, but it should not extend or overload cluster_nodes. Media-specific node state belongs in plugin resources such as MediaEndpoint and EndpointSnapshot.

AshNostrSync uses eventual consistency and last-write-wins object replacement. That is a good match for endpoint declarations, current health snapshots, and stream sessions. It is a poor match for media chunks or distributed counters.

Control Responsibilities

Legion is the preferred control plane for the MVP. It provisions infrastructure, starts and stops external media processes, scales on-demand fanout nodes, writes desired media topology into Sender through plugin management methods, and may update endpoint source links when topology changes.

Sender is the runtime playback contract. It stores and syncs streams, generations, media endpoints, renditions, endpoint snapshots, and bounded history. It serves the visitor player and playback metadata, runs local media sessions for local origin-ingest and HLS-edge endpoints, and emits node-local telemetry that the host metrics subsystem can export and roll up. Standalone Sender viewer metrics use player heartbeat events plus HLS playlist reloads tagged with a vsid query parameter; requests without vsid fall back to one viewer per non-loopback IP. When Tribes marks a request as coming through a trusted proxy, Sender serves the media but does not feed its internal viewer metric, because the proxy layer is expected to provide the authoritative viewer count.

This split keeps orchestration decisions close to Legion, which already manages machines, DNS, firewalls, TLS, and deployment state. It also keeps viewer requests independent from Legion availability: once Sender has synced state, any capable Tribes node can answer playback requests.

Sender exposes control operations through declared plugin management_methods. These are invoked via Tribes' /api/admin/management plugin.call method and are admin-only in the initial host contract. /plugins-api/tribe-one-sender/... remains reserved for viewer/player-facing JSON APIs such as playback metadata and player telemetry. HLS media is served from /sender/hls/.... Sender includes a standalone file-serving plug for this path; Vinyl can sit in front of it as a cache or serve the same spool directory directly.

Planes

Synced Metadata Plane

Replicated via Ash resources using extensions: [AshNostrSync].

This plane contains:

• stream definitions
• live generation state
• media endpoint declarations
• rendition metadata
• current endpoint snapshots
• bounded endpoint history and host metrics rollups
• policy/config required by every node to make playback decisions

This data must be compact, JSON-serializable, replay-safe, and correct under full-state replacement.

Node-Local Runtime Plane

Supervised local processes on each node.

This plane contains:

• RTMP listener or external RTMP sidecar supervisor
• ffmpeg or Membrane pipelines
• HLS spool directories
• local cache state
• active local viewer sessions
• raw telemetry samples
• host-exported Prometheus samples and node-local VictoriaMetrics data
• process lifecycle and restart state

Most of this should not be synced directly. Nodes publish summarized snapshots into the synced metadata plane.

For external ingest or fanout nodes that do not run Sender, Legion owns process lifecycle. Sender models those nodes as MediaEndpoint records and accepts endpoint, rendition, and snapshot updates.

Media Byte Plane

HTTP/RTMP traffic and cacheable media objects.

This plane contains:

• RTMP ingest
• HLS master playlists
• HLS media playlists
• MPEG-TS or fMP4 segments
• LL-HLS parts, preload hints, and blocking reloads
• pull-through HLS between nodes
• Vinyl or other edge cache behavior

This plane should use normal media protocols, cache headers, and direct HTTP routing.

Core Resources

These names are provisional but describe the intended shape.

Stream

Stable user/admin-facing stream object.

Fields:

• id
• slug
• title
• description
• visibility
• owner_id
• latency_mode: hls or ll_hls
• recording_policy
• default_rendition_policy
• deleted_at

StreamKey

Secret-bearing ingest credential. This needs careful projection design.

Fields:

• id
• stream_id
• label
• key_hash
• status
• last_used_at
• deleted_at

The raw stream key should never be synced. Only a hash or verifier should be persisted. If the key verifier is sensitive enough to matter across trusted nodes, document that it is inside the Tribes cluster trust boundary.

StreamGeneration

One live run of a stream.

Fields:

• id
• stream_id
• status: pending, live, degraded, ending, ended, failed
• started_at
• ended_at
• active_ingest_endpoint_id
• primary_origin_endpoint_id
• generation_token
• playlist_namespace
• error
• deleted_at

The generation id should be part of every HLS path. This avoids cache collisions between separate live sessions for the same stream.

MediaEndpoint

A media-capable node or service.

Fields:

• id
• type: tribes_origin, tribes_edge, external_origin, external_edge
• node_pubkey for Tribes endpoint types, backed by cluster_nodes.pubkey
• source_endpoint_id, required for edge endpoint types and nil for origins
• display_name
• status: enabled, disabled, draining
• public_base_url
• internal_base_url
• rtmp_ingest_url
• hls_base_url
• metadata
• deleted_at

External mux/fanout nodes are first-class endpoints even when they do not run Tribes or this plugin.

Rendition

Variant/rendition metadata for a generation.

Fields:

• id
• generation_id
• name
• width
• height
• video_bitrate
• audio_bitrate
• codec
• container: mpegts, fmp4
• playlist_path
• status
• deleted_at

EndpointSnapshot

Current live operational state, one row per endpoint and optionally stream/generation.

Fields:

• id
• endpoint_id
• stream_id
• generation_id
• node_pubkey
• health: healthy, degraded, unreachable, unknown
• role
• viewer_count
• ingress_bps
• egress_bps
• segment_lag_ms
• playlist_age_ms
• last_segment_at
• last_seen_at
• capacity
• load
• error
• updated_at

Each endpoint should update only its own snapshot rows. Avoid shared counters; LWW snapshots converge predictably.

EndpointHistory

Bounded rollup stats.

Fields:

• id
• endpoint_id
• stream_id
• generation_id
• bucket: minute, hour
• bucket_started_at
• samples: JSON array or aggregate object
• deleted_at

Keep this small. For early versions, prefer a single JSON ring buffer per endpoint/generation over a high-cardinality synced time-series table.

Historic operational metrics should normally use Tribes' host metrics subsystem instead of new Sender-owned synced time-series tables. Sender can declare compact plugin rollups in Tribes.Plugin.Spec.metrics; Tribes exports matching telemetry through /metrics, scrapes it into node-local VictoriaMetrics, writes compact synced Tribes.Metrics.Rollup rows, and exposes those rollups to Legion through metrics_rollups.list and to plugin code through Tribes.Plugin.Services.Metrics.

Playback And HLS Distribution

For the current implementation, viewer playback uses the same node that served the player and playback metadata. Legion controls node-to-node HLS distribution by updating each edge endpoint's source_endpoint_id. Sender can later build a graph from endpoint parent links if more complex routing is needed.

Flow:

1. Browser loads the plugin player page on any Tribes node.
2. Player requests /plugins-api/tribe-one-sender/streams/:stream_id/playback.
3. The local node reads StreamGeneration and generation-level Rendition rows.
4. The local node returns local /sender/hls/... HLS URLs.
5. The browser plays through Video.js v10 beta using its HTML custom-elements player.

For tribes_edge endpoints, Sender can run a basic pull-through session that mirrors the configured source endpoint's live playlist and referenced segments into the local spool. This keeps playback URLs node-local while leaving node-to-node routing decisions with Legion.

Endpoint snapshots still matter for monitoring and future routing. A snapshot with last_seen_at older than the configured threshold should be treated as degraded or unreachable by orchestration and future route selection.

Legion-Driven Single-Node MVP

In the minimal deployment:

Legion -> Sender management API on node A
OBS -> RTMP ingest on node A -> ffmpeg/Membrane -> HLS spool on node A -> browser viewers

The same node is:

• RTMP ingest endpoint
• HLS origin
• player edge
• stats reporter

Even in this case, create Stream, StreamKey, StreamGeneration, MediaEndpoint, Rendition, and EndpointSnapshot records. That avoids a rewrite when moving to multi-node.

Minimal control flow:

1. Legion creates or updates the stream through Sender management methods.
2. Legion creates a one-time stream key for OBS.
3. Legion upserts the local Tribes node as a media endpoint.
4. Legion calls stream.start with mode origin_ingest.
5. Sender creates the generation/rendition state and starts a local MediaSession through the configured media backend.
6. The creator connects OBS. If OBS disconnects, Sender keeps the generation active and restarts the local media backend so OBS can reconnect to the same stream.
7. Viewers load Sender's web player and request playback metadata from /plugins-api/tribe-one-sender/....
8. Legion calls stream.stop when the event ends.

Standalone Sender web controls can later call the same service layer directly, but they are not part of the MVP dependency path.

Multi-Node Tribes Setup

In a simple multi-node deployment:

OBS -> RTMP ingest on node A -> HLS origin on node A
                                      |
                                      +-> node B pull-through HLS edge
                                      +-> node C pull-through HLS edge

Every Tribes node receives synced endpoint source links and can answer playback requests from local state. A node may:

• serve locally if it has the segment
• pull through from the origin if configured as an edge

Regular HLS origin-to-edge pull-through is implemented and covered by the Docker origin/edge E2E scenario. For LL-HLS, cache/proxy behavior must be more conservative because blocking reloads, partial segments, and preload hints are latency-sensitive.

Legion may provision node B/C as normal Tribes nodes or external fanout nodes. For external nodes, Legion starts/stops their media processes and reports their endpoint/rendition/snapshot state back to Sender through management methods.

External Mux Scenario

Future deployment:

OBS -> external mux/transcode node -> 1080p/720p/480p HLS origins
                                      |
                                      +-> Tribes node A
                                      +-> Tribes node B
                                      +-> external HLS fanout node

The external mux is represented as MediaEndpoint(type: :external_origin).

Stats can enter Tribes by:

• Legion reporting snapshots through Sender management methods,
• polling from Tribes nodes, or
• a signed/token-authenticated stats POST from the external node to a plugin API route.

The receiving Tribes node validates the payload and writes synced EndpointSnapshot/EndpointHistory records. External nodes should not be AshNostrSync writers unless they are full trusted Tribes nodes.

HLS Storage and Caching

HLS output should be namespaced by generation:

/hls/:stream_id/:generation_id/master.m3u8
/hls/:stream_id/:generation_id/:rendition/media.m3u8
/hls/:stream_id/:generation_id/:rendition/segment-00001.ts
/hls/:stream_id/:generation_id/:rendition/part-00001.1.m4s

Cache policy:

• master playlists: short TTL or no-store
• live media playlists: very short TTL or no-store
• completed segments: immutable/long TTL
• LL-HLS parts: short but cacheable only when finalized
• preload hints and blocking reload responses: pass through or cache with extreme care
• 404 for future segments/parts: do not cache

Sender uses the same HLS timing config for origin and edge paths: hls_time, hls_list_size, hls_delete_threshold, and ended_retention_ms. Origin-ingest sessions pass the live-window settings to ffmpeg, which performs live segment deletion while the stream is running. Pull-through edge sessions mirror that behavior by keeping currently referenced segments plus the configured number of recently unreferenced segments. When an origin or edge session is explicitly stopped, Sender keeps the generation spool available for ended_retention_ms and then removes the generation directory.

Vinyl can sit in front of plugin HLS routes or external HLS origins. The plugin should emit cache headers that make Vinyl useful without requiring Vinyl-specific code.

Media Backend

Start with a backend abstraction:

defmodule TribeOne.TribesPlugin.Sender.MediaBackend do
  @callback start_ingest(StreamGeneration.t(), keyword()) :: {:ok, term()} | {:error, term()}
  @callback stop_ingest(StreamGeneration.t(), keyword()) :: :ok | {:error, term()}
  @callback health(term()) :: map()
end

Initial implementation uses supervised ffmpeg through MuonTrap because it is operationally mature for RTMP ingest, transmuxing, transcoding, and HLS packaging.

Membrane remains attractive for Elixir-level stream routing and process supervision. It should be evaluated behind the same backend contract rather than hard-coded into the architecture before the MVP is proven.

Media sessions are role-aware. The supported local roles are origin_ingest, where a Tribes node accepts the creator's RTMP input and produces HLS output, and hls_edge, where a Tribes node mirrors a source endpoint's HLS playlist and segments into the local spool. Future roles can cover external-origin tracking or fanout without renaming the admin API, which should use stream lifecycle methods such as stream.start, stream.stop, and stream.status.

The generation lifecycle is admin-owned: stream.start opens the stream window and stream.stop closes it. A creator source disconnect or an ffmpeg process exit should not end the generation by itself. The local MediaSession keeps the generation active, reports restarting while the backend is down, and restarts the backend with backoff so OBS can reconnect to the same stream whenever possible. Initial backend startup failure is different: if no backend can be started at all, stream.start fails and the generation is marked failed so it does not block the next start attempt.

Management API

Sender declares plugin management methods in its runtime spec. Tribes exposes them through /api/admin/management using the host plugin.call method. Legion should use these methods to mutate Sender's streaming contract and to start or stop local media sessions on Tribes nodes.

Initial methods:

• capabilities: return Sender management API version, supported roles, and supported endpoint types.
• stream.get_default: fetch the default stream.
• stream.ensure_default: create the default stream if it does not exist.
• stream.update_default: update the default stream.
• stream_key.create: create a stream key and return the raw key once. This method must be marked sensitive_response?: true.
• media_endpoints.upsert: create or update a media endpoint.
• renditions.upsert: create or update rendition metadata for a generation.
• stream.start: start a stream generation. Supported modes are origin_ingest and hls_edge.
• stream.stop: end a stream generation and stop local media if this node owns it.
• stream.status: return current stream/generation/session status.
• endpoint_snapshots.report: accept validated current stats for an endpoint, especially from Legion-managed external nodes.

The MVP product behavior is one active stream per Sender deployment. Internally Sender still uses resource ids for streams and generations so history, cache-safe HLS paths, and future expansion remain straightforward.

Management methods should update Ash resources through the existing domain interfaces rather than ad hoc database writes. They are admin-only for the initial Tribes contract.

Player

The player should be a plugin-owned JS/TS bundle.

Sender uses Video.js v10 beta through @videojs/html and the HTML/custom-elements integration. The React package is intentionally avoided for the first implementation because the plugin asset pipeline is plain JS/CSS.

Expected behavior:

• request playback info from the plugin API
• render the Video.js v10 live video skin
• keep source selection in Sender's playback API
• handle redirects/retries on stale manifests
• expose basic QoE metrics back to the plugin, such as startup time, stalls, selected rendition, and fatal player errors

Player telemetry should first be node-local and sampled. Only aggregate summaries should be synced.

Metrics

Sender should build on Tribes' host metrics subsystem rather than writing raw time series itself. The host owns /metrics, VictoriaMetrics queries, the rollup worker, synced Tribes.Metrics.Rollup rows, and the management/API surfaces for compact rollups.

Sender-owned metric work should be limited to:

• emitting low-cardinality telemetry events for live viewers, ingress/egress bitrate, playlist age, segment lag, player startup, stalls, and fatal playback errors
• declaring plugin metrics in Tribes.Plugin.Spec.metrics so Tribes can export and roll up selected values under rollup.plugins["tribe-one-sender"]
• reading compact rollups through Tribes.Plugin.Services.Metrics when the player UI or admin UI needs historic node health
• treating Vinyl, HAProxy, node-exporter, and vmagent metrics as host/service metrics; Sender may derive plugin meaning from them, but should not own their raw storage

For direct standalone playback, Sender estimates live viewers from explicit viewer sessions: browser players send a vsid query parameter on HLS playlist requests and include the same session id in player telemetry. For plain HLS clients without a session id, Sender falls back to counting one viewer per recent non-loopback IP. Requests identified by Tribes as trusted-proxy traffic are excluded from Sender's internal viewer estimate so Vinyl or another proxy can own that metric.

Direct raw VictoriaMetrics access is host-owned and not part of Sender's public plugin contract.

Dependencies

Required or likely runtime dependencies:

• ffmpeg: first media backend for RTMP ingest, HLS output, and transcoding.
• MuonTrap: process wrapper for local ffmpeg supervision.
• A local filesystem spool directory with enough write IOPS for live segments.
• Phoenix/Plug routes from the Tribes plugin API for playback metadata and HLS serving/proxying.
• @videojs/html: Video.js v10 beta HTML/custom-elements player. This currently pulls in hls.js transitively for HLS-capable variants.

Likely Elixir dependencies:

• ash, ash_postgres, and ash_nostr_sync are provided by the Tribes host/test setup and should be used for synced resources.
• jason/JSON support is already available in the host stack for JSON payloads.
• telemetry is already in the host stack and should be used for local metrics.
• Tribes metrics services are host-provided: /metrics, VictoriaMetrics querying, synced rollups, and Tribes.Plugin.Services.Metrics.

Packaging notes:

• Sender's Nix flake packages the plugin as a packaged Tribes plugin using the host tribes-plugin-package.nix builder.
• The Nix/Docker E2E image bundles a Tribes release with Sender installed, ffmpeg on PATH, and the MuonTrap BEAM dependency compiled into the plugin package.
• The current Docker E2E scenario runs one origin node and one edge node from the same image, calls stream.start on both nodes before RTMP ingest, pushes media with host ffmpeg, and verifies HLS output from the edge node.

Optional/future dependencies:

• Membrane libraries for RTMP/HLS/transcoding if the Membrane backend proves practical.
• Bandit/Phoenix static serving is already host-side; a dedicated local file server is optional if segment throughput exceeds what plugin routes should handle.
• Vinyl as an edge cache in deployment, not as an Elixir library dependency.

Security

• Stream keys must be random, revocable, and stored only as verifiers/hashes.
• RTMP ingest URLs should be unguessable and should not expose raw database ids alone.
• Playback URLs should enforce stream visibility before returning a manifest URL.
• External stats POSTs need an explicit trust mechanism: shared token, signed payload, mTLS, or orchestrator-provisioned credential.
• HLS segment paths should be generation-scoped to avoid leaking stale live content through cache reuse.
• Admin/orchestrator operations should go through Tribes plugin management methods, Ash actions, and policies, not ad hoc DB writes.

Open Questions

• Exact stream key verifier format.
• Exact Sender management method request/response payloads for Legion.
• Whether plugin HLS routes serve files directly or delegate to a local static server in production.
• Whether regular HLS and LL-HLS share one output layout from day one.
• How much historic stats should be synced before the data becomes too noisy for AshNostrSync.
• Whether Membrane can cover the desired RTMP and LL-HLS path robustly enough to replace or complement ffmpeg.