tribes-plugin-sender/docs/progress.md

# Sender Implementation Progress

This file tracks the planned implementation from plugin template to clustered streaming plugin.

## Phase 0: Architecture and Repo Preparation

- [x] Capture planned architecture in `docs/arch.md`.
- [x] Create this implementation checklist.
- [x] Rename user-facing copy from generic "My Plugin" to Sender/Streaming.
- [x] Update `manifest.json` description, capabilities, `migrations`, and `children`.
- [x] Enable the plugin supervision tree in `mix.exs`.
- [x] Decide initial media backend: supervised `ffmpeg` first, Membrane later behind a behaviour.
- [x] Decide MVP control plane: Legion drives admin/orchestration through Tribes plugin management methods.
- [x] Decide local HLS spool root config key: `config :tribe_one_sender, TribeOne.TribesPlugin.Sender.HLS, spool_root: ...`.
- [x] Document required runtime binaries in deployment packaging: at least `ffmpeg`.

## Phase 1: Ash Domain and Synced Schema

- [x] Add `TribeOne.TribesPlugin.Sender.Streaming` Ash domain.
- [x] Register the domain through the plugin spec `ash_domains`.
- [x] Add migrations for plugin tables.
- [x] Add `Stream` resource with AshNostrSync.
- [x] Add `StreamKey` resource with only hashed/verifier key material.
- [x] Add `StreamGeneration` resource with AshNostrSync.
- [x] Add `MediaEndpoint` resource with AshNostrSync.
- [x] Model endpoint parent links for HLS distribution instead of a separate route graph.
- [x] Add `Rendition` resource with AshNostrSync.
- [x] Add `EndpointSnapshot` resource with AshNostrSync.
- [x] Add a bounded `EndpointHistory` resource or defer it behind a JSON ring buffer.
- [x] Add code interfaces for common actions instead of calling `Ash.*` directly from web modules.
- [x] Add tests for resource creation, sync actions, and soft-delete behavior.

## Phase 2: Single-Node RTMP to HLS MVP

- [x] Add `TribeOne.TribesPlugin.Sender.MediaBackend` behaviour.
- [x] Implement `TribeOne.TribesPlugin.Sender.MediaBackend.FFmpeg`.
- [x] Add role-aware `TribeOne.TribesPlugin.Sender.MediaSession` runtime owner for `origin_ingest`.
- [x] Run FFmpeg through `MuonTrap` instead of bare Erlang ports.
- [x] Supervise ingest pipeline processes.
- [x] Generate a stream key and validate RTMP ingest attempts.
- [x] Declare Sender plugin management methods for Legion:
      `capabilities`, `stream.*`, `stream_key.create`,
      `media_endpoints.upsert`, `renditions.upsert`,
      and `endpoint_snapshots.report`.
- [x] Add management handlers for default stream get/create/update.
- [x] Add management handler for one-time stream key creation, marked sensitive.
- [x] Add management handlers for endpoint, route, and rendition upserts.
- [x] Add `stream.start` management handler with initial `origin_ingest` mode.
- [x] Add `stream.stop` and `stream.status` management handlers.
- [x] Start a `StreamGeneration` when Legion starts a stream.
- [x] Write HLS output to `spool_root/streams/:stream_id/:generation_id`.
- [x] Create initial `Rendition` rows from configured output variants.
- [x] Keep a started generation active across source disconnects/backend exits until admin stop.
- [x] Mark generation `live` once the first playable manifest exists.
- [x] Mark generation `ended` on explicit admin stop.
- [x] Mark generation `failed` when startup cannot create an initial backend session.
- [x] Prune live edge spools and clean stopped generation spools according to retention config.
- [x] Add tests around backend command construction and lifecycle state changes.

## Phase 3: HLS Serving and Playback API

- [x] Add plugin API route for playback metadata.
- [x] Add plugin API route or plug for HLS playlist/segment serving.
- [ ] Enforce stream visibility before returning playback data.
- [x] Add cache headers suitable for live HLS.
- [x] Avoid caching future-segment 404 responses.
- [x] Return generation-scoped URLs only.
- [x] Add basic player LiveView page.
- [x] Replace placeholder JS with a player bootstrap.
- [x] Use TypeScript for the player bootstrap.
- [x] Add Video.js v10 beta to the assets pipeline.
- [x] Render playback with Video.js v10 HTML/custom-elements player.
- [x] Report basic player errors and startup metrics to a plugin API endpoint.
- [x] Add page/API tests for playback authorization and not-found behavior.

## Phase 4: Local Stats and Snapshots

- [x] Track local active viewers by endpoint/generation.
- [x] Track ingress/egress bitrate estimates.
- [x] Track playlist age and last segment timestamp.
- [x] Publish `EndpointSnapshot` rows from the local node only.
- [x] Treat stale local stats as degraded before publishing.
- [x] Add local telemetry events for ingest, segment generation, playback, and errors.
- [x] Add a periodic snapshot worker.
- [x] Add tests for snapshot staleness and ownership rules.
- [x] Align monitoring plan with Tribes' host metrics subsystem: `/metrics`, node-local VictoriaMetrics, synced rollups, and plugin metric declarations.
- [x] Declare initial Sender plugin `viewer_count` metric and map it into host-exported plugin metrics.
- [ ] Add compact rollups for ingress/egress bitrate, playlist age, and segment lag when those local samples are mature enough.

## Phase 5: Cluster-Aware HLS Distribution

- [x] Accept Legion-authored endpoint source links through management methods.
- [x] Fill edge-node local HLS spools from each endpoint's configured source endpoint.
- [ ] Serve HLS from `/sender/hls/` through Vinyl.
- [x] Keep viewer playback on the current node that served the player/API.
- [ ] Add configurable health thresholds.
- [ ] Add tests for endpoint source-link behavior under healthy, degraded, stale, and overloaded endpoints.
- [ ] Make every Tribes node able to answer playback metadata for any synced live stream.

## Phase 6: Pull-Through HLS Edge Mode

- [x] Add endpoint type `tribes_edge`/`external_edge`.
- [x] Implement pull-through fetch from an upstream HLS origin.
- [x] Cache finalized segments locally.
- [x] Keep live playlist caching conservative.
- [ ] Add upstream failover based on endpoint source links.
- [x] Emit cache headers compatible with Vinyl.
- [x] Verify regular HLS origin/edge behavior before enabling LL-HLS edge behavior.
- [ ] Add tests for cache hit, miss, upstream failure, and stale manifest handling.

## Phase 7: External Mux and Fanout Nodes

- [x] Model external mux nodes as `MediaEndpoint(type: :external_origin)`.
- [x] Model external HLS repeaters as `MediaEndpoint(type: :external_edge)`.
- [ ] Accept Legion-reported external endpoint snapshots through management API.
- [ ] Document Legion-owned lifecycle for external media processes.
- [ ] Add signed/token-authenticated stats ingestion API.
- [ ] Add optional polling for external HLS origin health.
- [ ] Validate external endpoint payloads before writing synced snapshots.
- [ ] Support multiple renditions produced by an external mux.
- [ ] Support endpoint source links where external mux fans out to Tribes nodes and external edge nodes.
- [ ] Add tests for external stats auth and endpoint source-link selection.

## Phase 8: LL-HLS

- [ ] Switch or add ffmpeg output mode for CMAF/fMP4 segments.
- [ ] Add LL-HLS playlist settings and target part duration.
- [ ] Handle partial segment serving.
- [ ] Review preload hint and blocking reload behavior.
- [ ] Tighten cache headers for parts and playlists.
- [ ] Add player low-latency config.
- [ ] Add metrics for live edge distance and rebuffering.
- [ ] Test Safari native LL-HLS behavior separately from Video.js' HLS engine path.

## Phase 9: Membrane Evaluation

- [ ] Identify Membrane packages needed for RTMP ingest, HLS output, and optional transcoding.
- [ ] Build a backend prototype behind `TribeOne.TribesPlugin.Sender.MediaBackend`.
- [ ] Compare operational behavior against ffmpeg: startup, failure handling, CPU, memory, latency.
- [ ] Decide whether Membrane becomes primary, optional, or deferred.
- [ ] Keep ffmpeg backend available unless Membrane fully covers production needs.

## Phase 10: Packaging and Operations

- [ ] Add runtime config for ports, spool root, retention, backend, and route thresholds.
- [x] Add Nix deployment packaging for `ffmpeg`.
- [x] Add Nix packaging for Sender as a packaged Tribes plugin.
- [x] Add single-image Docker E2E packaging for a two-node origin/edge scenario.
- [ ] Add spool directory creation and permission checks on startup.
- [ ] Add health checks for required binaries and writable spool.
- [ ] Add optional standalone Sender admin UI for stream setup, endpoint status, and current live sessions.
- [ ] Add runbook docs for OBS configuration.
- [ ] Add runbook docs for Legion-driven single-node deployment.
- [ ] Add runbook docs for multi-node HLS edge deployment.
- [ ] Add runbook docs for external mux/fanout deployment.

## External Dependencies

Required for the first useful MVP:

- [x] `ffmpeg` runtime binary in Nix/Docker packaging.
- [x] Video.js v10 beta HTML player package, `@videojs/html`.
- [ ] Writable local spool directory.
- [ ] Legion client support for Sender plugin management methods.

Already available through the Tribes host or current plugin setup:

- [x] Phoenix/Plug routing for plugin pages and API routes.
- [x] Tribes plugin management methods over `/api/admin/management`.
- [x] Ash and AshPostgres for persisted resources.
- [x] AshNostrSync for cluster-replicated plugin state.
- [x] Telemetry primitives.
- [x] Tribes metrics subsystem: `/metrics`, VictoriaMetrics client, rollup worker, synced rollups, admin API, and plugin metrics contract.

Optional or future:

- [ ] Membrane packages for BEAM-native media routing.
- [x] `MuonTrap` or similar for stricter external process supervision.
- [ ] Vinyl in deployment for HLS caching.
- [x] Declare Vinyl authoritative viewer and playlist rollups for proxied/cached playback.
- [ ] External mux/transcode service for multi-bitrate production.