tribes-supertest/README.md

# tribes-supertest

Real integration scenarios for Tribes deployments, driven through a checked-out Legion CLI.

## Overview

This repo runs real deployment scenarios against cloud infrastructure and verifies what the deployed nodes actually booted and exposed.

- `../legion_kk` is the only required external project.
- Legion is treated as the deployment authority.
- supertest records what Legion asked Guix to install and what the nodes actually booted.

The runner currently invokes Legion's headless CLI directly:

- CLI entrypoint: `../legion_kk/src/engine/cli-main.ts`
- State: isolated per run under `.state/supertest/...`
- Artifacts: command logs, sanitized Legion state, and remote node diagnostics

See `docs/arch.md` for the topology / hook / test block / scenario architecture and extension guidance.

## Requirements

- This repo checked out locally
- `../legion_kk` checked out beside it
- Node/npm available in the current shell
- Legion dependencies available in `../legion_kk`
- A usable Legion kexec installer default, normally the generated mirror pin in `../legion_kk`

Cloud/provider credentials must be present in the environment:

- `LEGION_UNLOCK_PASSWORD`
- `HCLOUD_TOKEN`
- `OVH_APP_KEY` or `OVH_APPLICATION_KEY`
- `OVH_APP_SECRET` or `OVH_APPLICATION_SECRET`
- `OVH_CONSUMER_KEY`
- `SCW_ACCESS_KEY`
- `SCW_SECRET_KEY`
- `SCW_DEFAULT_PROJECT_ID`

Optional but commonly useful:

- `SCW_DEFAULT_ZONE`
- `OVH_ENDPOINT`
- `SUPERTEST_KEEP_NODES=1`
- `SUPERTEST_KEXEC_IMAGE=/abs/path/to/guix-kexec-installer.tar.gz` or `SUPERTEST_KEXEC_IMAGE=https://mirror.example/tribes-1/guix-kexec-installer-x86_64-linux-latest.tar.gz`
- `SUPERTEST_CERT_MODE=staging` (uses Let’s Encrypt staging ACME with relaxed test TLS verification) or `SUPERTEST_CERT_MODE=self-signed` (skips ACME and keeps self-signed edge certs)
- `hcloud` and `scw` CLI tooling in the shell for manual inspection or intervention
- Chromium available in the shell for Playwright-backed web-login blocks; the devenv shell provides this via `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH`

## Install

Install this repo's dependencies:

```bash
npm install
```

If you want to check the project locally before a live run:

```bash
npm run typecheck
npm test
npm run build
```

Check the Guix substitute servers before spending cloud time:

```bash
node --import tsx scripts/check-substitutes.ts
node --import tsx scripts/check-substitutes.ts --plugin sender
```

Delete leftover cloud resources without using Legion state:

```bash
scripts/cleanup-cloud-resources
scripts/cleanup-cloud-resources --dry-run
```

## Basic Usage

Use the development CLI from the repo shell to list registered groups and scenarios:

```bash
devenv shell supertest group list
devenv shell supertest scenario list
```

Run a scenario directly:

```bash
devenv shell supertest scenario run single-node-init
```

Keep created nodes around for inspection:

```bash
devenv shell supertest scenario run cluster-lifecycle --keep-nodes
```

Groups run several scenarios sequentially and fail fast on the first failing
scenario. All scenarios in a group share one `SUPERTEST_RUN_ID`, so their
artifacts land in sibling directories under the same run.

```bash
devenv shell supertest group run tribes
devenv shell supertest group run all
```

See `devenv shell supertest help` for generated group/scenario listings. For
built-artifact usage, run `npm run build` and invoke `./dist/cli.js` with the
same subcommands.

For architecture/debugging work, developer inspection lists are also available:

```bash
devenv shell supertest topology list
devenv shell supertest block list
```

Reusable account/DNS blocks are registered for future scenarios:

- `admin-account-create` creates a hosted admin account through Legion CLI, generating a password when omitted and printing created credentials in the final run report.
- `admin-account-delete` deletes an admin account by `id` or `username` and verifies it no longer appears in Legion CLI output.
- `admin-web-login` uses Playwright/Chromium to log into `https://<node-or-hostname>`.
- `dns-validation` resolves A/AAAA records by querying authoritative nameservers directly.

## Dev Branch Helper

For rapid `guix-tribes` dev-channel iteration, use:

```bash
scripts/test-dev-branch --plugin supertest single-node-plugin-rollout-rollback
```

The helper updates and pushes the hard-coded `guix-tribes` `supertest-dev`
branch, exports the required `SUPERTEST_GUIX_TRIBES_*` environment, and runs
the scenario with `SUPERTEST_CERT_MODE=self-signed` by default. Pass
`--cert-mode staging` to exercise the ACME flow against Let’s Encrypt staging
without burning production issuance limits, or `--cert-mode acme` for the normal
production ACME path. It verifies that the pinned `tribes` and optional
`tribes-plugin-$NAME` commits are already reachable from their `origin` remotes;
it does not push those source repos.

Useful subcommands:

```bash
scripts/test-dev-branch prepare --plugin supertest
scripts/test-dev-branch reset
scripts/test-dev-branch env
scripts/test-dev-branch ssh <node>
scripts/test-dev-branch rpc <node> -- 'Node.self()'
```

Use `--tribes-repo` or `--plugin-repo` to point at a clean worktree when the
main checkout contains unrelated local work.

## Implemented Scenarios

- `single-node-init`
  Provisions one Hetzner init node and captures deployed channels, service status, and NBDE state.
- `manual-node-init`
  Imports one existing Ubuntu-compatible host from `SUPERTEST_MANUAL_HOST_IP`, `SUPERTEST_MANUAL_USERNAME`, and `SUPERTEST_MANUAL_PASSWORD`, then captures deployed channels, service status, and NBDE state.
- `single-node-plugin-rollout-rollback`
  Provisions one Hetzner init node, applies a plugin rollout through the public admin API, and rolls back to the pre-rollout generation.
- `single-node-sender`
  Provisions one Hetzner init node, installs the `sender` plugin when needed, starts RTMP ingest through Legion, publishes an audio test stream, verifies HLS playlist and segment output, and stops the stream.
- `cluster-sender-fanout-reboot`
  Builds a three-node OVH/Hetzner/Scaleway cluster, installs the `sender` plugin when needed, starts one origin plus two HLS edges through Legion, publishes a 6 Mbit/s video test stream, runs HLS viewers against every node, verifies Sender viewer-count rollups through the admin API, reboots one edge, and verifies recovery. Set `SUPERTEST_SENDER_VIEWERS_PER_NODE` to override the default of 3 viewers per node.
- `cluster-dns-sender-fanout-reboot`
  Runs the same Sender fanout/reboot flow with `SUPERTEST_DNS_DOMAIN` planning enabled, DNS node/apex assertions, and edge certificate SAN validation before Sender exercise.
- `cluster-kobold-public-private`
  Builds a two-node Hetzner/Scaleway cluster, rolls out Trust-backed Kobold (the preview must resolve both `tribe-one-kobold` and `tribe-one-trust`), creates public and private datasets, and verifies Trust-gated access: public read-only after a Trust hello, private denial before grants, explicit private read/write grants, and that private datasets stay local to their origin.
- `cluster-plugin-integrated-rollout`
  Rolls out the `supertest` plugin on a Hetzner init node, seeds plugin data, then adds Scaleway and OVH join nodes and verifies the new nodes install with the active plugin target already integrated (schema ready, seed data synced, and the plugin-enabled system target present on every node).
- `cluster-plugin-rollout-sync-split-brain`
  Builds a three-node Hetzner/Scaleway/OVH cluster, applies the `supertest` plugin rollout, validates synced table writes and cluster pubsub across a temporary sync-port partition/rejoin, and rolls back every node.
- `cluster-lifecycle`
  Builds a mixed Hetzner/Scaleway cluster, removes a node, reconciles NBDE, adds a replacement node, and reconciles again.

Rollout cross-repo implementation notes + progress tracker:

- `docs/ROLLOUT_CROSS_REPO_PLAN_PROGRESS.md`

## Artifacts

Each run writes to:

```text
.state/supertest/<run-id>-<scenario>/
```

Inside a scenario directory you will typically find:

- `config-summary.json`
- `scenario.json`
- `legion-checkout.json`
- `commands/`
- `snapshots/`

Snapshots include:

- `nodes-list.json`
- `providers-list.json`
- `legion-state.raw.json`
- `legion-state.sanitized.json`
- `remote/<node-id>/...`

Remote diagnostics currently include checks such as:

- `guix system describe`
- `herd status tribes`
- `herd status postgres`
- `curl http://127.0.0.1:4000/healthz`
- `/root/legion/tribes-admin.sh bootstrap-ready`
- node public key
- LUKS UUID
- local boot-key presence
- clevis bindings
- peer Tang reachability

## Important Notes

- supertest uses isolated Legion state per run. It does not reuse your normal Legion desktop state.
- `guix system describe` is the relevant proof for the installed system channels.
- `guix describe` is not sufficient for that check and should not be used for scenario assertions here.
- With `SUPERTEST_KEEP_NODES=1`, cleanup is skipped on purpose.

## Cleanup

If a kept run leaves nodes behind, destroy them using the same isolated Legion state directory that created them.

Example:

```bash
env \
  LEGION_STATE_DIR=.state/supertest/<run>/single-node-init/legion-state \
  LEGION_CACHE_DIR=.state/supertest/<run>/single-node-init/legion-cache \
  LEGION_APP_ROOT=../legion_kk \
  LEGION_UNLOCK_PASSWORD="$LEGION_UNLOCK_PASSWORD" \
  node --import tsx ../legion_kk/src/engine/cli-main.ts \
  nodes destroy --materialize <node-id>
```

To inspect remaining tracked nodes in that state:

```bash
env \
  LEGION_STATE_DIR=.state/supertest/<run>/single-node-init/legion-state \
  LEGION_CACHE_DIR=.state/supertest/<run>/single-node-init/legion-cache \
  LEGION_APP_ROOT=../legion_kk \
  LEGION_UNLOCK_PASSWORD="$LEGION_UNLOCK_PASSWORD" \
  node --import tsx ../legion_kk/src/engine/cli-main.ts \
  nodes list --json
```

If a run has to be aborted hard, or Legion state no longer matches provider
reality, wipe provider-side resources directly:

```bash
scripts/cleanup-cloud-resources
```

This helper does not read or delete Legion state or supertest artifacts. It
uses the provider CLIs from the current shell to remove billable runtime
resources such as instances, volumes, snapshots, IP allocations, and load
balancers. By default it preserves non-test SSH keys; use `--all-keys` if the
provider credentials' project should be fully cleared.

## Manual Intervention

The normal control path is Legion's CLI, but it is useful to have provider tooling available for inspection or emergency cleanup.

- `hcloud`
  Useful for checking server status, IPs, volumes, rescue state, and deleting instances directly if Legion state and provider reality diverge.
- `scw`
  Useful for checking Scaleway instances, IPs, volumes, bootscripts, and deleting or inspecting resources outside the test runner.

These tools are optional for normal runs, but they are practical when:

- a run is kept with `SUPERTEST_KEEP_NODES=1`
- a deployment fails halfway through and you want provider-side visibility
- Legion state needs to be compared against provider-side reality
- cleanup has to be completed manually

## Useful Environment Variables

- `SUPERTEST_RUN_ID`
  Override the generated run id.
- `SUPERTEST_ARTIFACT_ROOT`
  Override the artifact directory root.
- `SUPERTEST_TRIBE_NAME`
  Override the configured tribe name.
- `SUPERTEST_CERT_MODE`
  `acme` (default), `staging`, or `self-signed`. `acme` uses the normal
  production ACME directory. `staging` uses the Let’s Encrypt staging directory
  through Legion's generated `legion-config.jsonc`, but keeps test probes on
  relaxed TLS verification because staging certificates are not publicly
  trusted. `self-signed` skips ACME and keeps self-signed edge certs.
  Supertest injects the selected mode as `LEGION_TEST_CERT_MODE` for Legion CLI
  commands.
- `SUPERTEST_HETZNER_INSTANCE`
  Override the Hetzner instance/offer selection.
- `SUPERTEST_OVH_INSTANCE`
  Override the OVH instance/offer selection.
- `SUPERTEST_SCALEWAY_INSTANCE`
  Override the Scaleway instance/offer selection.
- `SUPERTEST_HETZNER_BOOT_MODE`
  Override Hetzner boot mode.
- `SUPERTEST_SCALEWAY_BOOT_MODE`
  Override Scaleway boot mode.
- `SUPERTEST_PLUGIN_NAME`
  Override the plugin name used by plugin rollout scenarios (defaults to `supertest`).