dev: E2E and bench runners via justfile

2026-03-19 13:49:12 +01:00
parent 833c85f4ac
commit e7a7460191
4 changed files with 188 additions and 29 deletions
--- a/README.md
+++ b/README.md
@@ -48,10 +48,23 @@ Current `supported_nips` list:
 - Elixir `~> 1.18`
 - Erlang/OTP 28
 - PostgreSQL (18 used in the dev environment; 16+ recommended)
+- [`just`](https://github.com/casey/just) for the command runner used in this repo
 - Docker or Podman plus Docker Compose support if you want to run the published container image

 ---

+## Command runner (`just`)
+
+This repo includes a `justfile` that provides a grouped command/subcommand CLI over common mix tasks and scripts.
+
+```bash
+just
+just help bench
+just help e2e
+```
+
+---
+
 ## Run locally

 ### 1) Prepare the database
@@ -97,9 +110,9 @@ ws://localhost:4413/relay
 Primary test entrypoints:

 - `mix test` for the ExUnit suite
- `mix test.marmot_e2e` for the Marmot client end-to-end suite
- `mix test.node_sync_e2e` for the two-node relay sync end-to-end suite
- `mix test.node_sync_docker_e2e` for the release-image Docker two-node relay sync suite
+- `just e2e marmot` for the Marmot client end-to-end suite
+- `just e2e node-sync` for the two-node relay sync end-to-end suite
+- `just e2e node-sync-docker` for the release-image Docker two-node relay sync suite

 The node-sync harnesses are driven by:

@@ -108,7 +121,7 @@ The node-sync harnesses are driven by:
 - [`scripts/node_sync_e2e.exs`](./scripts/node_sync_e2e.exs)
 - [`compose.node-sync-e2e.yaml`](./compose.node-sync-e2e.yaml)

-`mix test.node_sync_e2e` runs two real Parrhesia nodes against separate PostgreSQL databases, verifies catch-up and live sync, restarts one node, and verifies persisted resume behavior. `mix test.node_sync_docker_e2e` runs the same scenario against the release Docker image.
+`just e2e node-sync` runs two real Parrhesia nodes against separate PostgreSQL databases, verifies catch-up and live sync, restarts one node, and verifies persisted resume behavior. `just e2e node-sync-docker` runs the same scenario against the release Docker image.

 GitHub CI currently runs the non-Docker node-sync e2e on the main Linux matrix job. The Docker node-sync e2e remains an explicit/manual check because it depends on release-image build/runtime fidelity and a working Docker host.

@@ -540,12 +553,12 @@ Notes:

 The benchmark compares two Parrhesia profiles, one backed by PostgreSQL and one backed by the in-memory adapter, against [`strfry`](https://github.com/hoytech/strfry) and [`nostr-rs-relay`](https://sr.ht/~gheartsfield/nostr-rs-relay/) using [`nostr-bench`](https://github.com/rnostr/nostr-bench). Benchmark runs also lift Parrhesia's relay-side limits by default so the benchmark client, not server guardrails, is the main bottleneck.

-`mix bench` is a sequential mixed-workload benchmark, not an isolated per-endpoint microbenchmark. Each relay instance runs `connect`, then `echo`, then `event`, then `req` against the same live process, so later phases measure against state and load created by earlier phases.
+`just bench compare` is a sequential mixed-workload benchmark, not an isolated per-endpoint microbenchmark. Each relay instance runs `connect`, then `echo`, then `event`, then `req` against the same live process, so later phases measure against state and load created by earlier phases.

 Run it with:

 ```bash
-mix bench
+just bench compare
 ```

 ### Cloud benchmark (Hetzner Cloud)
@@ -553,7 +566,8 @@ mix bench
 For distributed runs (one server node + multiple client nodes), use:

 ```bash
-./scripts/run_bench_cloud.sh
+just bench cloud
+# or: ./scripts/run_bench_cloud.sh
 ```

 or invoke the orchestrator directly:
@@ -572,7 +586,8 @@ Example:

 ```bash
 export HCLOUD_TOKEN=...
-./scripts/run_bench_cloud.sh --quick
+just bench cloud-quick
+# or: ./scripts/run_bench_cloud.sh --quick
 ```

 Outputs:
@@ -584,13 +599,13 @@ Useful history/render commands:

 ```bash
 # List available machines and runs in history
-./scripts/run_bench_update.sh --list
+just bench list

 # Regenerate chart + README table for a machine
-./scripts/run_bench_update.sh <machine_id>
+just bench update <machine_id>

 # Regenerate from all machines
-./scripts/run_bench_update.sh all
+just bench update all
 ```

 Current comparison results:
@@ -625,11 +640,11 @@ mix precommit
 Additional external CLI end-to-end checks with `nak`:

 ```bash
-mix test.nak_e2e
+just e2e nak
 ```

 For Marmot client end-to-end checks (TypeScript/Node suite using `marmot-ts`, included in `precommit`):

 ```bash
-mix test.marmot_e2e
+just e2e marmot
 ```
--- a/78
+++ b/78
@@ -0,0 +1,78 @@
+set shell := ["bash", "-euo", "pipefail", "-c"]
+set script-interpreter := ["bash", "-euo", "pipefail"]
+
+repo_root := justfile_directory()
+
+# Show curated command help (same as `just help`).
+default:
+  @just help
+
+# Show top-level or topic-specific help.
+help topic="":
+  @cd "{{repo_root}}" && ./scripts/just_help.sh "{{topic}}"
+
+# Raw e2e harness commands.
+[script]
+e2e subcommand="help" *args:
+  cd "{{repo_root}}"
+  subcommand="{{subcommand}}"
+
+  if [[ -z "$subcommand" || "$subcommand" == "help" ]]; then
+    just help e2e
+  elif [[ "$subcommand" == "nak" ]]; then
+    ./scripts/run_nak_e2e.sh {{args}}
+  elif [[ "$subcommand" == "marmot" ]]; then
+    ./scripts/run_marmot_e2e.sh {{args}}
+  elif [[ "$subcommand" == "node-sync" ]]; then
+    ./scripts/run_node_sync_e2e.sh {{args}}
+  elif [[ "$subcommand" == "node-sync-docker" ]]; then
+    ./scripts/run_node_sync_docker_e2e.sh {{args}}
+  elif [[ "$subcommand" == "suite" ]]; then
+    if [[ -z "{{args}}" ]]; then
+      echo "usage: just e2e suite <suite-name> <command> [args...]" >&2
+      exit 1
+    fi
+    ./scripts/run_e2e_suite.sh {{args}}
+  else
+    echo "Unknown e2e subcommand: $subcommand" >&2
+    just help e2e
+    exit 1
+  fi
+
+# Benchmark flows (local/cloud/history + direct relay targets).
+[script]
+bench subcommand="help" *args:
+  cd "{{repo_root}}"
+  subcommand="{{subcommand}}"
+
+  if [[ -z "$subcommand" || "$subcommand" == "help" ]]; then
+    just help bench
+  elif [[ "$subcommand" == "compare" ]]; then
+    ./scripts/run_bench_compare.sh {{args}}
+  elif [[ "$subcommand" == "collect" ]]; then
+    ./scripts/run_bench_collect.sh {{args}}
+  elif [[ "$subcommand" == "update" ]]; then
+    ./scripts/run_bench_update.sh {{args}}
+  elif [[ "$subcommand" == "list" ]]; then
+    ./scripts/run_bench_update.sh --list {{args}}
+  elif [[ "$subcommand" == "at" ]]; then
+    if [[ -z "{{args}}" ]]; then
+      echo "usage: just bench at <git-ref>" >&2
+      exit 1
+    fi
+    ./scripts/run_bench_at_ref.sh {{args}}
+  elif [[ "$subcommand" == "cloud" ]]; then
+    ./scripts/run_bench_cloud.sh {{args}}
+  elif [[ "$subcommand" == "cloud-quick" ]]; then
+    ./scripts/run_bench_cloud.sh --quick {{args}}
+  elif [[ "$subcommand" == "relay" ]]; then
+    ./scripts/run_nostr_bench.sh {{args}}
+  elif [[ "$subcommand" == "relay-strfry" ]]; then
+    ./scripts/run_nostr_bench_strfry.sh {{args}}
+  elif [[ "$subcommand" == "relay-nostr-rs" ]]; then
+    ./scripts/run_nostr_bench_nostr_rs_relay.sh {{args}}
+  else
+    echo "Unknown bench subcommand: $subcommand" >&2
+    just help bench
+    exit 1
+  fi
--- a/mix.exs
+++ b/mix.exs
@@ -28,11 +28,7 @@ defmodule Parrhesia.MixProject do
  def cli do
    [
      preferred_envs: [
-        precommit: :test,
-        bench: :test,
-        "bench.collect": :test,
-        "bench.update": :test,
-        "bench.at": :test
+        precommit: :test
      ]
    ]
  end
@@ -74,15 +70,6 @@ defmodule Parrhesia.MixProject do
      "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
      "ecto.reset": ["ecto.drop", "ecto.setup"],
      test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"],
-      "test.nak_e2e": ["cmd ./scripts/run_nak_e2e.sh"],
-      "test.marmot_e2e": ["cmd ./scripts/run_marmot_e2e.sh"],
-      "test.node_sync_e2e": ["cmd ./scripts/run_node_sync_e2e.sh"],
-      "test.node_sync_docker_e2e": ["cmd ./scripts/run_node_sync_docker_e2e.sh"],
-      bench: ["cmd ./scripts/run_bench_compare.sh"],
-      "bench.collect": ["cmd ./scripts/run_bench_collect.sh"],
-      "bench.update": ["cmd ./scripts/run_bench_update.sh"],
-      "bench.at": ["cmd ./scripts/run_bench_at_ref.sh"],
-      # cov: ["cmd mix coveralls.lcov"],
      lint: ["format --check-formatted", "credo"],
      precommit: [
        "format",
@@ -90,8 +77,7 @@ defmodule Parrhesia.MixProject do
        "credo --strict --all",
        "deps.unlock --unused",
        "test",
-        # "test.nak_e2e",
-        "test.marmot_e2e"
+        "cmd just e2e marmot"
      ]
    ]
  end
--- a/scripts/just_help.sh
+++ b/scripts/just_help.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+topic="${1:-}"
+
+if [[ -z "$topic" || "$topic" == "root" || "$topic" == "all" ]]; then
+  cat <<'EOF'
+Parrhesia command runner
+
+Usage:
+  just <group> <subcommand> [args...]
+  just help [group]
+
+Command groups:
+  e2e <subcommand>              End-to-end harness entrypoints
+  bench <subcommand>            Benchmark tasks (local/cloud/history/chart)
+
+Notes:
+  - Keep using mix aliases for core project workflows:
+      mix setup
+      mix test
+      mix lint
+      mix precommit
+
+Examples:
+  just help bench
+  just e2e marmot
+  just e2e node-sync
+  just bench compare
+  just bench cloud --clients 3 --runs 3
+EOF
+  exit 0
+fi
+
+if [[ "$topic" == "e2e" ]]; then
+  cat <<'EOF'
+E2E commands
+
+  just e2e nak                  CLI e2e tests via nak
+  just e2e marmot               Marmot client e2e tests
+  just e2e node-sync            Local two-node sync harness
+  just e2e node-sync-docker     Docker two-node sync harness
+
+Advanced:
+  just e2e suite <name> <cmd...>
+    -> runs scripts/run_e2e_suite.sh directly
+EOF
+  exit 0
+fi
+
+if [[ "$topic" == "bench" ]]; then
+  cat <<'EOF'
+Benchmark commands
+
+  just bench compare            Local benchmark comparison table only
+  just bench collect            Append run results to bench/history.jsonl
+  just bench update             Regenerate chart + README table
+  just bench list               List machines/runs from history
+  just bench at <git-ref>       Run collect benchmark at git ref
+  just bench cloud [args...]    Cloud benchmark wrapper
+  just bench cloud-quick        Cloud smoke profile
+
+Relay-target helpers:
+  just bench relay [all|connect|echo|event|req] [nostr-bench-args...]
+  just bench relay-strfry [...]
+  just bench relay-nostr-rs [...]
+
+Examples:
+  just bench compare
+  just bench collect
+  just bench update --machine all
+  just bench at v0.5.0
+  just bench cloud --clients 3 --runs 3
+EOF
+  exit 0
+fi
+
+echo "Unknown help topic: $topic" >&2
+echo "Run: just help" >&2
+exit 1