diff --git a/README.md b/README.md index 780aa5a..75cfca0 100644 --- a/README.md +++ b/README.md @@ -48,10 +48,23 @@ Current `supported_nips` list: - Elixir `~> 1.18` - Erlang/OTP 28 - PostgreSQL (18 used in the dev environment; 16+ recommended) +- [`just`](https://github.com/casey/just) for the command runner used in this repo - Docker or Podman plus Docker Compose support if you want to run the published container image --- +## Command runner (`just`) + +This repo includes a `justfile` that provides a grouped command/subcommand CLI over common mix tasks and scripts. + +```bash +just +just help bench +just help e2e +``` + +--- + ## Run locally ### 1) Prepare the database @@ -97,9 +110,9 @@ ws://localhost:4413/relay Primary test entrypoints: - `mix test` for the ExUnit suite -- `mix test.marmot_e2e` for the Marmot client end-to-end suite -- `mix test.node_sync_e2e` for the two-node relay sync end-to-end suite -- `mix test.node_sync_docker_e2e` for the release-image Docker two-node relay sync suite +- `just e2e marmot` for the Marmot client end-to-end suite +- `just e2e node-sync` for the two-node relay sync end-to-end suite +- `just e2e node-sync-docker` for the release-image Docker two-node relay sync suite The node-sync harnesses are driven by: @@ -108,7 +121,7 @@ The node-sync harnesses are driven by: - [`scripts/node_sync_e2e.exs`](./scripts/node_sync_e2e.exs) - [`compose.node-sync-e2e.yaml`](./compose.node-sync-e2e.yaml) -`mix test.node_sync_e2e` runs two real Parrhesia nodes against separate PostgreSQL databases, verifies catch-up and live sync, restarts one node, and verifies persisted resume behavior. `mix test.node_sync_docker_e2e` runs the same scenario against the release Docker image. +`just e2e node-sync` runs two real Parrhesia nodes against separate PostgreSQL databases, verifies catch-up and live sync, restarts one node, and verifies persisted resume behavior. `just e2e node-sync-docker` runs the same scenario against the release Docker image. GitHub CI currently runs the non-Docker node-sync e2e on the main Linux matrix job. The Docker node-sync e2e remains an explicit/manual check because it depends on release-image build/runtime fidelity and a working Docker host. @@ -540,12 +553,12 @@ Notes: The benchmark compares two Parrhesia profiles, one backed by PostgreSQL and one backed by the in-memory adapter, against [`strfry`](https://github.com/hoytech/strfry) and [`nostr-rs-relay`](https://sr.ht/~gheartsfield/nostr-rs-relay/) using [`nostr-bench`](https://github.com/rnostr/nostr-bench). Benchmark runs also lift Parrhesia's relay-side limits by default so the benchmark client, not server guardrails, is the main bottleneck. -`mix bench` is a sequential mixed-workload benchmark, not an isolated per-endpoint microbenchmark. Each relay instance runs `connect`, then `echo`, then `event`, then `req` against the same live process, so later phases measure against state and load created by earlier phases. +`just bench compare` is a sequential mixed-workload benchmark, not an isolated per-endpoint microbenchmark. Each relay instance runs `connect`, then `echo`, then `event`, then `req` against the same live process, so later phases measure against state and load created by earlier phases. Run it with: ```bash -mix bench +just bench compare ``` ### Cloud benchmark (Hetzner Cloud) @@ -553,7 +566,8 @@ mix bench For distributed runs (one server node + multiple client nodes), use: ```bash -./scripts/run_bench_cloud.sh +just bench cloud +# or: ./scripts/run_bench_cloud.sh ``` or invoke the orchestrator directly: @@ -572,7 +586,8 @@ Example: ```bash export HCLOUD_TOKEN=... -./scripts/run_bench_cloud.sh --quick +just bench cloud-quick +# or: ./scripts/run_bench_cloud.sh --quick ``` Outputs: @@ -584,13 +599,13 @@ Useful history/render commands: ```bash # List available machines and runs in history -./scripts/run_bench_update.sh --list +just bench list # Regenerate chart + README table for a machine -./scripts/run_bench_update.sh +just bench update # Regenerate from all machines -./scripts/run_bench_update.sh all +just bench update all ``` Current comparison results: @@ -625,11 +640,11 @@ mix precommit Additional external CLI end-to-end checks with `nak`: ```bash -mix test.nak_e2e +just e2e nak ``` For Marmot client end-to-end checks (TypeScript/Node suite using `marmot-ts`, included in `precommit`): ```bash -mix test.marmot_e2e +just e2e marmot ``` diff --git a/justfile b/justfile new file mode 100644 index 0000000..7f06cd9 --- /dev/null +++ b/justfile @@ -0,0 +1,78 @@ +set shell := ["bash", "-euo", "pipefail", "-c"] +set script-interpreter := ["bash", "-euo", "pipefail"] + +repo_root := justfile_directory() + +# Show curated command help (same as `just help`). +default: + @just help + +# Show top-level or topic-specific help. +help topic="": + @cd "{{repo_root}}" && ./scripts/just_help.sh "{{topic}}" + +# Raw e2e harness commands. +[script] +e2e subcommand="help" *args: + cd "{{repo_root}}" + subcommand="{{subcommand}}" + + if [[ -z "$subcommand" || "$subcommand" == "help" ]]; then + just help e2e + elif [[ "$subcommand" == "nak" ]]; then + ./scripts/run_nak_e2e.sh {{args}} + elif [[ "$subcommand" == "marmot" ]]; then + ./scripts/run_marmot_e2e.sh {{args}} + elif [[ "$subcommand" == "node-sync" ]]; then + ./scripts/run_node_sync_e2e.sh {{args}} + elif [[ "$subcommand" == "node-sync-docker" ]]; then + ./scripts/run_node_sync_docker_e2e.sh {{args}} + elif [[ "$subcommand" == "suite" ]]; then + if [[ -z "{{args}}" ]]; then + echo "usage: just e2e suite [args...]" >&2 + exit 1 + fi + ./scripts/run_e2e_suite.sh {{args}} + else + echo "Unknown e2e subcommand: $subcommand" >&2 + just help e2e + exit 1 + fi + +# Benchmark flows (local/cloud/history + direct relay targets). +[script] +bench subcommand="help" *args: + cd "{{repo_root}}" + subcommand="{{subcommand}}" + + if [[ -z "$subcommand" || "$subcommand" == "help" ]]; then + just help bench + elif [[ "$subcommand" == "compare" ]]; then + ./scripts/run_bench_compare.sh {{args}} + elif [[ "$subcommand" == "collect" ]]; then + ./scripts/run_bench_collect.sh {{args}} + elif [[ "$subcommand" == "update" ]]; then + ./scripts/run_bench_update.sh {{args}} + elif [[ "$subcommand" == "list" ]]; then + ./scripts/run_bench_update.sh --list {{args}} + elif [[ "$subcommand" == "at" ]]; then + if [[ -z "{{args}}" ]]; then + echo "usage: just bench at " >&2 + exit 1 + fi + ./scripts/run_bench_at_ref.sh {{args}} + elif [[ "$subcommand" == "cloud" ]]; then + ./scripts/run_bench_cloud.sh {{args}} + elif [[ "$subcommand" == "cloud-quick" ]]; then + ./scripts/run_bench_cloud.sh --quick {{args}} + elif [[ "$subcommand" == "relay" ]]; then + ./scripts/run_nostr_bench.sh {{args}} + elif [[ "$subcommand" == "relay-strfry" ]]; then + ./scripts/run_nostr_bench_strfry.sh {{args}} + elif [[ "$subcommand" == "relay-nostr-rs" ]]; then + ./scripts/run_nostr_bench_nostr_rs_relay.sh {{args}} + else + echo "Unknown bench subcommand: $subcommand" >&2 + just help bench + exit 1 + fi diff --git a/mix.exs b/mix.exs index c1d655a..cfee547 100644 --- a/mix.exs +++ b/mix.exs @@ -28,11 +28,7 @@ defmodule Parrhesia.MixProject do def cli do [ preferred_envs: [ - precommit: :test, - bench: :test, - "bench.collect": :test, - "bench.update": :test, - "bench.at": :test + precommit: :test ] ] end @@ -74,15 +70,6 @@ defmodule Parrhesia.MixProject do "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"], "ecto.reset": ["ecto.drop", "ecto.setup"], test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"], - "test.nak_e2e": ["cmd ./scripts/run_nak_e2e.sh"], - "test.marmot_e2e": ["cmd ./scripts/run_marmot_e2e.sh"], - "test.node_sync_e2e": ["cmd ./scripts/run_node_sync_e2e.sh"], - "test.node_sync_docker_e2e": ["cmd ./scripts/run_node_sync_docker_e2e.sh"], - bench: ["cmd ./scripts/run_bench_compare.sh"], - "bench.collect": ["cmd ./scripts/run_bench_collect.sh"], - "bench.update": ["cmd ./scripts/run_bench_update.sh"], - "bench.at": ["cmd ./scripts/run_bench_at_ref.sh"], - # cov: ["cmd mix coveralls.lcov"], lint: ["format --check-formatted", "credo"], precommit: [ "format", @@ -90,8 +77,7 @@ defmodule Parrhesia.MixProject do "credo --strict --all", "deps.unlock --unused", "test", - # "test.nak_e2e", - "test.marmot_e2e" + "cmd just e2e marmot" ] ] end diff --git a/scripts/just_help.sh b/scripts/just_help.sh new file mode 100755 index 0000000..4c6cbea --- /dev/null +++ b/scripts/just_help.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +set -euo pipefail + +topic="${1:-}" + +if [[ -z "$topic" || "$topic" == "root" || "$topic" == "all" ]]; then + cat <<'EOF' +Parrhesia command runner + +Usage: + just [args...] + just help [group] + +Command groups: + e2e End-to-end harness entrypoints + bench Benchmark tasks (local/cloud/history/chart) + +Notes: + - Keep using mix aliases for core project workflows: + mix setup + mix test + mix lint + mix precommit + +Examples: + just help bench + just e2e marmot + just e2e node-sync + just bench compare + just bench cloud --clients 3 --runs 3 +EOF + exit 0 +fi + +if [[ "$topic" == "e2e" ]]; then + cat <<'EOF' +E2E commands + + just e2e nak CLI e2e tests via nak + just e2e marmot Marmot client e2e tests + just e2e node-sync Local two-node sync harness + just e2e node-sync-docker Docker two-node sync harness + +Advanced: + just e2e suite + -> runs scripts/run_e2e_suite.sh directly +EOF + exit 0 +fi + +if [[ "$topic" == "bench" ]]; then + cat <<'EOF' +Benchmark commands + + just bench compare Local benchmark comparison table only + just bench collect Append run results to bench/history.jsonl + just bench update Regenerate chart + README table + just bench list List machines/runs from history + just bench at Run collect benchmark at git ref + just bench cloud [args...] Cloud benchmark wrapper + just bench cloud-quick Cloud smoke profile + +Relay-target helpers: + just bench relay [all|connect|echo|event|req] [nostr-bench-args...] + just bench relay-strfry [...] + just bench relay-nostr-rs [...] + +Examples: + just bench compare + just bench collect + just bench update --machine all + just bench at v0.5.0 + just bench cloud --clients 3 --runs 3 +EOF + exit 0 +fi + +echo "Unknown help topic: $topic" >&2 +echo "Run: just help" >&2 +exit 1