Management UI

Purpose	REST API and React frontend for rules/areas configuration and archive search history. The backend is the sole write path for config:rules / config:areas in Redis — every message processor polls the corresponding :version key every 30 seconds and hot-reloads on change
Auth	None (single-instance, trusted-network deployment)
Reads/writes	Redis; optionally publishes a minimal MQTT presence for Home Assistant (see MQTT and Home Assistant)

Named "management" to leave room for a future, separate UI focused on viewing live aircraft movement rather than editing configuration.

Status

The React rules editor, areas editor, and archive search history page (frontend/) are all built. The Dockerfile is a multi-stage build — a node stage produces the static frontend bundle, and the final stage runs both uvicorn (bound to 127.0.0.1:8000, not exposed outside the container) and nginx, started by entrypoint.sh. nginx serves the built frontend at / (with a try_files fallback to index.html for client-side routing) and proxies /api/* to uvicorn over HTTPS on the container's 443. docker-compose.management-ui.yaml publishes it on the host as MANAGEMENT_UI_HTTPS_PORT (default 18443, set in .env by scripts/install.sh) — see TLS below.

Both base images (node:26-slim for the frontend stage, python:3.14-slim for the final stage) are pinned to an explicit @sha256:<digest> so Dependabot surfaces a rebuild PR when either tag is re-pushed upstream. Resolve a current digest with docker buildx imagetools inspect <image>:<tag>; a CI guard in .github/workflows/run-tests.yaml fails the build on any unpinned FROM line.

TLS

Served over HTTPS on MANAGEMENT_UI_HTTPS_PORT (default 18443, e.g. https://<host>:18443) with a self-signed certificate by default. There is no plain-HTTP listener: a plain http:// request to that same port gets a 301 to https:// on the same host and port (nginx's error_page 497). There is no authentication either way (see the table above) — TLS here is about encrypting traffic on the LAN, not access control.

• Certificate generation: scripts/install.sh generates a ~10-year self-signed cert (openssl req -x509 ...) the first time you install or re-run it for the management-ui role, writing cert.pem/key.pem into ./data/management-ui/tls/ (bind-mounted read-only into the container at /etc/nginx/tls/ — see docker-compose.management-ui.yaml). The certificate's SAN covers localhost, 127.0.0.1, the host's detected hostname/LAN IP, and an optional extra hostname/IP you're prompted for during install. Generation is idempotent — re-running install.sh never overwrites an existing cert.
• Bring your own certificate: drop your own cert.pem/key.pem into ./data/management-ui/tls/ (matching those exact filenames) before running scripts/install.sh — the installer leaves an existing pair alone entirely and never generates a replacement.
• First-visit browser warning: since the certificate is self-signed, every browser shows an untrusted-certificate warning the first time you visit. This is expected — click through it (or add an exception), or import cert.pem into your OS/browser's trust store if you'd rather not see it again. There's no ACME/Let's Encrypt integration — this deployment model is LAN-only, with no public DNS name or open inbound 80/443 for ACME's HTTP-01/TLS-ALPN-01 challenges to work against.
• Replacing a certificate: overwrite cert.pem/key.pem in ./data/management-ui/tls/ and restart the container (docker compose -f docker-compose.management-ui.yaml restart management-ui) — nginx has no separate reload step wired up for this.

Endpoints

Per-item CRUD, not bulk replace — every rule and area is independently addressable by identifier. config:rules / config:areas in Redis still each store one JSON blob (the full array/collection — that's what message processors poll and hot-reload), so every write below is read-full-array, splice, validate, write-back, not a partial update in Redis itself.

Method	Path	Description
GET	/api/rules	All rules. 200 with the array (empty if none configured). Every rule also carries triggered_lifetime and triggered_last_30_days — response-only, computed fresh from the rule_triggers:* Redis keys the message processor writes (a true trailing 30-day window, summed daily keys), never accepted on write
GET	/api/rules/{identifier}	One rule, same two triggered_* fields as the list. 200, or 404 if not found
POST	/api/rules	Add one new rule. 201 with the created rule, 409 if identifier already exists, 400 on validation failure
PUT	/api/rules/{identifier}	Replace one existing rule. 200 with the updated rule, 404 if identifier doesn't exist yet (use POST to create), 400 if the body's identifier doesn't match the path or on validation failure
DELETE	/api/rules/{identifier}	Remove one rule. 204, or 404 if not found. Also deletes the rule's rule_triggers:* counter keys, so a later rule reusing the identifier starts at zero
GET	/api/areas	All areas, flattened: [{identifier, name, geometry}, ...] — not a GeoJSON FeatureCollection; see below
GET	/api/areas/{identifier}	One area. 200, or 404 if not found
POST	/api/areas	Add one new area. 201, 409 if identifier already exists, 400 on validation failure
PUT	/api/areas/{identifier}	Replace one existing area. 200, 404 if identifier doesn't exist yet, 400 on mismatch/validation failure
DELETE	/api/areas/{identifier}	Remove one area. 204, or 404 if not found
GET	/api/aircraft?icao_hex={hex} or ?registration={reg}	Aircraft enrichment lookup by one or the other (not both, not neither). 200, 400/422 if both/neither query param is given, 404 if not found, 503 if the backing RediSearch index isn't ready yet (registration lookup only — see below)
GET	/api/operators/{designator}	One operator's enrichment. 200, or 404 if not found
GET	/api/airports/{code}	Airport metadata, by 4-char ICAO or 3-char IATA code. 200, 404 if not found, 503 if the backing RediSearch index isn't ready yet (IATA lookup only — see below)
GET	/api/routes/{ident}	Route lookup by flight ident, with best-effort operator enrichment. 200 (route + operator, or operator-only with origin/destination/stops omitted if only the ICAO airline-designator prefix resolves), or 404 if neither resolves

Both Rule and Area require identifier (routing key, no spaces — message-processor/rules_engine.py rejects a spaced rule identifier outright; an area with a spaced or missing identifier is silently dropped by the engine's existing lenient per-feature area loading, which this backend turns into an explicit 400 by checking the identifier actually survived the reload). name is a separate, optional free-text display label that can contain spaces.

The registration-based /api/aircraft lookup and the IATA-code branch of /api/airports/{code} go through RediSearch (_search_one() in main.py); their 503 above is the safety net described in Reference-Data Search Index Bootstrap below, not an error you should normally see in practice. icao_hex-based aircraft lookups and 4-char-ICAO airport lookups are plain Redis key reads and never return 503.

Every successful write recomputes the full array/collection, computes a SHA-256 hash of it, and writes both to Redis (config:rules / config:areas and their :version counterparts) — :version is what message processors actually poll; they never read config:rules / config:areas itself unless the hash has changed.

Areas: flattened API shape vs. GeoJSON storage

The API's Area shape ({identifier, name, geometry}) is not what's stored in Redis or handed to RulesEngine — config:areas is still a GeoJSON FeatureCollection, each Feature with properties: {identifier, name} and the area's geometry, since that's what RulesEngine and the message processor expect. This backend translates between the two shapes at the API boundary (_area_to_feature/_feature_to_area in main.py) — clients of this API never see the FeatureCollection wrapper.

An area referenced by a rule's area condition is matched by identifier, not name (e.g. {"type": "area", "value": "LI"} matches an area whose identifier is "LI", regardless of its display name). A rule referencing an area identifier that doesn't exist fails with 400 — areas and rules validate against the same in-process RulesEngine instance, so save the area before a rule that references it.

The full request/response schema is in specs/openapi.yaml, exported from this app's own OpenAPI document (see below). Rule/Condition/Area shapes there roughly match SkyFollower-legacy's rules.example.json / areas.example.geojson conventions (condition values are strings even for numeric fields, e.g. altitude "10000"), and ARE the actual route parameter types for create/update — a request body that doesn't match (wrong type, an operator not valid for a condition's type, a missing field) gets a 422 from FastAPI/Pydantic before the route function ever runs. Condition is a type-discriminated union of one model per condition type, so each one's operator is restricted to only the values that type actually supports (e.g. date only accepts minimum/ maximum), rather than a single flat model allowing all 5 operators everywhere. RulesEngine remains a second, independent validation layer underneath this one — not made redundant by it, since it's the only validation applied to config:rules/config:areas written some other way than through this API (a hand-edited Redis value, a restored backup, a future integration), and it enforces things a single condition's fields can't express on their own (e.g. an area condition's value must name an area that actually exists in config:areas).

Frontend (frontend/)

React (TypeScript) + Vite + Tailwind CSS + React Router. No other UI component library, to keep dependencies minimal for ARM builds.

• Layout.tsx / components/SideNav.tsx — app shell: side nav + routed content area. Rules and Areas are both entries in SideNav.tsx's section list; a future section adds another entry there without touching Layout.tsx.
• hooks/useToast.ts + components/ToastContainer.tsx — shared success/error toast notifications.
• components/ConfirmModal.tsx — generic confirm dialog (discard unsaved changes, delete confirmation).
• api/client.ts — thin fetch wrapper: JSON parsing and {"detail": "..."} error surfacing from any 4xx/5xx response.
• api/rules.ts — typed client for /api/rules/*, plus the Rule/Condition types and the OPERATORS_BY_TYPE map used to filter each condition's operator dropdown by its type.
• views/RulesView.tsx — rule list (each row showing a red "Not Enabled" pill when disabled, or a hollow gray "Inactive" pill when enabled but a date condition isn't currently satisfied) + selected rule's form; owns save/discard/delete state and the two confirm flows. Enabling/disabling a rule is only done by editing its Enabled checkbox and saving -- no inline toggle in the list. "Export all" downloads every rule as a rules.json array (client-side, no round trip); "Import" (ImportRuleModal
  • lib/ruleImport.ts) takes that array back — auto-suffixing identifier collisions, skipping a rule whose area condition references an area not present here, and, for matched_rules, creating then verifying: a reference to a rule that doesn't exist and isn't part of the same import gets the rule ejected (deleted) transitively, while a legitimate cyclic pair inside one file survives.
• components/RuleForm.tsx / components/ConditionForm.tsx — the rule editor and its per-condition, type-aware value input (number, hex, wake-turbulence dropdown sourced alphabetically and title-cased for display, heading min/max pair, matched_rules checkbox list with includes/excludes operator labels, area dropdown sourced from GET /api/areas, and the date condition's datetime input, which converts a local datetime-local value to UTC and appends Z before saving -- the UI always saves the YYYY-MM-DDTHH:MMZ form; the backend's date-only YYYY-MM-DD format is still accepted if written some other way, e.g. directly via the API).
• api/areas.ts — typed client for /api/areas/*.
• views/AreasView.tsx — MapLibre GL JS + @mapbox/mapbox-gl-draw (works against maplibre-gl's Mapbox-GL-compatible API; there is no scoped @maplibre/maplibre-gl-draw package). Existing areas load as Draw features on mount, map bounds auto-fit to them, and each polygon gets a centroid label showing its name. Drawing a new polygon opens components/AreaNameModal.tsx to collect identifier/name, then saves immediately via POST /api/areas (no separate "Save All" step -- the backend is per-item CRUD, not a bulk replace, so each area is created/ updated/deleted independently, the same as RulesView.tsx). Selecting an existing area (from the side list or by clicking its polygon) enters Draw's direct_select mode for vertex editing and shows an inline Name field + Save/Discard/Delete in the side list; geometry edits on the map and name edits in the list share the same dirty/save/discard state.
• api/archiveSearch.ts — typed client for /api/archive/search/* and /api/archive/flights/{token}. downloadArchiveFlight fetches via Blob + a synthetic <a download> click rather than a plain navigation, specifically so an expired/invalid token's 400 can be caught and shown as a toast instead of the browser just rendering a raw JSON error page.
• views/HistoryView.tsx — search list (left) + selected search's results (right on desktop; the same content renders inline as an accordion under the selected list entry on mobile instead, since there's no persistent side-by-side space below the md: breakpoint). Polls GET /api/archive/search every few seconds while any visible search is RUNNING, so status pills update to COMPLETE/FAILED/ ABORTED live. Results (once COMPLETE) are fetched and cached per-page client-side, keyed by {uuid}:{page}, so paging back to an already-viewed page never re-fetches. components/NewSearchModal.tsx collects name + a raw SQL where_clause (plain <textarea>, not a syntax-highlighted editor) alongside a persistent column-reference legend, since the whole point of that field is writing SQL against exact column names from memory.

Client-side validation (validateRule/validateCondition in RuleForm.tsx) mirrors message-processor/rules_engine.py's per-type checks as a fast-fail UX nicety — the backend's 400 response is still the source of truth.

cd management-ui/frontend
npm install
npm run dev    # Vite dev server on :5173, proxying /api to localhost:8000
npm run build  # type-checks (tsc -b) then builds the static bundle to dist/

Configuration

Reads its configuration from environment variables via shared/config.py's load_config("redis", "s3", "athena", "mqtt"), interpolated by Compose from this host's .env (written by scripts/install.sh). management-ui is a separate compose file/project from core even when co-located on the same host (see docker-compose.management-ui.yaml), so its REDIS_HOST typically points at localhost rather than the redis service name core's own components use.

Variable	Required	Default	Description
MANAGEMENT_UI_HTTPS_PORT	❌	18443	Host port the HTTPS UI is published on (compose only; the container always listens on 443). Plain HTTP sent to it is redirected to HTTPS on the same port
REDIS_HOST	✅	—
REDIS_PORT	❌	6379
REDIS_PASSWORD	✅	—	Redis requires authentication; see shared/redis_client.py's build_redis_client()
S3_BUCKET	✅	—	S3 bucket the archive is written to (see Archive Search)
AWS_DEFAULT_REGION	✅	—	boto3's own variable name -- no credentials are ever passed in code, so every client picks up the default credential chain
AWS_ACCESS_KEY_ID	✅	—	boto3's own variable name
AWS_SECRET_ACCESS_KEY	✅	—	boto3's own variable name
ATHENA_WORKGROUP	❌	skyfollower	Athena workgroup to run queries against (see Archive Search)
ATHENA_DATABASE	❌	skyfollower	Glue database name
ATHENA_TABLE	❌	archive_flights	Glue table name, matching specs/aws/cloudformation.yaml's GlueTableName default
MQTT_HOST	❌	—	Leave unset to disable MQTT entirely (see MQTT and Home Assistant)
MQTT_PORT	❌	1883
MQTT_USERNAME	❌	—	Optional MQTT auth; leave unset for an anonymous broker
MQTT_PASSWORD	❌	—
LOG_LEVEL	❌	info	"debug" for verbose output

MQTT and Home Assistant

MQTT is entirely optional -- leave MQTT_HOST unset and the backend never opens a broker connection. When it is set, the backend publishes a minimal presence and nothing else: there is no telemetry loop, no periodic publish, and no operational stats. The client connects once and stays connected.

On every connect/reconnect it publishes, all retained:

Topic	Payload	Purpose
SkyFollower/management-ui/status	ONLINE	Liveness. A Last Will and Testament flips this to OFFLINE if the connection drops uncleanly; a clean shutdown publishes OFFLINE explicitly.
SkyFollower/management-ui/statistic/started_at	UTC ISO-8601 timestamp	Process start time.
SkyFollower/management-ui/statistic/version	Version string (dev if the image was built without a VERSION)	The running image version.
homeassistant/sensor/SkyFollower_management-ui_started_at/config	HA MQTT discovery config	Registers a "SkyFollower Management UI" device (whose sw_version carries the running version) with a single Start Time sensor (device_class: timestamp).

The running version is carried in the discovery device block's sw_version and as the plain statistic/version topic, so there is deliberately no standalone version sensor entity -- the same choice the receiver and archive processor make.

The "update available" entity for this component is not published here. A separate health component polls GHCR for the latest released image tag, compares it against the statistic/version value above, and publishes the HA update entity on this component's behalf.

The client is started and cleanly stopped as part of the FastAPI application lifespan, alongside the Redis/S3/Athena clients.

Archive Search

Athena/Glue query layer over the archive's Parquet index (see Archive Processor's Parquet Index section and specs/data-dictionary.yaml's archive_parquet_index record for the 9 underlying columns), with a History page (views/HistoryView.tsx) in the frontend below.

Method	Path	Description
POST	/api/archive/search	Start a search: {name, where_clause, start_date?, end_date?} body (start_date/end_date are optional UTC calendar dates, YYYY-MM-DD; see Partition pruning below). HTTP 202 with {uuid} immediately — the query itself runs in the background (see Background polling below), unless the resolved date range comes up empty, in which case the search is recorded already COMPLETE with zero rows and no Athena query is ever started. HTTP 400 if an explicit start_date is after end_date. HTTP 502 if Athena rejects the query outright (e.g. a permissions mismatch)
GET	/api/archive/search	List all current search records: {uuid, name, status, submitted_at, expires_at, error} each (error is only ever set for FAILED/ABORTED)
GET	/api/archive/search/{uuid}	One search record, including its where_clause and the RESOLVED start_date/end_date actually queried (null for a record predating this field). HTTP 404 if not found
GET	/api/archive/search/{uuid}/results?page={n}	One page of a COMPLETE search's cached results (page size is caller-selectable, 25-500, default 100): {rows: [...], total_rows, truncated}. rows/total_rows only ever reflect the first 500 matching rows — see Result retrieval and pagination — truncated is true when more than 500 rows actually matched. HTTP 404 if not found, HTTP 400 if the search isn't COMPLETE yet or page is past the cached results, HTTP 502 if Athena fails to return the result window (e.g. a permissions mismatch)
GET	/api/archive/search/{uuid}/download	Full result set, every row, direct from S3 — see Download below. HTTP 307 redirect to a short-lived presigned URL. HTTP 400 if the search isn't COMPLETE yet, HTTP 404 if not found, HTTP 502 if Athena/S3 fails
DELETE	/api/archive/search/{uuid}	Delete a search record — best-effort-cancels the Athena query if still RUNNING, deletes the Athena result file from S3 if COMPLETE, always deletes the Redis record. HTTP 204, or HTTP 404 if not found
GET	/api/archive/flights/{token}	Download one flight's full archived record (gzipped JSON) by its opaque, encrypted fetch token — never a raw S3 key. HTTP 400 on an invalid/expired token, HTTP 502 if S3 fails to return the object (e.g. it's already gone, or a permissions mismatch)

Every HTTP 502 above shares the same _AWS_ERROR response contract: any AWS-side failure (Athena or S3) is caught and surfaced as a single opaque HTTP 502, never the raw upstream AWS status code — so a 404/NoSuchKey and a 403/AccessDenied from S3 are indistinguishable to the caller, both just HTTP 502.

Query construction: a raw, user-supplied WHERE clause

where_clause is genuinely user-authored SQL, not a set of structured filter fields — a deliberate, informed trade-off given this tool's actual exposure (single operator, no auth, not internet-facing), not an oversight. The backend always controls the rest of the statement:

SELECT icao_hex, registration, type_designator, military, operator_designator,
       ident, first_message, last_message, s3_key
FROM {athena.database}.{athena.table}
WHERE ({partition_predicate}) AND ({where_clause})

where_clause only ever fills the second, parenthesized WHERE fragment — the SELECT list, FROM table, partition_predicate, and both sets of parentheses are never influenced by user input, and there's no way to reach a different table. Still bounded by more than the application layer alone: the querying IAM identity is already read-only on just this one table (see AWS Setup below), and Athena executes one statement per start_query_execution call, so no ;-chained multi-statement injection is possible regardless of content. On top of that, a cheap early rejection (before ever calling Athena) 400s a where_clause containing ; or a DDL/DML keyword (DROP/CREATE/ALTER/INSERT/DELETE/UPDATE/GRANT, matched word-boundary so ident = 'INSERT1' doesn't false-positive) — not a real security boundary given the IAM scoping above, purely so a mistake produces an instant, clear 400 instead of a slower, more opaque Athena AccessDenied.

s3_key is selected from Athena (the backend needs it server-side to mint each row's fetch token and derive its flight UUID — see "Flight fetch" below) but is never included in any HTTP response body a browser receives.

Partition pruning

The Glue table's year/month/day partition columns use partition projection with no crawler (specs/aws/cloudformation.yaml) — Athena generates and LISTs every one of the ~29,000 projected partitions (2022,2100 × 12 × 31) at query time unless the query's own WHERE clause narrows them. Without a predicate on those columns, every search — even one matching a single icao_hex — scans the entire projected range, measured at ~40 seconds against the live archive account regardless of how little data actually matches.

partition_predicate above is always backend-generated and always injected — there is no way to run a search without it. It is built from three constraints, intersected to the tightest overlap:

1. The archive's own bounds — _ARCHIVE_EPOCH (built from shared/glue_projection.YEAR_RANGE's lower bound, so it can never widen past what the Glue table actually projects) through tomorrow UTC.
2. An operator-supplied start_date/end_date on the search request (both optional; a missing bound defaults to the corresponding side of #1). UTC calendar dates, since that's what the partition columns themselves encode.
3. Whatever where_clause's own first_message/last_message predicates can prove (_derive_bounds) — e.g. a clause containing first_message >= timestamp '2026-01-01 00:00:00' derives a lower bound with no operator action needed. Widened by one day each side as timezone/boundary insurance. Only sound inside a pure AND conjunction — an OR or a NOT anywhere in the clause makes the derivation unsound, so either one bails out to "no bound derivable" rather than risk excluding a row Athena would otherwise have matched. Parsed with sqlglot rather than a regex specifically so a column reference is never confused with the same text appearing inside a string literal.

If the three constraints don't overlap (e.g. an explicit start_date after what where_clause could ever match), the search is recorded as an already-COMPLETE, zero-row result with no Athena query ever run — a real, correct answer to a range that provably can't match anything, not an error. This is distinct from an explicit start_date after end_date on the request itself, which is a 400 at creation time.

The RESOLVED range (after intersection) is persisted on the search record and returned by GET /api/archive/search/{uuid} so it stays visible next to results that may have come from a derived, not just a typed, range. Resubmitting a search reuses its persisted resolved range verbatim, rather than re-resolving "tomorrow" against whatever the clock reads at resubmit time.

_partition_predicate always emits the coarsest clause that exactly covers a span — a whole year, then whole months within a year, then a day range within a month — rather than one clause per day, so a multi-year range doesn't degenerate into thousands of ORed day clauses.

Background polling

One daemon thread per in-flight search, started by the POST handler right after start_query_execution returns. Polls Athena's GetQueryExecution on exponential backoff (1s, 2s, 4s, 8s, 16s, then capped at 30s) for up to 2 minutes wall-clock total. If the deadline is hit without reaching a terminal state, the thread calls stop_query_execution and marks the search ABORTED — deliberately distinct from FAILED (Athena itself reported the query failed) so a broken Glue table setup doesn't just look like a slow query in the log. On every process startup, any record still RUNNING is immediately marked ABORTED — its polling thread died with the previous process, nothing is left alive to ever finish that job.

Every write the polling thread makes to Redis is a conditional SET ... XX KEEPTTL (only if the key still exists, and never resets the fixed 7-day TTL) — this is the thread-resurrection guard: if DELETE removes the record between this thread's last read and its next write, that write becomes a silent no-op instead of resurrecting a record the user already deleted.

Listing active searches

GET /api/archive/search and the startup reconciliation sweep above both need every current archive_search:{uuid} record. A Redis SET (archive_search:index, holding just each search's uuid, no TTL of its own) is the source of truth for "what searches currently exist" — SADD on create, SREM on delete, SMEMBERS + one GET per uuid to list.

A uuid whose backing archive_search:{uuid} key has already expired (the fixed 7-day TTL) has no way to notify the index that it's gone on its own, so the index self-heals opportunistically instead: any list/ reconcile call that GETs a member and finds it already gone SREMs that stale uuid right there, pruning it the next time anyone asks.

Result retrieval and pagination

Redis stores a pointer (the Athena QueryExecutionId) to the result set, not the result data itself — the polling thread never fetches or caches rows at completion time. Two independent read paths exist over that same pointer, deliberately shaped differently:

• The paged view (GET .../results, this section) is served from a bounded, in-process cache holding at most the first 500 matching rows — fast, and no repeat Athena/S3 call per page.
• Download (GET .../download, see Download below) always goes straight to S3 via a presigned URL, for every result size, with the backend never reading the bytes at all.

Rows for the paged view are only fetched the first time a user actually opens a completed search's results, via one GetQueryResults call bounded at MaxResults=501 — not the full result CSV Athena also writes to S3 (evaluated and rejected: downloading and parsing the whole file scales with the match count, not a fixed cap, which is exactly the unbounded- memory failure mode this design avoids — an ordinary search with no date range against a multi-year, multi-million-flight archive can match hundreds of thousands of rows). Requesting one row past the 500-row cap makes "did more than 500 rows match" answerable from that single call: getting back 501 data rows (after skipping GetQueryResults' own column- header row) means truncated is true on every page's response, and only the first 500 are ever cached or paginated. Building all 500 rows (fetch tokens included) happens once, up front, on the first request — 500 Fernet encryptions per search is negligible; it's what keeps every page after the first a pure cache hit with zero further Athena calls.

The cached window is a bounded in-process LRU (OrderedDict, move-to-end on access, capped at 10 concurrently-cached searches) keyed by search UUID — 10 × 500 rows is roughly 4MB worst case, a fixed, small ceiling regardless of how many rows actually matched (versus roughly 800MB for a single large, ungapped search under the old whole-CSV design). Raising either the 500-row cap or the 10-search cap without reconsidering the other reopens the unbounded-memory failure this pair exists to close. The cache lives only in process memory and is wiped on restart: a page request for a search that was mid-viewing when the container restarted is just a cache miss (one slower request, cached again from there), not an error or data loss. Requesting a page beyond what's cached (e.g. page 6 at 100 rows per page, with only 500 rows ever cached) is a clean HTTP 400, never a silently empty page.

Download

GET /api/archive/search/{uuid}/download is the only way to get every matching row — 3 or 400,000 — with no size threshold and no separate "large result" code path. It never routes through the 500-row cache above, and the backend never reads the result bytes at any point:

1. A second, independently-submitted Athena query reuses the exact same partition_predicate and where_clause the original search resolved, but with a different SELECT list — s3_key is never selected. The flight uuid is derived from it in SQL instead, via regexp_extract(s3_key, '([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\.json\.gz$', 1) — anchored on the UUID immediately before .json.gz, so it matches both the legacy {icao_hex}_{ident}_{uuid}.json.gz key shape and the current, simplified {uuid}.json.gz shape (see the archive processor's S3 key format). This query runs at most once per search — its QueryExecutionId and eventual S3 output location are persisted on the search's Redis record, so a second click presigns the same object with no new Athena call at all. A search that's never downloaded never pays for this query.
2. Once that query succeeds, its own S3 output object — a plain CSV, uuid first, the same eight other columns in the same order the paged view exposes — is presigned (ExpiresIn ~15 minutes) and the endpoint returns HTTP 307 straight to it. Because the download re-runs the search rather than reusing its original query execution, a flight archived after the search was submitted can show up in the download that wasn't in the paged view — only affects the most recent days; older partitions are immutable once compacted.
3. The frontend follows the redirect via a real browser navigation (an <a>/window.open, never fetch()) specifically so this needs no S3 CORS configuration: a plain navigation isn't subject to the same-origin read restrictions a script-readable fetch() response would be.

The object S3 ultimately serves contains no bucket name, no date-folder prefix, and no .json.gz suffix anywhere in it — the storage layout is absent from the object entirely (via the SELECT list above) rather than stripped in transit, which is the property that actually matters: a backend step that stripped it after the fact would still have had it in hand at some point.

Flight fetch: encryption, not a raw or encoded key

The browser must never receive a real s3_key in any form — a reversible encoding (e.g. base64) is equivalent to sending the plaintext key. Each result row's token is its s3_key, encrypted with Fernet (authenticated symmetric encryption from Python's cryptography library) — a tampered/forged token simply fails to decrypt (400) rather than decrypting to some attacker-chosen key. The Fernet key itself is ephemeral: generated fresh via Fernet.generate_key() at every process startup, held only in memory, never written to .env or disk. A restart invalidates every token minted by the previous process — a search still open in a browser tab across a restart gets a 400 on "view flight" until the page is refreshed (which re-fetches results from the now-running process, producing freshly-encrypted tokens). Acceptable for a low-traffic tool with infrequent deploys; every restart is effectively a free, automatic key rotation. This assumes a single running instance of this component, matching this project's actual deployment model (no horizontal scaling anywhere in its design).

TTLs

• archive_search:{uuid} Redis records: 7 days from creation, fixed — never refreshed on access/viewing, or deleted early via DELETE.
• archive_search:index: no TTL — it's a small, actively-maintained set of uuids (see Listing active searches above), not a data record with its own expiry.
• Athena query-result files in S3: 8 days, one day longer than the Redis TTL so Redis's own pointer always drops before the file it references actually disappears (an S3 lifecycle rule on the query-results prefix — part of AWS Setup below, not something this backend enforces itself). This covers the download query's own output file too — deleting a search only ever cleans up its original search query's result file (see the DELETE row above); a download result file, if one was ever generated, is left for the same 8-day lifecycle rule to age out rather than cleaned up explicitly.
• Download presigned URLs: 15 minutes (_DOWNLOAD_PRESIGN_TTL_SECONDS) — short enough that a leaked link is only briefly useful, long enough to cover a slow save-as dialog or a large file transfer.

AWS Setup

Nothing described here exists yet in a fresh AWS account. Like archive-processor and archive-compaction, this backend never calls a Glue, IAM, or Athena provisioning API itself. All of it — both S3 buckets, the Glue database/table, the Athena workgroup, and this backend's own IAM identity — is created by the one-shot aws-setup container, which deploys a CloudFormation stack from specs/aws/cloudformation.yaml. scripts/install.sh runs it (with --outputs-only) when you install the management-ui role, and pre-fills the AWS prompts from the stack outputs. See docs/aws-configuration.md.

This backend's identity is read-only on Athena and Glue, and — because CloudFormation resolves AWS::AccountId as a pseudo-parameter — those statements are now scoped to exactly this workgroup, database, and table, not Resource: "*". Its S3 access: s3:GetObject on index/* and flights/* in the archive bucket, bucket-level s3:ListBucket on the archive bucket (Athena enumerates partition prefixes with the caller's credentials — this cannot move to the results bucket), and full read/write/multipart access to the dedicated Athena results bucket.

Backing up config:rules/config:areas

config:rules and config:areas are the only two Redis keys in the whole schema holding user-authored/curated state with no automatic regeneration path (every other key is either repopulated by a runner on its next scheduled run, or transient operational state that naturally rebuilds from live traffic). Redis's own AOF (docker-compose.core.yaml's redis-data volume) is the only persistence for them otherwise, so this backend keeps a second, independent copy:

• On every successful save (_save_rules_array/_save_areas_array in main.py), the same JSON body just written to Redis is also written atomically (temp file + os.replace, so a crash mid-write can't leave a truncated file) to $DATA_DIR/rules-backup.json / $DATA_DIR/areas-backup.json.
• At startup, before loading rules/areas into RulesEngine, if a Redis key is missing but its backup file exists and parses as valid JSON, the file's content is restored into Redis — the key and its :version hash in one transaction — and the restore is logged. If the key already has data in Redis, the backup file is never consulted — Redis is always the source of truth when it has data. A missing or corrupt backup file at startup is logged and skipped, not a crash (same empty-array behavior as today if nothing can be restored).
• Also at startup, if config:rules exists but config:rules:version does not (a deployment from before the version key existed, a partially restored volume, a manual seed), the backend computes sha256(body) and sets it. Without this a message processor polls forever with redis.get("config:rules:version") == None and never loads the rules that are sitting right there. Same for config:areas. An existing version key is left untouched even if it doesn't match the body.
• Every write of a config:* body and its :version hash — on save and in both reconcile branches above — goes through one MULTI/EXEC transaction, so a failure between the two can never leave them skewed.
• message-processor/rules_engine.py treats the version key as a fast-path signal only and re-hashes the body when the key is missing, skewed, or the engine is holding no rules — so a restore or a version-key fill-in is picked up on the next 30-second poll like any other change.

$DATA_DIR defaults to /app/data, matching docker-compose.management-ui.yaml's ./data/management-ui:/app/data bind mount (same convention as the message processor's and archive processor's /app/data mounts), and can be overridden with the DATA_DIR environment variable.

Reference-Data Search Index Bootstrap

The three RediSearch indices backing reference-data lookups (idx:aircraft:mictronics, idx:aircraft:registry, idx:airport) are each normally created lazily by the data runner that owns them, on that runner's first successful run. On a fresh install (or before that runner's first scheduled run), the index simply doesn't exist yet, and any lookup that reaches it used to surface a raw Redis error: No such index ... as an unhelpful 500.

lifespan() now ensures all three indices exist (empty, if unpopulated) unconditionally at startup, mirroring each runner's own lazy-creation pattern (_ensure_search_index() in main.py) — a lookup against an unpopulated index is then a normal 404 (no match) instead of an error. _search_one() keeps a safety net for any "no such index" condition that still slips through (e.g. a future .ft(...) call site lifespan() doesn't cover): it returns 503 with an actionable message rather than the raw exception text. redis-py/RediSearch expose no dedicated exception type for "index doesn't exist" — it surfaces as a generic ResponseError/RedisError whose message is literally "No such index <name>", so detection is by message text, not exception type (a code comment next to the check flags this as inherently a little fragile against future Redis/RediSearch wording changes). A genuine connectivity/ auth failure is unaffected — still 500 with the raw message.

Regenerating specs/openapi.yaml

cd management-ui/backend
python -c "
import yaml
import main
print(yaml.dump(main.app.openapi(), sort_keys=False, allow_unicode=True))
" > ../../specs/openapi.yaml

(app.openapi() only builds the schema from route declarations — it doesn't touch Redis or read any configuration, so neither a running Redis nor a configured environment is required to regenerate the spec.)

info.version in the generated document is always 9999.99.99 on main — the release workflow substitutes the real version at build time, matching specs/asyncapi.yaml's convention.