Core Health

Purpose	Standalone, always-on RabbitMQ + Redis monitor. Polls RabbitMQ's Management HTTP API and Redis's INFO/MEMORY STATS on its own connections and publishes curated MQTT/Home Assistant telemetry for both, replacing per-component RabbitMQ queue-depth self-polling
Run frequency	Always-on, three independent poll loops (RabbitMQ, Redis, and a once-daily container-registry check for the per-component "update available" entities)
Reads/writes	RabbitMQ Management API (read-only), Redis (INFO/MEMORY STATS plus plain key reads, both via the same default-user credential every other component uses), the container registry's public tags API (read-only, anonymous) — no direct RabbitMQ AMQP connection, no S3

How it works

Two independent background loops, neither blocking the other:

• RabbitMQ, every 10s: GET /api/overview (broker-wide connection count), GET /api/nodes (per-node memory/disk alarm flags — /api/overview itself doesn't carry these despite being the endpoint originally named for this data point during design; polling /api/nodes too is what actually answers it), GET /api/queues/%2F (every queue in one call), and GET /api/exchanges/%2F/skyfollower-adsb (the skyfollower-adsb exchange's own aggregate publish velocity — shared/rabbitmq_topology.py's ADSB_EXCHANGE constant, no discovery needed). The queue list is filtered to SkyFollower's own queues via shared/rabbitmq_topology.py's SKYFOLLOWER_RABBITMQ_RESOURCE_PATTERN — the exact same regex scripts/install.sh's provision_rabbitmq_users() uses to scope the application user's permissions, so the two definitions of "what SkyFollower owns" can't drift apart (bash can't import a Python constant, so the two copies are kept identical by hand — see the comments at each site).
• Redis, every 60s: INFO everything + MEMORY STATS, via the same default-user credential every other component authenticates with (see Credentials for why this isn't a separate scoped ACL user). No keyspace SCAN/--bigkeys ever — deliberately out of scope for this recurring loop (real cost against a large keyspace for a shape that doesn't change fast); the two commands above cover every field this component publishes.

Every entity core-health publishes for a queue or for Redis uses its own availability topic (SkyFollower/core-health/status), not the topic the device otherwise belongs to — a message processor being offline doesn't mean its queue's RabbitMQ-observed stats are unavailable (the broker still reports them), and vice versa. Queue/broker/Redis entities also carry expire_after (three poll intervals), so a poll failure that leaves a retained MQTT value in place doesn't read as still-fresh in Home Assistant indefinitely.

Where entities land

A Home Assistant device is just whatever entities share one device.identifier — core-health doesn't need to import or call into any other component's code, only know its device-identifier convention:

Queue	Device	Identifier
skyfollower-message-processor-{id}	That processor's own existing device	SkyFollower_message_processor_{id} (matches message-processor/main.py)
skyfollower-archive	The archive processor's own existing device	SkyFollower_archive (matches archive-processor/main.py)
skyfollower-adsb-unroutable, anything else SkyFollower-owned with no natural owner	New SkyFollower Core device	SkyFollower_Core

Broker-wide RabbitMQ stats and all of Redis also land on the SkyFollower Core device. Per-queue entity unique_id/object_ids are suffixed _queue_{field} (e.g. SkyFollower_message_processor_{id}_queue_consumers) so they can never collide with the owning component's own entities on the same device (e.g. SkyFollower_message_processor_{id}_processing_time_hwm_ms).

Archive queue missing detection

Not every deployment installs archive-processor, so the skyfollower-archive queue being absent from the polled /api/queues/%2F list is a normal, valid state — not an error. But it's also indistinguishable, from an operator's perspective, from "core-health hasn't polled yet" unless something says so explicitly. Every RabbitMQ poll checks the (already-filtered) SkyFollower-owned queue list for skyfollower-archive's presence and publishes archive_queue_missing (rabbitmq_connections_total's sibling on the SkyFollower Core device) accordingly — True when it's expected but not found, False once it exists. This is scoped specifically to skyfollower-archive, the one queue every deployment eventually has; it deliberately does not attempt to distinguish "archive-processor never installed" from "installed, then the queue got deleted/misconfigured" — both look identical via the Management API, and the retained flag clears itself automatically the next time skyfollower-archive shows up in the poll either way.

Counter mimicry (message-processor and the receiver)

core-health also reads and publishes, on message-processor's and the receiver's behalf, a handful of their own Redis-backed application counters — using those components' exact existing topic paths, unique_id/object_id, and device blocks, so nothing on the wire distinguishes core-health publishing these from the owning component publishing them itself:

• message-processor: registration_misses_{hour,today,lifetime}, operator_misses_{today,lifetime}, total_messages_processed_{hour,today,lifetime} — enumerated for free from the same skyfollower-message-processor-{id} queue list already polled above, no separate discovery mechanism.
• receiver: messages_{host}_{port}_total_{hour,today} per source connection — discovered via one cheap SMEMBERS per RabbitMQ poll cycle against a small Redis index set of currently-claimed receiver names (never a keyspace scan), then one GET per receiver for its registration entry (its source list). core-health is the sole publisher of these two (value and HA discovery); the receiver only feeds the Redis counters and never publishes those topics itself, so with no Redis configured they don't exist at all. The sibling messages_{host}_{port}_total_lifetime sensor is not core-health's — the receiver publishes that one directly from its own in-memory counter (it resets on a receiver restart by design), and it is never written to Redis.

A missing period key means the count is genuinely zero, not unavailable — mirroring message-processor's own _redis_counter() precedent. This is distinct from core-health's own Redis connectivity failing outright, which is a real "skip this tick, let the entity age out" case (see _redis_counter_or_none() in main.py).

Update-available entities

core-health also publishes one Home Assistant update entity per SkyFollower component — its running image version versus the newest release published to the container registry — nested under that component's own existing Home Assistant device. It is an availability indicator only: the discovery config carries no command_topic, no payload_install, and no INSTALL supported feature, so Home Assistant shows that a newer version exists but offers no in-place upgrade button. Applying it stays a manual docker compose pull.

Which components are running, and what image to check, is learned entirely from self-registration — never inferred or pattern-matched. Every MQTT-enabled component publishes one retained SkyFollower/register/{id} message (shared/mqtt_register.py's publish_register(), called right alongside its own Home Assistant discovery publish) carrying its GHCR image name and its discovery device block verbatim. The image name is baked into the image at build time exactly the way VERSION/GIT_COMMIT already are: every Dockerfile declares ARG IMAGE=unknown / ENV COMPONENT_IMAGE=$IMAGE, and build-container-images.yaml passes an IMAGE build-arg of skyfollower- followed by that workflow's own matrix.name value — the same bare name its own discover-images job already computed as the single source of truth for what each component's published image is called, so there is no second, hand-maintained mapping of component identity to image name anywhere in this component's own code to drift out of sync with that workflow. core-health subscribes to SkyFollower/register/+ and reads image and device straight out of each payload. A component that clears its retained registration drops out of the registry and its update entity is cleared with it; an image that never registered itself is ignored — there is nothing running to compare against.

The registry poll is deliberately slow — once a day (shared/timing.py's GHCR_VERSION_CHECK_INTERVAL_SECONDS), plus one pass GHCR_VERSION_CHECK_STARTUP_DELAY_SECONDS after startup so the entities aren't blank until the following day. Published release tags only move on a release, and a single pass fetches an anonymous bearer token plus a tags list for every distinct image. The token is memoised inside shared/version_check.py and reused across the whole pass; tag ordering is a YYYY.MM.BB integer-tuple comparison ("2026.9.9" sorts before "2026.9.10", which a string compare gets wrong) — see shared/version_check.py. Every lookup failure — a network error, a rate-limit answer, an image with no release tags yet — is turned into "keep the last-known latest version" rather than blanking the entity, matching core-health's best-effort telemetry everywhere else.

State is a single retained JSON blob per component ({"installed_version": ..., "latest_version": ..., "release_url": ...}, release_url being the GitHub release page for the latest version, omitted until the first successful poll) that Home Assistant parses natively, published at SkyFollower/register/{id}/update — right alongside the registration that drives it — re-published on change, on every core-health MQTT (re)connect, and on each daily poll. The update entity's own availability is core-health's, not the owning component's (the registration payload carries no availability topic of its own), the same choice already made for the message-processor/receiver counter mimicry above. management-ui and map are covered the same as everything else now that they publish a minimal Home Assistant presence of their own (shared/mqtt_presence.py, which self-registers on their behalf in the same call).

Redis keys

message-processor's counter-writing side (operator_misses, total_messages_processed, and the reset-mechanism fix for registration_misses) has since landed, and this component's reading side was reconciled to it: metrics_registration_misses_key()/ metrics_operator_misses_key()/metrics_total_messages_processed_key() are all real shared/redis_keys.py builders now, imported directly rather than shimmed locally. message-processor is write-only for these three counters (accumulates in memory, flushes to Redis on its own telemetry cadence — see message-processor/README.md) — this component remains the only one that ever publishes them over MQTT/Home Assistant.

The receiver's identity/registration mechanism has landed too, and this component's reading side is reconciled to it in the same way: receiver_registry_index_key()/receiver_registration_key()/ receiver_message_count_key() are all real shared/redis_keys.py builders now, imported directly rather than shimmed locally. receiver is write-only for these (accumulates in memory, flushes to Redis on its own telemetry cadence — see receiver/README.md) — this component remains the only one that ever publishes them over MQTT/Home Assistant. No provisional local key shims remain in main.py.

Configuration

Reads its configuration from environment variables via shared/config.py's load_config("rabbitmq_management", "redis", "mqtt"), interpolated by Compose from this host's .env (written by scripts/install.sh).

Variable	Required	Default	Description
RABBITMQ_HOST	✅	—	Same broker every other component connects to
RABBITMQ_MANAGEMENT_PORT	❌	15672	RabbitMQ's Management HTTP API port, not the AMQP port
RABBITMQ_MONITORING_USERNAME	✅	—	RabbitMQ's built-in monitoring-tagged user (see Credentials)
RABBITMQ_MONITORING_PASSWORD	✅	—
REDIS_HOST	✅	—	Same Redis every other component connects to
REDIS_PORT	❌	6379
REDIS_PASSWORD	✅	—	The same default-user credential every other component uses — for both INFO/MEMORY introspection and plain key reads (application counters); see Credentials for why this isn't a separate scoped user
MQTT_HOST	❌	—	Leave unset to disable MQTT entirely (core-health still polls, but publishes nothing)
MQTT_PORT	❌	1883
MQTT_USERNAME	❌	—	Optional MQTT auth; leave unset for an anonymous broker
MQTT_PASSWORD	❌	—
LOG_LEVEL	❌	info	"debug" for verbose output

Credentials

• RabbitMQ: a new user tagged monitoring — RabbitMQ's built-in tag for broker-wide, read-only Management API visibility, provisioned by scripts/install.sh once the rabbitmq container reports healthy (see provision_rabbitmq_users()). No per-resource permission is granted (or needed): the monitoring tag alone is what grants visibility, so this user's configure/write/read permissions are all set to match nothing.
• Redis: no new credential — core-health authenticates as the same "default" user via REDIS_PASSWORD every other component already uses, for both INFO/MEMORY and plain key reads. A separate, scoped (INFO/MEMORY-only) ACL user was designed and considered — it would have been the first Redis ACL user in this repo — but dropped: combining a new ACL user with the redis service's existing --requirepass mechanism was confirmed live to silently disable password authentication on the default user entirely (not a startup error — Redis accepts an unauthenticated connection with no AUTH needed at all), unless the ACL file is pre-seeded with the default user's own credentials before Redis's very first boot. That bootstrapping complexity and risk wasn't worth it for a restriction this component's own code already honors by simply never calling a write command — the same trust model every other component (message-processor, receiver, archive-processor) already operates under today.

MQTT

Published continuously (all retained) under SkyFollower/core-health/..., plus mimicked topics under the owning component's own root for the counter passthrough fields described above:

Topic	Published from
SkyFollower/core-health/status	ONLINE/OFFLINE, this component's own availability (LWT)
SkyFollower/core-health/statistic/{started_at,rabbitmq_connected,redis_connected}	General
SkyFollower/core-health/statistic/{rules_version,areas_version}	Canonical rules/areas config version (last 8 chars of config:rules:version / config:areas:version from Redis) — compare against each processor's own rules_version/areas_version sensor. Skipped (retained value left alone) while the key is absent or Redis is unreadable
SkyFollower/core-health/rabbitmq/statistic/{field}	Broker-wide RabbitMQ stats
SkyFollower/core-health/redis/statistic/{field}	Redis stats
SkyFollower/core-health/message-processor/{id}/statistic/{field}	A processor's queue stats
SkyFollower/core-health/archive/statistic/{field}	The archive queue's stats
SkyFollower/core-health/queue/{queue}/statistic/{field}	Any other SkyFollower-owned queue's stats (e.g. skyfollower-adsb-unroutable)
SkyFollower/message-processor/{id}/statistic/{field}	Mimicked message-processor counters (exact existing topic)
SkyFollower/receiver/{name}/statistic/{field}	Mimicked receiver counters (exact existing topic)
SkyFollower/register/{id}/update	Per-component "update available" state blob ({installed_version, latest_version, release_url}), published by core-health — see Update-available entities
homeassistant/update/{id}_update/config	Per-component update entity discovery config, published by core-health (empty retained payload clears it)
SkyFollower/register/{id}	Self-registration — subscribed here, published by every other MQTT-enabled component itself, not core-health (shared/mqtt_register.py)

Home Assistant autodiscovery configs are published on every MQTT (re)connect for the static broker-wide/Redis/general entities, and opportunistically the first time a given queue/counter/receiver/component is seen during this process's lifetime (dynamic entities aren't known in advance — a message processor, receiver, or whole component can appear or disappear while core-health keeps running).

Deployment

Deployed via docker-compose.core.yaml, always-on alongside rabbitmq/redis/ofelia (restart: unless-stopped, no profiles:). No depends_on — both clients simply retry on their own poll cadence until each backend is reachable, and this component keeps no state of its own (a pure poller/republisher), so it has no data volume, only the same tmpfs-mounted healthcheck heartbeat file every other long-running component uses.