Files

Billy D. 7b77d6c29f docs(adr): ADR-0021 notification architecture

- Define ntfy as central notification hub
- Standardize topic naming (gitea-ci, alertmanager-alerts, etc.)
- Document Alertmanager integration
- Design ntfy-to-Discord bridge architecture (future work)

2026-02-02 11:54:08 -05:00

5.5 KiB

Raw Blame History

ADR-0021: Notification Architecture

Status

Proposed

Context

The homelab infrastructure generates notifications from multiple sources:

CI/CD pipelines (Gitea Actions) - build success/failure
Alertmanager - Prometheus alerts for critical/warning conditions
Gatus - Service health monitoring
Flux - GitOps reconciliation events

Currently, ntfy serves as the primary notification hub, but there are several issues:

Topic inconsistency: CI workflows were posting to builds while documentation (ADR-0015) specified gitea-ci
No Alertmanager integration: Critical Prometheus alerts had no delivery mechanism
Discord integration desire: Team wants notifications forwarded to Discord for visibility

Decision

1. ntfy as the Notification Hub

ntfy will serve as the central notification aggregation point. All internal services publish to ntfy topics via the internal Kubernetes service URL:

http://ntfy-svc.observability.svc.cluster.local/<topic>

This keeps ntfy auth-protected externally while allowing internal services to publish freely.

2. Standardized Topics

Topic	Source	Description
`gitea-ci`	Gitea Actions	CI/CD build notifications
`alertmanager-alerts`	Alertmanager	Prometheus critical/warning alerts
`gatus`	Gatus	Service health status changes
`flux`	Flux	GitOps reconciliation events

3. Alertmanager Integration

Alertmanager is configured to forward alerts to ntfy using the built-in tpl=alertmanager template:

receivers:
  - name: ntfy-critical
    webhookConfigs:
      - url: "http://ntfy-svc.observability.svc.cluster.local/alertmanager-alerts?tpl=alertmanager&priority=urgent&tags=rotating_light"
        sendResolved: true
  - name: ntfy-warning
    webhookConfigs:
      - url: "http://ntfy-svc.observability.svc.cluster.local/alertmanager-alerts?tpl=alertmanager&priority=high&tags=warning"
        sendResolved: true

Routes direct alerts based on severity:

severity=critical → ntfy-critical receiver
severity=warning → ntfy-warning receiver

4. Discord Integration (Future)

Discord integration will be implemented as a dedicated bridge service that:

Subscribes to ntfy topics via SSE/WebSocket
Transforms ntfy message format to Discord embed format
Forwards to Discord webhook URL (stored in Vault at kv/data/discord)

Design Options

Option A: Sidecar Container (Simple)

Alpine container with curl/jq
Subscribes to ntfy JSON stream
Transforms and POSTs to Discord
Pros: Simple, no custom code
Cons: Shell script fragility, limited error handling

Option B: Dedicated Python Service (Recommended)

Small Python service using httpx or aiohttp
Proper reconnection logic and error handling
Configurable topic-to-channel mapping
Health endpoint for monitoring
Pros: Robust, testable, maintainable
Cons: Requires building/publishing container image

Option C: ntfy Actions (Limited)

Configure ntfy server with upstream-base-url or actions
Pros: Built into ntfy
Cons: ntfy doesn't natively support Discord webhook format

Recommended Architecture (Option B)

┌─────────────────┐     ┌──────────────────┐     ┌─────────────┐
│ CI/Alertmanager │────▶│      ntfy        │────▶│  ntfy App   │
│ Gatus/Flux      │     │  (notification   │     │  (mobile)   │
└─────────────────┘     │      hub)        │     └─────────────┘
                        └────────┬─────────┘
                                 │ SSE subscribe
                                 ▼
                        ┌──────────────────┐     ┌─────────────┐
                        │ ntfy-discord-    │────▶│  Discord    │
                        │ bridge           │     │  (webhook)  │
                        └──────────────────┘     └─────────────┘

The bridge service would be a new repo (ntfy-discord-bridge) following the same patterns as other Python services in the homelab.

Consequences

Positive

Single source of truth: All notifications flow through ntfy
Auth protection maintained: External ntfy access requires Authentik auth
Flexible routing: Can subscribe to specific topics per destination
Separation of concerns: Discord bridge is independent, can be disabled without affecting ntfy

Negative

Additional service: Discord bridge adds operational overhead
Latency: Two-hop delivery (source → ntfy → Discord) adds minimal latency

Neutral

Topic naming must be documented and followed consistently
Discord webhook URL must be maintained in Vault

Implementation Checklist

Standardize CI notifications to gitea-ci topic
Configure Alertmanager → ntfy for critical/warning alerts
Create ntfy-discord-bridge repository
Implement bridge service with proper error handling
Add ExternalSecret for Discord webhook from Vault
Deploy bridge to observability namespace
Document topic-to-Discord-channel mapping

ADR-0015: CI Notifications and Semantic Versioning
ADR-0020: Internal Registry for CI/CD

5.5 KiB Raw Blame History