diff --git a/docs/adr/ADR-0022-ntfy-discord-bridge.md b/docs/adr/ADR-0022-ntfy-discord-bridge.md index 4f502ae..7f17afc 100644 --- a/docs/adr/ADR-0022-ntfy-discord-bridge.md +++ b/docs/adr/ADR-0022-ntfy-discord-bridge.md @@ -2,7 +2,7 @@ ## Status -Proposed +Accepted ## Context @@ -18,7 +18,7 @@ ntfy does not natively support Discord webhook format. Discord expects a specifi ### Architecture -A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Discord: +A dedicated Go microservice (`ntfy-discord`) will bridge ntfy to Discord: ``` ┌─────────────────┐ ┌──────────────────┐ ┌─────────────┐ @@ -29,21 +29,28 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc │ SSE/JSON stream ▼ ┌──────────────────┐ ┌─────────────┐ - │ ntfy-discord- │────▶│ Discord │ - │ bridge │ │ Webhook │ + │ ntfy-discord │────▶│ Discord │ + │ (Go) │ │ Webhook │ └──────────────────┘ └─────────────┘ ``` ### Service Design -**Repository**: `ntfy-discord-bridge` +**Repository**: `ntfy-discord` **Technology Stack**: -- Python 3.12+ -- `httpx` for async HTTP (SSE subscription + Discord POST) -- `pydantic` for configuration validation -- `structlog` for structured logging -- Poetry/uv for dependency management +- Go 1.22+ +- `fsnotify` for hot reload of secrets/config +- Standard library `net/http` for SSE subscription +- `slog` for structured logging +- Scratch/distroless base image (~10MB final image) + +**Why Go over Python**: +- **Smaller images**: ~10MB vs ~150MB+ for Python +- **Cloud native**: Single static binary, no runtime dependencies +- **Memory efficient**: Lower RSS, ideal for always-on bridge +- **Concurrency**: Goroutines for SSE handling and webhook delivery +- **Compile-time safety**: Catch errors before deployment **Core Features**: @@ -52,24 +59,61 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc 3. **Message Transformation**: Convert ntfy format to Discord embed format 4. **Priority Mapping**: Map ntfy priorities to Discord embed colors 5. **Topic Routing**: Configure which topics go to which Discord channels/webhooks -6. **Health Endpoint**: `/health` for Kubernetes probes +6. **Hot Reload**: Watch mounted secrets/configmaps with fsnotify, reload without restart +7. **Health Endpoint**: `/health` and `/ready` for Kubernetes probes +8. **Metrics**: Prometheus metrics at `/metrics` + +### Hot Reload Implementation + +Kubernetes mounts secrets as symlinked files that update atomically. The bridge uses `fsnotify` to watch for changes: + +```go +// Watch for secret changes and reload config +func (b *Bridge) watchSecrets(ctx context.Context, secretPath string) { + watcher, _ := fsnotify.NewWatcher() + defer watcher.Close() + + watcher.Add(secretPath) + + for { + select { + case event := <-watcher.Events: + if event.Has(fsnotify.Write) || event.Has(fsnotify.Create) { + slog.Info("secret changed, reloading config") + b.reloadConfig(secretPath) + } + case <-ctx.Done(): + return + } + } +} +``` + +This allows ExternalSecrets to rotate the Discord webhook URL without pod restarts. ### Configuration -Environment variables or ConfigMap: +Configuration via environment variables and mounted secrets: ```yaml -NTFY_URL: "http://ntfy-svc.observability.svc.cluster.local" -DISCORD_WEBHOOK_URL: "${DISCORD_WEBHOOK_URL}" # From Vault via ExternalSecret +# Environment variables (ConfigMap) +NTFY_URL: "http://ntfy.observability.svc.cluster.local" +NTFY_TOPICS: "gitea-ci,alertmanager-alerts,flux-deployments,gatus" +LOG_LEVEL: "info" +METRICS_ENABLED: "true" -# Topic routing (optional - defaults to single webhook) -TOPIC_WEBHOOKS: | - gitea-ci: ${DISCORD_CI_WEBHOOK} - alertmanager-alerts: ${DISCORD_ALERTS_WEBHOOK} - deployments: ${DISCORD_DEPLOYMENTS_WEBHOOK} +# Mounted secret (hot-reloadable) +/secrets/discord-webhook-url # Single webhook for all topics +# OR for topic routing: +/secrets/topic-webhooks.yaml # YAML mapping topics to webhooks +``` -# Topics to subscribe to (comma-separated) -NTFY_TOPICS: "gitea-ci,alertmanager-alerts,deployments,gatus" +Topic routing file (optional): +```yaml +gitea-ci: "https://discord.com/api/webhooks/xxx/ci" +alertmanager-alerts: "https://discord.com/api/webhooks/xxx/alerts" +flux-deployments: "https://discord.com/api/webhooks/xxx/deploys" +default: "https://discord.com/api/webhooks/xxx/general" ``` ### Message Transformation @@ -95,8 +139,7 @@ Discord embed: "description": "ray-serve-apps published to PyPI", "color": 3066993, "fields": [ - {"name": "Topic", "value": "gitea-ci", "inline": true}, - {"name": "Tags", "value": "package", "inline": true} + {"name": "Topic", "value": "gitea-ci", "inline": true} ], "timestamp": "2026-02-02T11:34:51Z", "footer": {"text": "ntfy"} @@ -128,44 +171,57 @@ Common ntfy tags are converted to Discord-friendly emojis in the title: apiVersion: apps/v1 kind: Deployment metadata: - name: ntfy-discord-bridge + name: ntfy-discord namespace: observability spec: replicas: 1 selector: matchLabels: - app: ntfy-discord-bridge + app: ntfy-discord template: + metadata: + labels: + app: ntfy-discord spec: containers: - name: bridge - image: registry.daviestechlabs.io/ntfy-discord-bridge:latest + image: gitea-http.gitea.svc.cluster.local:3000/daviestechlabs/ntfy-discord:latest env: - name: NTFY_URL - value: "http://ntfy-svc.observability.svc.cluster.local" + value: "http://ntfy.observability.svc.cluster.local" - name: NTFY_TOPICS - value: "gitea-ci,alertmanager-alerts,deployments" - - name: DISCORD_WEBHOOK_URL - valueFrom: - secretKeyRef: - name: discord-webhook-secret - key: webhook-url + value: "gitea-ci,alertmanager-alerts,flux-deployments" + - name: SECRETS_PATH + value: "/secrets" ports: - containerPort: 8080 - name: health + name: http + volumeMounts: + - name: discord-secrets + mountPath: /secrets + readOnly: true livenessProbe: httpGet: path: /health - port: health + port: http initialDelaySeconds: 5 periodSeconds: 30 + readinessProbe: + httpGet: + path: /ready + port: http + periodSeconds: 10 resources: limits: - cpu: 100m - memory: 128Mi + cpu: 50m + memory: 32Mi requests: - cpu: 10m - memory: 64Mi + cpu: 5m + memory: 16Mi + volumes: + - name: discord-secrets + secret: + secretName: discord-webhook-secret ``` ### Secret Management @@ -192,28 +248,32 @@ spec: property: webhook_url ``` +When ExternalSecrets refreshes and updates the secret, the bridge detects the file change and reloads without restart. + ### Error Handling 1. **Connection Loss**: Exponential backoff (1s, 2s, 4s, ... max 60s) 2. **Discord Rate Limits**: Respect `Retry-After` header, queue messages 3. **Invalid Messages**: Log and skip, don't crash 4. **Webhook Errors**: Log error, continue processing other messages +5. **Config Reload Errors**: Log error, keep using previous config ## Consequences ### Positive -- **Robust**: Proper reconnection and error handling vs shell script -- **Testable**: Python service with unit tests -- **Observable**: Structured logging, health endpoints -- **Flexible**: Easy to add topic routing, filtering, rate limiting -- **Consistent**: Follows same patterns as other Python services (handler-base, etc.) +- **Tiny footprint**: ~10MB image, 16MB memory +- **Hot reload**: Secrets update without pod restart +- **Robust**: Proper reconnection and error handling +- **Observable**: Structured logging, Prometheus metrics, health endpoints +- **Fast startup**: <100ms cold start +- **Cloud native**: Static binary, distroless image ### Negative +- **Go learning curve**: Different patterns than Python services - **Operational Overhead**: Another service to maintain -- **Build Pipeline**: Requires CI/CD for container image -- **Latency**: Adds ~100ms to notification delivery +- **Latency**: Adds ~50-100ms to notification delivery ### Neutral @@ -222,13 +282,14 @@ spec: ## Implementation Checklist -- [ ] Create `ntfy-discord-bridge` repository -- [ ] Implement core bridge logic with httpx -- [ ] Add reconnection with exponential backoff +- [x] Create `ntfy-discord` repository +- [ ] Implement core bridge logic +- [ ] Add SSE client with reconnection - [ ] Implement message transformation -- [ ] Add health endpoint +- [ ] Add fsnotify hot reload for secrets +- [ ] Add health/ready/metrics endpoints - [ ] Write unit tests -- [ ] Create Dockerfile +- [ ] Create multi-stage Dockerfile (scratch base) - [ ] Set up CI/CD pipeline (Gitea Actions) - [ ] Add ExternalSecret for Discord webhook - [ ] Create Kubernetes manifests