docs(adr): update ADR-0022 to use Go with hot reload

- Switch from Python to Go for smaller images (~10MB vs ~150MB) - Add fsnotify for hot reload of secrets without pod restart - Update status from Proposed to Accepted - Add Prometheus metrics endpoint - Update resource limits (32Mi vs 128Mi) - Mark repository creation as complete
2026-02-02 17:49:34 -05:00
parent add1b5b71e
commit 5e9f589311
1 changed files with 112 additions and 51 deletions
--- a/docs/adr/ADR-0022-ntfy-discord-bridge.md
+++ b/docs/adr/ADR-0022-ntfy-discord-bridge.md
@@ -2,7 +2,7 @@

 ## Status

-Proposed
+Accepted

 ## Context

@@ -18,7 +18,7 @@ ntfy does not natively support Discord webhook format. Discord expects a specifi

 ### Architecture

-A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Discord:
+A dedicated Go microservice (`ntfy-discord`) will bridge ntfy to Discord:

 ```
 ┌─────────────────┐     ┌──────────────────┐     ┌─────────────┐
@@ -29,21 +29,28 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc
                                 │ SSE/JSON stream
                                 ▼
                        ┌──────────────────┐     ┌─────────────┐
-                        │ ntfy-discord-    │────▶│  Discord    │
-                        │ bridge           │     │  Webhook    │
+                        │  ntfy-discord    │────▶│  Discord    │
+                        │     (Go)         │     │  Webhook    │
                        └──────────────────┘     └─────────────┘
 ```

 ### Service Design

-**Repository**: `ntfy-discord-bridge`
+**Repository**: `ntfy-discord`

 **Technology Stack**:
- Python 3.12+
- `httpx` for async HTTP (SSE subscription + Discord POST)
- `pydantic` for configuration validation
- `structlog` for structured logging
- Poetry/uv for dependency management
+- Go 1.22+
+- `fsnotify` for hot reload of secrets/config
+- Standard library `net/http` for SSE subscription
+- `slog` for structured logging
+- Scratch/distroless base image (~10MB final image)
+
+**Why Go over Python**:
+- **Smaller images**: ~10MB vs ~150MB+ for Python
+- **Cloud native**: Single static binary, no runtime dependencies
+- **Memory efficient**: Lower RSS, ideal for always-on bridge
+- **Concurrency**: Goroutines for SSE handling and webhook delivery
+- **Compile-time safety**: Catch errors before deployment

 **Core Features**:

@@ -52,24 +59,61 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc
 3. **Message Transformation**: Convert ntfy format to Discord embed format
 4. **Priority Mapping**: Map ntfy priorities to Discord embed colors
 5. **Topic Routing**: Configure which topics go to which Discord channels/webhooks
-6. **Health Endpoint**: `/health` for Kubernetes probes
+6. **Hot Reload**: Watch mounted secrets/configmaps with fsnotify, reload without restart
+7. **Health Endpoint**: `/health` and `/ready` for Kubernetes probes
+8. **Metrics**: Prometheus metrics at `/metrics`
+
+### Hot Reload Implementation
+
+Kubernetes mounts secrets as symlinked files that update atomically. The bridge uses `fsnotify` to watch for changes:
+
+```go
+// Watch for secret changes and reload config
+func (b *Bridge) watchSecrets(ctx context.Context, secretPath string) {
+    watcher, _ := fsnotify.NewWatcher()
+    defer watcher.Close()
+    
+    watcher.Add(secretPath)
+    
+    for {
+        select {
+        case event := <-watcher.Events:
+            if event.Has(fsnotify.Write) || event.Has(fsnotify.Create) {
+                slog.Info("secret changed, reloading config")
+                b.reloadConfig(secretPath)
+            }
+        case <-ctx.Done():
+            return
+        }
+    }
+}
+```
+
+This allows ExternalSecrets to rotate the Discord webhook URL without pod restarts.

 ### Configuration

-Environment variables or ConfigMap:
+Configuration via environment variables and mounted secrets:

 ```yaml
-NTFY_URL: "http://ntfy-svc.observability.svc.cluster.local"
-DISCORD_WEBHOOK_URL: "${DISCORD_WEBHOOK_URL}"  # From Vault via ExternalSecret
+# Environment variables (ConfigMap)
+NTFY_URL: "http://ntfy.observability.svc.cluster.local"
+NTFY_TOPICS: "gitea-ci,alertmanager-alerts,flux-deployments,gatus"
+LOG_LEVEL: "info"
+METRICS_ENABLED: "true"

-# Topic routing (optional - defaults to single webhook)
-TOPIC_WEBHOOKS: |
-  gitea-ci: ${DISCORD_CI_WEBHOOK}
-  alertmanager-alerts: ${DISCORD_ALERTS_WEBHOOK}
-  deployments: ${DISCORD_DEPLOYMENTS_WEBHOOK}
+# Mounted secret (hot-reloadable)
+/secrets/discord-webhook-url  # Single webhook for all topics
+# OR for topic routing:
+/secrets/topic-webhooks.yaml  # YAML mapping topics to webhooks
+```

-# Topics to subscribe to (comma-separated)
-NTFY_TOPICS: "gitea-ci,alertmanager-alerts,deployments,gatus"
+Topic routing file (optional):
+```yaml
+gitea-ci: "https://discord.com/api/webhooks/xxx/ci"
+alertmanager-alerts: "https://discord.com/api/webhooks/xxx/alerts"
+flux-deployments: "https://discord.com/api/webhooks/xxx/deploys"
+default: "https://discord.com/api/webhooks/xxx/general"
 ```

 ### Message Transformation
@@ -95,8 +139,7 @@ Discord embed:
    "description": "ray-serve-apps published to PyPI",
    "color": 3066993,
    "fields": [
-      {"name": "Topic", "value": "gitea-ci", "inline": true},
-      {"name": "Tags", "value": "package", "inline": true}
+      {"name": "Topic", "value": "gitea-ci", "inline": true}
    ],
    "timestamp": "2026-02-02T11:34:51Z",
    "footer": {"text": "ntfy"}
@@ -128,44 +171,57 @@ Common ntfy tags are converted to Discord-friendly emojis in the title:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: ntfy-discord-bridge
+  name: ntfy-discord
  namespace: observability
 spec:
  replicas: 1
  selector:
    matchLabels:
-      app: ntfy-discord-bridge
+      app: ntfy-discord
  template:
+    metadata:
+      labels:
+        app: ntfy-discord
    spec:
      containers:
        - name: bridge
-          image: registry.daviestechlabs.io/ntfy-discord-bridge:latest
+          image: gitea-http.gitea.svc.cluster.local:3000/daviestechlabs/ntfy-discord:latest
          env:
            - name: NTFY_URL
-              value: "http://ntfy-svc.observability.svc.cluster.local"
+              value: "http://ntfy.observability.svc.cluster.local"
            - name: NTFY_TOPICS
-              value: "gitea-ci,alertmanager-alerts,deployments"
-            - name: DISCORD_WEBHOOK_URL
-              valueFrom:
-                secretKeyRef:
-                  name: discord-webhook-secret
-                  key: webhook-url
+              value: "gitea-ci,alertmanager-alerts,flux-deployments"
+            - name: SECRETS_PATH
+              value: "/secrets"
          ports:
            - containerPort: 8080
-              name: health
+              name: http
+          volumeMounts:
+            - name: discord-secrets
+              mountPath: /secrets
+              readOnly: true
          livenessProbe:
            httpGet:
              path: /health
-              port: health
+              port: http
            initialDelaySeconds: 5
            periodSeconds: 30
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: http
+            periodSeconds: 10
          resources:
            limits:
-              cpu: 100m
-              memory: 128Mi
+              cpu: 50m
+              memory: 32Mi
            requests:
-              cpu: 10m
-              memory: 64Mi
+              cpu: 5m
+              memory: 16Mi
+      volumes:
+        - name: discord-secrets
+          secret:
+            secretName: discord-webhook-secret
 ```

 ### Secret Management
@@ -192,28 +248,32 @@ spec:
        property: webhook_url
 ```

+When ExternalSecrets refreshes and updates the secret, the bridge detects the file change and reloads without restart.
+
 ### Error Handling

 1. **Connection Loss**: Exponential backoff (1s, 2s, 4s, ... max 60s)
 2. **Discord Rate Limits**: Respect `Retry-After` header, queue messages
 3. **Invalid Messages**: Log and skip, don't crash
 4. **Webhook Errors**: Log error, continue processing other messages
+5. **Config Reload Errors**: Log error, keep using previous config

 ## Consequences

 ### Positive

- **Robust**: Proper reconnection and error handling vs shell script
- **Testable**: Python service with unit tests
- **Observable**: Structured logging, health endpoints
- **Flexible**: Easy to add topic routing, filtering, rate limiting
- **Consistent**: Follows same patterns as other Python services (handler-base, etc.)
+- **Tiny footprint**: ~10MB image, 16MB memory
+- **Hot reload**: Secrets update without pod restart
+- **Robust**: Proper reconnection and error handling
+- **Observable**: Structured logging, Prometheus metrics, health endpoints
+- **Fast startup**: <100ms cold start
+- **Cloud native**: Static binary, distroless image

 ### Negative

+- **Go learning curve**: Different patterns than Python services
 - **Operational Overhead**: Another service to maintain
- **Build Pipeline**: Requires CI/CD for container image
- **Latency**: Adds ~100ms to notification delivery
+- **Latency**: Adds ~50-100ms to notification delivery

 ### Neutral

@@ -222,13 +282,14 @@ spec:

 ## Implementation Checklist

- [ ] Create `ntfy-discord-bridge` repository
- [ ] Implement core bridge logic with httpx
- [ ] Add reconnection with exponential backoff
+- [x] Create `ntfy-discord` repository
+- [ ] Implement core bridge logic
+- [ ] Add SSE client with reconnection
 - [ ] Implement message transformation
- [ ] Add health endpoint
+- [ ] Add fsnotify hot reload for secrets
+- [ ] Add health/ready/metrics endpoints
 - [ ] Write unit tests
- [ ] Create Dockerfile
+- [ ] Create multi-stage Dockerfile (scratch base)
 - [ ] Set up CI/CD pipeline (Gitea Actions)
 - [ ] Add ExternalSecret for Discord webhook
 - [ ] Create Kubernetes manifests