daviestechlabs/homelab-design
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(adr): update ADR-0022 to use Go with hot reload

Browse Source 
- Switch from Python to Go for smaller images (~10MB vs ~150MB)
- Add fsnotify for hot reload of secrets without pod restart
- Update status from Proposed to Accepted
- Add Prometheus metrics endpoint
- Update resource limits (32Mi vs 128Mi)
- Mark repository creation as complete
This commit is contained in:
Billy Davies
2026-02-02 17:49:34 -05:00
parent add1b5b71e
commit 5e9f589311
1 changed files with 112 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

163
docs/adr/ADR-0022-ntfy-discord-bridge.md
View File

@@ -2,7 +2,7 @@
## Status ## Status
Proposed Accepted
## Context ## Context
@@ -18,7 +18,7 @@ ntfy does not natively support Discord webhook format. Discord expects a specifi
### Architecture ### Architecture
A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Discord: A dedicated Go microservice (`ntfy-discord`) will bridge ntfy to Discord:
``` ```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────┐ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────┐
@@ -29,21 +29,28 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc
│ SSE/JSON stream │ SSE/JSON stream
▼ ▼
┌──────────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌─────────────┐
│ ntfy-discord- │────▶│ Discord │ │ ntfy-discord │────▶│ Discord │
│ bridge │ │ Webhook │ │ (Go) │ │ Webhook │
└──────────────────┘ └─────────────┘ └──────────────────┘ └─────────────┘
``` ```
### Service Design ### Service Design
**Repository**: `ntfy-discord-bridge` **Repository**: `ntfy-discord`
**Technology Stack**: **Technology Stack**:
- Python 3.12+ - Go 1.22+
- `httpx` for async HTTP (SSE subscription + Discord POST) - `fsnotify` for hot reload of secrets/config
- `pydantic` for configuration validation - Standard library `net/http` for SSE subscription
- `structlog` for structured logging - `slog` for structured logging
- Poetry/uv for dependency management - Scratch/distroless base image (~10MB final image)
**Why Go over Python**:
- **Smaller images**: ~10MB vs ~150MB+ for Python
- **Cloud native**: Single static binary, no runtime dependencies
- **Memory efficient**: Lower RSS, ideal for always-on bridge
- **Concurrency**: Goroutines for SSE handling and webhook delivery
- **Compile-time safety**: Catch errors before deployment
**Core Features**: **Core Features**:
@@ -52,24 +59,61 @@ A dedicated Python microservice (`ntfy-discord-bridge`) will bridge ntfy to Disc
3. **Message Transformation**: Convert ntfy format to Discord embed format 3. **Message Transformation**: Convert ntfy format to Discord embed format
4. **Priority Mapping**: Map ntfy priorities to Discord embed colors 4. **Priority Mapping**: Map ntfy priorities to Discord embed colors
5. **Topic Routing**: Configure which topics go to which Discord channels/webhooks 5. **Topic Routing**: Configure which topics go to which Discord channels/webhooks
6. **Health Endpoint**: `/health` for Kubernetes probes 6. **Hot Reload**: Watch mounted secrets/configmaps with fsnotify, reload without restart
7. **Health Endpoint**: `/health` and `/ready` for Kubernetes probes
8. **Metrics**: Prometheus metrics at `/metrics`
### Hot Reload Implementation
Kubernetes mounts secrets as symlinked files that update atomically. The bridge uses `fsnotify` to watch for changes:
```go
// Watch for secret changes and reload config
func (b *Bridge) watchSecrets(ctx context.Context, secretPath string) {
watcher, _ := fsnotify.NewWatcher()
defer watcher.Close()
watcher.Add(secretPath)
for {
select {
case event := <-watcher.Events:
if event.Has(fsnotify.Write) || event.Has(fsnotify.Create) {
slog.Info("secret changed, reloading config")
b.reloadConfig(secretPath)
}
case <-ctx.Done():
return
}
}
}
```
This allows ExternalSecrets to rotate the Discord webhook URL without pod restarts.
### Configuration ### Configuration
Environment variables or ConfigMap: Configuration via environment variables and mounted secrets:
```yaml ```yaml
NTFY_URL: "http://ntfy-svc.observability.svc.cluster.local" # Environment variables (ConfigMap)
DISCORD_WEBHOOK_URL: "${DISCORD_WEBHOOK_URL}" # From Vault via ExternalSecret NTFY_URL: "http://ntfy.observability.svc.cluster.local"
NTFY_TOPICS: "gitea-ci,alertmanager-alerts,flux-deployments,gatus"
LOG_LEVEL: "info"
METRICS_ENABLED: "true"
# Topic routing (optional - defaults to single webhook) # Mounted secret (hot-reloadable)
TOPIC_WEBHOOKS: | /secrets/discord-webhook-url # Single webhook for all topics
gitea-ci: ${DISCORD_CI_WEBHOOK} # OR for topic routing:
alertmanager-alerts: ${DISCORD_ALERTS_WEBHOOK} /secrets/topic-webhooks.yaml # YAML mapping topics to webhooks
deployments: ${DISCORD_DEPLOYMENTS_WEBHOOK} ```
# Topics to subscribe to (comma-separated) Topic routing file (optional):
NTFY_TOPICS: "gitea-ci,alertmanager-alerts,deployments,gatus" ```yaml
gitea-ci: "https://discord.com/api/webhooks/xxx/ci"
alertmanager-alerts: "https://discord.com/api/webhooks/xxx/alerts"
flux-deployments: "https://discord.com/api/webhooks/xxx/deploys"
default: "https://discord.com/api/webhooks/xxx/general"
``` ```
### Message Transformation ### Message Transformation
@@ -95,8 +139,7 @@ Discord embed:
"description": "ray-serve-apps published to PyPI", "description": "ray-serve-apps published to PyPI",
"color": 3066993, "color": 3066993,
"fields": [ "fields": [
{"name": "Topic", "value": "gitea-ci", "inline": true}, {"name": "Topic", "value": "gitea-ci", "inline": true}
{"name": "Tags", "value": "package", "inline": true}
], ],
"timestamp": "2026-02-02T11:34:51Z", "timestamp": "2026-02-02T11:34:51Z",
"footer": {"text": "ntfy"} "footer": {"text": "ntfy"}
@@ -128,44 +171,57 @@ Common ntfy tags are converted to Discord-friendly emojis in the title:
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: ntfy-discord-bridge name: ntfy-discord
namespace: observability namespace: observability
spec: spec:
replicas: 1 replicas: 1
selector: selector:
matchLabels: matchLabels:
app: ntfy-discord-bridge app: ntfy-discord
template: template:
metadata:
labels:
app: ntfy-discord
spec: spec:
containers: containers:
- name: bridge - name: bridge
image: registry.daviestechlabs.io/ntfy-discord-bridge:latest image: gitea-http.gitea.svc.cluster.local:3000/daviestechlabs/ntfy-discord:latest
env: env:
- name: NTFY_URL - name: NTFY_URL
value: "http://ntfy-svc.observability.svc.cluster.local" value: "http://ntfy.observability.svc.cluster.local"
- name: NTFY_TOPICS - name: NTFY_TOPICS
value: "gitea-ci,alertmanager-alerts,deployments" value: "gitea-ci,alertmanager-alerts,flux-deployments"
- name: DISCORD_WEBHOOK_URL - name: SECRETS_PATH
valueFrom: value: "/secrets"
secretKeyRef:
name: discord-webhook-secret
key: webhook-url
ports: ports:
- containerPort: 8080 - containerPort: 8080
name: health name: http
volumeMounts:
- name: discord-secrets
mountPath: /secrets
readOnly: true
livenessProbe: livenessProbe:
httpGet: httpGet:
path: /health path: /health
port: health port: http
initialDelaySeconds: 5 initialDelaySeconds: 5
periodSeconds: 30 periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: http
periodSeconds: 10
resources: resources:
limits: limits:
cpu: 100m cpu: 50m
memory: 128Mi memory: 32Mi
requests: requests:
cpu: 10m cpu: 5m
memory: 64Mi memory: 16Mi
volumes:
- name: discord-secrets
secret:
secretName: discord-webhook-secret
``` ```
### Secret Management ### Secret Management
@@ -192,28 +248,32 @@ spec:
property: webhook_url property: webhook_url
``` ```
When ExternalSecrets refreshes and updates the secret, the bridge detects the file change and reloads without restart.
### Error Handling ### Error Handling
1. **Connection Loss**: Exponential backoff (1s, 2s, 4s, ... max 60s) 1. **Connection Loss**: Exponential backoff (1s, 2s, 4s, ... max 60s)
2. **Discord Rate Limits**: Respect `Retry-After` header, queue messages 2. **Discord Rate Limits**: Respect `Retry-After` header, queue messages
3. **Invalid Messages**: Log and skip, don't crash 3. **Invalid Messages**: Log and skip, don't crash
4. **Webhook Errors**: Log error, continue processing other messages 4. **Webhook Errors**: Log error, continue processing other messages
5. **Config Reload Errors**: Log error, keep using previous config
## Consequences ## Consequences
### Positive ### Positive
- **Robust**: Proper reconnection and error handling vs shell script - **Tiny footprint**: ~10MB image, 16MB memory
- **Testable**: Python service with unit tests - **Hot reload**: Secrets update without pod restart
- **Observable**: Structured logging, health endpoints - **Robust**: Proper reconnection and error handling
- **Flexible**: Easy to add topic routing, filtering, rate limiting - **Observable**: Structured logging, Prometheus metrics, health endpoints
- **Consistent**: Follows same patterns as other Python services (handler-base, etc.) - **Fast startup**: <100ms cold start
- **Cloud native**: Static binary, distroless image
### Negative ### Negative
- **Go learning curve**: Different patterns than Python services
- **Operational Overhead**: Another service to maintain - **Operational Overhead**: Another service to maintain
- **Build Pipeline**: Requires CI/CD for container image - **Latency**: Adds ~50-100ms to notification delivery
- **Latency**: Adds ~100ms to notification delivery
### Neutral ### Neutral
@@ -222,13 +282,14 @@ spec:
## Implementation Checklist ## Implementation Checklist
- [ ] Create `ntfy-discord-bridge` repository - [x] Create `ntfy-discord` repository
- [ ] Implement core bridge logic with httpx - [ ] Implement core bridge logic
- [ ] Add reconnection with exponential backoff - [ ] Add SSE client with reconnection
- [ ] Implement message transformation - [ ] Implement message transformation
- [ ] Add health endpoint - [ ] Add fsnotify hot reload for secrets
- [ ] Add health/ready/metrics endpoints
- [ ] Write unit tests - [ ] Write unit tests
- [ ] Create Dockerfile - [ ] Create multi-stage Dockerfile (scratch base)
- [ ] Set up CI/CD pipeline (Gitea Actions) - [ ] Set up CI/CD pipeline (Gitea Actions)
- [ ] Add ExternalSecret for Discord webhook - [ ] Add ExternalSecret for Discord webhook
- [ ] Create Kubernetes manifests - [ ] Create Kubernetes manifests