chore: Consolidate ADRs into decisions/ directory
- Added ADR-0016: Affine email verification strategy - Moved ADRs 0019-0024 from docs/adr/ to decisions/ - Renamed to consistent format (removed ADR- prefix)
This commit is contained in:
131
decisions/0021-notification-architecture.md
Normal file
131
decisions/0021-notification-architecture.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# ADR-0021: Notification Architecture
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The homelab infrastructure generates notifications from multiple sources:
|
||||
|
||||
1. **CI/CD pipelines** (Gitea Actions) - build success/failure
|
||||
2. **Alertmanager** - Prometheus alerts for critical/warning conditions
|
||||
3. **Gatus** - Service health monitoring
|
||||
4. **Flux** - GitOps reconciliation events
|
||||
5. **Service readiness** - Notifications when deployments complete successfully
|
||||
|
||||
Currently, ntfy serves as the primary notification hub, but there are several issues:
|
||||
|
||||
- **Topic inconsistency**: CI workflows were posting to `builds` while documentation (ADR-0015) specified `gitea-ci`
|
||||
- **No Alertmanager integration**: Critical Prometheus alerts had no delivery mechanism
|
||||
- **No service readiness notifications**: No visibility when services come online after deployment
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. ntfy as the Notification Hub
|
||||
|
||||
ntfy will serve as the central notification aggregation point. All internal services publish to ntfy topics via the internal Kubernetes service URL:
|
||||
|
||||
```
|
||||
http://ntfy-svc.observability.svc.cluster.local/<topic>
|
||||
```
|
||||
|
||||
This keeps ntfy auth-protected externally while allowing internal services to publish freely.
|
||||
|
||||
### 2. Standardized Topics
|
||||
|
||||
| Topic | Source | Description |
|
||||
|-------|--------|-------------|
|
||||
| `gitea-ci` | Gitea Actions | CI/CD build notifications |
|
||||
| `alertmanager-alerts` | Alertmanager | Prometheus critical/warning alerts |
|
||||
| `gatus` | Gatus | Service health status changes |
|
||||
| `flux` | Flux | GitOps reconciliation events |
|
||||
| `deployments` | Flux/Argo | Service deployment completions |
|
||||
|
||||
### 3. Alertmanager Integration
|
||||
|
||||
Alertmanager is configured to forward alerts to ntfy using the built-in `tpl=alertmanager` template:
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
- name: ntfy-critical
|
||||
webhookConfigs:
|
||||
- url: "http://ntfy-svc.observability.svc.cluster.local/alertmanager-alerts?tpl=alertmanager&priority=urgent&tags=rotating_light"
|
||||
sendResolved: true
|
||||
- name: ntfy-warning
|
||||
webhookConfigs:
|
||||
- url: "http://ntfy-svc.observability.svc.cluster.local/alertmanager-alerts?tpl=alertmanager&priority=high&tags=warning"
|
||||
sendResolved: true
|
||||
```
|
||||
|
||||
Routes direct alerts based on severity:
|
||||
- `severity=critical` → `ntfy-critical` receiver
|
||||
- `severity=warning` → `ntfy-warning` receiver
|
||||
|
||||
### 4. Service Readiness Notifications
|
||||
|
||||
To provide visibility when services are fully operational after deployment:
|
||||
|
||||
**Option A: Flux Notification Controller**
|
||||
Configure Flux's notification-controller to send alerts when Kustomizations/HelmReleases succeed:
|
||||
|
||||
```yaml
|
||||
apiVersion: notification.toolkit.fluxcd.io/v1beta3
|
||||
kind: Provider
|
||||
metadata:
|
||||
name: ntfy-deployments
|
||||
spec:
|
||||
type: generic-hmac # or generic
|
||||
address: http://ntfy-svc.observability.svc.cluster.local/deployments
|
||||
---
|
||||
apiVersion: notification.toolkit.fluxcd.io/v1beta3
|
||||
kind: Alert
|
||||
metadata:
|
||||
name: deployment-success
|
||||
spec:
|
||||
providerRef:
|
||||
name: ntfy-deployments
|
||||
eventSeverity: info
|
||||
eventSources:
|
||||
- kind: Kustomization
|
||||
name: '*'
|
||||
- kind: HelmRelease
|
||||
name: '*'
|
||||
inclusionList:
|
||||
- ".*succeeded.*"
|
||||
```
|
||||
|
||||
**Option B: Argo Workflows Post-Deploy Hook**
|
||||
For Argo-managed deployments, add a notification step at workflow completion.
|
||||
|
||||
**Recommendation**: Use Flux Notification Controller (Option A) as it's already part of the GitOps stack and provides native integration.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **Single source of truth**: All notifications flow through ntfy
|
||||
- **Auth protection maintained**: External ntfy access requires Authentik auth
|
||||
- **Deployment visibility**: Know when services are ready without watching logs
|
||||
- **Consistent topic naming**: All sources follow documented conventions
|
||||
|
||||
### Negative
|
||||
|
||||
- **Configuration overhead**: Each notification source requires explicit configuration
|
||||
|
||||
### Neutral
|
||||
|
||||
- Topic naming must be documented and followed consistently
|
||||
- Future Discord integration addressed in ADR-0022
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] Standardize CI notifications to `gitea-ci` topic
|
||||
- [x] Configure Alertmanager → ntfy for critical/warning alerts
|
||||
- [ ] Configure Flux notification-controller for deployment notifications
|
||||
- [ ] Add `deployments` topic subscription to ntfy app
|
||||
|
||||
## Related
|
||||
|
||||
- ADR-0015: CI Notifications and Semantic Versioning
|
||||
- ADR-0022: ntfy-Discord Bridge Service
|
||||
Reference in New Issue
Block a user