196 lines
5.6 KiB
Markdown
196 lines
5.6 KiB
Markdown
# ARM64 Raspberry Pi Worker Node Strategy
|
|
|
|
* Status: accepted
|
|
* Date: 2026-02-05
|
|
* Deciders: Billy
|
|
* Technical Story: Integrate Raspberry Pi nodes into the Kubernetes cluster
|
|
|
|
## Context and Problem Statement
|
|
|
|
The homelab cluster includes 5 Raspberry Pi 4/5 nodes (ARM64 architecture) alongside x86_64 servers. These low-power nodes provide:
|
|
- Additional compute capacity for lightweight workloads
|
|
- Geographic distribution within the home network
|
|
- Learning platform for multi-architecture Kubernetes
|
|
|
|
However, ARM64 nodes have constraints:
|
|
- No GPU acceleration
|
|
- Lower CPU/memory than x86_64 servers
|
|
- Some container images lack ARM64 support
|
|
- Limited local storage
|
|
|
|
How do we effectively integrate ARM64 nodes while avoiding scheduling failures?
|
|
|
|
## Decision Drivers
|
|
|
|
* Maximize utilization of ARM64 compute
|
|
* Prevent ARM-incompatible workloads from scheduling
|
|
* Maintain cluster stability
|
|
* Support multi-arch container images
|
|
* Minimize operational overhead
|
|
|
|
## Considered Options
|
|
|
|
1. **Node labels + affinity for workload placement**
|
|
2. **Separate ARM64-only namespace**
|
|
3. **Taints to exclude from general scheduling**
|
|
4. **ARM64 nodes for specific workload types only**
|
|
|
|
## Decision Outcome
|
|
|
|
Chosen option: **Option 1 + Option 4 hybrid** - Use node labels with affinity rules, and designate ARM64 nodes for specific workload categories.
|
|
|
|
ARM64 nodes handle:
|
|
- Lightweight control plane components (where multi-arch images exist)
|
|
- Velero node-agent (backup DaemonSet)
|
|
- Node-level monitoring (Prometheus node-exporter)
|
|
- Future: Edge/IoT workloads
|
|
|
|
### Positive Consequences
|
|
|
|
* Clear workload segmentation
|
|
* No scheduling failures from arch mismatch
|
|
* Efficient use of low-power nodes
|
|
* Room for future ARM-specific workloads
|
|
* Cost-effective cluster expansion
|
|
|
|
### Negative Consequences
|
|
|
|
* Some nodes may be underutilized
|
|
* Must maintain multi-arch image awareness
|
|
* Additional scheduling complexity
|
|
|
|
## Cluster Composition
|
|
|
|
| Node | Architecture | Role | Instance Type |
|
|
|------|--------------|------|---------------|
|
|
| bruenor | amd64 | control-plane | - |
|
|
| catti | amd64 | control-plane | - |
|
|
| storm | amd64 | control-plane | - |
|
|
| khelben | amd64 | GPU worker (Strix Halo) | - |
|
|
| elminster | amd64 | GPU worker (NVIDIA) | - |
|
|
| drizzt | amd64 | GPU worker (RDNA2) | - |
|
|
| danilo | amd64 | GPU worker (Intel Arc) | - |
|
|
| regis | amd64 | worker | - |
|
|
| wulfgar | amd64 | worker | - |
|
|
| **durnan** | **arm64** | worker | raspberry-pi |
|
|
| **elaith** | **arm64** | worker | raspberry-pi |
|
|
| **jarlaxle** | **arm64** | worker | raspberry-pi |
|
|
| **mirt** | **arm64** | worker | raspberry-pi |
|
|
| **volo** | **arm64** | worker | raspberry-pi |
|
|
|
|
## Node Labels
|
|
|
|
```yaml
|
|
# Applied via Talos machine config or kubectl
|
|
labels:
|
|
kubernetes.io/arch: arm64
|
|
kubernetes.io/os: linux
|
|
node.kubernetes.io/instance-type: raspberry-pi
|
|
kubernetes.io/storage: none # No Longhorn on Pis
|
|
```
|
|
|
|
## Workload Placement
|
|
|
|
### DaemonSets (Run Everywhere)
|
|
|
|
These run on all nodes including ARM64:
|
|
|
|
| DaemonSet | Namespace | Multi-arch |
|
|
|-----------|-----------|------------|
|
|
| velero-node-agent | velero | ✅ |
|
|
| cilium-agent | kube-system | ✅ |
|
|
| node-exporter | observability | ✅ |
|
|
|
|
### ARM64-Excluded Workloads
|
|
|
|
These explicitly exclude ARM64 via node affinity:
|
|
|
|
```yaml
|
|
spec:
|
|
affinity:
|
|
nodeAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
nodeSelectorTerms:
|
|
- matchExpressions:
|
|
- key: kubernetes.io/arch
|
|
operator: In
|
|
values:
|
|
- amd64
|
|
```
|
|
|
|
| Workload Type | Reason for Exclusion |
|
|
|---------------|----------------------|
|
|
| GPU workloads | No GPU on Pis |
|
|
| Longhorn | Pis have no storage label |
|
|
| Heavy databases | Insufficient resources |
|
|
| Most HelmReleases | Image compatibility |
|
|
|
|
### ARM64-Compatible Light Workloads
|
|
|
|
Potential future workloads for ARM64 nodes:
|
|
|
|
| Workload | Use Case |
|
|
|----------|----------|
|
|
| MQTT broker | IoT message routing |
|
|
| Pi-hole | DNS ad blocking |
|
|
| Home Assistant | Home automation |
|
|
| Lightweight proxies | Traffic routing |
|
|
|
|
## Storage Exclusion
|
|
|
|
ARM64 nodes are excluded from Longhorn:
|
|
|
|
```yaml
|
|
# Longhorn Helm values
|
|
defaultSettings:
|
|
systemManagedComponentsNodeSelector: "kubernetes.io/arch:amd64"
|
|
```
|
|
|
|
Node label:
|
|
```yaml
|
|
kubernetes.io/storage: none
|
|
```
|
|
|
|
## Resource Constraints
|
|
|
|
| Node Type | CPU | Memory | Typical Available |
|
|
|-----------|-----|--------|-------------------|
|
|
| Raspberry Pi 4 | 4 cores | 4-8GB | 3 cores, 3GB |
|
|
| Raspberry Pi 5 | 4 cores | 8GB | 3.5 cores, 6GB |
|
|
|
|
## Multi-Architecture Image Strategy
|
|
|
|
For workloads that should run on ARM64:
|
|
|
|
1. **Use multi-arch base images** (e.g., `alpine`, `debian`)
|
|
2. **Build with Docker buildx**:
|
|
```bash
|
|
docker buildx build --platform linux/amd64,linux/arm64 -t myimage:latest .
|
|
```
|
|
3. **Verify arch support** before deployment
|
|
|
|
## Monitoring ARM64 Nodes
|
|
|
|
```promql
|
|
# Node resource usage by architecture
|
|
sum by (node, arch) (
|
|
node_memory_MemAvailable_bytes{}
|
|
* on(node) group_left(arch)
|
|
kube_node_labels{label_kubernetes_io_arch!=""}
|
|
)
|
|
```
|
|
|
|
## Future Considerations
|
|
|
|
- **Edge workloads**: ARM64 nodes ideal for edge compute patterns
|
|
- **IoT integration**: MQTT, sensor data collection
|
|
- **Scale-out**: Add more Pis for lightweight workload capacity
|
|
- **ARM64 ML inference**: Some models support ARM (TensorFlow Lite)
|
|
|
|
## Links
|
|
|
|
* [Kubernetes Multi-Architecture](https://kubernetes.io/docs/concepts/containers/images/#multi-architecture-images)
|
|
* [Talos on Raspberry Pi](https://talos.dev/v1.12/talos-guides/install/single-board-computers/rpi_generic/)
|
|
* Related: [ADR-0002](0002-use-talos-linux.md) - Use Talos Linux
|
|
* Related: [ADR-0026](0026-storage-strategy.md) - Storage Strategy
|