updating to match everything in my homelab.

2026-02-05 16:13:53 -05:00
parent f8787379c5
commit 80fb911e22
30 changed files with 3107 additions and 7 deletions
--- a/decisions/0031-gitea-cicd-strategy.md
+++ b/decisions/0031-gitea-cicd-strategy.md
@@ -0,0 +1,301 @@
+# Gitea CI/CD Pipeline Strategy
+
+* Status: accepted
+* Date: 2026-02-04
+* Deciders: Billy
+* Technical Story: Establish CI/CD patterns for building and publishing container images via Gitea Actions
+
+## Context and Problem Statement
+
+The homelab uses Gitea as the Git hosting platform. Applications need automated CI/CD pipelines to build container images, run tests, and publish artifacts. Gitea Actions provides GitHub Actions-compatible workflow execution.
+
+How do we configure CI/CD pipelines that work reliably with the homelab's self-hosted infrastructure including private container registry, rootless Docker-in-Docker runners, and internal services?
+
+## Decision Drivers
+
+* Self-hosted - no external CI/CD dependencies
+* Container registry integration - push to Gitea's built-in registry
+* Rootless security - runners don't require privileged containers
+* Internal networking - leverage cluster service discovery
+* Semantic versioning - automated version bumps based on commit messages
+
+## Considered Options
+
+1. **Gitea Actions with rootless DinD runners**
+2. **External CI/CD (GitHub Actions, GitLab CI)**
+3. **Self-hosted Jenkins/Drone**
+4. **Tekton Pipelines**
+
+## Decision Outcome
+
+Chosen option: **Option 1 - Gitea Actions with rootless DinD runners**
+
+Gitea Actions provides GitHub Actions compatibility, runs inside the cluster with access to internal services, and supports rootless Docker-in-Docker for secure container builds.
+
+### Positive Consequences
+
+* GitHub Actions syntax familiarity
+* In-cluster access to internal services
+* Built-in container registry integration
+* No external dependencies
+* Rootless execution for security
+
+### Negative Consequences
+
+* Some GitHub Actions may not work (org-specific actions)
+* Rootless DinD has some limitations
+* Self-hosted maintenance burden
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Developer Push                                  │
+└──────────────────────────────────┬──────────────────────────────────────────┘
+                                   │
+                                   ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Gitea Server                                    │
+│                    (git.daviestechlabs.io)                                  │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                        Actions Trigger                               │    │
+│  │  • Push to main branch                                              │    │
+│  │  • Pull request                                                     │    │
+│  │  • Tag creation                                                     │    │
+│  │  • workflow_dispatch                                                │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+└──────────────────────────────────┬──────────────────────────────────────────┘
+                                   │
+                                   ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                         Gitea Actions Runner                                 │
+│                    (rootless Docker-in-Docker)                              │
+│                                                                             │
+│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                     │
+│  │  Checkout   │───▶│   Buildx    │───▶│    Push     │                     │
+│  │             │    │   Build     │    │  Registry   │                     │
+│  └─────────────┘    └─────────────┘    └──────┬──────┘                     │
+│                                               │                             │
+└───────────────────────────────────────────────┼─────────────────────────────┘
+                                                │
+                                                ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                         Gitea Container Registry                             │
+│              (gitea-http.gitea.svc.cluster.local:3000)                      │
+│                                                                             │
+│  Images:                                                                    │
+│  • daviestechlabs/ray-worker-nvidia:v1.0.1                                 │
+│  • daviestechlabs/ray-worker-rdna2:v1.0.1                                  │
+│  • daviestechlabs/ray-worker-strixhalo:v1.0.1                              │
+│  • daviestechlabs/ray-worker-intel:v1.0.1                                  │
+│  • daviestechlabs/ntfy-discord:latest                                      │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Runner Configuration
+
+### Rootless Docker-in-Docker
+
+The runner uses rootless Docker for security:
+
+```yaml
+# Runner deployment uses rootless DinD
+# No privileged containers required
+# No sudo access in workflows
+```
+
+### Runner Registration
+
+Runners must be registered with **project-scoped tokens**, not instance tokens:
+
+1. Go to **Repository → Settings → Actions → Runners**
+2. Create new runner with project token
+3. Use token for runner registration
+
+**Common mistake:** Using instance-level token causes jobs not to be picked up.
+
+## Registry Authentication
+
+### Internal HTTP Endpoint
+
+Use internal cluster DNS for registry access. This avoids:
+- Cloudflare tunnel 100MB upload limit
+- TLS certificate issues
+- External network latency
+
+```yaml
+env:
+  REGISTRY: gitea-http.gitea.svc.cluster.local:3000/daviestechlabs
+  REGISTRY_HOST: gitea-http.gitea.svc.cluster.local:3000
+```
+
+### Buildx Configuration
+
+Configure buildx to use HTTP for internal registry:
+
+```yaml
+- name: Set up Docker Buildx
+  uses: docker/setup-buildx-action@v3
+  with:
+    buildkitd-config-inline: |
+      [registry."gitea-http.gitea.svc.cluster.local:3000"]
+        http = true
+        insecure = true
+```
+
+### Credential Configuration
+
+For rootless DinD, create docker config directly (no `docker login` - it defaults to HTTPS):
+
+```yaml
+- name: Configure Gitea Registry Auth
+  if: github.event_name != 'pull_request'
+  run: |
+    AUTH=$(echo -n "${{ secrets.REGISTRY_USER }}:${{ secrets.REGISTRY_TOKEN }}" | base64 -w0)
+    mkdir -p ~/.docker
+    cat > ~/.docker/config.json << EOF
+    {
+      "auths": {
+        "${{ env.REGISTRY_HOST }}": {
+          "auth": "$AUTH"
+        }
+      }
+    }
+    EOF
+```
+
+**Important:** Buildx reads `~/.docker/config.json` for authentication during push. Do NOT use `docker login` for HTTP registries as it defaults to HTTPS.
+
+### Required Secrets
+
+Configure in **Repository → Settings → Actions → Secrets**:
+
+| Secret | Purpose |
+|--------|---------|
+| `REGISTRY_USER` | Gitea username with package write access |
+| `REGISTRY_TOKEN` | Gitea access token with `write:package` scope |
+| `DOCKERHUB_TOKEN` | (Optional) Docker Hub token for rate limit bypass |
+
+## Semantic Versioning
+
+### Commit Message Conventions
+
+Version bumps are determined from commit message prefixes:
+
+| Prefix | Bump Type | Example |
+|--------|-----------|---------|
+| `major:` or `BREAKING CHANGE` | Major (x.0.0) | `major: Remove deprecated API` |
+| `minor:`, `feat:`, `feature:` | Minor (0.x.0) | `feat: Add new endpoint` |
+| (anything else) | Patch (0.0.x) | `fix: Correct typo` |
+
+### Version Calculation
+
+```yaml
+- name: Calculate semantic version
+  id: version
+  run: |
+    LATEST=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0")
+    VERSION=${LATEST#v}
+    IFS='.' read -r MAJOR MINOR PATCH <<< "$VERSION"
+    
+    MSG="${{ github.event.head_commit.message }}"
+    if echo "$MSG" | grep -qiE "^major:|BREAKING CHANGE"; then
+      MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0
+    elif echo "$MSG" | grep -qiE "^(minor:|feat:|feature:)"; then
+      MINOR=$((MINOR + 1)); PATCH=0
+    else
+      PATCH=$((PATCH + 1))
+    fi
+    
+    echo "version=v${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_OUTPUT
+```
+
+### Automatic Tagging
+
+After successful builds, create and push a git tag:
+
+```yaml
+- name: Create and push tag
+  run: |
+    git config user.name "gitea-actions[bot]"
+    git config user.email "actions@git.daviestechlabs.io"
+    git tag -a "$VERSION" -m "Release $VERSION ($BUMP)"
+    git push origin "$VERSION"
+```
+
+## Notifications
+
+### ntfy Integration
+
+Send build status to ntfy for notifications:
+
+```yaml
+- name: Notify on success
+  run: |
+    curl -s \
+      -H "Title: ✅ Images Built: ${{ gitea.repository }}" \
+      -H "Priority: default" \
+      -H "Tags: white_check_mark,docker" \
+      -d "Version: ${{ needs.determine-version.outputs.version }}" \
+      http://ntfy.observability.svc.cluster.local:80/gitea-ci
+```
+
+## Skip Patterns
+
+### Commit Message Skip Flags
+
+| Flag | Effect |
+|------|--------|
+| `[skip images]` | Skip all image builds |
+| `[ray-serve only]` | Skip worker images |
+| `[skip ci]` | Skip entire workflow |
+
+### Path-based Triggers
+
+Only run on relevant file changes:
+
+```yaml
+on:
+  push:
+    paths:
+      - 'dockerfiles/**'
+      - '.gitea/workflows/build-push.yaml'
+```
+
+## Troubleshooting
+
+### Common Issues
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| Jobs not picked up | Instance token instead of project token | Re-register with project-scoped token |
+| 401 Unauthorized | Missing or wrong registry credentials | Check REGISTRY_USER and REGISTRY_TOKEN secrets |
+| "http: server gave HTTP response to HTTPS client" | Using `docker login` with HTTP registry | Create config.json directly, don't use docker login |
+| Cloudflare 100MB upload limit | Using external endpoint for large images | Use internal HTTP endpoint |
+| TLS certificate error | Using HTTPS with self-signed cert | Use internal HTTP endpoint with buildkitd http=true |
+| sudo not found | Rootless DinD has no sudo | Use user-space configuration methods |
+| "must contain at least one job without dependencies" | All jobs have `needs` | Ensure at least one job has no `needs` clause |
+
+### Debugging
+
+1. Check runner logs in Gitea Actions UI
+2. Add debug output: `echo "::debug::Variable=$VAR"`
+3. Use `actions/debug-output` step for verbose logging
+
+## Workflow Template
+
+See [kuberay-images/.gitea/workflows/build-push.yaml](https://git.daviestechlabs.io/daviestechlabs/kuberay-images/src/branch/main/.gitea/workflows/build-push.yaml) for complete example.
+
+## Future Enhancements
+
+1. **Caching improvements** - Persistent layer cache across builds
+2. **Multi-arch builds** - ARM64 support for Raspberry Pi
+3. **Security scanning** - Trivy integration in CI
+4. **Signed images** - Cosign for image signatures
+5. **SLSA provenance** - Supply chain attestations
+
+## References
+
+* [Gitea Actions Documentation](https://docs.gitea.com/usage/actions/overview)
+* [Docker Buildx Documentation](https://docs.docker.com/build/buildx/)
+* [Semantic Versioning](https://semver.org/)