homelab-design/decisions/0031-gitea-cicd-strategy.md

# Gitea CI/CD Pipeline Strategy

* Status: accepted
* Date: 2026-02-04
* Deciders: Billy
* Technical Story: Establish CI/CD patterns for building and publishing container images via Gitea Actions

## Context and Problem Statement

The homelab uses Gitea as the Git hosting platform. Applications need automated CI/CD pipelines to build container images, run tests, and publish artifacts. Gitea Actions provides GitHub Actions-compatible workflow execution.

How do we configure CI/CD pipelines that work reliably with the homelab's self-hosted infrastructure including private container registry, rootless Docker-in-Docker runners, and internal services?

## Decision Drivers

* Self-hosted - no external CI/CD dependencies
* Container registry integration - push to Gitea's built-in registry
* Rootless security - runners don't require privileged containers
* Internal networking - leverage cluster service discovery
* Semantic versioning - automated version bumps based on commit messages

## Considered Options

1. **Gitea Actions with rootless DinD runners**
2. **External CI/CD (GitHub Actions, GitLab CI)**
3. **Self-hosted Jenkins/Drone**
4. **Tekton Pipelines**

## Decision Outcome

Chosen option: **Option 1 - Gitea Actions with rootless DinD runners**

Gitea Actions provides GitHub Actions compatibility, runs inside the cluster with access to internal services, and supports rootless Docker-in-Docker for secure container builds.

### Positive Consequences

* GitHub Actions syntax familiarity
* In-cluster access to internal services
* Built-in container registry integration
* No external dependencies
* Rootless execution for security

### Negative Consequences

* Some GitHub Actions may not work (org-specific actions)
* Rootless DinD has some limitations
* Self-hosted maintenance burden

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Developer Push                                  │
└──────────────────────────────────┬──────────────────────────────────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Gitea Server                                    │
│                    (git.daviestechlabs.io)                                  │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                        Actions Trigger                               │    │
│  │  • Push to main branch                                              │    │
│  │  • Pull request                                                     │    │
│  │  • Tag creation                                                     │    │
│  │  • workflow_dispatch                                                │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└──────────────────────────────────┬──────────────────────────────────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Gitea Actions Runner                                 │
│                    (rootless Docker-in-Docker)                              │
│                                                                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                     │
│  │  Checkout   │───▶│   Buildx    │───▶│    Push     │                     │
│  │             │    │   Build     │    │  Registry   │                     │
│  └─────────────┘    └─────────────┘    └──────┬──────┘                     │
│                                               │                             │
└───────────────────────────────────────────────┼─────────────────────────────┘
                                                │
                                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Gitea Container Registry                             │
│              (gitea-http.gitea.svc.cluster.local:3000)                      │
│                                                                             │
│  Images:                                                                    │
│  • daviestechlabs/ray-worker-nvidia:v1.0.1                                 │
│  • daviestechlabs/ray-worker-rdna2:v1.0.1                                  │
│  • daviestechlabs/ray-worker-strixhalo:v1.0.1                              │
│  • daviestechlabs/ray-worker-intel:v1.0.1                                  │
│  • daviestechlabs/ntfy-discord:latest                                      │
└─────────────────────────────────────────────────────────────────────────────┘
```

## Runner Configuration

### Rootless Docker-in-Docker

The runner uses rootless Docker for security:

```yaml
# Runner deployment uses rootless DinD
# No privileged containers required
# No sudo access in workflows
```

### Runner Registration

Runners must be registered with **project-scoped tokens**, not instance tokens:

1. Go to **Repository → Settings → Actions → Runners**
2. Create new runner with project token
3. Use token for runner registration

**Common mistake:** Using instance-level token causes jobs not to be picked up.

## Registry Authentication

### Internal HTTP Endpoint

Use internal cluster DNS for registry access. This avoids:
- Cloudflare tunnel 100MB upload limit
- TLS certificate issues
- External network latency

```yaml
env:
  REGISTRY: gitea-http.gitea.svc.cluster.local:3000/daviestechlabs
  REGISTRY_HOST: gitea-http.gitea.svc.cluster.local:3000
```

### Buildx Configuration

Configure buildx to use HTTP for internal registry:

```yaml
- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3
  with:
    buildkitd-config-inline: |
      [registry."gitea-http.gitea.svc.cluster.local:3000"]
        http = true
        insecure = true
```

### Credential Configuration

For rootless DinD, create docker config directly (no `docker login` - it defaults to HTTPS):

```yaml
- name: Configure Gitea Registry Auth
  if: github.event_name != 'pull_request'
  run: |
    AUTH=$(echo -n "${{ secrets.REGISTRY_USER }}:${{ secrets.REGISTRY_TOKEN }}" | base64 -w0)
    mkdir -p ~/.docker
    cat > ~/.docker/config.json << EOF
    {
      "auths": {
        "${{ env.REGISTRY_HOST }}": {
          "auth": "$AUTH"
        }
      }
    }
    EOF
```

**Important:** Buildx reads `~/.docker/config.json` for authentication during push. Do NOT use `docker login` for HTTP registries as it defaults to HTTPS.

### Required Secrets

Configure in **Repository → Settings → Actions → Secrets**:

| Secret | Purpose |
|--------|---------|
| `REGISTRY_USER` | Gitea username with package write access |
| `REGISTRY_TOKEN` | Gitea access token with `write:package` scope |
| `DOCKERHUB_TOKEN` | (Optional) Docker Hub token for rate limit bypass |

## Semantic Versioning

### Commit Message Conventions

Version bumps are determined from commit message prefixes:

| Prefix | Bump Type | Example |
|--------|-----------|---------|
| `major:` or `BREAKING CHANGE` | Major (x.0.0) | `major: Remove deprecated API` |
| `minor:`, `feat:`, `feature:` | Minor (0.x.0) | `feat: Add new endpoint` |
| (anything else) | Patch (0.0.x) | `fix: Correct typo` |

### Version Calculation

```yaml
- name: Calculate semantic version
  id: version
  run: |
    LATEST=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0")
    VERSION=${LATEST#v}
    IFS='.' read -r MAJOR MINOR PATCH <<< "$VERSION"
    
    MSG="${{ github.event.head_commit.message }}"
    if echo "$MSG" | grep -qiE "^major:|BREAKING CHANGE"; then
      MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0
    elif echo "$MSG" | grep -qiE "^(minor:|feat:|feature:)"; then
      MINOR=$((MINOR + 1)); PATCH=0
    else
      PATCH=$((PATCH + 1))
    fi
    
    echo "version=v${MAJOR}.${MINOR}.${PATCH}" >> $GITHUB_OUTPUT
```

### Automatic Tagging

After successful builds, create and push a git tag:

```yaml
- name: Create and push tag
  run: |
    git config user.name "gitea-actions[bot]"
    git config user.email "actions@git.daviestechlabs.io"
    git tag -a "$VERSION" -m "Release $VERSION ($BUMP)"
    git push origin "$VERSION"
```

## Notifications

### ntfy Integration

Send build status to ntfy for notifications:

```yaml
- name: Notify on success
  run: |
    curl -s \
      -H "Title: ✅ Images Built: ${{ gitea.repository }}" \
      -H "Priority: default" \
      -H "Tags: white_check_mark,docker" \
      -d "Version: ${{ needs.determine-version.outputs.version }}" \
      http://ntfy.observability.svc.cluster.local:80/gitea-ci
```

## Skip Patterns

### Commit Message Skip Flags

| Flag | Effect |
|------|--------|
| `[skip images]` | Skip all image builds |
| `[ray-serve only]` | Skip worker images |
| `[skip ci]` | Skip entire workflow |

### Path-based Triggers

Only run on relevant file changes:

```yaml
on:
  push:
    paths:
      - 'dockerfiles/**'
      - '.gitea/workflows/build-push.yaml'
```

## Troubleshooting

### Common Issues

| Issue | Cause | Solution |
|-------|-------|----------|
| Jobs not picked up | Instance token instead of project token | Re-register with project-scoped token |
| 401 Unauthorized | Missing or wrong registry credentials | Check REGISTRY_USER and REGISTRY_TOKEN secrets |
| "http: server gave HTTP response to HTTPS client" | Using `docker login` with HTTP registry | Create config.json directly, don't use docker login |
| Cloudflare 100MB upload limit | Using external endpoint for large images | Use internal HTTP endpoint |
| TLS certificate error | Using HTTPS with self-signed cert | Use internal HTTP endpoint with buildkitd http=true |
| sudo not found | Rootless DinD has no sudo | Use user-space configuration methods |
| "must contain at least one job without dependencies" | All jobs have `needs` | Ensure at least one job has no `needs` clause |

### Debugging

1. Check runner logs in Gitea Actions UI
2. Add debug output: `echo "::debug::Variable=$VAR"`
3. Use `actions/debug-output` step for verbose logging

## Workflow Template

See [kuberay-images/.gitea/workflows/build-push.yaml](https://git.daviestechlabs.io/daviestechlabs/kuberay-images/src/branch/main/.gitea/workflows/build-push.yaml) for complete example.

## Build Performance Tuning

GPU worker images are 20-30GB+ due to ROCm/CUDA/PyTorch layers. Several optimizations
are in place to avoid multi-hour rebuild/push cycles on every change.

### Registry-Based BuildKit Cache

Use `type=registry` cache (not `type=gha`, which is a no-op on Gitea runners):

```yaml
cache-from: type=registry,ref=${{ env.REGISTRY }}/image:buildcache
cache-to: type=registry,ref=${{ env.REGISTRY }}/image:buildcache,mode=max,image-manifest=true,compression=zstd
```

- `mode=max` caches all intermediate layers, not just the final image
- `compression=zstd` is faster than gzip with comparable ratios
- Cache is stored in the Gitea container registry alongside images
- Only changed layers are rebuilt and pushed on subsequent builds

### Docker Daemon Tuning

The runner's DinD daemon.json is configured for parallel transfers:

```json
{
  "max-concurrent-uploads": 10,
  "max-concurrent-downloads": 10,
  "features": {
    "containerd-snapshotter": true
  }
}
```

Defaults are only 3 concurrent uploads — insufficient for images with many large layers.

### Persistent DinD Layer Cache

The runner mounts a 100Gi Longhorn PVC at `/home/rootless/.local/share/docker` to
persist Docker's layer cache across pod restarts. Without this, every runner restart
forces re-download of 10-20GB base images (ROCm, Ray, PyTorch).

| Volume | Storage Class | Size | Purpose |
|--------|---------------|------|---------|
| `gitea-runner-data` | nfs-slow | 5Gi | Runner state, workspace |
| `gitea-runner-docker-cache` | longhorn | 100Gi | Docker layer cache |

## Future Enhancements

1. **Multi-arch builds** - ARM64 support for Raspberry Pi
2. **Security scanning** - Trivy integration in CI
3. **Signed images** - Cosign for image signatures
4. **SLSA provenance** - Supply chain attestations

## References

* [Gitea Actions Documentation](https://docs.gitea.com/usage/actions/overview)
* [Docker Buildx Documentation](https://docs.docker.com/build/buildx/)
* [Semantic Versioning](https://semver.org/)