Billy D.
aeb1b749be
Replace Python implementation with Go for smaller container images. Uses handler-base Go module for NATS, health, and telemetry. - main.go: pipeline bridge with Argo/Kubeflow HTTP submission - main_test.go: 8 tests covering helpers and HTTP submit functions - Dockerfile: multi-stage golang:1.25-alpine → scratch - CI: Gitea Actions with lint/test/release/docker/notify
44 lines
1.1 KiB
Markdown
44 lines
1.1 KiB
Markdown
# pipeline-bridge
|
|
|
|
Bridges NATS events to Argo Workflows and Kubeflow Pipelines. Subscribes to
|
|
`ai.pipeline.trigger` and submits workflow runs to the appropriate engine,
|
|
publishing status updates to `ai.pipeline.status.{request_id}`.
|
|
|
|
## Supported Pipelines
|
|
|
|
| Name | Engine | Template / Pipeline ID |
|
|
|------|--------|----------------------|
|
|
| document-ingestion | Argo | document-ingestion |
|
|
| batch-inference | Argo | batch-inference |
|
|
| model-evaluation | Argo | model-evaluation |
|
|
| rag-query | Kubeflow | rag-pipeline |
|
|
| voice-pipeline | Kubeflow | voice-pipeline |
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `KUBEFLOW_HOST` | `http://ml-pipeline.kubeflow.svc.cluster.local:8888` | Kubeflow Pipelines API |
|
|
| `ARGO_HOST` | `http://argo-server.argo.svc.cluster.local:2746` | Argo Server API |
|
|
| `ARGO_NAMESPACE` | `ai-ml` | Namespace for Argo workflows |
|
|
|
|
Plus all standard handler-base settings (`NATS_URL`, `OTEL_*`, `HEALTH_PORT`, etc.).
|
|
|
|
## Build
|
|
|
|
```bash
|
|
go build -o pipeline-bridge .
|
|
```
|
|
|
|
## Test
|
|
|
|
```bash
|
|
go test -v -race ./...
|
|
```
|
|
|
|
## Docker
|
|
|
|
```bash
|
|
docker build -t pipeline-bridge .
|
|
```
|