daviestechlabs/pipeline-bridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rewrite pipeline-bridge in Go

Browse Source 
Replace Python implementation with Go for smaller container images.
Uses handler-base Go module for NATS, health, and telemetry.

- main.go: pipeline bridge with Argo/Kubeflow HTTP submission
- main_test.go: 8 tests covering helpers and HTTP submit functions
- Dockerfile: multi-stage golang:1.25-alpine → scratch
- CI: Gitea Actions with lint/test/release/docker/notify
This commit is contained in:
Billy Davies
2026-02-19 17:52:31 -05:00
parent da6e96b9f6
commit aeb1b749be
14 changed files with 769 additions and 3433 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
README.md
View File

@@ -1,105 +1,43 @@
# Pipeline Bridge
# pipeline-bridge
Bridges NATS events to Kubeflow Pipelines and Argo Workflows.
## Overview
The Pipeline Bridge listens for pipeline trigger requests on NATS and submits them to the appropriate workflow engine (Argo Workflows or Kubeflow Pipelines). It monitors execution and publishes status updates back to NATS.
## NATS Subjects
| Subject | Direction | Description |
|---------|-----------|-------------|
| `ai.pipeline.trigger` | Subscribe | Pipeline trigger requests |
| `ai.pipeline.status.{request_id}` | Publish | Pipeline status updates |
Bridges NATS events to Argo Workflows and Kubeflow Pipelines. Subscribes to
`ai.pipeline.trigger` and submits workflow runs to the appropriate engine,
publishing status updates to `ai.pipeline.status.{request_id}`.
## Supported Pipelines
| Pipeline | Engine | Description |
|----------|--------|-------------|
| `document-ingestion` | Argo | Ingest documents into Milvus |
| `batch-inference` | Argo | Run batch LLM inference |
| `model-evaluation` | Argo | Evaluate model performance |
| `rag-query` | Kubeflow | Execute RAG query pipeline |
| `voice-pipeline` | Kubeflow | Full voice assistant pipeline |
| Name | Engine | Template / Pipeline ID |
|------|--------|----------------------|
| document-ingestion | Argo | document-ingestion |
| batch-inference | Argo | batch-inference |
| model-evaluation | Argo | model-evaluation |
| rag-query | Kubeflow | rag-pipeline |
| voice-pipeline | Kubeflow | voice-pipeline |
## Request Format
```json
{
"request_id": "uuid",
"pipeline": "document-ingestion",
"parameters": {
"source-url": "s3://bucket/docs/",
"collection-name": "knowledge_base"
}
}
```
## Response Format
```json
{
"request_id": "uuid",
"status": "submitted",
"pipeline": "document-ingestion",
"engine": "argo",
"run_id": "document-ingestion-abc123",
"message": "Pipeline submitted successfully",
"timestamp": "2026-01-03T12:00:00Z"
}
```
## Status Updates
The bridge publishes status updates as the workflow progresses:
- `submitted` - Workflow created
- `pending` - Waiting to start
- `running` - In progress
- `succeeded` - Completed successfully
- `failed` - Failed
- `error` - System error
## Implementation
The pipeline bridge uses the [handler-base](https://git.daviestechlabs.io/daviestechlabs/handler-base) library for standardized NATS handling, telemetry, and health checks.
## Environment Variables
## Configuration
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://nats.ai-ml.svc.cluster.local:4222` | NATS server URL |
| `KUBEFLOW_HOST` | `http://ml-pipeline.kubeflow.svc.cluster.local:8888` | Kubeflow Pipelines API |
| `ARGO_HOST` | `http://argo-server.argo.svc.cluster.local:2746` | Argo Workflows API |
| `ARGO_NAMESPACE` | `ai-ml` | Namespace for Argo Workflows |
| `ARGO_HOST` | `http://argo-server.argo.svc.cluster.local:2746` | Argo Server API |
| `ARGO_NAMESPACE` | `ai-ml` | Namespace for Argo workflows |
## Building
Plus all standard handler-base settings (`NATS_URL`, `OTEL_*`, `HEALTH_PORT`, etc.).
## Build
```bash
docker build -t pipeline-bridge:latest .
# With specific handler-base tag
docker build --build-arg BASE_TAG=latest -t pipeline-bridge:latest .
go build -o pipeline-bridge .
```
## Testing
## Test
```bash
# Port-forward NATS
kubectl port-forward -n ai-ml svc/nats 4222:4222
# Trigger document ingestion
nats pub ai.pipeline.trigger '{
"request_id": "test-1",
"pipeline": "document-ingestion",
"parameters": {"source-url": "https://example.com/docs.txt"}
}'
# Monitor status
nats sub "ai.pipeline.status.>"
go test -v -race ./...
```
## License
## Docker
MIT
```bash
docker build -t pipeline-bridge .
```