Kubeflow Pipelines

Kubeflow Pipeline definitions for the DaviesTechLabs AI/ML platform.

Pipelines

Pipeline	Description	Output
voice_pipeline.py	STT → RAG → LLM → TTS	voice_pipeline.yaml
document_ingestion_pipeline.py	Ingest docs → chunk → embed → Milvus	document_ingestion.yaml
document_ingestion_mlflow_pipeline.py	Same with MLflow tracking	document_ingestion_mlflow.yaml
evaluation_pipeline.py	Evaluate models against benchmarks	evaluation.yaml

Usage

Compile Pipelines

pip install kfp==2.12.1

# Compile all
python voice_pipeline.py
python document_ingestion_pipeline.py
python evaluation_pipeline.py

# Or compile individually
python -c "from kfp import compiler; from voice_pipeline import voice_assistant_pipeline; compiler.Compiler().compile(voice_assistant_pipeline, 'voice.yaml')"

Upload to Kubeflow

Upload the generated .yaml files to Kubeflow Pipelines UI or use the SDK:

import kfp
client = kfp.Client(host='http://kubeflow.ai-ml.svc.cluster.local/pipeline')
client.upload_pipeline('voice_pipeline.yaml', pipeline_name='Voice Assistant')

Pipeline Details

voice_pipeline

Full voice assistant with RAG:

1. Transcribe audio (Whisper)
2. Generate embeddings (BGE)
3. Search Milvus
4. Rerank documents (BGE Reranker)
5. Generate response (vLLM)
6. Synthesize speech (XTTS)

document_ingestion_pipeline

Ingest documents into vector DB:

1. Extract text (PDF, DOCX, HTML, TXT)
2. Chunk with overlap (tiktoken)
3. Generate embeddings
4. Store in Milvus collection

evaluation_pipeline

Benchmark model quality:

1. Load eval dataset (MMLU, etc.)
2. Run inference
3. Calculate metrics (accuracy, F1)
4. Log to MLflow

Integration

kfp-sync-job.yaml

Kubernetes Job to sync compiled pipelines to Kubeflow:

kubectl apply -f kfp-sync-job.yaml

From Argo Workflows

Pipelines can be triggered from Argo via the kfp-integration workflow in the argo repo.

Service Endpoints

Service	Endpoint
Whisper STT	http://whisper-predictor.ai-ml.svc.cluster.local
Embeddings	http://embeddings-predictor.ai-ml.svc.cluster.local
Reranker	http://reranker-predictor.ai-ml.svc.cluster.local
vLLM	http://llm-draft.ai-ml.svc.cluster.local:8000
TTS	http://tts-predictor.ai-ml.svc.cluster.local
Milvus	milvus.ai-ml.svc.cluster.local:19530

Related

• argo - Argo Workflows for training
• voice-assistant - Real-time voice handler
• homelab-design - Architecture docs