Billy D.
c26e4e5ef0
- voice_pipeline: STT → RAG → LLM → TTS - document_ingestion_pipeline: Extract → Chunk → Embed → Milvus - document_ingestion_mlflow_pipeline: With MLflow tracking - evaluation_pipeline: Model benchmarking - kfp-sync-job: K8s job to sync pipelines
98 lines
2.8 KiB
Markdown
98 lines
2.8 KiB
Markdown
# Kubeflow Pipelines
|
|
|
|
Kubeflow Pipeline definitions for the DaviesTechLabs AI/ML platform.
|
|
|
|
## Pipelines
|
|
|
|
| Pipeline | Description | Output |
|
|
|----------|-------------|--------|
|
|
| `voice_pipeline.py` | STT → RAG → LLM → TTS | voice_pipeline.yaml |
|
|
| `document_ingestion_pipeline.py` | Ingest docs → chunk → embed → Milvus | document_ingestion.yaml |
|
|
| `document_ingestion_mlflow_pipeline.py` | Same with MLflow tracking | document_ingestion_mlflow.yaml |
|
|
| `evaluation_pipeline.py` | Evaluate models against benchmarks | evaluation.yaml |
|
|
|
|
## Usage
|
|
|
|
### Compile Pipelines
|
|
|
|
```bash
|
|
pip install kfp==2.12.1
|
|
|
|
# Compile all
|
|
python voice_pipeline.py
|
|
python document_ingestion_pipeline.py
|
|
python evaluation_pipeline.py
|
|
|
|
# Or compile individually
|
|
python -c "from kfp import compiler; from voice_pipeline import voice_assistant_pipeline; compiler.Compiler().compile(voice_assistant_pipeline, 'voice.yaml')"
|
|
```
|
|
|
|
### Upload to Kubeflow
|
|
|
|
Upload the generated `.yaml` files to Kubeflow Pipelines UI or use the SDK:
|
|
|
|
```python
|
|
import kfp
|
|
client = kfp.Client(host='http://kubeflow.ai-ml.svc.cluster.local/pipeline')
|
|
client.upload_pipeline('voice_pipeline.yaml', pipeline_name='Voice Assistant')
|
|
```
|
|
|
|
## Pipeline Details
|
|
|
|
### voice_pipeline
|
|
|
|
Full voice assistant with RAG:
|
|
1. Transcribe audio (Whisper)
|
|
2. Generate embeddings (BGE)
|
|
3. Search Milvus
|
|
4. Rerank documents (BGE Reranker)
|
|
5. Generate response (vLLM)
|
|
6. Synthesize speech (XTTS)
|
|
|
|
### document_ingestion_pipeline
|
|
|
|
Ingest documents into vector DB:
|
|
1. Extract text (PDF, DOCX, HTML, TXT)
|
|
2. Chunk with overlap (tiktoken)
|
|
3. Generate embeddings
|
|
4. Store in Milvus collection
|
|
|
|
### evaluation_pipeline
|
|
|
|
Benchmark model quality:
|
|
1. Load eval dataset (MMLU, etc.)
|
|
2. Run inference
|
|
3. Calculate metrics (accuracy, F1)
|
|
4. Log to MLflow
|
|
|
|
## Integration
|
|
|
|
### kfp-sync-job.yaml
|
|
|
|
Kubernetes Job to sync compiled pipelines to Kubeflow:
|
|
|
|
```bash
|
|
kubectl apply -f kfp-sync-job.yaml
|
|
```
|
|
|
|
### From Argo Workflows
|
|
|
|
Pipelines can be triggered from Argo via the kfp-integration workflow in the [argo](https://git.daviestechlabs.io/daviestechlabs/argo) repo.
|
|
|
|
## Service Endpoints
|
|
|
|
| Service | Endpoint |
|
|
|---------|----------|
|
|
| Whisper STT | `http://whisper-predictor.ai-ml.svc.cluster.local` |
|
|
| Embeddings | `http://embeddings-predictor.ai-ml.svc.cluster.local` |
|
|
| Reranker | `http://reranker-predictor.ai-ml.svc.cluster.local` |
|
|
| vLLM | `http://llm-draft.ai-ml.svc.cluster.local:8000` |
|
|
| TTS | `http://tts-predictor.ai-ml.svc.cluster.local` |
|
|
| Milvus | `milvus.ai-ml.svc.cluster.local:19530` |
|
|
|
|
## Related
|
|
|
|
- [argo](https://git.daviestechlabs.io/daviestechlabs/argo) - Argo Workflows for training
|
|
- [voice-assistant](https://git.daviestechlabs.io/daviestechlabs/voice-assistant) - Real-time voice handler
|
|
- [homelab-design](https://git.daviestechlabs.io/daviestechlabs/homelab-design) - Architecture docs
|