Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Comparative analysis of Hosted API vs. self-hosted advantages.
• Architectures for on-premises, edge, and air-gapped systems.
• Evaluating single-model vs. multi-model portfolio approaches.
• Cost-effectiveness analysis per decision unit.

Hardware Sizing and Capacity Planning

• Correlating memory footprint with parameter counts.
• Trade-offs between CPU-only execution and GPU acceleration.
• Planning for concurrency and throughput requirements.
• Reserving headroom for increasing task complexity.

Serving Technology Stacks

• Evaluating vLLM and SGLang capabilities.
• Utilizing llama.cpp and GGUF for execution efficiency.
• Leveraging Ollama for streamlined local development.
• Selecting stacks based on decision workload characteristics.

Quantization and Calibration Preservation

• Exploring precision formats and associated memory savings.
• Assessing quality degradation and calibration loss.
• Implementing per-tensor and importance-aware quantization methods.
• Conducting acceptance testing following quantization.

Latency Optimization

• Distincting between median and tail latency impacts.
• Managing latency fluctuations with varying context lengths.
• Optimizing warm starts and model caching strategies.
• Batching multiple decision queries for efficiency.

Edge and Resource-Constrained Environments

• Running small decision models on CPU-only hardware.
• Deployment strategies for Apple Silicon and embedded systems.
• Ensuring offline functionality and robust update protocols.
• Managing weight distribution in air-gapped settings.

Containerization and Scalability

• Designing optimized images for model serving.
• Managing volumes and cache configurations.
• Implementing horizontal scaling and intelligent routing.
• Defining resource limits and GPU sharing policies.

Observability and Monitoring

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions over time.
• Recording decision outcomes and escalation paths.
• Setting alerts for calibration drift detection.

Version Control and Deployment Rollouts

• Managing model and adapter versioning schemes.
• Executing shadow and canary deployment strategies.
• Pinpointing model files for audit compliance.
• Establishing robust rollback procedures.

Security and Multi-Tenancy Management

• Implementing authentication and rate limiting controls.
• Ensuring tenant isolation for prompts and logs.
• Enforcing network isolation and egress controls.
• Maintaining audit logs for all decisions.