Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Key risks associated with cloud LLMs: data retention policies, input training usage, and external jurisdiction issues.
- Overview of Ollama’s architecture: model server, registry, and OpenAI-compatible API layer.
- Evaluation against alternatives like vLLM, llama.cpp, and Text Generation Inference.
- Licensing considerations for models including Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with CUDA and ROCm backend support.
- Configuring CPU-only fallbacks and optimizing AVX/AVX2 instructions.
- Deploying via Docker and mapping persistent volumes for data storage.
- Configuring multi-GPU environments and strategies for VRAM allocation.
Model Management
- Retrieving models from the Ollama registry, e.g., ollama pull llama3.
- Importing GGUF format models from HuggingFace and TheBloke repositories.
- Analyzing quantization tradeoffs across levels such as Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent loading limitations.
Custom Modelfiles
- Mastering Modelfile syntax: FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning generation parameters including temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce role-specific behaviors.
- Publishing and managing custom models within a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Integrating with frameworks like LangChain, LlamaIndex, and proprietary applications.
- Implementing authentication and rate limiting through reverse proxies.
Performance Optimization
- Managing context window sizes and KV cache efficiency.
- Optimizing batch inference and parallel request processing.
- Allocating CPU threads and ensuring NUMA-aware scheduling.
- Monitoring GPU utilization metrics and memory pressure states.
Security and Compliance
- Establishing network isolation for model serving endpoints.
- Implementing input filtering and output moderation workflows.
- Maintaining comprehensive audit logs for prompts and completions.
- Verifying model provenance and integrity via hash checking.
Requirements
- Intermediate proficiency in Linux administration and container management.
- High-level understanding of machine learning concepts and transformer architectures.
- Working knowledge of REST APIs and JSON data structures.
Target Audience
- AI engineers and developers seeking to migrate away from cloud-based LLM APIs.
- Organizations restricted from using cloud models due to data sensitivity policies.
- Government and defense entities requiring air-gapped language model capabilities.
