Ship production AI systems & LLM apps.
2 topics
Before working with LLMs and production AI, you need ML fundamentals. Understand supervised learning (classification and regression with labeled data), unsupervised learning (clustering, dimensionality reduction), and the ML workflow: data collection, feature engineering, model selection, training, evaluation (accuracy, precision, recall, F1, AUC-ROC), and deployment. Know key algorithms conceptually: linear/logistic regression, decision trees, random forests, gradient boosting (XGBoost), k-means clustering, and neural networks. Understand bias-variance tradeoff, overfitting vs underfitting, cross-validation, and hyperparameter tuning. You do not need to implement these from scratch — scikit-learn provides production-ready implementations — but understanding how they work helps you debug and improve AI systems.
2 resources
Deep learning uses neural networks with multiple layers to learn complex patterns. Understand: perceptrons and activation functions, backpropagation, CNNs (Convolutional Neural Networks for images), RNNs/LSTMs (sequential data), and most importantly — Transformers. The Transformer architecture (attention mechanism, self-attention, multi-head attention, positional encoding) powers modern AI. GPT (decoder-only, autoregressive text generation), BERT (encoder-only, understanding), and T5 (encoder-decoder, sequence-to-sequence) are the key architectures. Learn PyTorch (the industry standard framework) — tensors, autograd, nn.Module, DataLoaders, and training loops. Hugging Face Transformers library provides pre-trained models for any task. Understanding tokenization (BPE, WordPiece) and embedding spaces is essential for working with LLMs.
Modern AI engineering often starts with API-based LLMs rather than training from scratch. Master the OpenAI API (GPT-4, GPT-4o), Anthropic API (Claude), Google (Gemini), and open-source alternatives (Llama, Mistral via Together AI, Groq, or self-hosted). Understand: chat completion endpoints, system/user/assistant message roles, temperature and top-p for controlling randomness, max tokens and context windows, streaming responses for better UX, function calling / tool use for structured outputs, and JSON mode for reliable parsing. Learn cost optimization — prompt caching, model selection (use smaller models for simple tasks), and batching. Handle rate limits, retries with exponential backoff, and fallback models gracefully. The LangChain and LlamaIndex libraries provide abstractions for common patterns.
Prompt engineering is the art and science of crafting inputs that get the best outputs from LLMs. Key techniques: zero-shot (direct instruction), few-shot (providing examples), chain-of-thought (asking the model to reason step by step), ReAct (reasoning + acting with tools), and structured output prompting (specifying JSON schemas). Write clear, specific system prompts that define the model's role, constraints, and output format. Use delimiters to separate instructions from content. Include edge case handling in prompts. For complex tasks, break them into subtasks with separate prompts (prompt chaining). Evaluate prompt quality systematically — build test sets with expected outputs and measure accuracy, relevance, and safety. Prompt versioning and A/B testing help iterate effectively.
RAG combines LLMs with external knowledge retrieval to ground responses in your data. The pipeline: chunk your documents into segments, generate embeddings (vector representations) using models like OpenAI ada-002 or open-source alternatives, store them in a vector database (Pinecone, Weaviate, Qdrant, pgvector, ChromaDB), and at query time retrieve the most relevant chunks using similarity search, then include them in the LLM prompt as context. Key optimization areas: chunking strategy (size, overlap, semantic boundaries), embedding model selection, retrieval quality (hybrid search combining vector + keyword, reranking with cross-encoders), prompt construction (how to present retrieved context), and evaluation (faithfulness, relevance, answer correctness). RAG lets you build chatbots over company docs, code search, customer support agents, and more — without fine-tuning.
Fine-tuning adapts a pre-trained LLM to your specific task or domain. Full fine-tuning updates all model weights (expensive, needs lots of data and GPUs). Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation) and QLoRA (quantized LoRA) update only a small fraction of parameters — achieving similar quality at a fraction of the cost. The workflow: prepare a training dataset (instruction-response pairs), choose a base model (Llama 3, Mistral, Phi), configure training (learning rate, epochs, batch size), train using Hugging Face TRL or Axolotl, evaluate on held-out test data, and deploy. Fine-tuning is best for: consistent output format/style, domain-specific knowledge, reducing prompt length (baking instructions into weights), and improving performance on specific tasks. Always try RAG and prompt engineering first — fine-tune only when they are not enough.
AI agents are LLMs that can use tools, make decisions, and take multi-step actions autonomously. An agent receives a goal, plans steps, executes tools (web search, database queries, API calls, code execution), observes results, and iterates. Frameworks like LangGraph, CrewAI, and AutoGen provide abstractions for building agents. Key concepts: tool definition (name, description, parameters), agent loop (think → act → observe → repeat), memory (short-term conversation context + long-term vector store), and safety guardrails (input/output filtering, action limits, human-in-the-loop for sensitive actions). Multi-agent systems use specialized agents that collaborate — a researcher agent, a coder agent, and a reviewer agent working together. Production agents need robust error handling, timeout management, cost controls, and observability.
Shipping AI to production requires engineering discipline beyond model development. Key concerns: latency optimization (streaming responses, caching frequent queries, model quantization, edge deployment), cost management (token budgets, model routing — use cheaper models for simple queries), reliability (fallback models, graceful degradation, retry logic), safety (content filtering, PII detection and redaction, output validation), evaluation (automated evals with rubrics, human evaluation, regression testing with golden datasets), and observability (logging prompts/responses, tracking latency/cost/quality metrics, tracing multi-step agent flows). Tools like LangSmith, Weights & Biases, and Braintrust help with evaluation and monitoring. Implement feature flags to gradually roll out AI features and A/B test different models/prompts.