Cheatsheet - MLOps

MLOps Fundamentals¶

• MLOps:: (1) Extension of DevOps to ML projects; (2) Tools & best practices for bringing ML into production; (3) Systems approach to ML

• ML Engineering: 90% engineering, 10% ML

• ML is systems engineering: Focus on reliability, not just accuracy

• Data quality > Model complexity: Garbage in, garbage out

• Monitor everything: Data drift, model drift, performance

• Iterate continuously: ML is never “set and forget”

• Align with business: Technical success $\neq$ business success

ML Research vs Production

Also important for Production but not so much for Research: Fairness & Interpretability.

MLOps Components

Business Objectives → Data Engineering → Model Training & Evaluation → Model Testing → Deployment → Monitoring → (Feedback Loop)

Data Flywheel

More data → Better model → More users → More data → ...

Deep Learning Fundamentals¶

• Core Concepts

  • Forward Pass: Input → Hidden Layers → Output

  • Backpropagation: Compute gradients using chain rule

  • Gradient Descent: Update weights: $w = w - \alpha \frac{\partial L}{\partial w}$

  • Loss Functions:

    • Regression: MSE = $\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n} (\hat{y}_i - y_i)^2$

    • Classification: Cross-Entropy $= -\sum_{i} y_i \log(\hat{y}_i)$

• Optimizers

  • SGD: Basic, needs tuning

  • Adam: Adaptive learning rates (most popular)

  • RMSprop: Adaptive, good for RNNs

• Regularization

  • Dropout: Randomly deactivate neurons (rate=0.2-0.5)

  • Weight Decay: L2 penalty on weights

  • BatchNorm: Normalize layer inputs

Neural Network Architectures

Training Process

1. Define: Input size, layers, activations (ReLU, Softmax), output, loss, optimizer, metrics

2. Train: Adjust weights via backpropagation to minimize loss

3. Evaluate: Check metrics on validation set

Data Engineering¶

• Data Processing

  • DAG: Directed Acyclic Graph (tasks = nodes, dependencies = edges)

  • Workflow Managers: Airflow (Python), Spark (distributed), Metaflow (AWS), Ploomber (lightweight)

  • ETL/ELT: Extract → Transform → Load

• Feature Stores

  • Purpose: Centralized feature repository

  • Benefits: Avoid recomputation, ensure training/serving consistency, share features

  • Examples: Feast, Tecton, Hopsworks

• Data Versioning

  • Git LFS: Large File Storage for Git

  • DVC: Data Version Control (like Git for data)

  • Others: Delta Lake, lakeFS, Oxen

Deployment¶

Model Compression

• Pruning: Remove unimportant weights

• Quantization: Reduce precision (float32 → int8)

• Knowledge Distillation: Train small model on large model’s predictions

• Low-rank Optimization: Factorize weight matrices

Serving Frameworks

TorchServe (PyTorch), TensorFlow Serving, Ray Serve, vLLM (LLMs), MLServer (framework-agnostic)

Monitoring & Continual Learning¶

ML models degrade over time → Monitoring is essential

Types of Distribution Shifts

• Monitoring Approaches

  • Operational Metrics: Latency, uptime, GPU usage

  • ML Metrics: Accuracy, drift, fairness

  • Business Metrics: ROI, sales, cost reduction

• Drift Detection Methods

  • Statistical Tests: Kolmogorov-Smirnov (KS), KL Divergence

  • Tools: Evidently AI, NannyML, DeepChecks, Fiddler, Whylogs, Alibi Detect

• Retraining Strategies

  • Periodic: Fixed schedule (e.g., weekly)

  • Event-driven: Triggered by drift detection

  • Continual Learning: Adapt without catastrophic forgetting

• Safe Deployment

  • A/B Testing: Compare two versions

  • Canary Release: Gradual rollout

  • Shadow Testing: Run new model alongside old

ML Product Management & Teams¶

ML Maturity Levels

1. Nascent: Exploring ML, no production

2. Emerging: First ML systems in production

3. Maturing: Multiple ML systems, standardized

4. ML-First: ML is core to business

Business Alignment

• ML Objectives: Accuracy, latency, throughput, model size

• Business Objectives: ROI, cost reduction, revenue growth, compliance

• Key: ML objectives must directly support business objectives