These skills do not affect the main grade but show the breadth of your profile.
3D Graphics & Computing
▼
Knows GPU computing basics: CUDA cores, memory hierarchy, kernel execution. Understands how PyTorch uses GPU for tensor operations and runs simple LLM inference on GPU.
Independently optimizes GPU utilization for LLM: mixed precision (FP16/BF16), memory management, CUDA stream optimization. Profiles GPU workloads using nvidia-smi and PyTorch profiler.
Designs custom CUDA kernels for LLM inference optimization: fused attention, quantized matmul, custom activation functions. Uses Triton for writing high-performance GPU operations.
Defines GPU programming standards for the LLM team. Establishes best practices for CUDA optimization, memory management, multi-GPU coordination. Conducts GPU-intensive code reviews.
AI Coding Assistants
▼
Uses Cursor for LLM development: AI-assisted prompt engineering, code generation.
Configures Cursor: custom rules for LLM patterns, RAG code generation, evaluation helpers.
Optimizes Cursor: LLM-specific workflows, prompt optimization, architecture suggestions.
Defines AI IDE standards: approved configs, security policies.
Knows GitHub Copilot basics: code autocomplete, function generation from comments. Uses Copilot to accelerate boilerplate code writing in ML projects and LLM scripts.
Effectively uses Copilot Chat and inline suggestions for ML tasks: generating data preprocessing, prompt templates, and API integrations. Understands Copilot limitations for specialized ML code.
Applies Copilot to accelerate LLM system development: generating training scripts, inference pipelines, and evaluation harnesses. Configures custom instructions for ML context and reviews AI suggestions.
Defines AI assistant usage strategy for the LLM team. Establishes guidelines for effective Copilot prompt engineering, trains team on best practices and AI code review.
Algorithms & Data Structures
▼
Knows basic sorting and search algorithms, understands Big-O notation for complexity analysis. Applies knowledge when analyzing performance of simple text data processing pipelines for LLM.
Independently analyzes algorithmic complexity of LLM pipeline components: tokenization, beam search, top-k sampling. Selects optimal data structures for caching embedding results.
Optimizes algorithms in critical LLM inference sections: KV-cache eviction, attention mechanism, batching strategies. Applies profiling to identify bottlenecks in inference pipelines.
Defines algorithmic standards for the LLM team. Conducts architectural reviews of complex inference engine algorithms, establishes computational complexity optimization guidelines.
Knows basic data structures: arrays, hash tables, trees. Understands their application when working with text corpora and tokenizer vocabularies in LLM context.
Independently selects data structures for LLM tasks: trie for tokenization, priority queue for beam search, bloom filter for corpus deduplication. Optimizes memory footprint.
Designs specialized data structures for LLM infrastructure: HNSW for vector search, paged KV-cache, custom hash maps for vocabulary lookup in high-throughput inference.
Defines data structure standards for the LLM platform. Conducts reviews of data structure efficiency in the inference engine, establishes guidelines for memory-efficient implementations.
API Management
▼
Documents LLM API: prompt templates, input/output formats, usage examples.
Creates LLM documentation: API reference, prompt engineering guides, evaluation reports.
Designs documentation: prompt library, model cards, best practice guides.
Defines documentation standards: mandatory prompt docs, model cards, review process.
API Protocols
▼
Understands the fundamentals of Server-Sent Events & Streaming. Applies basic practices in daily work. Follows recommendations from the team and documentation.
Independently designs and implements Server-Sent Events and Streaming. Understands best practices and patterns. Writes API documentation and tests.
Designs API architecture with Server-Sent Events and Streaming for production systems. Defines versioning, error handling, and pagination standards. Optimizes performance.
Defines API strategy at product level. Establishes design standards. Conducts API design reviews. Coordinates cross-team API interaction.
Application Security
▼
Understands security for LLM: prompt injection awareness, data leakage prevention. Applies basic safety measures.
Implements LLM security: prompt injection defense, output filtering, PII detection. Deploys guardrails.
Designs LLM security: comprehensive prompt security, adversarial testing, data classification.
Defines LLM security standards: mandatory safety testing, guardrails requirements, security review.
Applies secure coding for LLM: API key management, input sanitization. Does not hardcode credentials.
Implements LLM security: prompt sanitization, output filtering, secure API integration. Protects PII.
Designs secure LLM systems: comprehensive safety pipelines, adversarial defense, compliance.
Defines security standards: LLM safety guidelines, mandatory review, compliance.
Background Jobs & Task Queues
▼
Uses Task Queues at basic level in transformers/vLLM. Performs simple tasks using ready templates. Understands basic concepts and follows team practices.
Independently implements tasks with Task Queues in transformers/vLLM. Understands internals and optimizes performance. Writes tests using eval harness.
Designs Task Queue solutions for production systems. Optimizes performance and scalability. Chooses between alternative approaches. Mentors the team.
Defines Task Queues architectural decisions at product level. Establishes standards. Conducts design reviews and defines technical roadmap.
Batch Processing
▼
Understands Pandas basics for LLM data preparation: text dataset loading and preprocessing, training corpus statistics analysis, and prompt/completion pair management. Cleans and formats text data for fine-tuning datasets. Follows team practices for data versioning and quality checks.
Implements LLM data processing pipelines with Pandas/Polars: text corpus cleaning and deduplication at scale, training data quality metrics computation, and evaluation dataset management. Uses Polars for high-performance text preprocessing on large corpora. Creates reproducible data preparation workflows for fine-tuning and evaluation.
Designs LLM data architecture with Pandas/Polars: training data pipeline infrastructure, evaluation benchmark management, and dataset versioning for model reproducibility. Implements data quality frameworks for detecting contamination, bias, and distribution shift. Creates organization-wide data preparation standards for LLM fine-tuning. Mentors team on scalable text processing.
Defines data engineering strategy. Establishes data platform. Coordinates data teams. Optimizes data mesh/data fabric approaches.
Caching
▼
Uses Redis at basic level in transformers/vLLM. Performs simple tasks using ready templates. Understands basic concepts and follows team practices.
Independently implements tasks with Redis in transformers/vLLM. Understands internals and optimizes performance. Writes tests using eval harness.
Designs Redis solutions for production systems. Optimizes performance and scalability. Chooses between alternative approaches. Mentors the team.
Defines Redis architectural decisions at product level. Establishes standards. Conducts design reviews and defines technical roadmap.
CI/CD
▼
Uses GitHub Actions for LLM: automated evaluation runs, prompt testing. Understands CI for ML.
Designs CI for LLM: automated benchmarks, regression testing, model deployment pipelines.
Defines CI/CD architecture: model evaluation pipelines, A/B deployment, safety testing.
Defines CI/CD standards: mandatory evaluation, safety gates, deployment governance.
Classical Machine Learning
▼
Knows Python basics: data types, functions, classes, modules. Writes scripts for LLM API interaction, text data processing, and basic ML task automation under mentor guidance.
Independently develops in Python for LLM projects: async/await for API calls, dataclasses for configurations, generators for data streaming. Writes unit tests and uses type hints.
Designs Python applications for LLM: high-performance async servers, memory-efficient data processing, custom PyTorch extensions. Optimizes performance through profiling and concurrency.
Defines Python standards for the LLM team. Establishes coding guidelines, project structure templates, CI/CD pipeline. Conducts architectural reviews of ML system Python components.
Clean Code & Refactoring
▼
Knows clean code basics and PEP 8 for Python. Writes readable code with type hints when developing scripts for data processing and LLM API calls under mentor guidance.
Independently applies code quality practices in LLM application development. Writes clean prompt engineering code with proper chain composition and fallback handling. Understands trade-offs between prompt complexity and response reliability. Reviews LLM integration code for token efficiency, error handling, and evaluation coverage.
Designs code quality standards for LLM projects: ML experiment structure, model configurations, reproducibility. Implements pre-commit hooks and CI/CD checks for ML codebases.
Defines code quality standards for the LLM team. Establishes guidelines for ML experiment organization, model and data versioning, conducts ML pipeline architectural reviews.
Cloud Providers
▼
Understands basic AWS concepts for LLM workloads — EC2 GPU instances for inference, S3 for model artifacts, and Bedrock for managed LLM access. Uses existing configurations for GPU instance provisioning and model deployment. Performs simple operations like monitoring inference endpoint health and checking GPU utilization under senior guidance.
Independently configures AWS services for LLM workloads — EC2 P4/P5 instances for inference, Bedrock for managed LLM access, and S3 for model artifact management. Writes IaC for GPU cluster provisioning with spot instance fallback strategies. Understands VPC networking for secure inference endpoints and IAM policies for model access control.
Designs AWS infrastructure solutions for production LLM serving — multi-GPU EC2 clusters with model parallelism, Bedrock integration for managed inference, and SageMaker endpoints for custom models. Optimizes cost through GPU instance reservation strategies, inference batching, and model quantization for reduced compute requirements. Implements security hardening for LLM workloads with model access governance, prompt logging, and data isolation.
Defines AWS infrastructure strategy for LLM serving platforms spanning GPU clusters, Bedrock integration, and model deployment automation. Establishes IaC standards for GPU instance management, model artifact lifecycle, and inference scaling policies. Conducts architecture reviews optimizing LLM serving costs and coordinates FinOps for GPU compute spending.
Code Review
▼
Participates in ML code review as reviewer: checks readability, variable naming, basic correctness. Learns to spot common errors in LLM data processing scripts.
Independently conducts code review of LLM pipelines: verifies tokenization correctness, embedding logic, and prompt templates. Identifies data leakage between train/test sets and reproducibility issues.
Conducts deep architecture reviews of LLM systems: inference pipelines, RAG components, fine-tuning scripts. Verifies correctness of distributed training, gradient accumulation, and mixed precision.
Defines code review standards for the LLM team. Creates checklists for ML code review, trains engineers on LLM project review specifics and common anti-patterns.
Concurrency & Parallelism
▼
Understands basic async programming in Python: asyncio for concurrent API calls, async streaming response handling, basic concurrent prompt execution patterns. Follows team conventions for async code in LLM pipeline components.
Independently applies async programming in LLM pipelines: concurrent API calls to model providers, async streaming response handling, non-blocking chain/agent execution. Understands trade-offs between sync and async patterns for LLM inference and retrieval.
Designs async architectures for LLM systems: concurrent multi-model inference, async streaming response aggregation, non-blocking RAG pipeline orchestration. Mentors team on async patterns for LLM latency optimization and throughput scaling.
Defines async programming standards for LLM team: streaming response architecture guidelines, concurrent inference design reviews, async RAG pipeline patterns. Establishes best practices for async patterns in LLM serving systems.
Understands concurrency for LLM: parallel API calls, concurrent prompt processing.
Implements parallel LLM processing: batch inference, concurrent RAG retrieval, parallel evaluation.
Designs concurrent LLM systems: parallel model inference, distributed RAG, pipeline parallelism.
Defines concurrency standards: parallel processing policies, rate limiting, resource management.
Containerization
▼
Knows Docker basics: Dockerfile, images, containers. Builds simple containers for LLM applications with Python dependencies and CUDA runtime under mentor guidance.
Independently creates optimized Docker images for LLM: multi-stage builds, CUDA toolkit, PyTorch with GPU support. Configures docker-compose for dev environments with vector DB and inference server.
Designs Docker strategy for LLM platforms: optimizing image sizes with models, GPU passthrough, shared memory for multi-GPU inference. Creates base images for the ML team.
Defines Docker standards for team LLM infrastructure. Establishes best practices for ML image building, model management in containers, GPU resource allocation, and security.
Database Optimization
▼
Understands indexing for LLM: vector indexes, embedding storage. Works with vector databases.
Designs indexing: vector index optimization (HNSW, IVF), hybrid search, metadata filtering.
Designs vector search architecture: multi-index strategies, re-ranking, performance optimization.
Defines indexing standards: vector DB selection, index requirements, performance targets.
Deep Learning
▼
Knows deep learning fundamentals: backpropagation, loss functions, optimizers (SGD, Adam). Understands neural network architecture and trains simple models on PyTorch under mentor guidance.
Independently trains and fine-tunes models with PyTorch: configures learning rate schedules, regularization, and data augmentation. Understands gradient flow in transformer architectures.
Designs custom training loops for LLM: mixed precision, gradient accumulation, distributed training. Diagnoses training issues: gradient vanishing/exploding, loss spikes, training instability.
Defines deep learning best practices for the LLM team. Establishes model training standards, conducts training configuration reviews, introduces training run monitoring systems.
Knows main neural network architectures: MLP, CNN, RNN, Transformer. Understands Transformer architecture specifics for language models: self-attention, positional encoding, feed-forward layers.
Independently analyzes and modifies neural network architectures for LLM tasks: adapter layers, custom attention patterns, mixture of experts. Implements architectural modifications in PyTorch.
Designs custom neural network architectures for LLM: efficient attention mechanisms, sparse transformers, multi-modal architectures. Conducts ablation studies to optimize architectural decisions.
Defines architectural standards for the LLM team. Establishes architecture selection guidelines, conducts architectural decision reviews, coordinates R&D on new architectures.
Understands PyTorch tensor operations and autograd for transformer models. Uses HuggingFace Transformers with PyTorch backend for inference. Loads pretrained LLMs, runs tokenization and text generation. Grasps attention mechanism basics in PyTorch.
Fine-tunes transformers with PyTorch using PEFT: LoRA, QLoRA, prefix tuning. Implements custom training loops for causal LM and seq2seq tasks. Works with tokenizers, attention masks and padding strategies. Integrates HuggingFace Trainer with DeepSpeed for efficient fine-tuning.
Architects LLM training infrastructure in PyTorch. Implements custom attention, positional encodings and model parallelism. Optimizes memory with gradient checkpointing, mixed precision and Flash Attention. Designs evaluation frameworks for LM quality. Mentors team on transformer internals.
Defines PyTorch-based LLM training standards: PEFT strategy selection, distributed training configs, evaluation benchmarks. Evaluates tools (vLLM, TensorRT-LLM) for production inference. Reviews fine-tuning architectures. Establishes best practices for reproducible LLM experiments.
Knows PyTorch basics: tensors, autograd, nn.Module, DataLoader. Uses PyTorch for training simple models and pre-trained LLM inference via Hugging Face Transformers.
Independently develops with PyTorch for LLM: custom datasets, training loops, mixed precision (torch.amp). Uses Hugging Face Accelerate for multi-GPU training and inference.
Designs advanced PyTorch components for LLM: custom attention layers, efficient inference via torch.compile, CUDA graphs. Optimizes training and inference performance at the framework level.
Defines PyTorch best practices for the LLM team. Establishes framework usage guidelines, custom extensions, performance optimization. Conducts PyTorch code reviews.
Knows transfer learning basics: pre-training, fine-tuning, feature extraction. Understands how pre-trained LLMs are used for downstream tasks and applies basic transfer learning approach.
Applies transfer learning techniques for LLM adaptation: LoRA, QLoRA, and prompt tuning. Fine-tunes foundation models on domain-specific corpora. Evaluates catastrophic forgetting and optimizes training efficiency.
Designs advanced transfer learning strategies: continual pre-training, multi-task transfer, cross-lingual transfer. Optimizes the trade-off between forgetting and adaptation for domain-specific models.
Defines transfer learning standards for the LLM team. Establishes guidelines for base model selection, transfer strategy, evaluation. Coordinates transfer learning experiments and model selection.
Distributed Tracing
▼
Understands OpenTelemetry for LLM: tracing inference requests, token usage metrics.
Instruments LLM systems: distributed tracing, prompt chain tracing, custom metrics.
Designs LLM observability: end-to-end tracing, quality metrics, cost tracking.
Defines observability standards for LLM serving platforms — establishes mandatory tracing for inference request flows, metric requirements for token throughput and GPU utilization, and structured logging for prompt processing and model performance monitoring.
File & Object Storage
▼
Uses S3/Object Storage at basic level in transformers/vLLM. Performs simple tasks using ready templates. Understands basic concepts and follows team practices.
Independently implements tasks with S3/Object Storage in transformers/vLLM. Understands internals and optimizes performance. Writes tests using eval harness.
Designs S3/Object Storage solutions for production systems. Optimizes performance and scalability. Chooses between alternative approaches. Mentors the team.
Defines S3/Object Storage architectural decisions at product level. Establishes standards. Conducts design reviews and defines technical roadmap.
Git & Workflows
▼
Knows Git basics: commit, branch, merge, pull request. Uses Git for versioning ML experiment code and LLM scripts, follows basic conventions when working in a team.
Independently manages complex Git workflows for ML projects: feature branches, rebasing, cherry-pick. Uses git-lfs for storing model checkpoints and large dataset files.
Designs Git strategy for LLM projects: branching model for ML experiments, integration with DVC for data versioning, automation via git hooks for ML artifact validation.
Defines Git best practices for the LLM team. Establishes branching standards, commit conventions for ML experiments, integration with experiment tracking and model registry.
GraphQL
▼
Understands GraphQL for LLM: query API for prompt management, metadata access.
Designs GraphQL for LLM: prompt template API, evaluation results queries, model metadata.
Designs LLM API architecture: unified prompt management, real-time evaluation, caching.
Defines GraphQL standards: schema design, access patterns.
gRPC
▼
Understands basics of gRPC & Protocol Buffers for LLM inference service communication. Reads proto definitions for model serving request/response schemas and streaming token generation. Follows team conventions for defining inference endpoint contracts and model metadata messages.
Independently designs gRPC services for LLM inference platforms — streaming token generation with server-side streaming, efficient proto schemas for prompt/completion messages with metadata, and load-balanced inference request routing. Writes comprehensive proto documentation and tests for model serving contracts.
Designs gRPC API architecture for production LLM serving platforms — token streaming with backpressure management, efficient proto schemas for multi-modal inference with embeddings, and versioned model serving contracts supporting A/B testing. Defines error handling for inference timeout and rate limiting scenarios and optimizes serialization for high-throughput token generation.
Defines API strategy at product level. Establishes design standards. Conducts API design reviews. Coordinates cross-team API interaction.
Infrastructure as Code
▼
Describes LLM infrastructure in Terraform: GPU instances, vector DB provisioning.
Designs Terraform for LLM: model serving infrastructure, vector DB clusters, monitoring.
Designs LLM infrastructure: multi-model deployment, auto-scaling, cost optimization.
Defines IaC standards: LLM infrastructure patterns, GPU management.
Integration Testing
▼
Tests LLM integrations: API integration, RAG pipeline testing. Verifies end-to-end flows.
Designs integration tests: RAG pipeline validation, tool use testing, multi-model integration.
Defines integration testing: comprehensive LLM system testing, safety validation, regression testing.
Introduces testing standards: mandatory integration tests, safety validation, quality gates.
Kubernetes & Orchestration
▼
Understands basic Kubernetes Core concepts. Uses existing configurations. Performs simple operations under senior guidance.
Independently configures Kubernetes deployments for LLM inference services with GPU scheduling and memory management. Writes IaC for model serving workloads using vLLM or TGI on Kubernetes with resource quotas. Understands Kubernetes networking for load balancing inference traffic and scaling based on token throughput metrics.
Designs Kubernetes infrastructure for production LLM serving at scale with multi-GPU scheduling and model parallelism. Implements cost-optimized GPU cluster management with mixed node pools and intelligent workload placement. Applies security hardening for model serving endpoints and implements efficient model artifact distribution across clusters.
Defines infrastructure strategy with Kubernetes Core. Establishes IaC standards. Conducts architecture review. Optimizes FinOps.
Knows Kubernetes basics: pods, services, deployments. Understands how to deploy LLM inference server in K8s and basic resource request concepts for GPU workloads.
Independently deploys LLM services in Kubernetes: configures GPU resource limits, node affinity for GPU nodes, HPA for inference autoscaling. Uses Helm charts for ML workloads.
Designs Kubernetes infrastructure for LLM platforms: multi-GPU scheduling, model serving with KServe/Triton, distributed training jobs with Kubeflow. Optimizes GPU utilization across the cluster.
Defines Kubernetes strategy for team LLM infrastructure. Establishes ML workload deployment standards, GPU scheduling policies, inference service monitoring and autoscaling.
LLM & Generative AI
▼
Understands basic agent framework concepts including tool-use patterns, chain composition, and memory types in LangChain and LlamaIndex. Follows team examples for building simple agents with predefined tools, structured output parsing, and basic conversation memory. Uses framework debugging tools to trace agent reasoning steps and identify issues in tool selection and response generation.
Independently builds agent systems using LangChain, LangGraph, and custom orchestration code with advanced tool integration and state management. Implements agent architectures including ReAct, plan-and-execute, and reflection patterns with configurable retry logic and error recovery. Evaluates and benchmarks framework performance for specific tasks — comparing agent strategies by accuracy, token usage, latency, and reliability under diverse input distributions.
Designs production-grade agent architectures using framework-agnostic patterns with pluggable LLM backends, tool registries, and observability integration. Implements advanced multi-agent systems with supervisor agents, specialized worker agents, and shared memory stores using LangGraph or custom state machines. Optimizes agent systems for production scale through intelligent caching, streaming execution, parallel tool calls, and cost-aware model routing strategies.
Defines agent framework architecture standards and evaluation methodologies for the organization's LLM engineering teams. Establishes best practices for agent testing, safety guardrails, cost management, and production monitoring across agent-based systems. Drives architectural decisions on framework selection, custom versus off-the-shelf agent infrastructure, and integration patterns with existing ML platform services.
Knows distributed training basics: DataParallel, model parallelism. Understands gradient synchronization concepts and runs simple multi-GPU training under mentor guidance on PyTorch.
Independently configures distributed training with DeepSpeed ZeRO and FSDP. Configures data parallel, pipeline parallel, and tensor parallel for models up to 7B parameters on GPU clusters.
Designs distributed training strategies for large LLM: 3D parallelism, ZeRO-3 offloading, activation checkpointing. Optimizes communication overhead and GPU utilization on 100+ GPUs.
Defines distributed training infrastructure for the LLM team. Establishes best practices for multi-node training configuration, monitoring and debugging distributed jobs on GPU clusters.
Knows text embeddings and vector database basics. Generates embeddings via sentence-transformers, stores and searches in ChromaDB. Understands cosine similarity and basic semantic search.
Independently designs embedding pipelines: model selection (OpenAI, Cohere, BGE), chunking strategies, and metadata filtering. Configures Pinecone/Weaviate for production workloads with recall optimization.
Designs scalable embedding infrastructure: hybrid search (dense + sparse), re-ranking, multi-vector retrieval. Optimizes latency and recall through fine-tuning embedding models and index tuning.
Defines embedding and vector DB strategy for the LLM platform. Establishes guidelines for embedding model selection, vector DB, index sharding, and retrieval quality monitoring.
Knows alignment basics: what is helpful, harmless, honest (HHH). Understands why alignment is needed for LLM and how it affects model behavior. Studies basic approaches under mentor guidance.
Independently implements alignment pipelines: preference dataset preparation, Constitutional AI prompts, DPO training. Evaluates alignment quality through human evaluation and automated benchmarks.
Designs comprehensive alignment strategies: multi-objective alignment, iterative RLHF, debate-based alignment. Develops custom evaluation frameworks for measuring alignment quality at scale.
Defines alignment standards for the LLM team. Establishes guidelines for alignment methodology, quality metrics, red-teaming processes. Coordinates alignment R&D and production integration.
Deploys LLM inference endpoints using managed services and monitors latency and error rates. Writes structured prompts with few-shot examples following team guidelines. Tracks token usage across requests, identifies costly queries, and applies basic prompt length optimization techniques.
Builds fine-tuning pipelines with dataset curation, training orchestration, and automated evaluation. Implements prompt templating systems supporting versioning and rollback across environments. Designs LLM evaluation frameworks with automated scoring, regression detection, and human-in-the-loop review workflows.
Designs scalable LLM serving infrastructure with model routing, adaptive batching, and multi-region deployment. Establishes organization-wide prompt engineering practices with governance and audit trails. Optimizes token budgets across services through semantic caching, prompt compression, and model distillation strategies.
Leads the LLM platform team, defining architecture standards for inference, fine-tuning, and evaluation infrastructure. Coordinates cross-team prompt engineering governance and token budget allocation. Drives vendor evaluation for foundation models, balancing capability, cost, and compliance requirements across the engineering organization.
Knows LLM deployment basics: REST API endpoint, model loading, basic serving. Deploys simple inference server on vLLM or text-generation-inference under mentor guidance.
Independently deploys LLM to production: configures vLLM with continuous batching, quantization (GPTQ/AWQ), and health checks. Implements monitoring of latency, throughput, and error rates.
Designs production LLM serving infrastructure: multi-model serving, A/B testing, canary deployments, auto-scaling. Optimizes latency (p50/p95/p99) and throughput under high load.
Defines LLM deployment strategy for the team. Establishes SLA for inference services, monitoring standards, rollback and incident response processes for LLM production systems.
Knows basic prompt engineering techniques: zero-shot, few-shot, chain-of-thought. Creates simple prompts for classification and text generation tasks, tests variations under mentor guidance.
Independently develops complex prompt templates: structured output, tool use, multi-step reasoning. Conducts systematic prompt optimization with A/B testing and quality metrics.
Designs production prompt engineering systems: prompt versioning, automated optimization, meta-prompting. Implements dynamic prompt construction based on context and user intent.
Defines prompt engineering standards for the LLM team. Establishes prompt libraries, prompt testing guidelines, review processes. Coordinates prompt optimization for production systems.
Knows LLM safety basics: jailbreaking, prompt injection, harmful content. Understands basic approaches to content filtering and safety classifiers for LLM applications.
Independently implements safety pipelines: input/output filtering, guardrails (NeMo Guardrails, Guardrails AI), red-teaming. Configures content moderation and PII detection for production LLM.
Designs comprehensive safety systems: multi-layer defense, adversarial robustness testing, dynamic safety policies. Develops custom safety classifiers and automated red-teaming frameworks.
Defines LLM safety standards for the team. Establishes safety testing protocols, incident response for safety events, governance processes. Coordinates safety R&D and compliance.
Knows basic LLM scaling concepts: scaling laws, compute-optimal training, emergent abilities. Understands trade-offs between model size and computational resources for different tasks.
Independently plans scaling strategies: compute budget calculation using scaling laws, model size vs data size selection. Optimizes training and inference costs for 7B-13B parameter models.
Designs scaling strategy for large LLM: multi-stage scaling plans, Chinchilla-optimal training, progressive training. Optimizes balance between model quality, training cost, and inference latency.
Defines scaling standards for the LLM team. Establishes guidelines for compute budgeting, model size selection, cost-benefit analysis. Coordinates scaling decisions for multiple products.
Knows RAG basics: retrieval + generation, basic pipeline with embedding and vector search. Builds simple RAG pipeline on LangChain with ChromaDB for QA task under mentor guidance.
Independently designs production RAG: advanced chunking, hybrid retrieval, re-ranking. Configures metadata filtering, conversation history, and source attribution. Evaluates quality via RAGAS.
Designs enterprise RAG systems: multi-source retrieval, agentic RAG, query routing. Optimizes retrieval quality through fine-tuning retrievers, custom re-rankers, and adaptive chunking strategies.
Defines RAG strategy for the LLM team. Establishes best practices for RAG architecture, data ingestion, quality monitoring. Coordinates RAG system as platform for multiple products.
Understands RAG pipeline components: embedding models, vector databases, and retrieval strategies. Implements basic document indexing and semantic search with FAISS or Pinecone for LLM augmentation.
Implements advanced RAG patterns: query decomposition, hypothetical document embeddings (HyDE), and multi-index retrieval. Optimizes chunk size and overlap for domain-specific corpora and evaluates retrieval quality with RAGAS metrics.
Designs scalable RAG platforms with graph-augmented retrieval, contextual compression, and adaptive chunking strategies. Implements knowledge graph integration for structured reasoning and mentors team on RAG evaluation methodology and hallucination mitigation.
Defines RAG Architecture strategy at team/product level. Establishes standards and best practices. Conducts reviews.
Knows RLHF basics: reward model, PPO, preference learning. Understands why RLHF is used for LLM alignment and studies basic concepts under mentor guidance.
Independently implements RLHF pipelines: preference data collection, reward model training, PPO training with trl library. Applies DPO as an alternative to PPO for more stable training.
Designs advanced RLHF systems: iterative RLHF, Constitutional AI, reward model ensembles. Optimizes RLHF pipelines for training stability and alignment quality.
Defines RLHF strategy for the LLM team. Establishes best practices for data collection, reward modeling, training stability. Coordinates RLHF experiments and production integration.
Knows Transformer basics: self-attention, multi-head attention, positional encoding, feed-forward layers. Understands encoder-decoder and decoder-only architectures and their application in LLM.
Independently analyzes and modifies Transformer architectures: RoPE, ALiBi, GQA, SwiGLU. Understands architectural differences between GPT, LLaMA, Mistral and their impact on performance.
Designs custom Transformer modifications: efficient attention (FlashAttention, sliding window), custom positional encoding, architectural search. Implements and evaluates novel architectural solutions.
Defines Transformer architecture standards for the LLM team. Establishes guidelines for architecture selection, new approach evaluation, R&D directions. Coordinates architectural experiments.
Knows vLLM basics: what is PagedAttention, continuous batching, inference serving. Launches vLLM server for pre-trained model inference with basic configuration under mentor guidance.
Independently configures vLLM for production: tensor parallelism, quantization (AWQ/GPTQ), GPU memory management. Optimizes throughput by tuning batch size and scheduling parameters.
Designs production vLLM infrastructure: multi-model serving, speculative decoding, custom sampling strategies. Optimizes latency and throughput through advanced configuration and hardware-specific tuning.
Defines vLLM deployment standards for the LLM team. Establishes guidelines for configuration, monitoring, capacity planning. Coordinates upgrades and migration between vLLM versions.
Logging
▼
Uses logging for LLM: logging prompts/completions, token usage tracking.
Implements LLM logging: structured prompt logging, evaluation metrics, cost tracking, safety logging.
Designs LLM observability: prompt analytics, quality tracking, cost dashboards.
Defines logging standards: prompt logging requirements, evaluation metrics, safety monitoring.
MCP & AI Tools
▼
Understands basic AI agent development concepts including the agent loop (observe-think-act), tool definition schemas, and prompt design for agent instructions. Follows team patterns for implementing simple agents with function calling, structured outputs, and basic error handling. Tests agent behavior using predefined scenarios and evaluates tool selection accuracy and response quality under guidance.
Independently develops AI agent systems with custom tool implementations, conversation state management, and multi-turn reasoning capabilities. Implements agent reliability patterns including retry with backoff, tool call validation, output verification loops, and graceful fallback to simpler strategies. Builds evaluation harnesses for agent systems measuring task completion, tool efficiency, cost per interaction, and safety boundary compliance across diverse input scenarios.
Designs production AI agent architectures with sophisticated reasoning capabilities — tree-of-thought planning, self-reflection and correction, and dynamic strategy adaptation based on task complexity analysis. Implements multi-agent systems with agent specialization, communication protocols, and consensus mechanisms for complex problem decomposition. Architects agent infrastructure for scale including distributed tool execution, persistent agent memory with vector stores, and real-time observability of agent reasoning chains and cost metrics.
Defines AI agent development standards, safety frameworks, and infrastructure architecture for the organization's LLM engineering practice. Establishes agent evaluation methodologies, production deployment requirements, and cost optimization strategies across agent-powered systems. Drives architectural decisions on agent infrastructure including runtime environments, tool marketplace governance, and cross-team agent capability reuse patterns.
Understands MCP architecture: client-server communication, capability negotiation, and transport layers. Configures MCP connections for LLM applications and tests basic tool invocations.
Implements MCP servers with dynamic tool registration, streaming responses, and context management. Optimizes tool descriptions for LLM comprehension and handles multi-turn tool-use conversations.
Designs scalable MCP infrastructure with server composition, tool orchestration, and fallback strategies. Implements protocol extensions for custom capabilities and mentors team on context window optimization with MCP tools.
Defines Model Context Protocol strategy at team/product level. Establishes standards and best practices. Conducts reviews.
Metrics & Monitoring
▼
Monitors LLM via Prometheus: latency, token usage, error rate. Views dashboards.
Creates LLM monitoring: quality metrics, cost dashboards, throughput tracking.
Designs LLM monitoring: quality regression detection, cost alerting, usage analytics.
Defines monitoring standards: mandatory LLM metrics, dashboards, alerting.
MLOps
▼
Logs prompt templates, model versions, and evaluation scores (BLEU, ROUGE, human ratings) in experiment tracking tools; follows team standards for recording LLM fine-tuning and inference experiments
Builds comprehensive experiment tracking for LLM workflows: versions prompt chains and RAG configurations, logs token usage and latency alongside quality metrics, and compares fine-tuning runs across different base models in W&B or MLflow
Architects experiment tracking systems for LLM platforms: designs custom logging for multi-stage pipelines (retrieval, generation, ranking), optimizes storage for large prompt/response artifacts, and mentors teams on tracking evaluation drift across model versions
Defines experiment tracking strategy for LLM product teams: standardizes tracking of prompt engineering iterations, model evaluations, and cost/quality trade-offs; builds team dashboards connecting experiment outcomes to product metrics and release decisions
Knows ML model deployment basics: REST API, model serialization, inference server. Deploys simple ML models via web framework with basic error handling under mentor guidance.
Independently deploys ML models to production: TorchServe, BentoML, model optimization (ONNX, TensorRT). Configures prediction monitoring, data drift detection, and A/B testing.
Designs ML serving infrastructure for the LLM ecosystem: unified serving layer for ML and LLM models, feature stores, online/offline prediction pipelines with low latency requirements.
Defines ML deployment standards for the LLM team. Establishes guidelines for model serving, versioning, rollback strategy. Coordinates infrastructure for different ML model types.
Knows experiment tracking basics: logging metrics, parameters, artifacts. Uses W&B or MLflow for tracking LLM training runs and fine-tuning experiments.
Independently organizes experiment tracking for LLM projects: structured projects in W&B, run comparison, hyperparameter sweeps. Versions datasets and model checkpoints.
Designs experiment tracking infrastructure for the LLM team: custom dashboards, automated reporting, CI/CD integration. Ensures reproducibility for all training and evaluation experiments.
Defines experiment tracking standards for the LLM team. Establishes guidelines for experiment organization, naming conventions, and mandatory logging. Integrates tracking with model registry.
Knows basic ML metrics: accuracy, precision, recall, F1. Computes metrics for classification and regression models used in LLM system preprocessing pipelines.
Independently conducts comprehensive ML model evaluation: confusion matrix, ROC-AUC, calibration plots. Evaluates auxiliary ML models in LLM pipelines: safety classifiers, intent detectors.
Designs evaluation frameworks for ML components of the LLM ecosystem: cross-validation strategies, statistical significance testing, fairness metrics. Automates regression testing.
Defines ML evaluation standards for the LLM team. Establishes guidelines for auxiliary ML model assessment, threshold selection, and A/B testing methodology.
Knows ML pipeline basics: data loading, preprocessing, training, evaluation. Builds simple pipelines for LLM fine-tuning data preparation using Hugging Face Datasets.
Independently builds production ML pipelines for LLM: data ingestion, cleaning, tokenization, training, evaluation. Uses Airflow or Prefect for orchestration, ensures idempotency.
Designs complex ML pipelines for LLM platforms: multi-stage data processing, continuous training, automated retraining triggers. Optimizes pipeline throughput and reliability.
Defines ML pipeline standards for the LLM team. Establishes guidelines for pipeline architecture, monitoring, error handling. Coordinates pipeline infrastructure for training and inference.
Logs LLM fine-tuning runs in MLflow, tracking training loss, perplexity, and evaluation scores. Records prompt templates, tokenizer configs, and adapter weights (LoRA, QLoRA) as artifacts. Uses MLflow UI to compare runs across different base models and hyperparameter sets.
Structures MLflow experiments for LLM evaluation pipelines, logging BLEU, ROUGE, and benchmark scores across prompt variations. Integrates with Hugging Face Trainer and DeepSpeed for automatic metric capture. Manages LLM versions in Model Registry with training data and quantization metadata.
Architects MLflow infrastructure for enterprise LLM development: tracking RLHF reward model iterations, managing adapter registries, and automating evaluation suites (MT-Bench, HumanEval). Builds custom MLflow flavors for serving quantized LLMs (GPTQ, AWQ) with latency and throughput tracking.
Sets MLflow strategy for LLM teams, standardizing tracking across fine-tuning, RLHF, and prompt engineering workflows. Establishes model registry policies for LLM versioning with adapter compatibility matrices. Reviews tracking practices for data lineage compliance and model card generation.
Understands basic monitoring concepts for large language models in production. Tracks response quality, token usage, latency, and hallucination rates using logging tools. Follows team practices for setting up alerts on LLM endpoint performance and prompt regression detection.
Independently builds monitoring pipelines for LLM applications. Tracks semantic drift, response quality scores, token cost trends, and prompt effectiveness using LangSmith or custom evaluation harnesses. Configures alerting on latency spikes, safety filter triggers, and output format violations.
Designs comprehensive monitoring platforms for LLM systems in production. Implements automated evaluation pipelines with LLM-as-judge, human feedback loops, and A/B test instrumentation. Builds cost optimization dashboards tracking token usage across model versions and prompt variants. Mentors team on LLMOps observability patterns.
Defines model monitoring strategy for LLM products at team level. Establishes evaluation frameworks combining automated metrics, human review workflows, and safety monitoring. Sets standards for cost tracking, latency budgets, and quality gates across prompt versions. Conducts reviews of monitoring coverage for all LLM-powered features.
Understands LLM serving basics: inference API setup (vLLM, TGI), prompt/completion endpoint configuration, and token-based billing considerations. Follows team practices for LLM deployment including context window management and response streaming setup.
Implements LLM serving solutions: KV-cache optimization for throughput, batching strategies (continuous batching, dynamic batching), and quantization for cost-efficient inference (GPTQ, AWQ, GGUF). Configures vLLM/TGI for production workloads. Implements streaming response infrastructure and token-level latency monitoring.
Designs LLM serving architecture: multi-model gateway with intelligent routing, speculative decoding for latency optimization, and disaggregated serving (prefill/decode separation). Implements cost optimization: token budget management, caching layers for repeated prompts, and model cascade strategies. Creates LLM serving benchmarks and capacity planning models. Mentors team on production LLM infrastructure.
Defines LLM serving strategy for the organization. Establishes inference cost management policies, serving SLA targets, and GPU infrastructure governance. Evaluates serving frameworks (vLLM, TGI, TensorRT-LLM). Conducts architecture reviews for LLM infrastructure. Drives adoption of cost-efficient LLM serving patterns.
Natural Language Processing
▼
Knows NLP basics: tokenization, stemming, NER, sentiment analysis. Understands how classic NLP tasks are solved with LLM and applies basic text preprocessing techniques.
Independently solves NLP tasks using LLM: text classification, NER, summarization, translation. Compares LLM approaches with classical methods, selects the optimal one for the task.
Designs comprehensive NLP systems based on LLM: multi-task learning, zero-shot transfer, domain adaptation. Optimizes quality through prompt engineering, fine-tuning, and ensemble approaches.
Defines NLP strategy for the LLM team. Establishes guidelines for approach selection (LLM vs classical NLP), evaluation methodology, domain adaptation strategies for various NLP tasks.
Knows tokenization basics: BPE, WordPiece, SentencePiece. Understands how tokenizer affects LLM quality and cost. Uses pre-trained tokenizers from Hugging Face for basic tasks.
Independently works with LLM tokenization: analyzes token distribution, optimizes input length, handles special tokens. Trains custom tokenizers on domain-specific corpora.
Designs tokenization strategies for LLM: multi-language tokenizer training, vocabulary extension, tokenizer-aware data preprocessing. Optimizes fertility rate and coverage for target domains.
Defines tokenization standards for the LLM team. Establishes guidelines for tokenizer selection and training, tokenization quality evaluation, and integration with training and inference pipelines.
Understands core Transformer architecture and attention mechanisms. Works with pre-trained LLMs using Hugging Face Transformers library for inference and basic prompt engineering. Follows established patterns for model loading, tokenization, and API integration.
Applies fine-tuning techniques (LoRA, QLoRA, PEFT) to adapt LLMs for specific tasks. Implements RAG pipelines combining retrieval with generation. Understands trade-offs between model quantization, context window size, and output quality for production use cases.
Architects production LLM systems with optimized serving (vLLM, TGI), model parallelism, and efficient batching strategies. Designs evaluation frameworks for model quality, safety, and bias detection. Implements advanced techniques: RLHF, constitutional AI, and chain-of-thought optimization.
Defines LLM platform strategy: model selection criteria, fine-tuning pipelines, and serving infrastructure standards. Establishes evaluation benchmarks, safety guardrails, and cost optimization practices. Reviews architectural decisions for RAG systems, agent frameworks, and multi-model orchestration.
Networking
▼
Understands networking for LLM: API endpoint communication, streaming responses, WebSocket.
Implements LLM networking: streaming API integration, connection pooling, load balancing.
Designs LLM network architecture: multi-model routing, failover, edge inference.
Defines networking standards: API integration patterns, failover policies.
OOP & Design Patterns
▼
Knows basic design patterns: Strategy, Factory, Observer. Applies them when writing simple LLM pipeline components: data loaders, prompt formatters.
Independently applies design patterns for LLM applications: chain of responsibility for prompt processing pipelines, strategy for interchangeable model providers, adapter for different embedding backends, observer for streaming response handling. Understands trade-offs between pattern complexity and LLM application maintainability.
Designs LLM system architecture using patterns: Plugin for model backends, Pipeline for inference chains, Registry for model/tokenizer management. Ensures extensibility.
Defines architectural patterns for the LLM platform. Establishes guidelines for applying patterns in ML systems, conducts design reviews of new inference pipeline components.
Understands basic OOP concepts in Python: classes, inheritance, protocols. Applies simple SOLID principles when structuring LLM pipeline components. Follows team patterns for prompt template classes and chain/agent wrapper design in LangChain-based projects.
Independently applies OOP/SOLID in LLM pipeline code: proper abstraction for prompt templates and chain components, interface-based model provider backends, single responsibility in retrieval/generation/evaluation modules. Understands trade-offs between OOP patterns and functional composition in LangChain/LlamaIndex codebases.
Applies OOP/SOLID in LLM application architecture: strategy pattern for swappable model providers, chain of responsibility for prompt processing pipelines, abstract interfaces for embedding and retrieval components. Designs extensible LLM frameworks supporting multi-model orchestration.
Defines OOP/SOLID standards for LLM team: chain/agent class hierarchy guidelines, model provider interface contracts, retrieval module architecture patterns. Conducts reviews of abstraction design in LLM pipeline components.
Relational Databases
▼
Understands basic PostgreSQL for LLM work: storing prompt templates and conversation history, pgvector basics for RAG embedding storage, basic queries for retrieval pipeline data management. Follows team conventions for LLM application database schemas.
Independently designs schemas for LLM applications: optimizes pgvector queries for RAG retrieval, implements efficient conversation history storage, configures indexing strategies for embedding similarity search. Understands trade-offs between PostgreSQL pgvector and dedicated vector databases for different RAG scales.
Designs PostgreSQL architecture for LLM application infrastructure: optimizes pgvector for production RAG at scale, implements hybrid retrieval strategies combining semantic and keyword search, configures database architecture for conversation state management. Mentors team on PostgreSQL patterns for production LLM applications.
Defines PostgreSQL data strategy for LLM products: establishes standards for pgvector usage and RAG data architecture, designs database patterns for conversation management at scale, drives adoption of PostgreSQL best practices for LLM application data infrastructure.
REST API
▼
Understands basic REST API concepts for LLM services: chat completion endpoints, streaming response handling, token usage tracking in API responses. Follows team conventions for LLM API integration and prompt endpoint design.
Independently designs REST APIs for LLM services: chat completion endpoints with streaming support, prompt management APIs, usage tracking and billing endpoints. Understands best practices for LLM API rate limiting, token-based pricing in responses, and context window management.
Designs API architecture for LLM services: streaming chat completion APIs, prompt management and versioning endpoints, usage metering and billing APIs. Defines API contracts for LLM platform components. Mentors team on designing scalable LLM inference APIs.
Defines API strategy for LLM services at product level: LLM inference API standards, prompt management API governance, usage metering endpoint policies. Conducts API architecture reviews for LLM platform services and establishes LLM API design practices.
Search Engines
▼
Uses Elasticsearch/OpenSearch at basic level in transformers/vLLM. Performs simple tasks using ready templates. Understands basic concepts and follows team practices.
Independently implements Elasticsearch/OpenSearch tasks in transformers/vLLM. Understands internals and optimizes performance. Writes tests using eval harness.
Designs Elasticsearch/OpenSearch solutions for production systems. Optimizes performance and scalability. Chooses between alternative approaches. Mentors the team.
Defines Elasticsearch/OpenSearch architectural decisions at product level. Establishes standards. Conducts design reviews and defines technical roadmap.
System Design
▼
Knows system design basics: client-server, REST API, databases. Understands simple LLM application architecture: API gateway, inference server, vector database.
Independently designs medium-complexity LLM systems: RAG service with caching, async processing, and message queues. Considers scalability, reliability, and cost when choosing architectural decisions.
Designs complex LLM systems: multi-model inference platforms, real-time RAG with streaming, distributed vector search. Ensures high availability, fault tolerance, and horizontal scalability.
Defines architectural standards for the LLM team. Establishes system design guidelines for ML systems, conducts architecture reviews, defines technical debt management strategy.
Type Systems
▼
Applies type safety for LLM: typed prompt templates, response schemas. Uses Pydantic.
Ensures type safety: structured output schemas, typed function calling, validation.
Designs type-safe LLM systems: schema-driven output, typed tool interfaces, contract validation.
Defines type safety standards: output schema requirements, validation policies.
Unit Testing
▼
Knows unit testing basics with pytest: fixtures, assertions, parameterization. Writes simple tests for LLM utilities: text preprocessing, prompt formatting, response parsing.
Independently writes comprehensive tests for LLM components: mocking API calls, testing tokenization and embedding pipelines. Uses pytest-asyncio for async code and coverage reporting.
Designs testing strategy for LLM systems: property-based testing for prompt templates, snapshot testing for model outputs, integration tests for RAG pipelines. Ensures high coverage.
Defines testing standards for the LLM team. Establishes guidelines for ML code testing, minimum coverage requirements, CI/CD integration. Trains the team on ML testing best practices.
Understands basic unit testing for LLM code: pytest for prompt template functions, mocking LLM API responses, assertion patterns for chain outputs. Follows team practices for testing retrieval and generation pipeline components independently.
Independently writes unit tests for LLM pipelines: pytest for prompt template logic, mocked LLM responses for chain testing, edge case coverage for context window overflow and token limits. Integrates LLM tests into CI/CD with deterministic mock responses and output validation.
Designs testing strategy for LLM systems: test pyramid for prompt/chain/agent components, evaluation-driven testing with LLM-as-judge, contract tests for model provider interfaces. Optimizes test execution with mock LLM responses and selective integration testing. Mentors team on testing non-deterministic AI outputs.
Defines testing strategy at product level for LLM systems: prompt evaluation testing standards, chain/agent testing governance, LLM output quality testing frameworks. Establishes shift-left testing culture with automated quality gates for LLM pipeline deployments.
Web Frameworks
▼
Uses Python Web Frameworks at basic level in transformers/vLLM. Performs simple tasks using ready templates. Understands basic concepts and follows team practices.
Independently implements tasks with Python Web Frameworks in transformers/vLLM. Understands internals and optimizes performance. Writes tests using eval harness.
Designs Python Web Framework solutions for production systems. Optimizes performance and scalability. Chooses between alternative approaches. Mentors the team.
Defines architecture for LLM serving APIs and prompt management platforms using FastAPI. Establishes standards for streaming response endpoints, token usage tracking, and guardrail middleware. Conducts design reviews and defines the roadmap for LLM gateway infrastructure.