Grade Calculator — DevLevelUp

Select a role to assess

Rate your skill proficiency — the system will calculate your recommended grade

Role

Skills Self-Assessment

Role: LLM Engineer

Assessed: 0 Total: 0

3D Graphics & Computing 1

▼

Classical Machine Learning 1

▼

Deep Learning 4

▼

Knows transfer learning basics: pre-training, fine-tuning, feature extraction. Understands how pre-trained LLMs are used for downstream tasks and applies basic transfer learning approach.

Applies transfer learning techniques for LLM adaptation: LoRA, QLoRA, and prompt tuning. Fine-tunes foundation models on domain-specific corpora. Evaluates catastrophic forgetting and optimizes training efficiency.

Defines transfer learning standards for the LLM team. Establishes guidelines for base model selection, transfer strategy, evaluation. Coordinates transfer learning experiments and model selection.

Shapes enterprise transfer learning strategy. Defines approaches to foundation model selection, org-wide transfer practices, and knowledge sharing between teams and products.

LLM & Generative AI 16

▼

Understands basic agent framework concepts including tool-use patterns, chain composition, and memory types in LangChain and LlamaIndex. Follows team examples for building simple agents with predefined tools, structured output parsing, and basic conversation memory. Uses framework debugging tools to trace agent reasoning steps and identify issues in tool selection and response generation.

Independently builds agent systems using LangChain, LangGraph, and custom orchestration code with advanced tool integration and state management. Implements agent architectures including ReAct, plan-and-execute, and reflection patterns with configurable retry logic and error recovery. Evaluates and benchmarks framework performance for specific tasks — comparing agent strategies by accuracy, token usage, latency, and reliability under diverse input distributions.

Defines agent framework architecture standards and evaluation methodologies for the organization's LLM engineering teams. Establishes best practices for agent testing, safety guardrails, cost management, and production monitoring across agent-based systems. Drives architectural decisions on framework selection, custom versus off-the-shelf agent infrastructure, and integration patterns with existing ML platform services.

Shapes the organization's agent infrastructure strategy, defining how agent frameworks integrate with the broader AI/ML platform. Drives research and innovation in agent architectures including self-evolving tool ecosystems, meta-learning agent strategies, and novel approaches to agent safety and alignment. Influences the LLM agent framework ecosystem through open-source contributions, research publications, and community leadership on production agent system design patterns.

Knows text embeddings and vector database basics. Generates embeddings via sentence-transformers, stores and searches in ChromaDB. Understands cosine similarity and basic semantic search.

Independently designs embedding pipelines: model selection (OpenAI, Cohere, BGE), chunking strategies, and metadata filtering. Configures Pinecone/Weaviate for production workloads with recall optimization.

Defines embedding and vector DB strategy for the LLM platform. Establishes guidelines for embedding model selection, vector DB, index sharding, and retrieval quality monitoring.

Shapes enterprise embedding infrastructure strategy. Defines approaches to centralized embedding services, managing billions of vectors, cost optimization, and retrieval quality at scale.

Knows alignment basics: what is helpful, harmless, honest (HHH). Understands why alignment is needed for LLM and how it affects model behavior. Studies basic approaches under mentor guidance.

Independently implements alignment pipelines: preference dataset preparation, Constitutional AI prompts, DPO training. Evaluates alignment quality through human evaluation and automated benchmarks.

Defines alignment standards for the LLM team. Establishes guidelines for alignment methodology, quality metrics, red-teaming processes. Coordinates alignment R&D and production integration.

Shapes enterprise alignment strategy for the organization. Defines alignment governance, research directions, and compliance with emerging regulations. Mentors leads on advanced alignment techniques.

Deploys LLM inference endpoints using managed services and monitors latency and error rates. Writes structured prompts with few-shot examples following team guidelines. Tracks token usage across requests, identifies costly queries, and applies basic prompt length optimization techniques.

Builds fine-tuning pipelines with dataset curation, training orchestration, and automated evaluation. Implements prompt templating systems supporting versioning and rollback across environments. Designs LLM evaluation frameworks with automated scoring, regression detection, and human-in-the-loop review workflows.

Leads the LLM platform team, defining architecture standards for inference, fine-tuning, and evaluation infrastructure. Coordinates cross-team prompt engineering governance and token budget allocation. Drives vendor evaluation for foundation models, balancing capability, cost, and compliance requirements across the engineering organization.

Defines the company-wide LLM engineering strategy spanning model selection, deployment topology, and cost governance. Architects next-generation LLM platforms supporting multi-model orchestration, continuous evaluation, and automated optimization. Drives industry partnerships and open-source contributions that advance the state of LLM infrastructure and tooling.

Knows RAG basics: retrieval + generation, basic pipeline with embedding and vector search. Builds simple RAG pipeline on LangChain with ChromaDB for QA task under mentor guidance.

Independently designs production RAG: advanced chunking, hybrid retrieval, re-ranking. Configures metadata filtering, conversation history, and source attribution. Evaluates quality via RAGAS.

Defines RAG strategy for the LLM team. Establishes best practices for RAG architecture, data ingestion, quality monitoring. Coordinates RAG system as platform for multiple products.

Shapes enterprise RAG platform for the organization. Defines approaches to unified knowledge bases, multi-tenant RAG, and governance. Ensures scalability and quality at the scale of millions of documents.

Knows Transformer basics: self-attention, multi-head attention, positional encoding, feed-forward layers. Understands encoder-decoder and decoder-only architectures and their application in LLM.

Independently analyzes and modifies Transformer architectures: RoPE, ALiBi, GQA, SwiGLU. Understands architectural differences between GPT, LLaMA, Mistral and their impact on performance.

Defines Transformer architecture standards for the LLM team. Establishes guidelines for architecture selection, new approach evaluation, R&D directions. Coordinates architectural experiments.

Shapes enterprise Transformer R&D strategy. Defines long-term architectural directions, evaluates emerging architectures (Mamba, RWKV), and plans transitions between architecture generations.

Natural Language Processing 3

▼

Understands core Transformer architecture and attention mechanisms. Works with pre-trained LLMs using Hugging Face Transformers library for inference and basic prompt engineering. Follows established patterns for model loading, tokenization, and API integration.

Applies fine-tuning techniques (LoRA, QLoRA, PEFT) to adapt LLMs for specific tasks. Implements RAG pipelines combining retrieval with generation. Understands trade-offs between model quantization, context window size, and output quality for production use cases.

Defines LLM platform strategy: model selection criteria, fine-tuning pipelines, and serving infrastructure standards. Establishes evaluation benchmarks, safety guardrails, and cost optimization practices. Reviews architectural decisions for RAG systems, agent frameworks, and multi-model orchestration.

Shapes organizational LLM strategy: evaluates foundation model providers, defines enterprise AI governance policies, and architects cross-team LLM infrastructure. Establishes standards for model risk management, cost forecasting, and responsible AI deployment. Mentors leads on building scalable AI platforms.

Prompt Engineering 1

▼

Additional skills (optional) 54

These skills do not affect the main grade but show the breadth of your profile.

AI Coding Assistants

▼

Algorithms & Data Structures

▼

API Management

▼

API Protocols

▼

Application Security

▼

Background Jobs & Task Queues

▼

Batch Processing

▼

Understands Pandas basics for LLM data preparation: text dataset loading and preprocessing, training corpus statistics analysis, and prompt/completion pair management. Cleans and formats text data for fine-tuning datasets. Follows team practices for data versioning and quality checks.

Implements LLM data processing pipelines with Pandas/Polars: text corpus cleaning and deduplication at scale, training data quality metrics computation, and evaluation dataset management. Uses Polars for high-performance text preprocessing on large corpora. Creates reproducible data preparation workflows for fine-tuning and evaluation.

Defines data engineering strategy. Establishes data platform. Coordinates data teams. Optimizes data mesh/data fabric approaches.

Caching

▼

CI/CD

▼

Clean Code & Refactoring

▼

Knows clean code basics and PEP 8 for Python. Writes readable code with type hints when developing scripts for data processing and LLM API calls under mentor guidance.

Independently applies code quality practices in LLM application development. Writes clean prompt engineering code with proper chain composition and fallback handling. Understands trade-offs between prompt complexity and response reliability. Reviews LLM integration code for token efficiency, error handling, and evaluation coverage.

Defines code quality standards for the LLM team. Establishes guidelines for ML experiment organization, model and data versioning, conducts ML pipeline architectural reviews.

Cloud Providers

▼

Understands basic AWS concepts for LLM workloads — EC2 GPU instances for inference, S3 for model artifacts, and Bedrock for managed LLM access. Uses existing configurations for GPU instance provisioning and model deployment. Performs simple operations like monitoring inference endpoint health and checking GPU utilization under senior guidance.

Independently configures AWS services for LLM workloads — EC2 P4/P5 instances for inference, Bedrock for managed LLM access, and S3 for model artifact management. Writes IaC for GPU cluster provisioning with spot instance fallback strategies. Understands VPC networking for secure inference endpoints and IAM policies for model access control.

Defines AWS infrastructure strategy for LLM serving platforms spanning GPU clusters, Bedrock integration, and model deployment automation. Establishes IaC standards for GPU instance management, model artifact lifecycle, and inference scaling policies. Conducts architecture reviews optimizing LLM serving costs and coordinates FinOps for GPU compute spending.

Code Review

▼

Concurrency & Parallelism

▼

Understands basic async programming in Python: asyncio for concurrent API calls, async streaming response handling, basic concurrent prompt execution patterns. Follows team conventions for async code in LLM pipeline components.

Independently applies async programming in LLM pipelines: concurrent API calls to model providers, async streaming response handling, non-blocking chain/agent execution. Understands trade-offs between sync and async patterns for LLM inference and retrieval.

Defines async programming standards for LLM team: streaming response architecture guidelines, concurrent inference design reviews, async RAG pipeline patterns. Establishes best practices for async patterns in LLM serving systems.

Containerization

▼

Database Optimization

▼

Deep Learning

▼

Understands PyTorch tensor operations and autograd for transformer models. Uses HuggingFace Transformers with PyTorch backend for inference. Loads pretrained LLMs, runs tokenization and text generation. Grasps attention mechanism basics in PyTorch.

Fine-tunes transformers with PyTorch using PEFT: LoRA, QLoRA, prefix tuning. Implements custom training loops for causal LM and seq2seq tasks. Works with tokenizers, attention masks and padding strategies. Integrates HuggingFace Trainer with DeepSpeed for efficient fine-tuning.

Defines PyTorch-based LLM training standards: PEFT strategy selection, distributed training configs, evaluation benchmarks. Evaluates tools (vLLM, TensorRT-LLM) for production inference. Reviews fine-tuning architectures. Establishes best practices for reproducible LLM experiments.

Distributed Tracing

▼

File & Object Storage

▼

Git & Workflows

▼

GraphQL

▼

gRPC

▼

Understands basics of gRPC & Protocol Buffers for LLM inference service communication. Reads proto definitions for model serving request/response schemas and streaming token generation. Follows team conventions for defining inference endpoint contracts and model metadata messages.

Independently designs gRPC services for LLM inference platforms — streaming token generation with server-side streaming, efficient proto schemas for prompt/completion messages with metadata, and load-balanced inference request routing. Writes comprehensive proto documentation and tests for model serving contracts.

Defines API strategy at product level. Establishes design standards. Conducts API design reviews. Coordinates cross-team API interaction.

Infrastructure as Code

▼

Integration Testing

▼

Kubernetes & Orchestration

▼

LLM & Generative AI

▼

Understands RAG pipeline components: embedding models, vector databases, and retrieval strategies. Implements basic document indexing and semantic search with FAISS or Pinecone for LLM augmentation.

Implements advanced RAG patterns: query decomposition, hypothetical document embeddings (HyDE), and multi-index retrieval. Optimizes chunk size and overlap for domain-specific corpora and evaluates retrieval quality with RAGAS metrics.

Defines RAG Architecture strategy at team/product level. Establishes standards and best practices. Conducts reviews.

Defines RAG Architecture strategy at organizational level. Establishes enterprise approaches. Mentors leads and architects.

Logging

▼

MCP & AI Tools

▼

Understands basic AI agent development concepts including the agent loop (observe-think-act), tool definition schemas, and prompt design for agent instructions. Follows team patterns for implementing simple agents with function calling, structured outputs, and basic error handling. Tests agent behavior using predefined scenarios and evaluates tool selection accuracy and response quality under guidance.

Independently develops AI agent systems with custom tool implementations, conversation state management, and multi-turn reasoning capabilities. Implements agent reliability patterns including retry with backoff, tool call validation, output verification loops, and graceful fallback to simpler strategies. Builds evaluation harnesses for agent systems measuring task completion, tool efficiency, cost per interaction, and safety boundary compliance across diverse input scenarios.

Defines AI agent development standards, safety frameworks, and infrastructure architecture for the organization's LLM engineering practice. Establishes agent evaluation methodologies, production deployment requirements, and cost optimization strategies across agent-powered systems. Drives architectural decisions on agent infrastructure including runtime environments, tool marketplace governance, and cross-team agent capability reuse patterns.

Shapes the organization's AI agent engineering vision, defining foundational architectures for autonomous and semi-autonomous agent systems. Drives research frontiers in agent development — emergent agent behaviors, self-evolving tool ecosystems, long-horizon planning with world models, and novel approaches to agent alignment and safety verification. Publishes research and influences the LLM agent engineering community through contributions to agent safety standards, evaluation benchmarks, and open-source agent infrastructure.

Understands MCP architecture: client-server communication, capability negotiation, and transport layers. Configures MCP connections for LLM applications and tests basic tool invocations.

Implements MCP servers with dynamic tool registration, streaming responses, and context management. Optimizes tool descriptions for LLM comprehension and handles multi-turn tool-use conversations.

Defines Model Context Protocol strategy at team/product level. Establishes standards and best practices. Conducts reviews.

Defines Model Context Protocol strategy at organizational level. Establishes enterprise approaches. Mentors leads and architects.

Metrics & Monitoring

▼

MLOps

▼

Logs prompt templates, model versions, and evaluation scores (BLEU, ROUGE, human ratings) in experiment tracking tools; follows team standards for recording LLM fine-tuning and inference experiments

Builds comprehensive experiment tracking for LLM workflows: versions prompt chains and RAG configurations, logs token usage and latency alongside quality metrics, and compares fine-tuning runs across different base models in W&B or MLflow

Defines experiment tracking strategy for LLM product teams: standardizes tracking of prompt engineering iterations, model evaluations, and cost/quality trade-offs; builds team dashboards connecting experiment outcomes to product metrics and release decisions

Logs LLM fine-tuning runs in MLflow, tracking training loss, perplexity, and evaluation scores. Records prompt templates, tokenizer configs, and adapter weights (LoRA, QLoRA) as artifacts. Uses MLflow UI to compare runs across different base models and hyperparameter sets.

Structures MLflow experiments for LLM evaluation pipelines, logging BLEU, ROUGE, and benchmark scores across prompt variations. Integrates with Hugging Face Trainer and DeepSpeed for automatic metric capture. Manages LLM versions in Model Registry with training data and quantization metadata.

Sets MLflow strategy for LLM teams, standardizing tracking across fine-tuning, RLHF, and prompt engineering workflows. Establishes model registry policies for LLM versioning with adapter compatibility matrices. Reviews tracking practices for data lineage compliance and model card generation.

Understands basic monitoring concepts for large language models in production. Tracks response quality, token usage, latency, and hallucination rates using logging tools. Follows team practices for setting up alerts on LLM endpoint performance and prompt regression detection.

Independently builds monitoring pipelines for LLM applications. Tracks semantic drift, response quality scores, token cost trends, and prompt effectiveness using LangSmith or custom evaluation harnesses. Configures alerting on latency spikes, safety filter triggers, and output format violations.

Defines model monitoring strategy for LLM products at team level. Establishes evaluation frameworks combining automated metrics, human review workflows, and safety monitoring. Sets standards for cost tracking, latency budgets, and quality gates across prompt versions. Conducts reviews of monitoring coverage for all LLM-powered features.

Understands LLM serving basics: inference API setup (vLLM, TGI), prompt/completion endpoint configuration, and token-based billing considerations. Follows team practices for LLM deployment including context window management and response streaming setup.

Implements LLM serving solutions: KV-cache optimization for throughput, batching strategies (continuous batching, dynamic batching), and quantization for cost-efficient inference (GPTQ, AWQ, GGUF). Configures vLLM/TGI for production workloads. Implements streaming response infrastructure and token-level latency monitoring.

Defines LLM serving strategy for the organization. Establishes inference cost management policies, serving SLA targets, and GPU infrastructure governance. Evaluates serving frameworks (vLLM, TGI, TensorRT-LLM). Conducts architecture reviews for LLM infrastructure. Drives adoption of cost-efficient LLM serving patterns.

Networking

▼

OOP & Design Patterns

▼

Knows basic design patterns: Strategy, Factory, Observer. Applies them when writing simple LLM pipeline components: data loaders, prompt formatters.

Independently applies design patterns for LLM applications: chain of responsibility for prompt processing pipelines, strategy for interchangeable model providers, adapter for different embedding backends, observer for streaming response handling. Understands trade-offs between pattern complexity and LLM application maintainability.

Defines architectural patterns for the LLM platform. Establishes guidelines for applying patterns in ML systems, conducts design reviews of new inference pipeline components.

Understands basic OOP concepts in Python: classes, inheritance, protocols. Applies simple SOLID principles when structuring LLM pipeline components. Follows team patterns for prompt template classes and chain/agent wrapper design in LangChain-based projects.

Independently applies OOP/SOLID in LLM pipeline code: proper abstraction for prompt templates and chain components, interface-based model provider backends, single responsibility in retrieval/generation/evaluation modules. Understands trade-offs between OOP patterns and functional composition in LangChain/LlamaIndex codebases.

Defines OOP/SOLID standards for LLM team: chain/agent class hierarchy guidelines, model provider interface contracts, retrieval module architecture patterns. Conducts reviews of abstraction design in LLM pipeline components.

Relational Databases

▼

Understands basic PostgreSQL for LLM work: storing prompt templates and conversation history, pgvector basics for RAG embedding storage, basic queries for retrieval pipeline data management. Follows team conventions for LLM application database schemas.

Independently designs schemas for LLM applications: optimizes pgvector queries for RAG retrieval, implements efficient conversation history storage, configures indexing strategies for embedding similarity search. Understands trade-offs between PostgreSQL pgvector and dedicated vector databases for different RAG scales.

Defines PostgreSQL data strategy for LLM products: establishes standards for pgvector usage and RAG data architecture, designs database patterns for conversation management at scale, drives adoption of PostgreSQL best practices for LLM application data infrastructure.

REST API

▼

Understands basic REST API concepts for LLM services: chat completion endpoints, streaming response handling, token usage tracking in API responses. Follows team conventions for LLM API integration and prompt endpoint design.

Independently designs REST APIs for LLM services: chat completion endpoints with streaming support, prompt management APIs, usage tracking and billing endpoints. Understands best practices for LLM API rate limiting, token-based pricing in responses, and context window management.

Defines API strategy for LLM services at product level: LLM inference API standards, prompt management API governance, usage metering endpoint policies. Conducts API architecture reviews for LLM platform services and establishes LLM API design practices.

Search Engines

▼

System Design

▼

Type Systems

▼

Unit Testing

▼

Understands basic unit testing for LLM code: pytest for prompt template functions, mocking LLM API responses, assertion patterns for chain outputs. Follows team practices for testing retrieval and generation pipeline components independently.

Independently writes unit tests for LLM pipelines: pytest for prompt template logic, mocked LLM responses for chain testing, edge case coverage for context window overflow and token limits. Integrates LLM tests into CI/CD with deterministic mock responses and output validation.

Defines testing strategy at product level for LLM systems: prompt evaluation testing standards, chain/agent testing governance, LLM output quality testing frameworks. Establishes shift-left testing culture with automated quality gates for LLM pipeline deployments.

Web Frameworks

▼