These skills do not affect the main grade but show the breadth of your profile.
AI Coding Assistants
▼
Uses AI for data engineering: generating SQL queries, PySpark transformations, debugging data pipelines. Verifies AI output.
Applies LLM for data work: SQL optimization, Airflow DAG generation, data quality rules. Critically evaluates AI suggestions.
Defines AI strategy for data team: AI-assisted data quality, automated documentation, pipeline generation.
Uses Cursor for data pipeline code autocompletion: generating SQL queries, dbt models, Airflow DAG boilerplate. Applies suggestions for pandas/Polars transformations.
Effectively works with Cursor in data projects: generates dbt macros, Airflow operators, data quality checks. Uses project context (dbt project.yml, requirements.txt) for better suggestions.
Shapes AI tool policy for data team: what to generate (SQL, configs, tests), what requires manual work (data modeling, schema design). Evaluates quality impact.
Uses Copilot for data code generation: SQL queries, pandas transformations, pytest fixtures for data testing. Understands when suggestions are correct. Writes docstrings for better suggestions.
Works productively with Copilot: generates dbt models from descriptions, Airflow DAGs, data validation rules. Uses Copilot Chat for SQL query analysis. Critically evaluates suggestions for correctness.
Implements Copilot in the team: configures policies, defines usage boundaries for data engineering. Evaluates ROI. Monitors security: credential leaks, SQL injection in generated code.
Alerting & On-Call
▼
Understands SLI/SLO for data: data freshness, completeness, accuracy metrics. Monitors basic data quality indicators.
Defines data SLI: freshness SLO, completeness targets, pipeline latency. Configures alerting on data quality metrics.
Defines data SLO standards: mandatory quality SLI, data freshness requirements, incident response for data issues.
Algorithms & Data Structures
▼
Understands basic algorithmic concepts for data pipelines: simple join strategies, basic partitioning concepts, sorting algorithms for data ordering. Follows team guidance on algorithm selection for ETL operations.
Independently applies algorithmic thinking in data engineering: selects appropriate join algorithms for distributed processing, understands shuffle operation impact on Spark performance, evaluates partitioning strategies for data distribution. Analyzes complexity of ETL pipeline operations at scale.
Establishes performance budgets for data pipelines. Conducts reviews of sorting and aggregation algorithms in ETL processes. Implements Spark job profiling and shuffle operation optimization.
Understands basic data structures for data pipelines: tabular data formats, JSON/YAML configuration structures, database schema concepts. Applies simple data structures following team patterns for ETL script development and data validation.
Independently selects appropriate data structures for data pipelines: partitioned datasets for efficient data processing, columnar formats for analytical queries, schema definitions for data contracts. Understands trade-offs between batch and streaming data container formats.
Defines data structure standards in pipeline code: dataclasses for schemas, TypedDict for configs, Pydantic models for validation. Reviews structure choices in context of memory-efficient batch processing.
API Management
▼
Documents data APIs and pipelines: describes data sources, transformations, output schemas. Writes README for data repos.
Creates data documentation: pipeline architecture docs, data dictionaries, lineage diagrams. Automates schema documentation.
Defines documentation standards: mandatory data docs, catalog requirements, review process.
Application Security
▼
Understands security for data: data access control, PII handling, encryption basics. Applies secure practices when working with sensitive data.
Applies data security: column-level encryption, data masking, access control policies. Scans pipelines for security vulnerabilities.
Defines data security standards: GDPR/compliance requirements, data protection policies, security review process.
Applies secure coding for data: secrets management in pipeline configs, secure connections to data sources. Never hardcodes credentials.
Implements security in data code: encrypted connections, secure credential storage, audit logging. Scans pipeline code for vulnerabilities.
Defines data security standards: secure pipeline guidelines, credentials management, audit requirements.
Authentication & Authorization
▼
Understands auth for data: API authentication for data sources, service accounts. Manages credentials for pipeline connections.
Implements auth for data pipelines: OAuth2 for API sources, service-to-service auth, token management in Airflow connections.
Defines auth standards for data: access control policies, credential rotation, audit requirements.
Background Jobs & Task Queues
▼
Uses Task Queues at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Task Queues in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Defines task orchestration strategy: Celery for micro-tasks vs Airflow for DAGs, choosing between push/pull models. Implements idempotency standards and queue monitoring.
Designs distributed processing architecture: task queues for near-real-time vs batch schedulers, hybrid approaches. Defines SLA and capacity planning for different priority levels.
Caching
▼
Uses Redis at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Redis in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Defines caching strategy in data pipelines: Redis for hot reference data, TTL policies for cache freshness. Establishes metrics and SLA for cache-dependent pipelines.
CI/CD
▼
Understands CI/CD for data pipelines: running dbt/SQL tests on PR, linting Python code. Reads failed build logs. Understands workflow triggers.
Configures CI/CD for data projects: testing dbt models, SQL validation, data contract checking in PR. Automates Airflow DAG deployment. Configures environments for staging/production.
Defines CI/CD standards for data team: mandatory checks (dbt test, SQL lint, contract validation), deployment strategy, release management for data pipelines.
Clean Code & Refactoring
▼
Understands basic code quality principles for data pipeline code. Follows team conventions for ETL script structure and SQL formatting. Writes simple, clean data transformation functions with proper logging. Accepts code review feedback on pipeline code organization.
Independently applies code quality practices in data pipeline development. Writes clean ETL/ELT code with proper error handling, idempotency, and logging. Understands trade-offs between pipeline complexity and maintainability. Reviews data pipeline code for data quality checks, schema evolution handling, and resource efficiency.
Shapes pipeline code quality standards: ruff/black for formatting, mypy for typing, pytest for transformation unit tests. Implements data contracts and schema validation in CI.
Cloud Providers
▼
Works with basic AWS services for data: S3 for storage, RDS for databases, IAM for access. Uses AWS CLI for file upload/download. Understands regions and availability zones.
Uses AWS data services: Glue for ETL, Athena for ad-hoc queries, Redshift for DWH, Step Functions for orchestration. Configures VPC, security groups for data pipeline components.
Defines AWS strategy for data platform: multi-account for prod/staging, Lake Formation for fine-grained access control, cost allocation by project. Chooses managed vs self-hosted.
Code Review
▼
Participates in data code reviews: checks SQL style, dbt model structure, naming conventions. Leaves constructive comments. Learns from feedback on own PRs.
Conducts quality reviews: checks SQL performance (joins, window functions), dbt materialization choice, data quality test coverage. Gives feedback on data modeling. Suggests optimizations.
Builds code review culture: defines checklist (SQL performance, test coverage, documentation), establishes SLA. Automates checks (dbt test, SQL lint, contract validation).
Concurrency & Parallelism
▼
Understands the fundamentals of Async Programming at a basic level. Applies simple concepts in work tasks using Python/SQL. Follows recommendations from senior developers when solving problems.
Independently applies async programming in data pipelines: concurrent data source ingestion, async ETL stage orchestration, non-blocking data validation checks. Understands trade-offs between synchronous and asynchronous pipeline patterns in Spark/Airflow.
Defines asynchronous processing strategy: asyncio for API calls in ETL, aiohttp for bulk loading, async generators for streaming. Implements backpressure handling standards in data pipelines.
Understands parallelism in data: Spark partitions, parallel ETL tasks, concurrent data loads. Manages basic parallelism in Airflow.
Implements parallel data processing: Spark parallel execution, Airflow task concurrency, parallel data ingestion. Optimizes partitioning.
Defines parallelism standards: resource allocation policies, concurrent pipeline limits, cluster sizing.
Containerization
▼
Runs data pipeline components in Docker: Airflow, Spark, PostgreSQL for local development. Understands Dockerfile, docker-compose. Builds images with Python dependencies for ETL tasks.
Creates production-ready containers for data tools: multi-stage builds for Spark jobs, image size optimization with Python packages (numpy, pandas). Configures docker-compose for full data stack.
Defines Docker standards for data team: base images, security scanning, registry management. Implements container-based testing for data pipelines.
Distributed Tracing
▼
Understands OpenTelemetry for data: pipeline execution tracing, processing metrics. Configures basic instrumentation.
Instruments data pipelines: distributed tracing through OpenTelemetry, custom metrics for throughput/latency, correlation IDs.
Defines observability standards for data: mandatory instrumentation, metric requirements, alerting policies.
E2E Testing
▼
Tests data pipelines end-to-end: verifies source-to-target data flow, output validation. Compares expected vs actual results.
Designs e2e data tests: full pipeline validation, data reconciliation, schema compatibility testing. Integrates into CI.
Implements e2e testing standards: mandatory pipeline validation, data quality requirements, testing infrastructure.
Git & Workflows
▼
Works with Git in data projects: commit, push, pull, branch for dbt models and Airflow DAGs. Writes meaningful commit messages. Resolves merge conflicts in SQL files and config files.
Uses Git advanced features: interactive rebase, cherry-pick for pipeline hotfixes. Works with monorepo (dbt + Airflow + configs). Configures .gitignore for data artifacts (parquet, csv).
Shapes Git workflow for data team: branching model, PR process with dbt CI, CODEOWNERS for critical models. Defines rules for mono/multi-repo data projects.
GraphQL
▼
Understands GraphQL for data access: query API for data catalog, metadata queries. Works with GraphQL-based data tools.
Designs GraphQL API for data: metadata queries, lineage exploration, data preview endpoints. Implements authorization.
Defines GraphQL standards for data: schema design, access patterns, performance requirements.
Integration Testing
▼
Tests data pipeline integration: source → transform → sink validation, connector testing. Verifies data flow integrity.
Designs integration tests: end-to-end pipeline testing, cross-system data validation, staging environment testing.
Implements integration testing standards: mandatory pipeline validation, data reconciliation, cross-team data quality testing.
Kubernetes & Orchestration
▼
Understands Kubernetes for running data workloads: pods, services, deployments. Deploys Airflow through Helm chart. Reads pod logs for debugging pipeline failures.
Configures Kubernetes for data tools: Airflow KubernetesExecutor, Spark on K8s, resource requests/limits for data jobs. Understands PV/PVC for persistent storage. Configures autoscaling for batch workloads.
Defines K8s strategy for data platform: namespace isolation, resource quotas per team, scheduling policies for batch vs streaming. Implements GitOps for infrastructure.
Logging
▼
Configures logging in data pipelines: Python logging for ETL scripts, structured JSON format. Logs processing stages, record counts, execution time. Does not log PII.
Designs logging for data platform: correlation ID for end-to-end tracking, structured logs with metadata (dag_id, task_id, run_id). Integrates with ELK/Loki for centralized log analysis.
Defines logging standards for data platform: mandatory fields (pipeline_id, dataset, stage), log levels policy, retention. Implements log-based monitoring for pipeline SLA.
Message Queues & Event Streaming
▼
Uses Apache Kafka at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Apache Kafka in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Defines Kafka strategy for data platform: topic naming conventions, retention policies, schema evolution rules. Implements Kafka Connect for integration with RDBMS, S3, Elasticsearch.
Designs organizational streaming platform: multi-cluster Kafka, MirrorMaker 2, tiered storage. Defines governance for event schemas and data contracts between producer/consumer teams.
Metrics & Monitoring
▼
Views dashboards in Grafana. Understands what metrics are (CPU, memory, latency). Can find the right dashboard and analyze a simple graph.
Adds custom metrics to applications (counter, gauge, histogram). Writes PromQL queries for dashboards. Creates Grafana dashboards. Configures basic alerts (high error rate, high latency).
Designs observability stack for services. Creates SLI/SLO dashboards. Writes complex PromQL (rate, histogram_quantile, recording rules). Configures Alertmanager with routing and silencing. Optimizes metric cardinality. Integrates with PagerDuty/OpsGenie.
OOP & Design Patterns
▼
Applies patterns in data engineering: ETL vs ELT pattern, batch vs streaming. Understands data pipeline patterns and their application.
Uses data patterns: slowly changing dimensions, change data capture, star/snowflake schema. Implements idempotent pipelines.
Defines data architectural patterns: reference data architectures, pattern catalog, design review criteria.
Understands the fundamentals of OOP & SOLID Principles at a basic level. Applies simple concepts in work tasks using Python/SQL. Follows recommendations from senior developers when solving problems.
Independently applies OOP/SOLID in data pipeline code: proper abstraction for ETL stages, interface-based connector design for data sources, single responsibility in transformation modules. Understands trade-offs between OOP patterns and declarative approaches in Spark/Airflow codebases.
Defines pipeline code structure standards: abstract classes for ETL components, Strategy pattern for different data sources, Factory for reader/writer. Trains team on applying SOLID in data engineering.
Prompt Engineering
▼
Crafts prompts for data code generation: describes tables, types, transformations. Gets working SQL queries and Python scripts for typical ETL tasks.
Writes structured prompts: generating dbt models with ref()/source(), Airflow DAGs with correct dependencies, data quality checks. Uses chain-of-thought for complex SQL.
Builds prompt library for data team: templates for typical tasks (ETL design, schema review, data quality), documentation templates. Defines AI automation boundaries.
REST API
▼
Works with data source REST APIs: GET requests for extraction, pagination, rate limit handling. Uses requests/httpx for external API integration in ETL pipelines.
Designs API source integrations: OAuth2 authorization, retry with exponential backoff, cursor-based pagination for large exports. Creates reusable API connectors for Airflow operators.
Defines API standards for data platform: contracts for data producers, webhook interfaces for event-driven ingestion, rate limiting for shared data API. Conducts API design reviews.
System Design
▼
Understands system design for data: distributed systems basics, CAP theorem, data replication. Evaluates scalability requirements.
Designs data systems: partitioning strategies, replication models, consistency trade-offs. Defines capacity requirements for pipelines.
Defines system design standards: architecture review criteria, scalability requirements, reliability guidelines for data systems.
Type Systems
▼
Applies type safety in data code: typed Python (mypy), schema definitions, data class models. Uses typed configurations.
Ensures type safety: Pydantic models for data validation, typed Spark DataFrames, strict schema enforcement.
Defines type safety standards: schema governance, validation requirements, typed pipeline policies.
Unit Testing
▼
Writes unit tests for ETL components: tests transformations, parsers, validators. Uses pytest for Python data pipeline code.
Tests data pipeline code: unit tests for Spark transformations, dbt models, Airflow operators. Mocks data sources and external systems.
Defines testing standards for data engineering: mandatory data tests, quality gates, pipeline validation requirements. Implements testing culture.
Web Frameworks
▼
Uses Python Web Frameworks at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Python Web Frameworks in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Defines data services architecture: FastAPI for metadata API, internal APIs for data contracts, webhook endpoints for event-driven pipelines. Chooses between sync/async approach for different data request types.
Designs data platform API layer: self-service data portal, metadata API, data marketplace. Defines API standards for data access through query engines (Trino, Presto).