AI Coding Assistants 1
▼
Uses Copilot for data code generation: SQL queries, pandas transformations, pytest fixtures for data testing. Understands when suggestions are correct. Writes docstrings for better suggestions.
Works productively with Copilot: generates dbt models from descriptions, Airflow DAGs, data validation rules. Uses Copilot Chat for SQL query analysis. Critically evaluates suggestions for correctness.
Maximizes productivity: generating complex SQL/dbt macros, Terraform modules, data quality rules. Builds workflow where AI accelerates routine (boilerplate), engineer focuses on architecture.
Implements Copilot in the team: configures policies, defines usage boundaries for data engineering. Evaluates ROI. Monitors security: credential leaks, SQL injection in generated code.
Background Jobs & Task Queues 1
▼
Uses Task Queues at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Task Queues in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Designs asynchronous processing through Celery/RQ: background tasks for data processing, priority queues for different SLAs, dead letter queues for failed tasks. Configures retry policies and monitoring.
Defines task orchestration strategy: Celery for micro-tasks vs Airflow for DAGs, choosing between push/pull models. Implements idempotency standards and queue monitoring.
Batch Processing 4
▼
Understands Apache Spark fundamentals for data engineering: RDD/DataFrame APIs, basic transformations and actions, reading/writing Parquet/CSV/JSON. Follows team patterns for PySpark job structure, SparkSession configuration, and cluster resource allocation.
Independently implements Spark data pipelines: optimizes shuffle operations and partitioning strategies, implements Structured Streaming for real-time ETL, manages Delta Lake tables with ACID transactions. Tunes Spark configurations for memory, parallelism, and cost efficiency.
Designs Spark-based data platform architecture: multi-tenant cluster management, cost-optimized workload scheduling with YARN/Kubernetes, and lakehouse architecture with Delta Lake/Iceberg. Implements data quality frameworks, CDC pipelines, and Spark application performance monitoring.
Defines Spark standards: coding guidelines, job submission patterns, resource allocation policies. Chooses between PySpark and Spark SQL by scenario. Implements unit testing for Spark jobs through chispa.
Creates dbt models: SELECT queries with ref() and source(), staging and mart models. Writes generic tests (unique, not_null). Understands DAG and dependencies between models.
Designs dbt project: custom macros, incremental models, snapshots for SCD Type 2. Configures environments (dev/staging/prod). Optimizes models through materialization selection.
Designs dbt architecture: multi-project setup, package management, custom generic tests. Implements unit tests for complex transformations. Optimizes performance: incremental + merge, partition-based.
Defines dbt standards: project structure, naming conventions, documentation requirements, PR review checklist. Implements dbt metrics layer and exposure definitions for data contracts.
Processes data through pandas: read_csv/read_parquet, filtering, grouping, merge. Understands DataFrame API. Works with data types and missing values (fillna, dropna).
Optimizes processing through pandas/Polars: chunked reading for large files, category dtype for memory, vectorized operations instead of iterrows. Migrates to Polars for performance-critical tasks.
Designs data processing: Polars for single-node high-performance, pandas for quick prototyping, PySpark for distributed. Selects tool by volume and processing pattern. Optimizes memory management.
Defines data processing standards: when pandas/Polars vs Spark, coding guidelines, testing patterns. Implements benchmarking for tool selection. Trains team on Polars adoption.
Writes SQL for ETL: INSERT INTO SELECT, MERGE for upserts, CTE for readable transformations. Uses window functions (ROW_NUMBER, LAG, LEAD) for data processing.
Designs SQL transformations: stored procedures for complex ETL, parameterized queries, temp tables for intermediate computations. Optimizes execution plans. Manages transaction control.
Designs SQL-based ETL architecture: ELT pattern (load-then-transform), incremental processing through merge/upsert, materialized views for performance. Integrates with dbt for version-controlled SQL.
Defines SQL standards for data team: style guide, review checklist, performance budgets. Chooses between SQL-based ETL (dbt) and code-based (PySpark) by scenario.
Caching 1
▼
Uses Redis at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Redis in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Uses Redis for data pipeline optimization: lookup table caching, deduplication through Redis Sets, rate limiting for API sources. Implements Redis Streams for lightweight event processing.
Defines caching strategy in data pipelines: Redis for hot reference data, TTL policies for cache freshness. Establishes metrics and SLA for cache-dependent pipelines.
CI/CD 1
▼
Understands CI/CD for data pipelines: running dbt/SQL tests on PR, linting Python code. Reads failed build logs. Understands workflow triggers.
Configures CI/CD for data projects: testing dbt models, SQL validation, data contract checking in PR. Automates Airflow DAG deployment. Configures environments for staging/production.
Designs CI/CD for data platform: pipeline testing (unit + integration), automated schema validation, blue/green deployment for Airflow. Implements data quality gates in deployment pipeline.
Defines CI/CD standards for data team: mandatory checks (dbt test, SQL lint, contract validation), deployment strategy, release management for data pipelines.
Clean Code & Refactoring 1
▼
Understands basic code quality principles for data pipeline code. Follows team conventions for ETL script structure and SQL formatting. Writes simple, clean data transformation functions with proper logging. Accepts code review feedback on pipeline code organization.
Independently applies code quality practices in data pipeline development. Writes clean ETL/ELT code with proper error handling, idempotency, and logging. Understands trade-offs between pipeline complexity and maintainability. Reviews data pipeline code for data quality checks, schema evolution handling, and resource efficiency.
Designs code quality standards for data pipelines: Airflow DAG structure, Spark job organization, ETL testing patterns. Refactors monolithic data transformations into modular, idempotent pipeline stages. Implements quality gates for data contract enforcement and pipeline reliability.
Shapes pipeline code quality standards: ruff/black for formatting, mypy for typing, pytest for transformation unit tests. Implements data contracts and schema validation in CI.
Code Review 1
▼
Participates in data code reviews: checks SQL style, dbt model structure, naming conventions. Leaves constructive comments. Learns from feedback on own PRs.
Conducts quality reviews: checks SQL performance (joins, window functions), dbt materialization choice, data quality test coverage. Gives feedback on data modeling. Suggests optimizations.
Conducts architectural reviews: evaluates pipeline design, data model decisions, schema evolution impact. Reviews Airflow DAG structure, Terraform changes for data infra.
Builds code review culture: defines checklist (SQL performance, test coverage, documentation), establishes SLA. Automates checks (dbt test, SQL lint, contract validation).
Data Governance 3
▼
Uses data catalog for dataset discovery: browses table descriptions, owners, lineage. Documents own tables: field descriptions, business glossary terms.
Configures data catalog: integration with metadata sources (Hive, Glue, dbt), automated harvesting. Creates business glossary. Tags data for classification (PII, financial).
Designs metadata management: DataHub/OpenMetadata/Amundsen setup, custom connectors for internal systems, automated lineage extraction from Spark/Airflow. Integrates catalog with data quality.
Defines data cataloging standards: mandatory metadata, ownership policy, data stewardship process. Implements data discovery workflow for self-service analytics.
Understands data contracts: schema definition, SLA (freshness, completeness). Follows established contracts when creating tables. Documents schema of own data products.
Creates data contracts: YAML/JSON schema definitions, quality checks, SLA metrics. Integrates contract validation into CI/CD. Configures alerting on contract violations.
Designs data contract framework: schema registry integration, automated validation pipeline, versioning strategy. Implements contract testing between producers and consumers.
Defines data contract standards: template, review process, enforcement policy. Coordinates between producer and consumer teams. Implements data-as-a-product culture.
Reads data lineage graphs: understands where data comes from and where it goes. Uses lineage for impact analysis when sources change. Documents dependencies of own models.
Configures automated lineage collection: Airflow/dbt/Spark integration with lineage system. Uses lineage for debugging data quality issues. Visualizes dependencies in DataHub/OpenMetadata.
Designs lineage infrastructure: column-level lineage, cross-system tracking (operational → analytical), custom extractors for internal tools. Integrates lineage with data catalog and quality monitoring.
Defines lineage standards: coverage, granularity (table vs column level), freshness. Uses lineage for impact analysis during schema changes. Implements lineage-based alerting.
Data Lakehouse 2
▼
Understands medallion architecture principles (bronze/silver/gold). Ingests data into raw landing zones using batch loaders and schema registries. Follows established patterns for Parquet/ORC file layout and catalog metadata.
Independently designs ETL pipelines across data lake zones with schema evolution support. Optimizes storage costs using lifecycle policies, compaction, and tiered storage. Implements data quality gates between medallion layers with automated validation.
Designs data architecture with Data Lake Architecture. Optimizes for big data. Implements data governance and quality frameworks.
Defines data lake standards: zone architecture (bronze/silver/gold), file formats, partition strategies. Implements access control and data classification. Coordinates between data producers and consumers.
Understands the fundamentals of Delta Lake / Apache Iceberg. Applies basic practices in daily work. Follows recommendations from the team and documentation.
Independently implements data pipelines with Delta Lake/Apache Iceberg. Optimizes performance. Ensures data quality.
Designs data architecture with Delta Lake/Apache Iceberg. Optimizes for big data. Implements data governance and quality frameworks.
Defines Delta Lake/Iceberg standards: table format selection, partitioning strategy, compaction schedules. Implements time-travel for data debugging and schema enforcement.
Data Modeling 2
▼
Designs data warehouse tables: star schema (fact + dimension), SCD types (Type 1, 2). Creates staging, intermediate and mart layers. Understands normalization and denormalization for analytics.
Designs dimensional models: Kimball methodology (conformed dimensions, bus matrix), Data Vault (hubs, links, satellites). Applies SCD Type 2 with effective dates. Models semi-structured data.
Designs enterprise data models: Data Vault 2.0 for flexibility, Anchor Modeling for high-change environments, wide tables for ClickHouse. Defines layered architecture: raw → staging → curated → mart.
Defines data modeling standards: naming conventions, documentation requirements, review process. Chooses approach (Kimball vs Inmon vs Data Vault) by context. Trains team on modeling best practices.
Performs data warehouse schema migrations: CREATE/ALTER TABLE through SQL scripts. Understands DDL idempotency. Tests migrations on dev environment before production.
Designs schema evolution for data pipelines: backward-compatible migrations, expand-contract for zero-downtime, versioning through Flyway/Alembic. Handles schema drift in sources.
Designs migration strategy: zero-downtime migrations for data warehouse, schema evolution in Parquet/Avro, backward compatibility in data contracts. Automates through CI/CD.
Defines migration standards: review process for schema changes, rollback strategies, compatibility matrix. Coordinates migrations between upstream producers and downstream consumers.
Data Orchestration 2
▼
Creates Airflow DAGs: PythonOperator, BashOperator, task dependencies. Understands execution date, catchup, schedule interval. Monitors runs in Airflow UI. Debugs failed tasks through logs.
Designs Airflow DAGs: dynamic task generation, XCom for data passing, TaskGroups for organization. Uses sensors, hooks for external system integration. Configures connections and variables.
Designs Airflow architecture: KubernetesExecutor for dynamic scaling, custom operators/hooks, DAG factory pattern for generation. Optimizes performance: pool management, priority weight, concurrency.
Defines Airflow standards: DAG structure, naming conventions, testing requirements, deployment workflow. Chooses between Airflow and alternatives (Dagster, Prefect) by scenario.
Uses Dagster or Prefect to build and schedule basic ETL pipelines. Understands assets, tasks, and flow concepts. Monitors pipeline runs and handles retries for transient failures.
Independently implements data pipelines with Dagster/Prefect. Optimizes performance. Ensures data quality.
Designs data architecture with Dagster/Prefect. Optimizes for big data. Implements data governance and quality frameworks.
Defines orchestration standards: Dagster vs Prefect vs Airflow selection by project, migration strategy. Evaluates software-defined assets (Dagster) vs task-based (Airflow) approaches.
Data Quality 1
▼
Writes basic data quality checks: NOT NULL, unique constraints, value ranges. Uses dbt tests or Great Expectations for validation. Understands metrics: completeness, accuracy, timeliness.
Configures data quality framework: Great Expectations/Soda for automated checks, custom expectations, alerting on failures. Monitors data freshness and volume anomalies.
Designs data quality system: multi-layer validation (source → staging → mart), anomaly detection (statistical), automated remediation. Integrates quality metrics into data catalog.
Defines data quality standards: SLA per dataset, quality dimensions (accuracy, completeness, consistency, timeliness), ownership model. Implements data quality scorecard.
Data Warehousing 1
▼
Sets up basic warehouse tables in Snowflake, BigQuery, or Redshift following team conventions. Implements simple partitioning and clustering strategies for common query patterns. Builds straightforward ELT pipelines that load raw data into staging areas and applies basic transformations for downstream consumption.
Designs DWH components: dimensional modeling per Kimball, SCD Types (1, 2, 3), aggregate tables. Configures incremental loading. Optimizes performance through distribution keys and sort keys.
Designs data warehouse architecture: multi-layer (staging → ODS → DWH → marts), Slowly Changing Dimensions, bridge tables for many-to-many. Selects cloud DWH (Redshift/BigQuery/Snowflake) by requirements.
Defines DWH standards: modeling methodology (Kimball vs Inmon), naming conventions, testing requirements. Coordinates between domain teams for conformed dimensions. Conducts architectural reviews.
Database Administration 2
▼
Understands backup importance in data pipelines: checkpoint files, intermediate results. Uses snapshots before destructive operations. Knows how to recover data from staging tables.
Configures backup for data pipeline artifacts: intermediate data versioning in S3, point-in-time recovery in PostgreSQL. Implements rollback mechanisms for ETL processes.
Designs disaster recovery for data platform: backup strategies for different storage (HDFS snapshots, S3 versioning, DB dumps). Implements time-travel in Delta Lake/Iceberg for data recovery.
Defines DR standards for data platform: RPO/RTO by data tier, retention policies, automated backup verification. Conducts DR drills and recovery testing.
Understands basic database replication concepts: primary-replica topology, synchronous vs asynchronous replication, and replication lag. Monitors replica health using standard tools and dashboards. Follows team runbooks for failover procedures and connection string configuration.
Configures and manages database replication for data pipelines: sets up read replicas for ETL offloading, handles schema migrations across replicated environments, and implements change data capture (CDC). Understands consistency trade-offs and designs data flows accounting for replication lag.
Designs high-availability data architectures with multi-region replication, cross-database CDC pipelines, and event-driven synchronization. Optimizes replication for large-scale analytical workloads with minimal production impact. Architects disaster recovery strategies with defined RPO/RTO targets.
Defines product-level data strategy. Establishes Replication and High Availability standards. Conducts data schema and scaling strategy reviews.
Database Optimization 2
▼
Creates indexes for accelerating extraction queries: B-tree for equality, composite for multi-column WHERE. Uses EXPLAIN to verify index usage in ETL queries.
Designs indexing strategy for ETL sources: partial indexes for active records, covering indexes for frequent extractions. Understands read/write performance trade-offs in OLTP sources.
Optimizes indexing for data pipelines: GIN for JSONB, BRIN for time-series, bloom filters in ClickHouse. Designs indexing strategy considering batch vs streaming access patterns.
Defines indexing standards: mandatory EXPLAIN review for extraction queries, automated missing index detection in slow query log. Balances read/write performance.
Optimizes SQL queries for ETL: avoids SELECT *, uses WHERE for source-side filtering. Understands pushdown predicates. Analyzes EXPLAIN for full table scans.
Optimizes extraction and transformation: predicate pushdown, partition pruning, choosing between JOIN and subquery. Profiles SQL queries in Airflow through query tags. Optimizes Spark SQL execution plans.
Designs optimal data access patterns: incremental extraction through watermarks, micro-batch vs full-load trade-offs. Optimizes Spark query plans: broadcast joins, AQE, partition coalescing.
Defines query performance standards: SLA on extraction time, resource budgets. Implements automated slow query monitoring and regression detection.
gRPC 1
▼
Understands Protobuf schemas for data serialization. Uses gRPC for high-performance data transfer between services. Works with .proto files for defining data contracts.
Designs Protobuf schemas for data pipelines: schema evolution with backward compatibility, nested messages for complex data. Uses gRPC streaming for real-time data transfer.
Designs gRPC interfaces for data services: bidirectional streaming for CDC, server streaming for bulk data delivery. Integrates Protobuf with Schema Registry for centralized schema management.
Defines serialization strategy: Protobuf vs Avro vs JSON for different scenarios (streaming vs batch, internal vs external). Implements schema management standards and compatibility checking.
Infrastructure as Code 1
▼
Uses Terraform for creating data infrastructure: S3 buckets, RDS instances, Glue catalogs. Understands state, plan, apply. Reads existing modules.
Writes Terraform modules for data stack: Redshift/BigQuery clusters, Kafka topics, Airflow MWAA. Manages environments through workspaces. Configures remote state in S3.
Designs IaC for data platform: reusable modules for data services (EMR, Glue, Kinesis), environment promotion pipeline. Automates data infrastructure provisioning.
Defines IaC standards for data platform: module library, naming conventions, change management process. Implements policy-as-code (Sentinel/OPA) for data infrastructure.
Kubernetes & Orchestration 1
▼
Understands Kubernetes for running data workloads: pods, services, deployments. Deploys Airflow through Helm chart. Reads pod logs for debugging pipeline failures.
Configures Kubernetes for data tools: Airflow KubernetesExecutor, Spark on K8s, resource requests/limits for data jobs. Understands PV/PVC for persistent storage. Configures autoscaling for batch workloads.
Designs Kubernetes infrastructure for data: Spark Operator, Airflow KubernetesExecutor with dynamic resource allocation, Argo Workflows for ML pipelines. Optimizes node pools for different workloads.
Defines K8s strategy for data platform: namespace isolation, resource quotas per team, scheduling policies for batch vs streaming. Implements GitOps for infrastructure.
Logging 1
▼
Configures logging in data pipelines: Python logging for ETL scripts, structured JSON format. Logs processing stages, record counts, execution time. Does not log PII.
Designs logging for data platform: correlation ID for end-to-end tracking, structured logs with metadata (dag_id, task_id, run_id). Integrates with ELK/Loki for centralized log analysis.
Designs observability for data pipelines: structured logging with data lineage context, metrics emission (records processed, data quality scores), anomaly alerting. Integrates with OpenTelemetry.
Defines logging standards for data platform: mandatory fields (pipeline_id, dataset, stage), log levels policy, retention. Implements log-based monitoring for pipeline SLA.
Message Queues & Event Streaming 1
▼
Uses Apache Kafka at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Apache Kafka in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Designs event-driven ingestion: Kafka producers for CDC, consumer groups for parallel processing, Schema Registry with Avro/Protobuf. Optimizes throughput: batching, compression, partition assignment.
Defines Kafka strategy for data platform: topic naming conventions, retention policies, schema evolution rules. Implements Kafka Connect for integration with RDBMS, S3, Elasticsearch.
Networking 1
▼
Understands networking basics for data pipelines: TCP/IP for database connections, DNS for service discovery. Configures connectivity between ETL and data sources through VPN/VPC peering.
Configures network connectivity for data infrastructure: VPC peering for cross-account access, PrivateLink for managed services, security groups for data pipeline components. Diagnoses connection issues.
Designs data platform network architecture: private connectivity to data sources, Transit Gateway for multi-VPC, Direct Connect for on-prem data centers. Optimizes network performance for bulk data transfer.
NoSQL Databases 2
▼
Writes basic CQL queries to read and insert data into Cassandra tables. Understands partition keys and clustering columns in existing schemas. Follows data modeling guidelines established by the team.
Designs Cassandra data models optimized for query-driven access patterns. Implements efficient batch operations and manages TTL-based data lifecycle. Tunes read/write consistency levels to balance latency and durability.
Architects multi-datacenter Cassandra deployments for real-time data pipelines. Optimizes cluster topology, compaction strategies, and partition distribution for petabyte-scale workloads. Designs migration strategies between schema versions.
Defines product-level data strategy. Establishes Apache Cassandra standards. Conducts data schema and scaling strategy reviews.
Writes analytical queries in ClickHouse: SELECT with GROUP BY, ORDER BY, LIMIT. Understands MergeTree engine and partitioning. Inserts data through INSERT and uses tabular formats.
Designs ClickHouse tables for analytical pipelines: engine selection (MergeTree, AggregatingMergeTree, ReplacingMergeTree), partitioning by date, materialized views for pre-aggregation. Optimizes insertion through batch inserts.
Designs ClickHouse architecture for data warehouse: distributed tables, sharding strategy, dictionary tables for joins. Optimizes queries: projections, skip indexes, query pipeline tuning. Configures Kafka engine for streaming ingestion.
Defines ClickHouse standards: schema design guidelines, naming conventions, monitoring through system tables. Chooses ClickHouse vs other OLAP (Druid, Pinot, DuckDB) by use case.
Relational Databases 1
▼
Writes SQL queries for data extraction: SELECT with JOIN, GROUP BY, window functions. Works with psycopg2/SQLAlchemy. Understands PostgreSQL types: JSONB, arrays, timestamps with timezone.
Optimizes extraction from PostgreSQL: COPY for bulk export, cursor-based pagination, partitioned tables. Configures logical replication for CDC. Designs staging tables for ETL.
Designs PostgreSQL as data pipeline source/sink: CDC through Debezium, materialized views for aggregates, FDW for federation. Optimizes VACUUM for high-write staging tables.
Defines PostgreSQL standards for data platform: when PostgreSQL vs analytical databases (ClickHouse/Redshift), connection pooling through PgBouncer for ETL workloads. Conducts extraction pattern reviews.
REST API 1
▼
Works with data source REST APIs: GET requests for extraction, pagination, rate limit handling. Uses requests/httpx for external API integration in ETL pipelines.
Designs API source integrations: OAuth2 authorization, retry with exponential backoff, cursor-based pagination for large exports. Creates reusable API connectors for Airflow operators.
Designs data platform API layer: REST for metadata catalog, webhook endpoints for event-driven ingestion. Implements data API with query parameters for self-service data access.
Defines API standards for data platform: contracts for data producers, webhook interfaces for event-driven ingestion, rate limiting for shared data API. Conducts API design reviews.
Search Engines 1
▼
Uses Elasticsearch/OpenSearch at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Elasticsearch/OpenSearch in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Designs search index for data catalog: mapping, custom analyzers for metadata, nested documents for lineage. Optimizes bulk indexing pipeline for millions of catalog records.
Defines search strategy for data platform: Elasticsearch for metadata discovery, full-text search across dataset descriptions. Implements ILM for index lifecycle management.
Stream Processing 1
▼
Understands Kafka Streams fundamentals including KStream/KTable duality. Writes simple stream consumers and producers for data pipeline ingestion stages.
Builds real-time ETL pipelines with Kafka Streams for data transformation and enrichment. Implements exactly-once semantics and monitors consumer lag across processing stages.
Designs end-to-end streaming data architectures with Kafka Streams for high-throughput pipelines. Orchestrates complex event processing, implements schema evolution strategies, and optimizes for backpressure handling.
Defines streaming standards: Kafka Streams vs Flink, windowing policies, state management. Implements consumer lag and processing latency monitoring. Chooses between exactly-once and at-least-once.
Web Frameworks 1
▼
Uses Python Web Frameworks at a basic level in Airflow/dbt. Performs simple tasks using established templates. Understands basic concepts and follows team practices.
Independently implements tasks with Python Web Frameworks in Airflow/dbt. Understands internals and optimizes performance. Writes tests with great_expectations.
Develops data API on FastAPI: endpoints for metadata access, catalog, lineage. Implements async request processing for data warehouse. Integrates with Airflow REST API for DAG management.
Defines data services architecture: FastAPI for metadata API, internal APIs for data contracts, webhook endpoints for event-driven pipelines. Chooses between sync/async approach for different data request types.