Data Engineer
Building and maintaining data pipelines, data warehouses, ensuring data availability and quality
Data Engineer is a role in the Data Engineering family. It has 64 skills across 5 levels (from Junior to Principal). 186 skills are mandatory. Key domains: Programming Fundamentals, Backend Development, Database Management.
Technology Stack
Focus by Level
Writing ETL scripts (Python/SQL). Working with Airflow DAGs. Loading data into warehouse. Monitoring pipelines. SQL queries for analysts.
Designing data pipelines. Working with Spark/Flink. Optimizing SQL queries on large datasets. Data quality checks. Working with data warehouse.
Data platform architecture. Designing data lake/lakehouse. Storage cost optimization. Designing real-time pipelines. Mentoring.
Data platform strategy. DataOps practices. Governance and lineage. Coordination with ML and Analytics. Data quality standards.
Enterprise data strategy. Multi-cloud data architecture. Data mesh. Cost optimization at scale. Vendor evaluation.
Skill Matrix
64 skills × 5 levels. Click on a cell for details.
AI-Assisted Development
4 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| GitHub Copilot | A | W | A | E | E |
| Cursor IDE | A | W | A | A | — |
| ChatGPT / Claude | A | W | A | E | E |
| Prompt Engineering for Code | A | W | A | E | — |
API & Integration
4 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| REST API Design | A | W | A | E | E |
| GraphQL Design | A | W | A | E | E |
| gRPC & Protocol Buffers | A | W | A | E | E |
| API Documentation | A | W | A | E | E |
Architecture & System Design
1 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| System Design Fundamentals | A | W | A | E | E |
Backend Development
6 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Python Web Frameworks | A | W | A | E | E |
| Apache Kafka | A | W | A | E | E |
| Redis | A | W | A | E | E |
| Task Queues | A | W | A | E | E |
| Elasticsearch / OpenSearch | A | W | A | E | E |
| S3 / Object Storage | A | W | A | E | E |
Cloud & Infrastructure
5 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Docker | A | W | A | E | E |
| Kubernetes Core | A | W | A | E | E |
| Terraform | A | W | A | E | E |
| AWS | A | W | A | E | E |
| Network Fundamentals | A | W | A | — | — |
Data Engineering
14 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Apache Spark | A | W | A | E | E |
| dbt | A | W | A | E | E |
| Pandas / Polars | A | W | A | E | E |
| SQL-based ETL | A | W | A | E | E |
| Stream Processing | A | W | A | E | E |
| Delta Lake / Apache Iceberg | A | W | A | E | E |
| Data Lake Architecture | A | W | A | E | E |
| Data Warehouse Design | A | W | A | E | E |
| Data Catalog | A | W | A | E | E |
| Data Lineage | A | W | A | E | E |
| Data Contracts | A | W | A | E | E |
| Data Quality | A | W | A | E | E |
| Apache Airflow | A | W | A | E | E |
| Dagster / Prefect | A | W | A | E | E |
Database Management
9 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| PostgreSQL | A | W | A | E | E |
| ClickHouse | A | W | A | E | E |
| Apache Cassandra | A | W | A | E | E |
| Database Indexing | A | W | A | E | E |
| Query Optimization | A | W | A | E | E |
| Replication & High Availability | A | W | A | E | E |
| Backup & Disaster Recovery | A | W | A | E | E |
| Data Modeling & Schema Design | A | W | A | E | E |
| Database Migrations | A | W | A | E | E |
DevOps & CI/CD
1 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| GitHub Actions / GitLab CI | A | W | A | E | E |
Observability & Monitoring
4 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Structured Logging | A | W | A | E | E |
| Prometheus & Grafana | A | W | A | — | — |
| OpenTelemetry | A | W | A | E | E |
| SLI / SLO / SLA | A | W | A | E | E |
Programming Fundamentals
8 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Algorithms & Complexity | A | W | A | E | E |
| Data Structures | A | W | A | E | E |
| OOP & SOLID Principles | A | W | A | E | E |
| Design Patterns | A | W | A | E | E |
| Multithreading | A | W | A | E | E |
| Async Programming | A | W | A | E | E |
| Code Quality & Refactoring | A | W | A | E | E |
| Type Safety & Type Systems | A | W | A | E | E |
Security
3 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| OWASP & Application Security | A | W | A | E | E |
| Secure Coding Practices | A | W | A | E | E |
| JWT / OAuth2 / OIDC | A | W | A | E | E |
Testing & QA
3 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Unit Testing | A | W | A | E | E |
| Integration Testing | A | W | A | E | E |
| E2E Testing | A | W | A | E | E |
Version Control & Collaboration
2 skills| Skills | Jun | Mid | Sen | Lead | Princ |
|---|---|---|---|---|---|
| Git Advanced | A | W | A | E | E |
| Code Review | A | W | A | E | E |
FAQ
What skills are needed for the Data Engineer role?
The Data Engineer role requires 64 skills, of which 186 are mandatory. Skills are distributed across 5 levels: from Junior to Principal. See full matrix.
How to advance to the next level in the Data Engineer role?
Use the Grade Calculator to assess your current level and get personalized recommendations. The system will show which skills need to be developed for the next level.
What tech stack is used in the Data Engineer role?
The stack includes 5 technologies at different levels. Python 3.11+, SQL, Apache Airflow, PostgreSQL/ClickHouse, pandas, Docker, Git, Python 3.12+, SQL advanced, Airflow, Spark/PySpark, ClickHouse/BigQuery, dbt, Kafka basics, S3/GCS, Python, Spark/Flink, ClickHouse/BigQuery/Snowflake, Kafka/Debezium, Delta Lake/Iceberg, Terraform, Kubernetes, Data lineage (OpenLineage)...
How does the community define requirements for the Data Engineer role?
Role requirements are shaped by the community through a proposal system. Any member can suggest changes that go through voting and expert review.