What a Lead } should know

Establishes performance budgets for data pipelines. Conducts reviews of sorting and aggregation algorithms in ETL processes. Implements Spark job profiling and shuffle operation optimization.

Defines asynchronous processing strategy: asyncio for API calls in ETL, aiohttp for bulk loading, async generators for streaming. Implements backpressure handling standards in data pipelines.

Shapes pipeline code quality standards: ruff/black for formatting, mypy for typing, pytest for transformation unit tests. Implements data contracts and schema validation in CI.

Defines data structure standards in pipeline code: dataclasses for schemas, TypedDict for configs, Pydantic models for validation. Reviews structure choices in context of memory-efficient batch processing.

Defines pipeline code structure standards: abstract classes for ETL components, Strategy pattern for different data sources, Factory for reader/writer. Trains team on applying SOLID in data engineering.

Defines Kafka strategy for data platform: topic naming conventions, retention policies, schema evolution rules. Implements Kafka Connect for integration with RDBMS, S3, Elasticsearch.

Defines search strategy for data platform: Elasticsearch for metadata discovery, full-text search across dataset descriptions. Implements ILM for index lifecycle management.

Defines data services architecture: FastAPI for metadata API, internal APIs for data contracts, webhook endpoints for event-driven pipelines. Chooses between sync/async approach for different data request types.

Defines caching strategy in data pipelines: Redis for hot reference data, TTL policies for cache freshness. Establishes metrics and SLA for cache-dependent pipelines.

Defines S3 strategy for data lake: naming conventions, access patterns, versioning policy. Implements bucket policies and cross-account access for multi-team data sharing.

Defines task orchestration strategy: Celery for micro-tasks vs Airflow for DAGs, choosing between push/pull models. Implements idempotency standards and queue monitoring.

Defines product-level data strategy. Establishes Apache Cassandra standards. Conducts data schema and scaling strategy reviews.

Defines DR standards for data platform: RPO/RTO by data tier, retention policies, automated backup verification. Conducts DR drills and recovery testing.

Defines ClickHouse standards: schema design guidelines, naming conventions, monitoring through system tables. Chooses ClickHouse vs other OLAP (Druid, Pinot, DuckDB) by use case.

Defines data modeling standards: naming conventions, documentation requirements, review process. Chooses approach (Kimball vs Inmon vs Data Vault) by context. Trains team on modeling best practices.

Defines indexing standards: mandatory EXPLAIN review for extraction queries, automated missing index detection in slow query log. Balances read/write performance.

Defines migration standards: review process for schema changes, rollback strategies, compatibility matrix. Coordinates migrations between upstream producers and downstream consumers.

Defines PostgreSQL standards for data platform: when PostgreSQL vs analytical databases (ClickHouse/Redshift), connection pooling through PgBouncer for ETL workloads. Conducts extraction pattern reviews.

Defines query performance standards: SLA on extraction time, resource budgets. Implements automated slow query monitoring and regression detection.

Defines product-level data strategy. Establishes Replication and High Availability standards. Conducts data schema and scaling strategy reviews.

Defines serialization strategy: Protobuf vs Avro vs JSON for different scenarios (streaming vs batch, internal vs external). Implements schema management standards and compatibility checking.

Defines API standards for data platform: contracts for data producers, webhook interfaces for event-driven ingestion, rate limiting for shared data API. Conducts API design reviews.

Defines AWS strategy for data platform: multi-account for prod/staging, Lake Formation for fine-grained access control, cost allocation by project. Chooses managed vs self-hosted.

Defines Docker standards for data team: base images, security scanning, registry management. Implements container-based testing for data pipelines.

Defines K8s strategy for data platform: namespace isolation, resource quotas per team, scheduling policies for batch vs streaming. Implements GitOps for infrastructure.

Defines IaC standards for data platform: module library, naming conventions, change management process. Implements policy-as-code (Sentinel/OPA) for data infrastructure.

Defines CI/CD standards for data team: mandatory checks (dbt test, SQL lint, contract validation), deployment strategy, release management for data pipelines.

Defines Airflow standards: DAG structure, naming conventions, testing requirements, deployment workflow. Chooses between Airflow and alternatives (Dagster, Prefect) by scenario.

Defines Spark standards: coding guidelines, job submission patterns, resource allocation policies. Chooses between PySpark and Spark SQL by scenario. Implements unit testing for Spark jobs through chispa.

Defines orchestration standards: Dagster vs Prefect vs Airflow selection by project, migration strategy. Evaluates software-defined assets (Dagster) vs task-based (Airflow) approaches.

Defines data cataloging standards: mandatory metadata, ownership policy, data stewardship process. Implements data discovery workflow for self-service analytics.

Defines data contract standards: template, review process, enforcement policy. Coordinates between producer and consumer teams. Implements data-as-a-product culture.

Defines data lake standards: zone architecture (bronze/silver/gold), file formats, partition strategies. Implements access control and data classification. Coordinates between data producers and consumers.

Defines lineage standards: coverage, granularity (table vs column level), freshness. Uses lineage for impact analysis during schema changes. Implements lineage-based alerting.

Defines data quality standards: SLA per dataset, quality dimensions (accuracy, completeness, consistency, timeliness), ownership model. Implements data quality scorecard.

Defines DWH standards: modeling methodology (Kimball vs Inmon), naming conventions, testing requirements. Coordinates between domain teams for conformed dimensions. Conducts architectural reviews.

Defines dbt standards: project structure, naming conventions, documentation requirements, PR review checklist. Implements dbt metrics layer and exposure definitions for data contracts.

Defines Delta Lake/Iceberg standards: table format selection, partitioning strategy, compaction schedules. Implements time-travel for data debugging and schema enforcement.

Defines data processing standards: when pandas/Polars vs Spark, coding guidelines, testing patterns. Implements benchmarking for tool selection. Trains team on Polars adoption.

Defines SQL standards for data team: style guide, review checklist, performance budgets. Chooses between SQL-based ETL (dbt) and code-based (PySpark) by scenario.

Defines streaming standards: Kafka Streams vs Flink, windowing policies, state management. Implements consumer lag and processing latency monitoring. Chooses between exactly-once and at-least-once.

Implements Copilot in the team: configures policies, defines usage boundaries for data engineering. Evaluates ROI. Monitors security: credential leaks, SQL injection in generated code.

Defines logging standards for data platform: mandatory fields (pipeline_id, dataset, stage), log levels policy, retention. Implements log-based monitoring for pipeline SLA.

Builds code review culture: defines checklist (SQL performance, test coverage, documentation), establishes SLA. Automates checks (dbt test, SQL lint, contract validation).

Shapes Git workflow for data team: branching model, PR process with dbt CI, CODEOWNERS for critical models. Defines rules for mono/multi-repo data projects.