What a Senior } should know

Applies algorithmic expertise to data platform: join algorithm selection for distributed data processing (broadcast vs sort-merge vs hash), shuffle optimization algorithms for Spark, incremental computation algorithms for streaming ETL. Designs efficient deduplication and data quality algorithms at scale.

Designs async architectures for data pipelines: concurrent data source ingestion at scale, async ETL orchestration with backpressure, non-blocking data quality validation. Mentors team on async patterns for data pipeline throughput.

Designs code quality standards for data pipelines: Airflow DAG structure, Spark job organization, ETL testing patterns. Refactors monolithic data transformations into modular, idempotent pipeline stages. Implements quality gates for data contract enforcement and pipeline reliability.

Selects optimal data structures for data platform workloads: partitioned Parquet/ORC columnar formats, bucketed hash distributions for join optimization, Delta Lake merge structures for upsert patterns. Optimizes Spark shuffle data structures for minimal memory overhead. Designs schema evolution strategies with backward/forward-compatible structures.

Applies OOP/SOLID in data platform architecture: abstract interfaces for data source connectors, strategy pattern for transformation engine selection, template method for standardized pipeline stages. Designs extensible data frameworks that support adding new data sources and transformations without modifying core pipeline logic.

Designs event-driven ingestion: Kafka producers for CDC, consumer groups for parallel processing, Schema Registry with Avro/Protobuf. Optimizes throughput: batching, compression, partition assignment.

Designs search index for data catalog: mapping, custom analyzers for metadata, nested documents for lineage. Optimizes bulk indexing pipeline for millions of catalog records.

Develops data API on FastAPI: endpoints for metadata access, catalog, lineage. Implements async request processing for data warehouse. Integrates with Airflow REST API for DAG management.

Uses Redis for data pipeline optimization: lookup table caching, deduplication through Redis Sets, rate limiting for API sources. Implements Redis Streams for lightweight event processing.

Designs data lake on S3: partitioning strategy (year/month/day), Parquet/ORC for analytics, lifecycle policies for archival. Optimizes costs through Intelligent-Tiering and Glacier.

Designs asynchronous processing through Celery/RQ: background tasks for data processing, priority queues for different SLAs, dead letter queues for failed tasks. Configures retry policies and monitoring.

Architects multi-datacenter Cassandra deployments for real-time data pipelines. Optimizes cluster topology, compaction strategies, and partition distribution for petabyte-scale workloads. Designs migration strategies between schema versions.

Designs disaster recovery for data platform: backup strategies for different storage (HDFS snapshots, S3 versioning, DB dumps). Implements time-travel in Delta Lake/Iceberg for data recovery.

Designs ClickHouse architecture for data warehouse: distributed tables, sharding strategy, dictionary tables for joins. Optimizes queries: projections, skip indexes, query pipeline tuning. Configures Kafka engine for streaming ingestion.

Designs enterprise data models: Data Vault 2.0 for flexibility, Anchor Modeling for high-change environments, wide tables for ClickHouse. Defines layered architecture: raw → staging → curated → mart.

Optimizes indexing for data pipelines: GIN for JSONB, BRIN for time-series, bloom filters in ClickHouse. Designs indexing strategy considering batch vs streaming access patterns.

Designs migration strategy: zero-downtime migrations for data warehouse, schema evolution in Parquet/Avro, backward compatibility in data contracts. Automates through CI/CD.

Designs PostgreSQL as data pipeline source/sink: CDC through Debezium, materialized views for aggregates, FDW for federation. Optimizes VACUUM for high-write staging tables.

Designs optimal data access patterns: incremental extraction through watermarks, micro-batch vs full-load trade-offs. Optimizes Spark query plans: broadcast joins, AQE, partition coalescing.

Designs high-availability data architectures with multi-region replication, cross-database CDC pipelines, and event-driven synchronization. Optimizes replication for large-scale analytical workloads with minimal production impact. Architects disaster recovery strategies with defined RPO/RTO targets.

Designs gRPC interfaces for data services: bidirectional streaming for CDC, server streaming for bulk data delivery. Integrates Protobuf with Schema Registry for centralized schema management.

Designs data platform API layer: REST for metadata catalog, webhook endpoints for event-driven ingestion. Implements data API with query parameters for self-service data access.

Designs data platform on AWS: Lake Formation for governance, EMR for Spark, Kinesis for streaming, Glue Catalog for metadata. Optimizes costs: reserved instances, spot for batch, S3 lifecycle.

Designs containerization for data platform: images for Airflow workers, Spark executors, dbt. Optimizes build pipeline: pip dependency caching, slim images. Configures local development environment.

Designs Kubernetes infrastructure for data: Spark Operator, Airflow KubernetesExecutor with dynamic resource allocation, Argo Workflows for ML pipelines. Optimizes node pools for different workloads.

Designs data platform network architecture: private connectivity to data sources, Transit Gateway for multi-VPC, Direct Connect for on-prem data centers. Optimizes network performance for bulk data transfer.

Designs IaC for data platform: reusable modules for data services (EMR, Glue, Kinesis), environment promotion pipeline. Automates data infrastructure provisioning.

Designs CI/CD for data platform: pipeline testing (unit + integration), automated schema validation, blue/green deployment for Airflow. Implements data quality gates in deployment pipeline.

Designs Airflow architecture: KubernetesExecutor for dynamic scaling, custom operators/hooks, DAG factory pattern for generation. Optimizes performance: pool management, priority weight, concurrency.

Designs Spark-based data platform architecture: multi-tenant cluster management, cost-optimized workload scheduling with YARN/Kubernetes, and lakehouse architecture with Delta Lake/Iceberg. Implements data quality frameworks, CDC pipelines, and Spark application performance monitoring.

Designs data architecture with Dagster/Prefect. Optimizes for big data. Implements data governance and quality frameworks.

Designs metadata management: DataHub/OpenMetadata/Amundsen setup, custom connectors for internal systems, automated lineage extraction from Spark/Airflow. Integrates catalog with data quality.

Designs data contract framework: schema registry integration, automated validation pipeline, versioning strategy. Implements contract testing between producers and consumers.

Designs data architecture with Data Lake Architecture. Optimizes for big data. Implements data governance and quality frameworks.

Designs lineage infrastructure: column-level lineage, cross-system tracking (operational → analytical), custom extractors for internal tools. Integrates lineage with data catalog and quality monitoring.

Designs data quality system: multi-layer validation (source → staging → mart), anomaly detection (statistical), automated remediation. Integrates quality metrics into data catalog.

Designs data warehouse architecture: multi-layer (staging → ODS → DWH → marts), Slowly Changing Dimensions, bridge tables for many-to-many. Selects cloud DWH (Redshift/BigQuery/Snowflake) by requirements.

Designs dbt architecture: multi-project setup, package management, custom generic tests. Implements unit tests for complex transformations. Optimizes performance: incremental + merge, partition-based.

Designs data architecture with Delta Lake/Apache Iceberg. Optimizes for big data. Implements data governance and quality frameworks.

Designs data processing: Polars for single-node high-performance, pandas for quick prototyping, PySpark for distributed. Selects tool by volume and processing pattern. Optimizes memory management.

Designs SQL-based ETL architecture: ELT pattern (load-then-transform), incremental processing through merge/upsert, materialized views for performance. Integrates with dbt for version-controlled SQL.

Designs end-to-end streaming data architectures with Kafka Streams for high-throughput pipelines. Orchestrates complex event processing, implements schema evolution strategies, and optimizes for backpressure handling.

Maximizes productivity: generating complex SQL/dbt macros, Terraform modules, data quality rules. Builds workflow where AI accelerates routine (boilerplate), engineer focuses on architecture.

Designs observability stack for services. Creates SLI/SLO dashboards. Writes complex PromQL (rate, histogram_quantile, recording rules). Configures Alertmanager with routing and silencing. Optimizes metric cardinality. Integrates with PagerDuty/OpsGenie.

Designs observability for data pipelines: structured logging with data lineage context, metrics emission (records processed, data quality scores), anomaly alerting. Integrates with OpenTelemetry.

Conducts architectural reviews: evaluates pipeline design, data model decisions, schema evolution impact. Reviews Airflow DAG structure, Terraform changes for data infra.

Defines branching strategy for data projects: trunk-based with feature flags for DAGs, protected branches for production models. Configures pre-commit hooks for SQL linting and dbt compile.