What a Principal } should know

Defines data platform performance strategy. Designs algorithms for petabyte-scale dataset processing: partitioning, bucketing, approximate algorithms (HyperLogLog, Count-Min Sketch). Balances throughput and latency at the architecture level.

Designs data platform async architecture: event-driven ingestion, async CDC, streaming pipelines with exactly-once semantics. Defines strategy for scaling real-time processing through Kafka Streams and Flink.

Defines data platform technical quality strategy. Plans migrations between Spark/Airflow versions. Establishes pipeline reliability metrics: SLA, data freshness, completeness.

Designs platform data models: columnar formats (Parquet, ORC) for analytics, row-based for transactions, graph structures for lineage. Defines serialization/deserialization strategy for cross-system compatibility.

Shapes data platform architectural model: plugin system for connectors, abstractions for data sources/sinks. Defines where OOP is justified and where functional style (PySpark transformations) is preferable.

Designs organizational streaming platform: multi-cluster Kafka, MirrorMaker 2, tiered storage. Defines governance for event schemas and data contracts between producer/consumer teams.

Designs data mesh search infrastructure: federated search across domain catalogs, ML ranking for data discovery, cross-cluster search.

Designs data platform API layer: self-service data portal, metadata API, data marketplace. Defines API standards for data access through query engines (Trino, Presto).

Designs data platform caching infrastructure: shared Redis for cross-pipeline lookup data, warming strategies during deployments, partitioning for scaling.

Designs object storage architecture: multi-region replication, S3-compatible storage (MinIO) for on-prem, cost optimization through storage classes. Defines governance for data retention.

Designs distributed processing architecture: task queues for near-real-time vs batch schedulers, hybrid approaches. Defines SLA and capacity planning for different priority levels.

Designs NoSQL strategy for data platform: Cassandra for high-write ingestion, ScyllaDB for low-latency lookups. Defines data modeling best practices for wide-column stores and Spark integration for batch analytics.

Designs data resilience strategy: multi-region replication, cross-cloud DR, cold/warm/hot backup tiers. Defines governance for data protection and compliance (GDPR right to delete).

Designs platform OLAP strategy: ClickHouse cluster topology, cross-DC replication, capacity planning. Defines when ClickHouse vs Snowflake/BigQuery. Plans data serving layer architecture.

Designs organizational data modeling strategy: domain-driven data models, canonical data model, data mesh domain boundaries. Defines governance for entity resolution and master data management.

Designs data platform indexing strategy: columnar indexes (Parquet row groups), zone maps, min/max statistics. Defines indexing approaches for data lake (Iceberg metadata) and data warehouse.

Designs schema management platform: schema registry, versioned data contracts, automated compatibility checking. Defines governance for schema evolution in data mesh.

Designs PostgreSQL strategy: logical replication for real-time CDC, Citus for distributed analytics, TimescaleDB for time-series pipelines. Defines PostgreSQL boundaries in data architecture.

Designs query optimization strategy: federated queries through Trino/Presto, query caching, materialized views. Defines SLA for query latency by type: ad-hoc, scheduled, interactive.

Designs platform data replication strategy: CDC through Debezium/Kafka Connect, cross-region replication for geo-distributed analytics, multi-master scenarios. Defines consistency guarantees for different data products.

Designs data platform serialization strategy: unified schema registry, cross-format compatibility, schema evolution governance. Defines when Protobuf vs Avro vs Parquet schemas.

Designs data mesh API strategy: domain data product API, federated data access layer, API gateway for data services. Defines standards for data-as-a-product interfaces.

Designs cloud data strategy: multi-cloud (AWS + GCP/Azure), data residency compliance, vendor lock-in mitigation. Defines reference architecture for cloud-native data platform.

Designs container strategy for data platform: runtime standardization for Spark/Airflow/dbt, image governance, registry architecture. Defines container vs serverless for different workloads.

Designs Kubernetes platform for data: multi-cluster for staging/production, spot instances for batch, dedicated nodes for streaming. Defines compute strategy and cost optimization.

Designs organizational IaC strategy: self-service infrastructure provisioning for data teams, cost controls, compliance automation. Defines multi-cloud infrastructure abstraction.

Designs CI/CD strategy for data platform: unified pipeline for different data tools (dbt, Airflow, Spark), environment management, data infrastructure provisioning. Defines release governance.

Designs orchestration strategy: Airflow for batch, event-driven for real-time, hybrid patterns. Defines multi-team governance, shared infrastructure, cost allocation.

Designs platform Spark strategy: EMR vs Databricks vs self-hosted, cluster sizing, dynamic allocation. Defines when Spark vs DuckDB vs Polars. Plans migration to Spark 4.0.

Designs next-gen orchestration: Dagster for software-defined data assets, Prefect for event-driven, hybrid with Airflow legacy. Defines migration path and coexistence strategy.

Designs organizational metadata platform: federated catalog for data mesh, automated metadata extraction, ML-powered data discovery. Defines metadata governance and interoperability standards.

Designs data contract platform: centralized registry, automated enforcement, compatibility checking. Defines governance model for inter-team data sharing in data mesh.

Designs data lakehouse architecture: unified storage layer, query engine federation (Trino/Spark), governance framework. Defines when lakehouse vs traditional DWH vs data mesh.

Designs organizational lineage platform: end-to-end lineage from operational systems to BI dashboards, automated compliance reporting, lineage-driven data governance.

Designs data quality platform: centralized quality monitoring, ML-based anomaly detection, quality-driven data trust scoring. Defines organization-wide data quality framework.

Designs organizational DWH strategy: centralized vs decentralized, semantic layer, cost management. Defines evolution path: traditional DWH → lakehouse → data mesh. Plans cross-platform migration.

Designs transformation strategy: dbt for SQL transformations, Spark for complex processing, dbt mesh for multi-team. Defines governance for shared models and cross-project dependencies.

Designs open table format strategy: Delta Lake vs Iceberg vs Hudi, catalog integration (Unity/Nessie), cross-engine interop. Defines migration path from Hive tables.

Defines local data processing strategy: DuckDB for ad-hoc analytics, Polars for batch ETL, Arrow for zero-copy data exchange. Designs unified API for different backends.

Designs transformation strategy: SQL for declarative ETL, Python for complex logic, hybrid approaches. Defines query engine selection (Trino, BigQuery, Redshift) by workload pattern.

Designs data platform streaming architecture: Kafka Streams for lightweight processing, Flink for complex CEP, hybrid batch+streaming. Defines lambda vs kappa architecture by scenario.

Defines AI-assisted data engineering strategy: tool selection, governance policies, quality gates for AI-generated code and SQL. Evaluates LLM-based data tools (text-to-SQL).

Shapes data platform observability strategy: unified logging for Airflow/Spark/dbt, data-specific metrics (freshness, completeness, volume), automated anomaly detection.

Defines organizational review standards: cross-team review for shared data models, architectural review board for platform changes. Builds engineering excellence culture.

Defines data platform versioning strategy: mono vs multi-repo for dbt/Airflow/infra, release management for data pipeline changes. Coordinates cross-team collaboration.