What a Senior } should know

Applies algorithmic thinking to SRE: anomaly detection algorithms for proactive incident prevention, load shedding algorithms for graceful degradation, capacity prediction algorithms based on historical trends. Designs efficient alerting algorithms minimizing noise while maintaining detection sensitivity.

Designs async architectures for SRE systems: concurrent monitoring at scale, async incident response orchestration, non-blocking alert correlation and routing. Mentors team on async patterns for operational system reliability.

Designs code quality standards for SRE tooling: runbook automation structure, observability pipeline code, chaos engineering scripts. Refactors incident management automation for reliability and auditability. Establishes review practices for operational code covering error budgets, SLO monitoring, and on-call tooling.

Selects optimal data structures for SRE tooling: time-series databases for metric storage, cardinality-efficient data structures for high-dimensional monitoring, ring buffers for recent event windows. Optimizes alert evaluation data structures for minimal latency on rule processing. Designs efficient data models for incident timeline reconstruction and post-mortem analysis.

Has deep expertise in diagnosing multithreading issues: analyzes thread contention and deadlock in production systems at scale, implements monitoring for concurrent system health indicators, optimizes thread configurations across service fleets. Mentors team on production concurrent system debugging and performance tuning.

Applies OOP/SOLID in SRE tooling architecture: abstract interfaces for monitoring backends, strategy pattern for alert routing, template method for standardized runbook automation. Designs extensible observability frameworks with clean separation between data collection, processing, and alerting layers.

Uses Kafka for SRE tasks: streaming metrics processing, alert aggregation, audit log collection. Configures consumer groups for distributed alert processing. Monitors Kafka cluster health.

Develops SRE services on Flask/FastAPI: status pages, runbook automation API, custom webhook receivers for alerting integration. Implements health-check endpoints for internal tooling.

Uses Redis for SRE tools: rate limiting for alert deduplication, distributed locks for deployments, caching for status page. Monitors Redis performance and memory usage.

Designs PostgreSQL reliability: streaming replication setup, automated failover (Patroni), point-in-time recovery. Optimizes for operational workloads: vacuum tuning, index maintenance, connection pooling.

Designs API for SRE platform: self-service infrastructure provisioning, SLO management API, incident orchestration. Implements idempotency for critical operations (rollback, scaling).

Designs AWS platform for reliability: multi-AZ architecture, cross-region DR, AWS Organizations for multi-account. Configures GuardDuty, Config Rules for security. Automates through AWS CDK/Lambda.

Designs container runtime standards: image security pipeline, registry management, runtime security (seccomp, AppArmor). Optimizes container performance: cgroup limits, OOM handling, filesystem layers.

Designs Helm strategy: library charts for shared components, umbrella charts for full stack deployment. Automates chart testing (helm unittest, ct lint). Implements GitOps workflow for Helm releases.

Designs Kubernetes reliability: pod disruption budgets, topology spread constraints, custom operators for automation. Configures multi-zone deployment, graceful shutdown. Optimizes cluster performance.

Designs K8s infrastructure for reliability: StatefulSets for stateful workloads, DaemonSets for node-level monitoring, Jobs/CronJobs for maintenance. Configures service mesh (Istio/Linkerd).

Designs load balancing architecture: global load balancing (Route53, Cloudflare), internal LB for microservices, gRPC balancing. Optimizes connection draining, session affinity, retry policies.

Designs network reliability: multi-AZ networking, failover strategies, BGP for multi-homing. Optimizes network performance: connection pooling, keep-alive tuning, TCP optimization.

Designs IaC for platform: reusable modules, composition patterns, Terragrunt for DRY configuration. Automates infrastructure drift detection. Implements blast radius reduction through workspace isolation.

Designs VPN architecture: hub-and-spoke vs mesh, Transit Gateway VPN attachments, automated tunnel management. Optimizes throughput. Plans migration to zero-trust (BeyondCorp).

Designs ArgoCD architecture for highly available service delivery with multi-cluster failover and disaster recovery patterns. Implements GitOps-driven reliability practices including automated canary analysis, SLO-based deployment gates, and chaos engineering integration for deployment validation. Optimizes ArgoCD for production reliability with controller HA, webhook-driven reconciliation, and drift alerting pipelines.

Designs CI/CD for platform: multi-stage deployments, canary/blue-green, rollback automation. Implements progressive delivery (Argo Rollouts, Flagger). Optimizes pipeline performance.

Designs GitOps for reliability: configures progressive delivery with SLO-based analysis gates (canary + SLI check → promote/rollback), implements automated rollback on SLO violation. Designs GitOps disaster recovery: rapid cluster state restoration from Git.

Designs chaos program linked with SRE practices: integrates chaos experiments into post-mortem follow-ups, creates continuous verification for critical paths. Implements sophisticated experiments: clock skew, DNS failures, TLS certificate expiry, cascading failure scenarios.

Designs incident response process: automated severity detection, runbook automation, war room orchestration. Implements SLO-based alerting for proactive incident detection.

Designs infrastructure security: network segmentation, runtime protection (Falco), vulnerability management pipeline. Implements security-as-code: policy enforcement, compliance scanning.

Designs secrets management: Vault HA cluster (Raft), dynamic secrets for databases, PKI infrastructure. Automates certificate rotation. Implements audit logging and compliance monitoring.

Maximizes productivity: generating complex Terraform modules, K8s operators, monitoring dashboards-as-code. Workflow: AI for boilerplate, engineer for architecture and security review.

Designs capacity management: predictive scaling, load testing for capacity validation, cost-aware autoscaling. Implements automated right-sizing recommendations. Models growth scenarios.

Designs DR architecture: active-passive vs active-active, pilot light, warm standby. Implements automated failover. Conducts chaos engineering for DR validation. Defines RTO/RPO by tier.

Designs high-load architecture: sharding strategies, CQRS for read-heavy workloads, queue-based decoupling. Optimizes: connection pooling, batch processing, async communication.

Designs distributed systems: CAP theorem trade-offs, consensus protocols, eventual consistency patterns. Defines data replication strategy. Designs cross-service communication (mesh, events).

Architects the observability platform integrating APM, logging, and tracing. Defines SLI/SLO for critical services and automates error-budget tracking. Leads post-mortem processes and drives reliability improvements based on APM insights.

Designs observability strategy with Continuous Profiling. Implements distributed tracing. Defines SLI/SLO. Conducts post-mortems.

Defines metrics framework: standard instrumentation library, metric naming conventions, cardinality management. Implements derived metrics through recording rules. Integrates with SLO tooling.

Designs ELK architecture: cluster sizing, ILM for retention management, cross-cluster search. Optimizes: index templates, mapping, ingest pipelines. Configures alerting through ElastAlert.

Designs the organization-wide logging strategy with Loki as the centralized log platform. Defines SLI/SLO based on log-derived metrics and automates error-budget alerting. Leads post-mortems leveraging Loki correlation with distributed traces and APM data.

Designs observability strategy with Jaeger / Grafana Tempo. Implements distributed tracing. Defines SLI/SLO. Conducts post-mortems.

Optimizes on-call: alert tuning for noise reduction, automated remediation for common issues. Designs runbook automation. Analyzes on-call metrics and creates improvement plan.

Designs OTel architecture: collector fleet management, sampling strategies, tail-based sampling. Optimizes overhead and storage costs. Implements custom processors for enrichment.

Designs Prometheus architecture: federation, Thanos/Mimir for long-term storage and global view. Optimizes cardinality, retention. Implements multi-cluster monitoring.

Designs SLO framework: multi-window burn rate alerting, SLO-based pages, error budget policies. Implements automated SLO reporting. Integrates SLO with deployment decisions.

Designs logging architecture: unified log format for all services, sampling strategy for high-volume, log-based metrics extraction. Integrates with OpenTelemetry logs API.

Conducts architectural review: evaluates infrastructure design, blast radius, failure modes. Reviews capacity planning, security implications, cost impact of changes.

Defines Git strategy for infrastructure: trunk-based for configs, release branches for infrastructure versions. Configures branch protection, CODEOWNERS for critical infrastructure.

Designs low-latency architecture: CDN placement, edge caching, connection pooling optimization, tail-latency management. Implements per-service latency budgets. Implements automated regression detection.