Skill Profile

Experiment Design

This skill defines expectations across roles and levels.

Machine Learning & AI Classical Machine Learning

Roles

where this skill appears

Levels

structured growth path

Mandatory requirements

the other 5 optional

Machine Learning & AI

Classical Machine Learning

2/22/2026

Choose your current level and compare expectations. The items below show what to cover to advance to the next level.

What is Expected at Each Level

The table shows how skill depth grows from Junior to Principal. Click a row to see details.

Role	Required	Description
Data Scientist		Understands the fundamentals of Experiment Design. Applies basic practices in daily work. Follows recommendations from the team and documentation.

Role	Required	Description
Data Scientist		Designs complex experiments: multi-variant tests, sequential testing, stratified randomization. Applies causal inference methods: difference-in-differences, instrumental variables. Calculates minimum detectable effect and plans experiment duration.

Role	Required	Description
Data Scientist		Designs experimentation platforms for systematic A/B testing of ML models. Applies advanced methods: switchback experiments, cluster randomization, synthetic control. Establishes guardrail metrics and early stopping rules for safe experiments.

Role	Required	Description
Data Scientist		Defines experimentation culture in the data science team. Establishes experiment design standards and review processes. Coordinates experimentation platform development and integration with ML pipelines.

Role	Required	Description
Data Scientist		Shapes experimentation strategy at organizational level. Defines investments in experimentation infrastructure. Publishes methodological research on experiment design in ML contexts.

Junior 1 requirements

Data Scientist

Understands the fundamentals of Experiment Design. Applies basic practices in daily work. Follows recommendations from the team and documentation.

Middle 1 requirements

Data Scientist

Designs complex experiments: multi-variant tests, sequential testing, stratified randomization. Applies causal inference methods: difference-in-differences, instrumental variables. Calculates minimum detectable effect and plans experiment duration.

Senior 1 requirements

Data Scientist

Designs experimentation platforms for systematic A/B testing of ML models. Applies advanced methods: switchback experiments, cluster randomization, synthetic control. Establishes guardrail metrics and early stopping rules for safe experiments.

Lead / Staff 1 requirements

Data Scientist

Defines experimentation culture in the data science team. Establishes experiment design standards and review processes. Coordinates experimentation platform development and integration with ML pipelines.

Principal 1 requirements

Data Scientist

Shapes experimentation strategy at organizational level. Defines investments in experimentation infrastructure. Publishes methodological research on experiment design in ML contexts.

Loading comments...