Experiment function

The experiment function enables you to split recipients into randomized cohorts within your workflows, routing each recipient down a specific branch based on a percentage-based distribution. This is useful for A/B testing notification content, gradually rolling out new notification strategies, or running experiments across your recipient base.

When a workflow run reaches an experiment step, Knock assigns the recipient to a cohort using the following process:

1. Cohort key. Knock hashes the cohort key (which defaults to the recipient ID) to produce a deterministic assignment. This means the same recipient will always be routed to the same cohort for a given step, ensuring consistency across workflow runs.
2. Percentage distribution. Each cohort is assigned a percentage weight. Knock uses these weights to determine which cohort the hashed key maps to. The percentages across all cohorts must sum to 100%.
3. Branch execution. Once assigned, the recipient proceeds through the steps defined in their cohort's branch.

Each experiment step starts with two cohorts by default. You can add additional cohorts or adjust the percentage distribution between them.

• Percentage weights. Set the percentage for each cohort to control the distribution of recipients. The total across all cohorts must equal 100%.
• Cohort key. By default, the cohort key is the recipient ID. You can customize this to any value from the workflow run scope, which is useful when you want the same user to be assigned to different cohorts in different contexts.

The experiment function is well-suited for:

Test different notification templates, copy, or channels to see which performs better. For example, split recipients 50/50 between two email templates to measure engagement.

Roll out a new notification strategy to a small percentage of recipients before expanding to all users.

Run multi-variant experiments by splitting recipients across three or more cohorts with different notification flows.

Test whether or not a particular message adds lift to your conversion goals. This type of test is a variation of A/B testing where one cohort receives a message and the other cohort does not. It's helpful in determining if a message is useful.

Experiment steps can be nested inside other experiment steps, regular branches, or any other workflow function. This enables you to create more complex experimental setups, such as splitting recipients into cohorts and then further splitting within each cohort.

You can debug experiment step execution in the workflow debugger. During a workflow run, the debugger shows which cohort was chosen for each recipient, including the cohort key value used for assignment and the percentage distribution across cohorts.