A/B testing
lastmod 2026-04-08created 2026-04-081 edits
In marketing and web analytics, A/B testing (bucket testing or split-run testing) is a comprehensive controlled experiment using two variants, A and B. It is a form of statistical hypothesis testing, or "two-sample hypothesis testing", as used in the field of statistics. In online fields such as web design (especially user experience design), the goal of A/B testing is to identify which changes to a web page increase or maximize an outcome of interest (for example, the click-through rate of a banner ad). Formally, the current web page is associated with the null hypothesis. A/B testing is a way to compare two versions of a single variable by testing a subject's response to variable B against variable A and determining which of the two variables is more effective.
As the name implies, two versions (A and B) are compared, which are identical except for one variation that might affect a user's behavior. Version A is called the currently used version (control), while some aspect of version B is modified (treatment). For example, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, since improvements to the profit margin in drop-off rates can represent a significant gain in sales. Not always, but sometimes significant improvements can be seen by testing elements such as text, layout, images, and colors.
Multivariate testing or multinomial testing is similar to A/B testing, but can test more than two versions at the same time or test more controls. It controls the use of more than two versions, or more at the same time. Simple A/B tests are not valid in observational, quasi-experimental, or other non-experimental situations, such as survey data, offline data, and other more complex phenomena.
Although A/B testing is similar in approach to the between-subjects design commonly used in a variety of research traditions, it has been marketed in certain niche areas as a change in philosophy and business strategy. A/B testing as a web development philosophy leads the field toward a broader movement of evidence-based practice. Because most marketing automation tools now commonly come with the ability to run A/B tests continuously, an advantage of A/B testing is that it is considered something that can be performed continuously in almost every area. This makes it possible to keep up with changing trends by updating websites and other tools using current resources.