What is a multivariate test?
A multivariate test is a process where variations of different elements can be evaluated simultaneously. Multivariate testing allows you to determine which combination of variations performs best.
Example of a multivariate test
One common use of multivariate testing is evaluating which variations of a website perform best. For example, perhaps you want to increase sign ups on a specific webpage. You could test two different titles, two different images, and two different calls to action.
A total of eight different versions (the maximum combination for these three elements) would be tested simultaneously to determine which version of the webpage produces the most sign ups.
How to calculate the variations in a multivariate test
Calculating the total number of variations in a multivariate test is a straightforward equation.
[# variations for element A] X [# variations for element B]… = # total possible variations
Using the multivariate test example above, the calculation would be 2 X 2 X 2 = 8.
Multivariate testing vs A/B testing
Multivariate testing allows you to see which variation of different elements perform best together. This can also be called multi-variable testing.
A/B testing compares just two variations - whether overall performance or single elements. For example, you might test a green call to action on the page [test A] against a red call to action on the same page [test B] to see which color gets the most clicks.
Another option with A/B testing is to compare two drastically different pages against each other. Even though there may be many different elements on the two pages, an A/B test only shows the overall performance of each page - not the individual elements. It’s worth noting that additional variations can be tested (i.e. A/B/C testing), but they still only compare the overall performance of each page - unlike multivariate which shows the relationships between varied elements.
Pros and cons of multivariate testing
Multivariate testing is an efficient way of evaluating different elements and possible combinations. This process can save valuable time that otherwise would have been spent on many iterations of A/B testing.
The primary limitation of multivariate testing is the high traffic requirement. The traffic will be evenly divided between all possible variations. So if you have eight possible variations, your traffic will be divided into eighths. The danger is when a webpage doesn’t receive high enough traffic to produce reliable results (i.e. statistical significance).
Also, multivariate testing isn’t applicable for certain types of change. For example, testing all the elements of a rebranded homepage design with the existing design wouldn’t make sense because of the radical variations between them.
In contrast, A/B tests compare the overall performance of each (radical) variation. A/B tests also allocate 50% of traffic to each variation since there are usually only two variations tested at the same time (unless performing A/B/C testing as mentioned above).