Split testing with Netlify with Phil Hawksworth

Creating and configuring split tests

Split testing, A/B testing, or multi-variant testing can be a valuable tool for testing different implementations of your site's features to determine which performs the best for your users.

Netlify's Split Testing allows you to silently serve different versions of your sites to your users without compromising performance which would prejudice your test results.

This video shows how to create and configure split tests, building upon branch deploys.

[00:00:00] Another powerful feature of Netlify is split testing. Now, you most might have encountered split testing under other names. You might know it as A/B testing or as multi-variant testing, but all of these things really are describing a way of serving different versions of your site to users at random so that you can then test different variants of your site. You might have, for instance, a different call to action that you might be wanting to test and see how well that converts when you get that out into the field. The way that many sites do that or many services do that is by using client-side JavaScript to manipulate the UI.

[00:00:40] They'll serve a version of the site and then a service will inject some JavaScript that perhaps manipulates either the styles of this or some aspect of the DOM so that you have the different versions that you can then test. Now, one drawback of that, doing a client-side, is that you're modifying the performance profile of the site. Really, we know that anything that might slow down performance, slow down rendering, is going to have an impact on conversion. What we'd really rather do is serve the exact same performance profile even though we've got different UI. Now, there's a way that we can do that with Netlify. We've covered it really, the fundamentals of it in the previous lesson.

[00:01:20] What we've done is we've looked at branch deploys in a previous lesson, and we're going to build on that now. This site, if we go and look at our deploys, you'll see that I've created a branch here called loud-cta. Let's have a look at that on this URL. Loud-cta is available on this URL. It's a little different to the production one. The main difference being that the "Buy Now" button is a lot louder. It's got a couple of exclamation marks. It's pink by default, whereas the production version, not so. It's a little bit quieter. It's black by default.

[00:02:00] Let's see if we can find ways to serve these two different versions of the site to the users without them realizing that something different is going on behind the scenes. We really want the same URL for both. The way that this happened on Netlify, it's thanks to Netlify Edge, which allows us to run a thin layer of logic right out on the edge nodes of the CDN that can shape the traffic from one branch to the other depending on rules that we create. Let's go and create some of those rules now.

[00:02:30] Back in our site administration, we go over to our split testing tab. I've created those branches already. We've just seen those on those URLs. Here, what we can do is we can choose which branches we will want to shape traffic between. I will have some traffic going to my loud call to action, some traffic going to the main branch, and we can choose how we like to shape that traffic. I'm going to go for a 50-50 split, and hope that I can demonstrate both of those to you as we get those running. Let's save that configuration, and then as soon as we're ready to start that test, hit Start Test.

[00:03:00] Now, that rule is running out on the CDN. If I go back to my production URL, I've got a 50-50% chance of getting one version or the other. Let's see what I get. Okay, I get the same production version. Let's see how the random gods favor us. We'll go to the same URL in an incognito window. Great. We're now in this version. In this session and this incognito window, we've come to the other branch. We've come to the loud call to action. You can see the different exclamation marks in there, the different copy and the slightly louder button. Let's look at that side by side. There we are, we see them both on the same URL but serving different branches behind the scenes. The way that we make sure that users will have a consistent experience and they always return to the same branch within their session is we create what's called branch affinity.

[00:03:50] Let's just make this a little larger so we can see. If we go and look at our dev tools and look at the cookies, we can see that there's been a cookie dropped here. The Netlify Edge network will drop a cookie whenever there's a split test running which purely allocates a random number between zero and one to make sure that this user always goes back to the same branch. The value here will correspond to the different branches that are available.

[00:04:20] We can do smart things with this. We'll come to that in a different lesson. For now, it's just key to know that behind the scenes, the CDN is dropping a cookie here so that users always get a consistent experience as they're navigating the site within their session.

Okay, let's turn this back off again. Let's just see how easy it is to turn the split test back off. It really is as simple as you'd expect. We click the Stop Test button and now that rule is gone from the CDN. When I make this request again, you might see, there we are. Our call to action is returned. It's not quite as shouty. We're still on the same URL, but now behind the scenes. We're serving things just from the main branch, just from that production branch.

[00:05:10] Split tests really are as simple as that to get up and running. We'll look in other videos about more complex things you can do with them and how we can do tracking and other important things to do with split testing. For now, there's the configuration. Have fun with it.


your progress