Looking for an answer? Just run a test!
A/B testing is one of the most effective ways to discover whether an idea will move the needle. But is it hard to get started? Absolutely not! In this article, we’re going to show you how to get started and how to take your testing methodology to the next level.
Why A/B Testing is important for your Shopify store
In short, A/B testing allows you to validate hypotheses. If you have an idea, an A/B test will help you discover whether that idea will have an impact on your business. That impact could be positive or negative, but regardless of the outcome, you will gain valuable insights about your customers, your experience, and your business.
Here are a few example tests you can run:
✅ Pricing Optimizations: Test different price points to maximize revenue.
✅ Product Page Layouts: Experiment with different content placements.
✅ Shipping & Discount Offers: See how various incentives impact conversions.
✅ Checkout Optimizations: Improve the final step of the buying process.
The good news is that you don’t need to be a developer or a statistician to reap the benefits of A/B testing. Let’s show you how you can get started and take it to the next level.
Some real examples from our portfolio
Before we dive into the how-to, let’s show you the results that have been achieved by some of our portfolio brands. Here are two examples from Smidge and Jonas Paul Eyewear.
Smidge
Smidge is a small-batch supplements company focused on clean ingredients, sustainable product development, and operational practices. While they've conducted dozens of A/B tests, this specific change yielded some instructive results.
Smidge wanted to test what would happen if they combined the quantity selector with the add-to-cart button on the same line. You might hypothesize that this change would lead to an increase in AOV by making it easier and more obvious to add additional units.
While the test was successful, the results didn’t show an increase in AOV. Instead, the change led to an increase in CvR, which ultimately resulted in higher revenue per session compared to the control, where the two options were on separate lines.
This is a great illustration of why it’s important to test your hypothesis with an A/B test. Without testing, you might have assumed the increase was due to more items being purchased per order, when in fact, it was due to more people converting.
Jonas Paul Eyewear
Jonas Paul creates prescription eyewear specifically designed for kids, offering stylish frames that can withstand anything your child throws at them. While the example above focused on adjusting the page’s content, Jonas Paul’s test went beyond the page—they ran a shipping experiment on their Home Try-On Kit.
Jonas Paul wanted to assess the total revenue impact of charging customers for shipping, rather than covering the cost. While it may seem obvious that conversion rates would decrease when charging for shipping, it’s not as clear which option will generate more revenue—and more importantly, more profit. Here’s what Jonas Paul discovered from the test:
Unsurprisingly, the test revealed that conversions decreased when customers were responsible for paying shipping fees. In a test like this, the goal is to determine whether the drop in sales volume is offset by the savings in shipping costs. Although this experiment didn’t result in a home run, it paved the way for continued shipping tests, which Jonas Paul continues to explore.
Future shipping experiments may include testing different shipping rates, varying thresholds for free shipping, and exploring how free shipping offers are displayed on the site.
In both examples above, the initial test inspired a new wave of experimentation, allowing both brands to continually improve and optimize their strategies to maximize profits.
How to set up an A/B test
Now that you’ve seen what’s possible, we will guide you through how easy setting up an A/B test can be. We are going to walk through a content test where we change the customer experience. While this is a relatively simple test, later we’ll explore how you can experiment with much more than just on-page content.
In this example, we will change the hero visual a customer sees when they land on the site. We’ll do this using the native visual editor in our preferred Shopify A/B testing app, Intelligems.
1. Create and name the test
To create a test, we’re going to give it a name {Different Hero Images} and a description that explains the specifics of what is being tested. In this case, we will make it clear that we want to test a product shot vs. a lifestyle shot.
2. Select how you want to build the different experiences
There are many ways to run a test like this on your on-site content. If you already have experiences created because they exist on different pages, you can quickly conduct a “Split URL Test” by simply directing a portion of traffic to each URL. Or, you might also have templates or full themes that you want to compare against each other.
In this example, we are going to assume you want to define the specific experiences via an editor.
3. Allocate the traffic split
Once you choose the type of test you want to create, simply move the slider to select how much traffic you want to send to each group.
You can also add more groups to the test if you want to do—you guessed it—an A/B/C test.
4. Decide who sees it
Next, we’ll edit the targeting (segmentation) options. These targeting options allow you to get specific about who should see the test.
In this example, the images are much more prominent on desktop than on mobile, so we’re going to restrict this test to desktop traffic only. If you want to have different requirements to see each group, you can also go deeper with “custom” targeting.
5. Build the experiences by adjusting your current site
Now for the fun part. Use the on-site editor to make the changes to your site without needing to write a line of code. Simply pick the text, image, or content block you want to edit or remove.
In this example, we are changing the hero for the product shot group and leaving the visual of all the prepared food in the lifestyle shot group.
You’re free to make as many changes to the experiences as you desire, but remember, the more you change, the harder it is to isolate the results of any one change.
6. Hit that start button
Be sure to press start in order for the changes to take effect on your live site.
7. Analyze the results
Depending on your store’s traffic and sales volume, the time it takes for a test to reach statistical significance will vary. But you can come check on the in-progress results anytime.
Use apps to build your testing capabilities
Using apps provides room to grow and evolve your testing capabilities. You can start with your classic content-based A/B tests and adapt over time to include pricing, promotions, offer tests, and more.
Intelligems is our preferred app when running A/B tests for the Shopify brands we service. This app gives us full control over:
-
Segmentation (who sees the test)
-
Personalization (where they see it)
-
Experimentation (how they experience it)
-
Evaluation (how you read the results)
When you have control over the entirety of the test, you can build and analyze with confidence.
This also allows us to test different growth levers beyond your classic content test, including:
-
Content experiences
-
Price
-
Promotions
-
AOV boosters like gifts with purchases and shipping thresholds
Other A/B testing apps we’ve used with our clients include Convert.com and Shoplift. Convert.com is great for advanced experimentation, but it is not Shopify-specific. Shoplift offers easy-to-use visual editing and is quickly expanding its testing capabilities.
A/B testing is more than conversion rate optimization—It’s profit growth
As many brands start their A/B testing journey, they begin by searching for conversion rate optimizations (CRO) across the site. This is a great way to stack up wins, but it’s far from the only way to A/B test.
It’s now easier than ever to see how your content tests are impacting more than just conversion. You can also see how the changes you make impact metrics like revenue per visitor. Visibility into these metrics can help you run a more profitable store.
Need help getting started with A/B testing or taking your tests to the next level? Chat with our experts!
