In the fast-changing digital world, it is difficult to assess the impact of new ideas. For more than 20 years, leading digital companies have been adopting a “test everything” approach based on A/B testing to make decisions that can generate significant value for the company. That is, before creating new content or developing any functionality, product, or service, A/B tests are conducted to evaluate its acceptance and impact.
One of the first A/B test cases most people know about was was Google’s experiment of testing 41 Shades of Blue , later known as “Fifty Shades”. Over the years, the benefits of this test have continued to be discussed.
In 2019, representatives from 13 large organizations (12 companies and Stanford University), which each currently run more than 100,000 tests per year, held the first summit to discuss the key challenges of A/B testing.
Taking advantage of this list of organizations, below, sorted alphabetically by company, you will find 5 examples of A/B tests performed by each of them. These examples show how A/B tests are the starting point for assessing the vast majority of changes that are implemented in the products and services they offer. At the end of the post you will find the information sources for the examples shown.
A/B TESTS EXAMPLES
Airbnb
Airbnb, the online portal for vacation rentals and travel experiences, uses A/B testing extensively to learn and make decisions at every step of its product development, from design to algorithms. They are equally important in shaping the user experience.
Search Ranking
Airbnb uses A/B testing to optimize the search ranking algorithm for its vacation rental listings. For example, in 2017, Airbnb tested a new search ranking algorithm that took into account factors such as the quality and completeness of a listing, host response rate, and rental location.
Pricing
Airbnb uses A/B testing to optimize the pricing of its vacation rentals. For instance, in 2018, the company tested a new pricing feature that offered hosts personalized price suggestions based on factors like demand and competition.
Guest Messaging
Airbnb uses A/B testing to optimize its guest messaging feature, which allows hosts to communicate with guests before and during their stay. For example, in 2020, the company tested a new messaging feature that allowed hosts to send automated messages to guests.
Nelio A/B Testing
Native Tests for WordPress
Use your WordPress page editor to create variants and run powerful tests with just a few clicks. No coding skills required.
User Interface
Airbnb uses A/B testing to optimize the user interface of its mobile app and website. For example, in 2019, the company tested a new interface design that made it easier for users to navigate and book vacation rentals.
Reviews
Airbnb uses A/B testing to optimize its review system, allowing guests to leave feedback on their stays. For example, in 2018, the company tested a new review system that allowed guests to rate their stays on a scale of 1-10, instead of the previous 5-star rating system.
Amazon
Amazon is known for its extensive use of A/B testing to optimize its products and services. Let’s look at some examples of how Amazon uses A/B testing.
Amazon.com
Amazon continually uses A/B testing to optimize the user experience on its online marketplace. For instance, in 2020, Amazon tested a new design for its product detail pages that was aimed to improve the visibility of customer reviews.
Amazon Prime Video
Amazon uses A/B testing to improve the user experience of its streaming video service. For example, in 2020, Amazon tested a new feature that allowed users to customize the order of their watchlist.
Amazon Web Services
Amazon uses A/B testing to optimize the performance and reliability of its cloud computing platform. For instance, in 2020, Amazon launched Amazon Personalize, which makes it easy to integrate personalized recommendations into existing websites, apps, email marketing systems, and more, including the option to create A/B tests.
Amazon Alexa
Amazon uses A/B testing to improve the user experience of its voice assistant. For example, they recently created an A/B test for Vocala (a company specializing in voice ads) to determine the ideal length of invitation for users to subscribe to the paid version of their clients.
Amazon Advertising
Amazon uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bid strategies to see what works best for their business.
Example of the same product page viewed from two different computers:
Booking
Booking.com, the online travel agency that allows users to book accommodation, flights, and rental cars, extensively uses A/B testing to optimize its platform and improve user experience. Everything goes through a previous A/B test before being launched on the market for the public.
Search Results
Booking.com uses A/B testing to optimize the search results page for its accommodation listings. For example, in 2020 the company tested a new search results page that featured larger images and more detailed information about listings. In addition, accommodation owners have a wide variety of products to experiment with.
User Interface
Booking.com uses A/B testing to optimize the user interface of its mobile app and website. For example, in 2019, the company tested a new design for its mobile app that made it easier for users to find and book accommodation.
Pricing
Booking.com uses A/B testing to optimize the pricing of its accommodation listings. For example, in 2018, the company tested a new pricing feature that offered personalized discounts to customers who had previously abandoned their bookings.
Reviews
Booking.com uses A/B testing to optimize its review system, allowing customers to leave feedback on their stays. For example, in 2018 the company tested a new review system that included more detailed information about properties, such as room sizes and amenities.
Website Layout
Booking.com uses A/B testing to optimize its website layout. For example, in 2017 the company tested a new layout for its home page that featured larger images and more prominent search bars.
Meta
Meta is known for using A/B testing to optimize its products and services.
News Feed
Facebook uses A/B testing to test and refine its News Feed algorithm, which determines what content appears in users’ feeds. For example, in 2018 Facebook tested a new algorithm that prioritized posts from family and friends over content from publishers.
Ad Targeting
Facebook uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bidding strategies to see what works best for their business.
Instagram Messaging
Meta conducted A/B tests to optimize its messaging app across its various platforms. For example, in 2020, before launching Instagram Messaging, it ran an A/B test to compare its usage on desktop vs. native apps, monitoring the number of users sending messages and the total messages sent on each device.
Profile Pages
Facebook uses A/B testing to test and refine its profile pages. For example, in 2018 Facebook tested a new layout for profile pages that included separate tabs for posts and photos.
Instagram Stories
Instagram tests every new feature and change to its app through A/B testing on small random samples of its user base before rolling it out to the world. For example, in August 2020 they A/B tested a set of fun, dynamic text styles followed by animations to give people more options to express themselves in Stories and Reels.
Google is well known for its extensive use of A/B testing across the board, as I have already commented at the beginning of this post. In fact, they have a website dedicated to the experiments that their team of developers create.
Google Search
Google regularly uses A/B tests to improve the search experience for users. For example, in 2020 Google tested a new design for search results that displayed website branding more prominently. The test was intended to help users better understand the source of the information they find on Google.
Google Ads
Google uses A/B testing to help advertisers optimize their ad campaigns. Advertisers can use A/B testing to test different ad copies, ad formats, and targeting options to see what works best for their business.
Google Maps
Google uses A/B testing to improve the Google Maps user experience. For example, in 2019 Google tested a new feature that allowed users to follow businesses and receive updates about them. The feature was tested on a small group of users to see how they interacted with it and if it improved engagement with local businesses.
Google Chrome
Google uses A/B testing to improve the Chrome user experience. For instance, in 2019 Google tested a new feature that automatically blocked resource-intensive ads on websites. The test was aimed at improving Chrome’s performance and reducing data usage.
Google Play Store
Google uses A/B testing to improve app discovery and installs on the Play Store. For example, in 2020 Google tested a new design for app icons in the Play Store that was intended to make it easier for users to find and install apps they’re interested in.
LinkedIn is a company that for years has been using A/B tests to improve its platform, content, and user experience.
Profile Pictures
LinkedIn uses A/B testing to help users choose the best profile image. In 2017, the company tested a feature that allowed users to A/B test their profile pictures and receive feedback on which one was more effective.
Messaging
LinkedIn uses A/B testing to optimize its messaging feature. For example, in 2018 the company tested a new feature that allowed users to share documents and presentations within their messages.
Job Postings
LinkedIn uses A/B testing to help employers optimize their job postings. Employers can use A/B testing to test different job titles, descriptions, and requirements to see what attracts the best candidates.
Advertising
LinkedIn uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bidding strategies to see what works best for their business.
Content
LinkedIn uses A/B testing to optimize its content recommendation algorithm. In 2018, the company tested a new feature that allowed users to follow hashtags and see related content in their feeds.
Lyft
Lyft, the popular rideshare company, regularly uses A/B testing to optimize its platform and improve user experience.
Ride Request Flow
Lyft uses A/B testing to optimize its ride request flow, which determines how users request rides on its platform. For example, in 2018 the company tested a new ride request flow that allowed users to see an upfront price estimate before booking a ride.
Driver Incentives
Lyft uses A/B testing to optimize its driver incentive programs, which are designed to encourage drivers to complete more rides on its platform. For example, in 2018 the company tested a new incentive program that offered bonuses to drivers who completed a certain number of trips in a week. And in 2022 it tested an earnings algorithm that allowed drivers to see destination and payment details before accepting a request.
Promotions
Lyft uses A/B testing to optimize its promotional offers, which are designed to attract new users to its platform. For example, in 2019, the company tested a new promotional offer that gave users a discount on their first five rides.
App Design
Lyft uses A/B testing to optimize the design of its mobile app, which is used by both drivers and riders. For example, in 2018 the company tested a new app design that made it easier for users to navigate between different screens.
Driver Ratings
Lyft uses A/B testing to optimize its driver rating system, which allows riders to rate their drivers after every ride. For example, in 2020 the company tested a new driver rating system that allowed passengers to rate drivers on a scale of 1-5, instead of just 1-3.
Microsoft
Microsoft also extensively uses A/B testing to optimize its products and services.
Microsoft Office
Microsoft uses A/B testing to improve the user experience of its Office suite of products, such as Word, Excel, and PowerPoint. For example, in 2020 Microsoft tested a new feature in Word that suggested synonyms for commonly used words to help users improve their writing.
Microsoft Bing
Microsoft uses A/B testing to improve the search experience on Bing. For example, in 2020 Microsoft tested a new design for the Bing search results page that aimed to make it easier for users to find the information they’re looking for.
Microsoft Teams
Microsoft uses A/B testing to improve the collaboration experience in Teams. For example, in 2020 Microsoft tested a new feature that allowed users to record and transcribe meetings to make it easier to review them later.
Microsoft Edge
Microsoft uses A/B testing to improve the user experience of its Edge browser. For example, in 2020 Microsoft tested a new feature that automatically grouped tabs based on their content to help users stay organized.
Microsoft Azure
Microsoft uses A/B testing to improve the performance and reliability of its Azure cloud computing platform. For example, in 2020 Microsoft tested a new feature that allowed users to set up automated failover for their applications in case of a service outage.
Netflix
Netflix is a company that has been using A/B testing to improve its platform, content, and user experience. Here are some examples and references of how Netflix uses A/B testing.
User Interface
Netflix uses A/B testing to test and refine its user interface. For example, in 2019 the company tested a new feature that allowed users to shuffle episodes of a TV show or watch them in order. In 2021 it tested various button texts to subscribe.
Content Recommendations
Netflix uses A/B testing to improve its content recommendation algorithm. For example, in 2018 the company tested a new feature that allowed users to hide titles they didn’t want to see in their recommendations.
Video Playback
Netflix uses A/B testing to optimize video playback. For example, in 2018 the company tested a new feature that allowed users to adjust the playback speed of videos.
Mobile Experience
Netflix uses A/B testing to improve your mobile experience. For example, in 2018 the company tested a new feature that allowed users to watch trailers for TV shows and movies directly from the app’s home screen.
Marketing and Advertising
Netflix uses A/B testing to test different marketing and advertising strategies. For example, in 2018 the company tested different trailers and promotional images for its hit series “Stranger Things” to see which ones performed the best.
Stanford University
Stanford University, one of the world’s leading research institutions, uses A/B testing to optimize its websites and online learning platforms.
Online Course Platform
Stanford’s online learning platform, Stanford Online, uses A/B testing to optimize the user experience and improve student engagement. For example, in 2019 the platform tested a new design for its course home page that highlighted the key features of each course and made it easy for students to enroll.
Admissions Website
Stanford uses A/B testing to optimize its admissions website, which prospective students use to learn more about the university and its programs. For instance, in 2018 the university tested a new design for its admissions website that emphasized the student experience and featured more personalized content.
Email Campaigns
Stanford uses A/B testing to optimize its email campaigns, which are used to communicate with current students and alumni. For example, in 2017 the university tested different subject lines and call-to-action buttons in its alumni newsletter to improve open rates and click-through rates.
Website Navigation
Stanford uses A/B testing to optimize navigation on its main website, which serves as a hub for information about the university and its programs. For example, in 2016 the university tested a new navigation menu that made it easier for users to find the information they were looking for. And currently they continue to encourage students to participate in navigation tests to redesign their websites.
Donation campaigns
Stanford uses A/B testing to optimize its donation campaigns, which are used to raise funds for the university’s various initiatives and programs. For example, in 2020 the university tested different variations of its donation landing page to improve the donation conversion rate. One test involved adding a personalized message from the dean of the business school, resulting in a 20% increase in donations.
Twitter is known for using A/B testing to improve its platform, features, and user experience.
Timeline
Twitter uses A/B testing to optimize its timeline algorithm, which determines which tweets appear in users’ feeds. For example, in 2018 Twitter tested a new algorithm that displayed tweets from users’ “favorite” accounts at the top of their feeds.
Ads
Twitter uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad formats, targeting options, and bidding strategies to see what works best for their business.
Notifications
Twitter uses A/B testing to optimize its notifications feature. For example, in 2018 the company tested a new feature that allowed users to receive notifications from accounts they don’t follow.
Explore
Twitter uses A/B testing to optimize its Explore feature, which allows users to discover new content on the platform. For example, in 2019 Twitter tested a new design for Explore that made it easier for users to find content based on their interests.
Tweets
Twitter uses A/B testing to help users write better tweets. For example, in 2018 the company tested a feature that suggested hashtags and handles to users as they typed their tweets.
Uber
Uber, the popular ride-hailing company, extensively uses A/B testing to optimize its platform and improve its user experience.
Surge Pricing
Uber uses A/B testing to optimize its surge pricing algorithm, which determines how much extra riders must pay during periods of high demand. For example, in 2018 the company tested a new surge pricing algorithm that took into account factors such as weather and traffic congestion.
Driver Incentives
Uber uses A/B testing to optimize its driver incentive programs, which are designed to encourage drivers to complete more rides on its platform. For example, in 2020 the company tested a new incentive program that offered bonuses to drivers who maintained high ratings from riders.
Onboarding Process
Uber uses A/B testing to optimize its driver and passenger onboarding processes, which determine how new users are introduced to its platform. For example, in 2019 the company tested a new passenger onboarding process that offered users a discount on their first ride.
App Design
Uber uses A/B testing to optimize the design of its mobile app, which is used by both drivers and riders. For example, in 2021 the company tested a new app design that made it easy for passengers to select their preferred ride type.
Ride Matching
Uber uses A/B testing to optimize its ride matching algorithm, which matches riders with available drivers in their area. For example, in 2018, the company tested a new ride matching algorithm that took into account factors such as traffic congestion and driver availability.
Yandex
Yandex also uses A/B testing to optimize its products and services.
Search
Yandex regularly uses A/B testing to improve the search experience for users. For example, in 2020 Yandex tested a new feature that allowed users to filter search results by their language proficiency.
Ads Campaigns
Yandex uses A/B testing to help advertisers optimize their ad campaigns. Advertisers can use A/B testing to test different ad copy, ad formats, and targeting options to see what works best for their business.
Browser
Yandex uses A/B testing to improve the user experience of its web browser. For example, in 2020 Yandex tested a new feature that allowed users to control the speed of video playback in the browser.
Taxi
Yandex uses A/B testing to improve the user experience of its ride-hailing service. For instance, in 2020, Yandex tested a new feature that allowed users to book a taxi in advance.
Music
Yandex uses A/B testing to improve the user experience of its music streaming service. For instance, in 2020, Yandex tested a new feature that allowed users to control the playback speed of songs.
Conclusion
Like all these companies, investing in A/B testing can greatly benefit your organization and you don’t need to be a large company to have reliable A/B testing results on your website.
A/B testing, and in particular, those created to improve your website, allow you to improve the user experience, increase conversions, reduce the bounce rate, save time and money, and all with the guarantee that you are making informed decisions.
References
In addition to consulting ChatGPT, these are some of the sources of information for the preparation of this post (links consulted on Feb 27, 2022):
- Airlock – Facebook’s mobile A/B testing framework
- Launching Instagram Messsaging on desktop
- Building text animations for Instagram Stories
- Meta Businees Help Center: About A/B Testing
- Google Ads Help Centre
- How Google Maps Uses A/B Testing with Gilad from Moburst
- Google Play Experiments: A Technical Deep Dive into the New Updates
- LinkedIn Engineering: A/B Testing Articles
- A Guide to Proper A/B Testing Strategy on LinkedIn Ads
- Driving Experimentation: How Lyft Fuels Amazin Customer Experimence
- Lyft Engineering: Experimentation in a Rideshareshing Marketplace
- Lyft testing new pay algorithm to lure drivers
- Microsoft Experimentation Platform
- The Surprising Power of Online Experiments
- Microsoft Case Study
- The Benefits of Controlled Experimentation at Scale
- The Verge: Netflix is experimenting with different episode orders for its new anthology show
- Verge: Netflix is testing a ‘random episode’ feature for TV shows
- Netflix Technology Blog: A/B Testing and Beyond: Improving the Netflix Streaming Experience with Experimentation and Data Science
- Trusted Reviews: Netflix is testing trailers between episodes and the internet is kicking off
- A/B Testing at Netflix, the typography of Stranger Things, and more
- Standford offers a preview of the redesign of its main website
- Standford Report: Help redesign Bechtel International Center’s website
- Here’s How Twitter’s ML Algorithms Rank The ‘Best’ Tweets on Your Timeline
- Twitter’s latests test lets users susbscribe to a tweet’s replies
- Twitter is Reportedly Working on a New Notifications Design to Reduce Confusion!
- Twitter introduces personalized news alerts ahead of the World Cup
- A hashtag recommendation system for twitter data streams
- Uber Is Testing a New Feature Where You Can Wait for a Lower Fare
- Uber knows you: how data optimizes our rides
- Uber and Lyft: How Driver Incentives Drive Retention
- Yandex Direct: A/B Testing Reliability Calculator
- Yandex: A/B testing is now available in Mobile Mediation
- Yandex Direct introduced a new approach to A/B-testing
- Yandex Browser Testing
- Online Evaluation for Effective Web Service Development
Featured image from freestocks on Unsplash.
Leave a Reply