Escritorio con dos portátiles

In the fast-changing digital world, it is difficult to assess the impact of new ideas. For more than 20 years, leading digital companies have been adopting a “test everything” approach based on A/B testing to make decisions that can generate significant value for the company. That is, before creating new content or developing any functionality, product, or service, A/B tests are conducted to evaluate its acceptance and impact.

One of the first A/B test cases most people know about was was Google’s experiment of testing 41 Shades of Blue , later known as “Fifty Shades”. Over the years, the benefits of this test have continued to be discussed.

Google shades of blue test
Google shades of blue test.

In 2019, representatives from 13 large organizations (12 companies and Stanford University), which each currently run more than 100,000 tests per year, held the first summit to discuss the key challenges of A/B testing.

Taking advantage of this list of organizations, below, sorted alphabetically by company, you will find 5 examples of A/B tests performed by each of them. These examples show how A/B tests are the starting point for assessing the vast majority of changes that are implemented in the products and services they offer. At the end of the post you will find the information sources for the examples shown.

Airbnb

Airbnb, the online portal for vacation rentals and travel experiences, uses A/B testing extensively to learn and make decisions at every step of its product development, from design to algorithms. They are equally important in shaping the user experience.

Search Ranking

Airbnb uses A/B testing to optimize the search ranking algorithm for its vacation rental listings. For example, in 2017, Airbnb tested a new search ranking algorithm that took into account factors such as the quality and completeness of a listing, host response rate, and rental location.

Pricing

Airbnb uses A/B testing to optimize the pricing of its vacation rentals. For instance, in 2018, the company tested a new pricing feature that offered hosts personalized price suggestions based on factors like demand and competition.

Example of price changes shown on Airbnb after an A/B test (source: Experiments at Airbnb).

Guest Messaging

Airbnb uses A/B testing to optimize its guest messaging feature, which allows hosts to communicate with guests before and during their stay. For example, in 2020, the company tested a new messaging feature that allowed hosts to send automated messages to guests.

User Interface

Airbnb uses A/B testing to optimize the user interface of its mobile app and website. For example, in 2019, the company tested a new interface design that made it easier for users to navigate and book vacation rentals.

Reviews

Airbnb uses A/B testing to optimize its review system, allowing guests to leave feedback on their stays. For example, in 2018, the company tested a new review system that allowed guests to rate their stays on a scale of 1-10, instead of the previous 5-star rating system.

Amazon

Amazon is known for its extensive use of A/B testing to optimize its products and services. Let’s look at some examples of how Amazon uses A/B testing.

Amazon.com

Amazon continually uses A/B testing to optimize the user experience on its online marketplace. For instance, in 2020, Amazon tested a new design for its product detail pages that was aimed to improve the visibility of customer reviews.

Amazon Prime Video

Amazon uses A/B testing to improve the user experience of its streaming video service. For example, in 2020, Amazon tested a new feature that allowed users to customize the order of their watchlist.

Amazon Web Services

Amazon uses A/B testing to optimize the performance and reliability of its cloud computing platform. For instance, in 2020, Amazon launched Amazon Personalize, which makes it easy to integrate personalized recommendations into existing websites, apps, email marketing systems, and more, including the option to create A/B tests.

Amazon Alexa

Amazon uses A/B testing to improve the user experience of its voice assistant. For example, they recently created an A/B test for Vocala (a company specializing in voice ads) to determine the ideal length of invitation for users to subscribe to the paid version of their clients.

Amazon Advertising

Amazon uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bid strategies to see what works best for their business.

Example of the same product page viewed from two different computers:

Source: BrillMark.
Source: BrillMark.

Booking

Booking.com, the online travel agency that allows users to book accommodation, flights, and rental cars, extensively uses A/B testing to optimize its platform and improve user experience. Everything goes through a previous A/B test before being launched on the market for the public.

Search Results

Booking.com uses A/B testing to optimize the search results page for its accommodation listings. For example, in 2020 the company tested a new search results page that featured larger images and more detailed information about listings. In addition, accommodation owners have a wide variety of products to experiment with.

User Interface

Booking.com uses A/B testing to optimize the user interface of its mobile app and website. For example, in 2019, the company tested a new design for its mobile app that made it easier for users to find and book accommodation.

Pricing

Booking.com uses A/B testing to optimize the pricing of its accommodation listings. For example, in 2018, the company tested a new pricing feature that offered personalized discounts to customers who had previously abandoned their bookings.

Reviews

Booking.com uses A/B testing to optimize its review system, allowing customers to leave feedback on their stays. For example, in 2018 the company tested a new review system that included more detailed information about properties, such as room sizes and amenities.

Website Layout

Booking.com uses A/B testing to optimize its website layout. For example, in 2017 the company tested a new layout for its home page that featured larger images and more prominent search bars.

Meta

Meta is known for using A/B testing to optimize its products and services.

News Feed

Facebook uses A/B testing to test and refine its News Feed algorithm, which determines what content appears in users’ feeds. For example, in 2018 Facebook tested a new algorithm that prioritized posts from family and friends over content from publishers.

Ad Targeting

Facebook uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bidding strategies to see what works best for their business.

Instagram Messaging

Meta conducted A/B tests to optimize its messaging app across its various platforms. For example, in 2020, before launching Instagram Messaging, it ran an A/B test to compare its usage on desktop vs. native apps, monitoring the number of users sending messages and the total messages sent on each device.

Profile Pages

Facebook uses A/B testing to test and refine its profile pages. For example, in 2018 Facebook tested a new layout for profile pages that included separate tabs for posts and photos.

Instagram Stories

Instagram tests every new feature and change to its app through A/B testing on small random samples of its user base before rolling it out to the world. For example, in August 2020 they A/B tested a set of fun, dynamic text styles followed by animations to give people more options to express themselves in Stories and Reels.

Google

Google is well known for its extensive use of A/B testing across the board, as I have already commented at the beginning of this post. In fact, they have a website dedicated to the experiments that their team of developers create.

Google Search

Google regularly uses A/B tests to improve the search experience for users. For example, in 2020 Google tested a new design for search results that displayed website branding more prominently. The test was intended to help users better understand the source of the information they find on Google.

Google Ads

Google uses A/B testing to help advertisers optimize their ad campaigns. Advertisers can use A/B testing to test different ad copies, ad formats, and targeting options to see what works best for their business.

Google Maps

Google uses A/B testing to improve the Google Maps user experience. For example, in 2019 Google tested a new feature that allowed users to follow businesses and receive updates about them. The feature was tested on a small group of users to see how they interacted with it and if it improved engagement with local businesses.

Google Chrome

Google uses A/B testing to improve the Chrome user experience. For instance, in 2019 Google tested a new feature that automatically blocked resource-intensive ads on websites. The test was aimed at improving Chrome’s performance and reducing data usage.

Google Play Store

Google uses A/B testing to improve app discovery and installs on the Play Store. For example, in 2020 Google tested a new design for app icons in the Play Store that was intended to make it easier for users to find and install apps they’re interested in.

LinkedIn

LinkedIn is a company that for years has been using A/B tests to improve its platform, content, and user experience.

Profile Pictures

LinkedIn uses A/B testing to help users choose the best profile image. In 2017, the company tested a feature that allowed users to A/B test their profile pictures and receive feedback on which one was more effective.

Messaging

LinkedIn uses A/B testing to optimize its messaging feature. For example, in 2018 the company tested a new feature that allowed users to share documents and presentations within their messages.

Job Postings

LinkedIn uses A/B testing to help employers optimize their job postings. Employers can use A/B testing to test different job titles, descriptions, and requirements to see what attracts the best candidates.

Advertising

LinkedIn uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad creatives, targeting options, and bidding strategies to see what works best for their business.

Content

LinkedIn uses A/B testing to optimize its content recommendation algorithm. In 2018, the company tested a new feature that allowed users to follow hashtags and see related content in their feeds.

Lyft

Lyft, the popular rideshare company, regularly uses A/B testing to optimize its platform and improve user experience.

Ride Request Flow

Lyft uses A/B testing to optimize its ride request flow, which determines how users request rides on its platform. For example, in 2018 the company tested a new ride request flow that allowed users to see an upfront price estimate before booking a ride.

Driver Incentives

Lyft uses A/B testing to optimize its driver incentive programs, which are designed to encourage drivers to complete more rides on its platform. For example, in 2018 the company tested a new incentive program that offered bonuses to drivers who completed a certain number of trips in a week. And in 2022 it tested an earnings algorithm that allowed drivers to see destination and payment details before accepting a request.

Promotions

Lyft uses A/B testing to optimize its promotional offers, which are designed to attract new users to its platform. For example, in 2019, the company tested a new promotional offer that gave users a discount on their first five rides.

App Design

Lyft uses A/B testing to optimize the design of its mobile app, which is used by both drivers and riders. For example, in 2018 the company tested a new app design that made it easier for users to navigate between different screens.

Driver Ratings

Lyft uses A/B testing to optimize its driver rating system, which allows riders to rate their drivers after every ride. For example, in 2020 the company tested a new driver rating system that allowed passengers to rate drivers on a scale of 1-5, instead of just 1-3.

Microsoft

Microsoft also extensively uses A/B testing to optimize its products and services.

Microsoft Office

Microsoft uses A/B testing to improve the user experience of its Office suite of products, such as Word, Excel, and PowerPoint. For example, in 2020 Microsoft tested a new feature in Word that suggested synonyms for commonly used words to help users improve their writing.

Microsoft Bing

Microsoft uses A/B testing to improve the search experience on Bing. For example, in 2020 Microsoft tested a new design for the Bing search results page that aimed to make it easier for users to find the information they’re looking for.

Microsoft Teams

Microsoft uses A/B testing to improve the collaboration experience in Teams. For example, in 2020 Microsoft tested a new feature that allowed users to record and transcribe meetings to make it easier to review them later.

Microsoft Edge

Microsoft uses A/B testing to improve the user experience of its Edge browser. For example, in 2020 Microsoft tested a new feature that automatically grouped tabs based on their content to help users stay organized.

Microsoft Azure

Microsoft uses A/B testing to improve the performance and reliability of its Azure cloud computing platform. For example, in 2020 Microsoft tested a new feature that allowed users to set up automated failover for their applications in case of a service outage.

Netflix

Netflix is a company that has been using A/B testing to improve its platform, content, and user experience. Here are some examples and references of how Netflix uses A/B testing.

User Interface

Netflix uses A/B testing to test and refine its user interface. For example, in 2019 the company tested a new feature that allowed users to shuffle episodes of a TV show or watch them in order. In 2021 it tested various button texts to subscribe.

Example of an A/B test performed on Netflix (Source: Netflix Technology Blog).

Content Recommendations

Netflix uses A/B testing to improve its content recommendation algorithm. For example, in 2018 the company tested a new feature that allowed users to hide titles they didn’t want to see in their recommendations.

Video Playback

Netflix uses A/B testing to optimize video playback. For example, in 2018 the company tested a new feature that allowed users to adjust the playback speed of videos.

Mobile Experience

Netflix uses A/B testing to improve your mobile experience. For example, in 2018 the company tested a new feature that allowed users to watch trailers for TV shows and movies directly from the app’s home screen.

Marketing and Advertising

Netflix uses A/B testing to test different marketing and advertising strategies. For example, in 2018 the company tested different trailers and promotional images for its hit series “Stranger Things” to see which ones performed the best.

Stanford University

Stanford University, one of the world’s leading research institutions, uses A/B testing to optimize its websites and online learning platforms.

Online Course Platform

Stanford’s online learning platform, Stanford Online, uses A/B testing to optimize the user experience and improve student engagement. For example, in 2019 the platform tested a new design for its course home page that highlighted the key features of each course and made it easy for students to enroll.

Admissions Website

Stanford uses A/B testing to optimize its admissions website, which prospective students use to learn more about the university and its programs. For instance, in 2018 the university tested a new design for its admissions website that emphasized the student experience and featured more personalized content.

Email Campaigns

Stanford uses A/B testing to optimize its email campaigns, which are used to communicate with current students and alumni. For example, in 2017 the university tested different subject lines and call-to-action buttons in its alumni newsletter to improve open rates and click-through rates.

Website Navigation

Stanford uses A/B testing to optimize navigation on its main website, which serves as a hub for information about the university and its programs. For example, in 2016 the university tested a new navigation menu that made it easier for users to find the information they were looking for. And currently they continue to encourage students to participate in navigation tests to redesign their websites.

Donation campaigns

Stanford uses A/B testing to optimize its donation campaigns, which are used to raise funds for the university’s various initiatives and programs. For example, in 2020 the university tested different variations of its donation landing page to improve the donation conversion rate. One test involved adding a personalized message from the dean of the business school, resulting in a 20% increase in donations.

Twitter

Twitter is known for using A/B testing to improve its platform, features, and user experience.

Timeline

Twitter uses A/B testing to optimize its timeline algorithm, which determines which tweets appear in users’ feeds. For example, in 2018 Twitter tested a new algorithm that displayed tweets from users’ “favorite” accounts at the top of their feeds.

Ads

Twitter uses A/B testing to help advertisers optimize their campaigns on its advertising platform. Advertisers can use A/B testing to test different ad formats, targeting options, and bidding strategies to see what works best for their business.

Notifications

Twitter uses A/B testing to optimize its notifications feature. For example, in 2018 the company tested a new feature that allowed users to receive notifications from accounts they don’t follow.

Explore

Twitter uses A/B testing to optimize its Explore feature, which allows users to discover new content on the platform. For example, in 2019 Twitter tested a new design for Explore that made it easier for users to find content based on their interests.

Tweets

Twitter uses A/B testing to help users write better tweets. For example, in 2018 the company tested a feature that suggested hashtags and handles to users as they typed their tweets.

Uber

Uber, the popular ride-hailing company, extensively uses A/B testing to optimize its platform and improve its user experience.

Surge Pricing

Uber uses A/B testing to optimize its surge pricing algorithm, which determines how much extra riders must pay during periods of high demand. For example, in 2018 the company tested a new surge pricing algorithm that took into account factors such as weather and traffic congestion.

Driver Incentives

Uber uses A/B testing to optimize its driver incentive programs, which are designed to encourage drivers to complete more rides on its platform. For example, in 2020 the company tested a new incentive program that offered bonuses to drivers who maintained high ratings from riders.

Onboarding Process

Uber uses A/B testing to optimize its driver and passenger onboarding processes, which determine how new users are introduced to its platform. For example, in 2019 the company tested a new passenger onboarding process that offered users a discount on their first ride.

App Design

Uber uses A/B testing to optimize the design of its mobile app, which is used by both drivers and riders. For example, in 2021 the company tested a new app design that made it easy for passengers to select their preferred ride type.

Ride Matching

Uber uses A/B testing to optimize its ride matching algorithm, which matches riders with available drivers in their area. For example, in 2018, the company tested a new ride matching algorithm that took into account factors such as traffic congestion and driver availability.

Yandex

Yandex also uses A/B testing to optimize its products and services.

Search

Yandex regularly uses A/B testing to improve the search experience for users. For example, in 2020 Yandex tested a new feature that allowed users to filter search results by their language proficiency.

Ads Campaigns

Yandex uses A/B testing to help advertisers optimize their ad campaigns. Advertisers can use A/B testing to test different ad copy, ad formats, and targeting options to see what works best for their business.

Browser

Yandex uses A/B testing to improve the user experience of its web browser. For example, in 2020 Yandex tested a new feature that allowed users to control the speed of video playback in the browser.

Taxi

Yandex uses A/B testing to improve the user experience of its ride-hailing service. For instance, in 2020, Yandex tested a new feature that allowed users to book a taxi in advance.

Music

Yandex uses A/B testing to improve the user experience of its music streaming service. For instance, in 2020, Yandex tested a new feature that allowed users to control the playback speed of songs.

Conclusion

Like all these companies, investing in A/B testing can greatly benefit your organization and you don’t need to be a large company to have reliable A/B testing results on your website.

A/B testing, and in particular, those created to improve your website, allow you to improve the user experience, increase conversions, reduce the bounce rate, save time and money, and all with the guarantee that you are making informed decisions.

References

In addition to consulting ChatGPT, these are some of the sources of information for the preparation of this post (links consulted on Feb 27, 2022):

Featured image from freestocks on Unsplash.

Leave a Reply

Your email address will not be published. Required fields are marked *

I have read and agree to the Nelio Software Privacy Policy

Your personal data will be located on SiteGround and will be treated by Nelio Software with the sole purpose of publishing this comment here. The legitimation is carried out through your express consent. Contact us to access, rectify, limit, or delete your data.