What is A/B testing on a dating site?
It is showing two versions of the same page or step to two random groups of people and comparing one result. Version A is what you have now. Version B has one change. If the groups are random and large enough, any difference in the result comes from the change.
A proper test has four parts, all decided before it starts:
- One change. A new headline, one fewer sign-up step, a different paywall message.
- One measure. Completed sign-ups, first messages sent or first payments. See conversion rate in our glossary.
- Random assignment. Each person lands in A or B by chance and stays there.
- A fixed sample size. The number of people you will wait for before you look at the answer.
Testing does not tell you what to build. It tells you which of two options did better on one measure, for the people who saw them.
What should a dating site test first?
Test the steps closest to sign-up and payment, because that is where a small site has enough traffic to learn anything. Work down this table in order.
| Area | Example test | Measure | Traffic needed |
|---|---|---|---|
| Landing page | Headline that names your audience against a general one | Sign-ups started | Lowest: every visitor counts |
| Sign-up flow | Photo upload during sign-up against after it | Sign-ups completed | Low |
| First session | Suggested first message against a blank box | Members who send a message | Medium |
| Paywall | What the upgrade screen says and when it appears | First payments | High: only some members reach it |
| Emails | Subject line or send time | Opens, then return visits | Medium |
| Price | Monthly price or plan lengths | Revenue per sign-up over months | Highest, and it carries legal risk (see below) |
Our guides to landing pages for dating sites and payer conversion suggest what to change at each step.
How many sign-ups do you need for a valid test?
More than most new sites expect: usually thousands in each group. The smaller the difference you want to detect, the more people you need.
Evan Miller, in his article on how not to run an A/B test, gives a rule of thumb for the number needed in each group: 16 times the variance, divided by the square of the smallest difference you want to detect. For a rate such as "share of sign-ups who pay", the variance is the rate times one minus the rate.
Here is a worked example. It uses the same starting figures as our revenue calculator (50 sign-ups a day, 10% of whom pay) and is an illustration, not a benchmark.
| Change you want to detect | Needed in each group | Total sign-ups | Time at 50 sign-ups a day |
|---|---|---|---|
| 10% paying up to 12% | 3,600 | 7,200 | About 5 months |
| 10% paying up to 11% | 14,400 | 28,800 | About 19 months |
The lesson is practical. A site with modest traffic can only detect large differences, so it should test bold changes, not small ones. A new headline, a different offer or a shorter form may move the result enough to see. A new button colour will not.
How long should a test run, and when can you stop?
Run it until you reach the sample size you fixed at the start, and not before. Stopping as soon as the dashboard shows a winner is the most common way to fool yourself.
Miller's article explains why. Significance figures assume you set the sample size in advance. If you check repeatedly and stop at the first significant result, the rate of false wins rises. In his example of a change that does nothing, checking after every observation produced a false positive 26.1% of the time, more than five times the 5% the dashboard claimed.
Three rules follow:
- Write down the sample size and the measure before the test starts.
- Run for whole weeks, so that weekdays and weekends are both covered.
- If you must be able to stop early, use a tool built for it. Miller points to sequential and Bayesian designs as methods that allow this.
Why are dating sites harder to test than online shops?
Because members affect each other, so the two groups in a test are not truly separate. A shopper who sees a new checkout does not change what another shopper sees. A dating member who gets a new feature changes the experience of everyone they contact.
Brenton McMenamin, writing about his work at OkCupid, set out the problem in an article for the company's tech blog (we read an archived copy). His example was video chat, which only works when both people have it. Give it to everyone and there is no control group. Give it only to pairs in the test group and it appears unevenly, which changes how people behave. His answer was to assign whole communities to a version, not individuals. He divided the US and Canada into 36 regions and found that about 95% of interactions stayed inside one region.
Academic work on two-sided marketplaces reaches the same conclusion. A 2022 paper by Li, Zhao, Johari and Weintraub, listed by Stanford Graduate School of Business, notes that people in a market compete with each other, which creates interference and a biased estimate.
In practice:
- Safe to test per person: anything a member does alone, such as landing pages, sign-up steps, paywall wording and emails.
- Test by region or not at all: anything that changes who sees whom, how many messages are sent or what the other person can do.
On a small site, the second kind is usually better judged by a careful before and after comparison, with the limits stated honestly.
Is it safe to test different prices on members?
Only with care, because price tests show different people different prices, and regulators and the press treat unexplained price differences as unfair.
Tinder is the best documented case. In February 2022 Engadget reported that Tinder would stop charging older members more for Tinder+, after a study by Mozilla and Consumers International found that members aged 30 to 49 were charged an average of 65.3% more than younger members in five of the six countries tested. In March 2024 the Dutch consumer authority, ACM, said Tinder had agreed to tell consumers clearly and in advance when a personalised discount is applied. ACM's position was that price personalisation is allowed only if the company is clear about it.
If you test prices:
- Assign prices at random, never by age, sex, sexual orientation, location within a country or any other personal trait.
- Test on new sign-ups only, so existing members do not see their price change.
- Honour the price each member was shown, including at renewal.
- Measure revenue per sign-up over several months, not first payments alone. A lower price can win in week one and lose over the year.
Our guide to dating subscription pricing covers what to charge. This section is general information, not legal advice.
Does A/B testing affect search rankings or app store listings?
Not if you follow the platforms' own rules. Google publishes guidance for tests on web pages, and Apple has a testing tool built into the App Store.
For websites, Google says:
- Do not show Googlebot one set of URLs and people another. That is cloaking.
- If the versions sit at different URLs, point the variants at the original with rel="canonical".
- Use a 302 (temporary) redirect for a test, not a 301.
- Run the test only as long as you need, then remove the test pages and scripts promptly.
For iPhone apps, Apple's product page optimisation lets you test up to three alternative versions of your store page against the original. You can vary the icon, screenshots and preview videos. A test runs for up to 90 days, and Apple recommends waiting until a version is shown to be better or worse with at least 90% confidence. Our guide to app store optimisation for dating apps goes further.
What can you test on a white label platform?
You can test everything you control: your adverts, landing pages, brand, wording and audience. The product behind the sign-up form belongs to the platform, so tests there are the platform's to run.
On Dating Partners, our sister company run by the same group as this site, the partner looks after the brand, domain, look and marketing, and the platform runs the software and payments. Dating Partners is for websites only; apps run through DatingAPI, also part of our group, where your app sits under your own store listing.
That split has one advantage and one limit. A platform that serves many brands has far more traffic to test product changes with than a single new site. But you cannot run your own paywall or pricing test unless the provider offers it. Ask any provider:
- Can I send traffic to two landing pages and see sign-ups and paying members for each?
- Do you test the sign-up flow and paywall yourselves, and do winning versions reach my brand?
- Can I change prices or offers for my brand, and can I test them?
Our partner console page shows what reporting our own platform gives partners. If your plan depends on running your own product experiments every week, a white label is the wrong fit and you need your own platform.
Questions
Which A/B testing tool should a dating site use?
Any tool that assigns visitors at random, shows each person the same version every time, and reports on sign-ups and payments, not only clicks. Check whether it sets cookies that need consent before you install it.
Is it worth testing button colours?
Rarely, on a small site. Small visual changes tend to produce small effects, and small effects need very large samples to detect. Test the offer, the headline and the number of steps first.
Can I run more than one test at a time?
Yes, if the tests sit on different steps and each assigns people at random. Apple allows only one product page test at a time on the App Store.
What is a holdout group?
A small share of members who keep the old version after you roll out a winner. Comparing them with everyone else over several months shows whether the gain lasts.
What should I do when a test shows no difference?
Keep the simpler or cheaper version and move on. A flat result is useful: it tells you that part of the journey is not where your problem is.
Sources
- Evan Miller: How Not To Run an A/B Test, checked 21 November 2026
- Brenton McMenamin (OkCupid): The pitfalls of A/B testing in social networks (archived copy), checked 21 November 2026
- Li, Zhao, Johari and Weintraub: Interference, Bias, and Variance in Two-Sided Marketplace Experimentation (Stanford GSB), checked 21 November 2026
- Google Search Central: Minimize A/B testing impact in Google Search, checked 21 November 2026
- Apple Developer: Product page optimization, checked 21 November 2026
- Engadget: Tinder will stop charging older users more for premium features, checked 21 November 2026
- ACM (Netherlands Authority for Consumers and Markets): Tinder and personalised prices, 7 March 2024, checked 21 November 2026
- Dating Partners partner terms, checked 21 November 2026
What changed
- : First published. Replaces our older guide on A/B testing for dating sites.