<!-- AI Moderation for Dating Sites: What It Can and Cannot Do | https://whitelabeldating.com/guides/ai-moderation-dating/ | Run by High Intent Media, which also runs Dating Partners and DatingAPI. -->

# AI moderation for dating sites and apps

> AI moderation uses software to screen photos, profile text, messages and behaviour on a dating site. It acts on the clear cases and sends the doubtful ones to a person. It is strong on nudity, spam and known scam patterns, and weak on context. Keep people in charge of bans and appeals, and test any tool on your own content.

## What does AI moderation do on a dating site?

AI moderation is software that scores photos, text and behaviour for risk, so that clear breaches are stopped at once and doubtful cases reach a person quickly. It applies the rules you have written. It does not make them.

On a dating site it usually does four jobs.

| Job | What the software looks at | Where it is reliable | Where it needs a person |
|---|---|---|---|
| Photo screening | Profile and chat images | Nudity, violence, images that match known illegal material | Swimwear, art, group photos, photos of someone else |
| Text screening | Profile text and messages | Spam, links, phone numbers, repeated scripts | Banter, sarcasm, slang, other languages |
| Scam and fake detection | Sign-up details, devices, message patterns | Mass messaging, copied scripts, bulk sign-ups | A slow, patient scam run by a real person |
| Report triage | Member reports | Sorting by type and urgency | Deciding who is telling the truth |

The value is in the sorting. Amazon says that with its Rekognition image service, human moderators typically review 1 to 5% of total volume, the part the software has already flagged. That is a supplier's figure, not a promise for your site, but it shows the purpose.

Our guide to a [content moderation workflow](/guides/content-moderation-workflow/) covers the full path from rules to appeals.

## How do the big dating apps use AI moderation?

They use it to act before a member has to report something. Three published examples show the pattern.

- **Blurring lewd images.** Bumble launched Private Detector on Bumble and Badoo in 2019. It blurs a suspected nude image in chat, warns the member, and lets them choose to view it, block it or report it. Bumble says it performs at more than 98% accuracy and has released an open-source version.
- **Blocking fake and scam accounts.** Bumble says its Deception Detector blocked 95% of the accounts it identified as spam or scam automatically in testing, with a human moderation team behind the model.
- **Prompting the sender.** Tinder's Are You Sure? uses AI to spot potentially offensive language as a member types and asks them to pause before sending. Match Group said in May 2021 that it reduced inappropriate language in sent messages by more than 10% in early testing.

All three are company figures. Two points stand out: the member stays in control of what they see, and people still sit behind the model.

## Where does AI moderation go wrong?

It goes wrong on context, and it goes wrong at scale. A model does not know that two members have been joking for a week, or that a phrase is affectionate in one community and abusive in another.

The common failures on dating services are:

- **False positives.** A beach photo is flagged as nudity, or a blunt message is flagged as harassment. Each one is a paying member wrongly restricted.
- **False negatives.** A scammer who writes well and moves slowly looks like a keen member.
- **Language gaps.** Tools trained mostly on English are weaker on other languages and on slang.
- **Uneven treatment.** A tool can flag one group of members more often than another. You only find this if you look.
- **Scores that are not probabilities.** A score shows the model's confidence, not how often it is right on your content.

Do not accept an accuracy percentage without asking what it was measured on. A supplier's test set says little about your members, your photo rules and your languages.

## Which decisions should stay with a human moderator?

Any decision that removes a member's access, and any decision that depends on context, should be made or confirmed by a person. Automation suits actions that are easy to undo.

A workable split uses the confidence score the tool returns:

1. **High score, low impact.** Act automatically. Blur the image, hold the photo back from display or block the link.
2. **Middle score.** Hold the content and put it in a review queue with the reason shown to the moderator.
3. **Low score.** Publish, and rely on reports and behaviour monitoring.
4. **Any account ban.** A person confirms it, except for the clearest spam and bulk sign-ups, and the member can appeal to a person.

Set the thresholds yourself. Amazon Rekognition, for example, returns moderation labels with a confidence of 50% or more unless you set a different minimum. A default like that is a starting point, not a policy. Rekognition can also route images to a human review step, which is the design to copy.

## What do the rules say about automated moderation?

The rules expect moderation that works, and they limit decisions made by software alone. Three sets of rules matter most to a dating operator in the UK and the EU.

- **Online Safety Act (UK).** Ofcom's illegal harms measures ask every service that lets members contact each other to have a content moderation function that reviews suspected illegal content and can take it down swiftly. Larger and riskier services also need written policies, performance targets, prioritisation, resourcing and training. Automated hash matching for child sexual abuse material is a measure for large services at medium or high risk, and for high-risk services with "more than 700,000 monthly active United Kingdom users" or that offer file storage or sharing. Ofcom counts a service as large above 7 million UK members a month. It names fraud, intimate image abuse and harassment as risks for dating services.
- **UK GDPR.** The Information Commissioner's Office (ICO) says a decision made solely by automated means that has a legal or similarly significant effect is restricted. Where one is allowed, people must be able to obtain human intervention, give their point of view and challenge the decision. The ICO also expects a data protection impact assessment and regular checks for accuracy and bias.
- **Digital Services Act (EU).** Article 17 requires a statement of reasons when content or an account is restricted, including information on any use of automated means. Article 20 says decisions on complaints must be taken under the supervision of qualified staff and "not solely on the basis of automated means".

Which duties apply depends on your size and where your members are. See our guides to the [Online Safety Act for dating sites](/guides/online-safety-act-dating/) and [DSA compliance](/guides/dsa-dating-compliance/). This page is general information, not legal advice.

## Which AI moderation tools can a dating operator use?

You can use general-purpose tools from the large cloud and AI companies, a specialist trust and safety supplier, or a dating platform that includes moderation. What they publish varies a great deal.

| Option | What it covers | What is published |
|---|---|---|
| Amazon Rekognition | Image and video moderation labels with confidence scores, plus a link to human review | Documentation is public. Pricing is by use; check Amazon's current price list. |
| OpenAI moderation endpoint | Text and images, scored from 0 to 1 across 13 categories such as harassment, sexual content and violence | OpenAI says the endpoint is free to use. |
| Perspective API (Jigsaw and Google) | Text toxicity scoring | Closing. The site says the service ends after 31 December 2026, so do not build on it. |
| Specialist trust and safety suppliers | Often text, images, behaviour and a review console in one product | Prices are usually not published; ask them. |
| A dating platform | Screening, review and enforcement run for you | Ask what is automated and what a person reviews. |

General tools were not built for dating conversations, so test before you trust them. If you run your own software, you also need the review queue, the audit log and the appeal route around the tool.

## How should you test an AI moderation tool before you rely on it?

Test it on your own content, with your own rules, before it takes any action.

- Build a sample of real, already-moderated photos and messages, including hard cases.
- Run the tool in flag-only mode and compare its flags with your moderators' decisions.
- Count wrong flags and misses separately for each category. They cost you different things.
- Check results by language, age group and gender to look for uneven treatment.
- Choose thresholds per category. Nudity and harassment do not need the same setting.
- Agree what happens when the tool is down: hold content, or publish and review later.
- Recheck every few months. Scammers change their scripts, and suppliers change their models.

Ask the supplier where content is processed, whether it is kept or used for training, and whether you can export decisions for audit. Our guide to [AI in dating](/guides/ai-in-dating/) covers the wider questions to ask an AI supplier.

## Who runs AI moderation on a white label dating platform?

The platform runs it, and every brand on the network gets the same checks. You do not choose a tool or staff a review queue.

Dating Partners, our sister company run by the same group as this site, lists four layers of member checks: identity checks, AI scam detection, human review of every profile and ongoing behaviour monitoring. Dating Partners is for dating websites only, and it does not accept adult or sugar dating brands.

DatingAPI, also run by our group, includes safety and moderation in its platform for dating apps.

This route does not suit an operator who wants to set its own moderation rules or train its own models. Whoever you use, ask what is automated, what a person reviews, how long reviews take and how a member appeals. Our [safety and moderation](/platform/safety-and-moderation/) page sets out what our own platform does.

## Common questions

### Can AI replace human moderators on a dating site?

No. AI cuts the amount of content people have to look at, but bans, appeals and anything that depends on context still need a trained person. Regulators and the EU's Digital Services Act expect a human route.

### Is AI moderation expensive?

The screening itself can be cheap. OpenAI says its moderation endpoint is free, and cloud image tools charge by use. The larger cost is the people who review flags and handle appeals.

### Should we tell members that AI screens their content?

Yes. Say so in your terms and privacy notice, and when you act on a flag, tell the member what was found and how to appeal. In the EU, a statement of reasons must say whether automated means were used.

### Does AI moderation read private messages?

It can. Bumble and Tinder both scan chat for lewd images or harmful language. If you do this, say so in your privacy notice and keep the purpose limited to safety.

### What is hash matching?

It compares the digital fingerprint of an uploaded image with a list of fingerprints of known illegal images. It finds exact or near copies of known material, and it does not judge new images.
