When Five Stars Don't Mean What They Used To

93% of consumers say online reviews influence their purchase decisions. That number alone tells you how much is at stake. Reviews shape where people go for healthcare, who they trust with their car, and whether a small business survives. When those reviews are faked, the whole system breaks down.

This project started from a simple question: how sophisticated does fake review fraud actually get? The answer, it turns out, is more sophisticated than most detection systems are built to handle.

The Problem with Simple Filters

Yelp already has a filter. It catches some things. But coordinated campaigns, where the same templated review gets posted across multiple accounts and businesses with just enough variation to dodge keyword matching, tend to slip through.

What we wanted to build was something that could catch not just obvious duplicates, but the subtler patterns too: reviews that share the same unusual narrative arc, reviews posted by accounts with no friends or review history, and reviews that appear at different businesses within a short time window. The kind of signals that look innocent in isolation but tell a clearer story together.

The Dataset

We focused on five-star reviews from two industries: Health and Medical, and Auto Repair, across 2020 and 2021. The dataset covered over 35,000 reviews across roughly 20,000 unique users and thousands of businesses across 14 to 15 states.

A few things stood out immediately. Between 39% and 44% of reviewers had zero friends on Yelp depending on the year and industry. Zero. That is a well-known signal for throwaway accounts. And despite restricting to five-star reviews only, sentiment analysis still showed a left-tailed skew, meaning the actual text did not always match the rating. That kind of disconnect between label and content is worth paying attention to.

How the Detection System Works

Rather than betting everything on one method, we layered four different approaches and combined them into a single weighted suspicion score.

Step 1: Similarity Detection (50% of the score)

We computed four similarity measures for every review pair within each industry pool. Two were embedding-based using the all-MiniLM-L6-v2 sentence transformer model, which catches semantic similarity even when the exact words differ. Two were TF-IDF based, which catches exact vocabulary overlap with higher weight on rare words.

Any pair exceeding a 0.9 threshold on any of the four measures got flagged. The embedding and TF-IDF approaches are only moderately correlated at around 0.66 to 0.73, which means they catch genuinely different things. Using both was the right call.

Step 2: Fraud Signals (35% of the score)

For every flagged pair we applied seven rule-based signals: same user posting from different states, reviews posted within seven days of each other, cross-state coordination between different users, word count anomalies outside the 5th to 95th percentile range, character-level near-identical text via SequenceMatcher, accounts with three or fewer total reviews, and marketing language patterns like 'call now' or 'free consultation'.

Step 3: Claude AI Scoring (15% of the score)

The top 500 flagged pairs by similarity score were sent to the Claude API for deeper review. Each review got scored individually across four dimensions: marketing language, generic praise, emotional manipulation, and incentivized content. Each pair was also evaluated together for coordinated campaign likelihood. The combined Claude score weighted coordinated likelihood at 50% and blended the individual review scores for the rest.

What We Actually Found

Out of more than 35,000 five-star reviews, the similarity filter flagged 137 pairs above the 0.9 threshold: 88 in Health and Medical, 49 in Auto Repair. Once fraud signals and AI scoring were layered on, 91% of those pairs scored above 0.60 on the composite meta-score, and 27 pairs crossed the high-confidence threshold of 0.70.

Total reviews analyzed 35,156 five-star reviews

Flagged pairs (similarity > 0.9) 137 pairs (88 Health, 49 Auto)

Scored above 0.60 meta-score 91% of flagged pairs

High-confidence pairs (> 0.70) 27 pairs

Most common fraud signal Low review count (throwaway accounts)

Meta-score range 0.49 to 0.85, mean 0.65

The Examples That Stuck With Us

Numbers tell part of the story. The actual review pairs tell the rest.

Score 0.81: Word-for-Word at The Joint Chiropractic

Same user. Same location. Two reviews posted 10 days apart. The text is completely identical down to the personal narrative about passing the bar exam, serving in the Air Force as an attorney, specific doctor names, and promotional hashtags. Not similar. Not paraphrased. Identical.

"You're not just a patient when you come to the Joint Chiropractic in Brandon, you become family!"

Both reviews. Same sentence. Same hashtags. The only explanation is deliberate re-posting to keep the review visible in Yelp's ranking algorithm.

Score 0.78: Template Editing Across Dental Practices in Indiana

Different users, different dental practices, about a month apart. The narrative structure is nearly identical: first-time patient, consultation turning into same-day wisdom tooth extraction, the same opening phrase 'first I'd like to start off by saying,' the same emotional arc, even the same punchline about it being a relaxing experience despite having teeth pulled. Only the doctor names and practice names were swapped out.

This is the pattern that keyword filters miss entirely. The vocabulary is different enough to pass. The meaning is a copy.

Score 0.78: One Word Changed Across 200+

Same user, same auto shop in Arizona, three weeks apart. The entire review is identical across roughly 200 words. The only change: one version says 'Certified ASE master automotive mechanic' and the other drops the word 'master.' That's it. Same Honda CR-V story, same $1,300 quote from competitors, same $160 final bill, same conclusion. This looks like someone affiliated with the business reposting the same review to reset its freshness in the algorithm.

Why Layering Signals Matters

One fraud signal on its own doesn't mean much. A low review count could just be a new user. Temporal proximity could be coincidence. But when multiple signals converge on the same pair, the picture gets clearer fast.

The most interesting technical finding was around the correlation structure of the four similarity measures. Within each method, the two distance metrics (cosine and Euclidean for embeddings, cosine and Jaccard for TF-IDF) are highly correlated at 0.9 or above, so you only really need one from each method. But between methods, correlation drops to 0.66 to 0.73. Embeddings catch semantic paraphrases that TF-IDF downweights. TF-IDF catches exact word reuse that embeddings treat as normal variation. The gap between 0.70 and 1.00 is where each method finds what the other misses.

What This Is and What It Is Not

This is a graduate course project built for CIS 509 at ASU. The findings are exploratory. Flagged pairs are suspicious, not confirmed fraud. Any real-world deployment would need human review, proper legal framing, and a lot more validation before anyone acts on results.

That said, the patterns are hard to explain away. Copy-pasted reviews with detailed personal narratives, template editing at scale across businesses, incentivized re-posting with referral codes baked into the text. These are not noise. Something real is happening in the long tail of five-star reviews, and simple filters are not finding it.

What We Would Do Differently

The biggest gap is that the system only sees what crosses the 0.9 threshold. Fraudulent reviews that are sufficiently varied to stay below that line are invisible. A future version would benefit from semi-supervised learning trained on confirmed fraud cases, which would let the threshold relax without drowning in false positives.

We would also want to analyze at the business level rather than just review pairs. A business where 15 of 20 reviewers have zero friends and half the reviews contain marketing language tells a different story than a business with one suspicious pair.

The full code, notebooks, and results export are available on GitHub: https://github.com/allllc/FakeYelpReviews

Here is the presentation