Retailers May Be Reading Sales Numbers Wrong

You are not wrong to obsess over year-on-year and same-store sales. Those numbers have kept a lot of retailers alive. But if you’re using them to judge whether a specific marketing program, loyalty initiative, or “AI-powered” marketing push is working, they are quietly misleading you.

The problem: Good metrics used for the wrong question

YoY and same store sales, sometimes known as comparable / comp sales, are excellent at answering, “Are we broadly healthy as a business?” Unfortunately, they are terrible at answering, “Did this particular action cause a change in shopper behavior?”

Think about everything that moves those top-line numbers at once: inflation, SNAP changes, competitive openings and closures, gas prices, calendar quirks like where holidays fall, and even the weather. All of that is swirling through your reports before a single shopper opens your email or sees your SMS.

So when you run a campaign and then look at YoY or same-store sales for the period, you’re not measuring the campaign. You’re measuring the entire world plus your campaign, and then quietly giving the campaign the credit.

The illusion of “campaign lift”

Suppose you run a two-week loyalty push with sharper digital offers and a few personalized recommendations. At the end of the period, you see:

• Store sales: +4% vs last year

• Loyalty sales: +7% vs last year

It’s tempting to say, “The loyalty campaign drove a 7% lift.” But what if, in your market:

• Overall grocery spending is up 4% because of inflation and wage growth

• A nearby competitor is remodeling and temporarily losing traffic

• SNAP disbursement timing lined up favorably with your promo period

If you do not know what would have happened without the campaign, you cannot honestly claim those numbers as impact.

You’re not being deceptive; the industry has trained you to look at these numbers this way. But this is exactly what “reading the numbers wrongly” looks like.

The question you actually care about

When you invest in marketing, personalization, or a new loyalty mechanic, the real question is simpler and much more specific:

Did this action make this group of shoppers buy more than they otherwise would have?

That is an incrementality question, not a YoY sales question.

Incrementality testing tries to isolate the effect of an action by comparing two similar groups of shoppers who lived through the same conditions.

Incrementality testing tries to isolate the effect of the action by comparing two similar groups of shoppers who lived through the same conditions; but only one group was exposed to the program. Same macro trends, same holidays, same gas prices, same local competitors; the only meaningful difference is your treatment.

Now you’re no longer asking, “Were sales up?” but “Were sales more up for the shoppers who got the program than for the ones who didn’t?”

A simple mental model for incrementality

You do not need a data science team to grasp the core idea.

1. Pick a group of shoppers you care about (for example, lapsed high-value shoppers or your top 30% by spend - at Goodlight AI, we group shoppers into segments like Champions and Hibernating).

2. Randomly split them into two groups: treatment and control.

3. Run your campaign only to the treatment group.

4. After a defined window (say 7–14 days), compare the outcomes.

Now imagine the results:

• Control group: +4% vs last year

• Treatment group: +4% vs last year

Your overall numbers look healthy, but your campaign did not move that segment at all. The universe lifted both groups equally; your program added nothing extra.

Now a different run:

• Control group: +4% vs last year

• Treatment group: +7% vs last year

The universe still gave you 4%. The extra 3% in the treatment group is the incremental lift—the part you can reasonably attribute to the campaign.

This is the number you actually want when you ask, “Did it work?”