Why your OOH brand lift results are statistically flawed

This approach transforms OOH from a medium measured by impressions and reach
into one evaluated by tangible business outcomes

Why your OOH brand lift results are statistically flawed

By Shawn Spooner, Global Chief Technology Officer, billups

Out-of-home measurement faces a fundamental challenge: distinguishing between ad effectiveness and misleading statistical artifacts. While your campaign may report a significant lift in store visits, spatial correlation could be silently skewing those results. Traditional methodologies that match control groups solely on demographics overlook a crucial factor – proximity to the advertised location naturally affects visit probability, regardless of ad exposure. This blind spot can lead to misleading performance metrics, making it difficult to accurately evaluate campaign impact or compare results across measurement partners.

The problem: When correlation doesn’t equal causation in OOH

There is a phenomenon in statistics known as spatial autocorrelation, which essentially measures the connection between a point in space and the outcome we want to measure. For example, if a window insert is placed on the door of a coffee chain location, seeing the ad could be highly connected to visiting the store, as you see it as you walk in and cannot easily avoid exposure. But can we say that exposure to the window ad caused the visit?

In most cases, we cannot confidently claim this, as the person was already walking into the store before they were exposed to the advertisement. Let’s expand on this idea and imagine moving the same advertisement onto a billboard a mile away. At this point, we can measure how connected the billboard is to visiting the coffee chain through its correlation coefficient (a statistical measure between -1 and 1 that indicates how strongly two variables are related).

This is a great start, but we must factor in another key component of targeting OOH: the audience. We choose an audience based on the belief that they are more likely to respond to the advertisement. One way to do this is to pair people who saw the ad to people who did not, by matching them on their demographic profile. This appears to be the logical thing to do; matches should be random, and the difference between the two groups should result in the lift.

The hidden statistical trap

In actuality, this method leaves much to be desired. The first big challenge is that even though it matches the demographics of the exposed device, the control device may be owned by someone very far away from the coffee chain, and thus they are far less likely to visit even if they had been exposed. This mismatch is one result of spatial correlation at work. One can take advantage of this fact and create results that are far from accurate, and that’s why our testing created thousands of synthetic OOH campaigns, all attempting to drive visitors to the hypothetical coffee retailer.

We proved that you can create a lift of about 100X just by carefully selecting units that are poor matches spatially but are perfect matches demographically, even when there is zero campaign effect. This type of manipulation is called P-hacking in traditional statistics (the practice of manipulating data analysis to find patterns that appear statistically significant but aren’t) and has resulted in many studies that reached invalid conclusions or in the worst case intentionally mislead.

What this means when comparing studies is that any attribution supplier who does not account for spatial correlation could be overstating the lift of an out-of-home campaign, in the worst case, by about 100 times. Spatial correlation is a double-edged sword, though: it is also possible to eliminate actual lift by picking control devices that are more spatially entangled than their treated counterparts.

A solution: unit-based control pools

There has been an important development in OOH measurement solutions, something we have been calling unit-based control pools.

The key idea is that when we are picking our campaign units, we measure how correlated they are with visits to the coffee chain locations. We also measure the audiences exposed and their probability to visit. At this point, the campaign units are not running any coffee-related messaging, and the exposure should not change the viewer’s behavior. We use this fact to find control units that match the spatial correlation and the audience composition, and we ensure that the visitation rates are the same across both groups.

Next, once the campaign starts, we look at the first time each device saw the campaign, and we pair them with a device that saw the control OOH at the same time. This subtle tweak ensures that each pairing between exposed and control has a highly comparable spatial correlation, base visitation probability, and audience breakdown.

Finally, we test this methodology both on synthetic data, where we show that it correctly recovers actual lift 98% of the time (the true lift was in the 98% highest posterior density interval, or HPDI), and on our past studies. We must show that in placebo campaigns where no intervention took place, there is no lift. Our results show a 0% lift in all placebo cases.

Precise measurement creates opportunities for meaningful optimization. By implementing unit-based control pools that account for both spatial and demographic factors, marketers can gain clearer insights into which placements truly drive consumer behavior. This approach transforms OOH from a medium measured by impressions and reach into one evaluated by tangible business outcomes. As the industry evolves, those who adopt scientifically rigorous measurement will stand out, armed with data that tells the true story of their campaign’s effectiveness.

Help us Help You by keeping the only Independent Voice for OOH.
If you read OOH Today on LinkedIn, thank you. To ensure you never miss a post
and stay unconstrained speech for the OOH/Billboard Industry,
click here to ⇒Subscribe. 

- Advertisement -