Quantum of Luck

I was surprised recently at hearing an interview with Olivier Giroud from Arsenal Football Club about a goal he scored New Year’s day against Crystal Palace. Giroud’s goal was an immediate candidate for Premier League goal of the year. Making a run inside the penalty box, Giroud received a waist-high cross behind his body. Reaching backwards with his left heal, Giroud deflected the ball forward and into the upper corner of the goal (a bodily contortion often called a “scorpion kick”).

After the match a reporter asked Giroud to describe the shot; Giroud replied that he “had maximum luck.” Bewildered by the candid acknowledgement, the reporter replied “luck?” Giroud affirmed “yeah, yeah, it’s just about luck.”

The interview had me wondering how much of Giroud’s success could be attributed to luck and how much credit should he take for himself? Would I be as “lucky” to make the same shot? I also wondered if it’s possible to quantify the amount of luck influencing any result.

I began with the English Oxford dictionary definition of luck: “success or failure apparently brought by chance rather than through one’s own actions.” The dictionary continued with a suitable example: “it was just luck that the first kick went in.”

This definition captures the role of chance but I wasn’t satisfied with the lack of quantitative guidance. Perhaps Giroud and I both need luck to make the shot but how could we need the same amount? I might attempt the shot 100 times and not score once. So I looked to probability theory and after some deliberation came up with the following definition:

“The amount of luck contributing to a result is the difference between the expected result and the actual result.”

For example, if the odds of winning a lottery are 1 in 100 and I win with a single ticket, then luck accounted for the difference between the number of tickets needed to expect a win (51 tickets) and the number of tickets I bought (1 ticket). Buying one ticket was the sum contribution of my efforts and luck had to account for the additional 50 tickets I should have had to buy. Whether the quantity of luck is “good” or “bad” is subjective but my hunch is that most people like to win the lottery.

I  put my definition to the test in the following ways:

First, should we consider it lucky when an expected result happens? If there’s a 70% chance of sunshine on my wedding day, would I be lucky to enjoy a sunny wedding? I don’t believe we should call an expected result “lucky” no matter how strongly we feel about the outcome. We could, on the other hand, call it unlucky if unexpected rain were to fall on the wedding. It may seem fortunate when the sun shines as expected, but I wouldn’t consider it luck. You simply appreciate when the expected happens.

What about my quantification of luck? Is luck proportional to the probability of outcomes? I think it’s correct to say that people are luckier at winning when the probabilities against them are greater. You need more luck to overcome 5000:1 odds than 2:1 odds. And at the craps table it takes the same amount of effort to roll the dice but you’ll need more luck to roll a 12 than a 7. I find probabilities a compelling way to quantify luck.

So what about Oliver Giroud? How “lucky” was he to make the shot? His result is harder to quantify than dice rolls and lotteries, but it’s not a shot that professional players make every day. In my case, luck may not be enough to score the goal. It would take something more like…divine intervention.

Three Common Pitfalls of Data Measurement

Comparison is an integral part of data analysis. Effective marketers compare changes to their data over time (i.e. how campaigns progress or customer sentiment evolves) and they conduct side-by-side comparisons of similar data elements (such as conversion rates for one offer versus another). But comparative analysis is only effective when you understand the data you are comparing and the relationships among different data elements. Every data-driven marketer should consider a couple of questions when conducting data analysis: “what, exactly, is my data measuring?” and “am I making valid comparisons?”

Previously I talked about the importance of knowing the relationship between your data and the real-world things your data describes. In this post I’m building on the concept of knowing what your data actually measures to help you make better decisions. The same principles apply whether you are a marketing practitioner using data to improve a specific function or a data scientist responsible for transforming data on behalf of an entire organization.

Each piece of data in your database represents a real-world concept. Oftentimes, these concepts are not well defined and their meaning is implied only through basic naming conventions. For example, it’s easy to assume that a “visitor” to your website is a living, breathing human being; but even that simple assumption is often wrong. Layered on top of the conceptual disparities are differences in measurement tools used to capture data in the first place — differences that are usually invisible to the end data consumer. Measurement technology is ultimately what defines the boundary between what is counted as a particular data element and what is not. For example, you could define a “video impression” as the press of a play button, the delivery of a video asset, initiation of video rendering in a player, or an arbitrary milestone through which a video asset has played. Each of these definitions yields a different outcome in the data that impacts your analysis and conclusions.

Establishing clear definitions for your data should be the primary pursuit, but given that much of the data you’ll work with is ambiguous, here are a few common pitfalls to avoid. These mistakes will lead to bad comparisons and poor decision making from your data.

You’re treating different things as if they were the same 

This may be the most common mistake when it comes to using data. Too often marketers make poor assumptions about what their data really represents. A good example is the treatment of browser cookies as individual consumers — a common marketing practice for years. This technique met with satisfaction despite inaccuracies because better solutions were hard to come by. Fortunately, better measurement solutions are emerging as personal consumer devices proliferate and marketers provide stronger incentives for consumers to share identity information.

Digital ad impressions are another example of ill-defined data elements. The “impression” label is misleading since ad impressions have historically been counted whether an ad is visible to a human subject or not. Better data is emerging in the form of “viewable” ad metrics to account for the considerable differences between viewable and non-viewable impressions, but unless your data consistently distinguishes the two your analyses may be skewed. The industry is rallying around standard definitions of viewability for different ad types but marketers should remain vigilant to the adherence of these standards by different data and technology vendors.

Marketers should also be mindful of differences in measurement granularity for the same or similar data elements. For example, adding weekly unique visitors throughout a month will not produce a count of monthly unique visitors. They’re not the same thing. You also cannot presume that your customer on ‘Digital Drive’ is a certain age and gender because your demographic-by-zip-code database suggests specific values given the address. Granularity plays a principal role in the application of time and location data but also applies to other data types.

You’re measuring the same things in different ways

There’s a difference between measuring the same thing and measuring the same way. Even if all your marketing technology vendors use the same definition for a data element in concept, each one will produce varying results due to differences in measurement capabilities. Differences in data collection code or technology, location and availability of data collection centers, and compatibility with consumer devices all play a role in determining how complete and accurate the data will be.

Aside from differences in measurement technology, it’s also important to understand when you’re measuring different events. In my previous video impression example, each of these four moments is a different event that should be understood separately. Variances in these metrics might be more valuable for technical diagnostics than for marketing analysis, but it’s still important when combining data across sources that you’re comparing the same events. Even events that occur milliseconds apart can produce sizable variances. IAB guidelines for Interactive Advertising treat a 10-percent variance between advertiser and publisher metrics as acceptable for measurement.

You’re analyzing incomplete or overlapping data

If you’re pulling data together across many sources the likelihood of having incomplete, orphaned, overlapping and redundant data is high. Before making comparisons ask questions like, “Does my campaign data include results from every DSP and media source used in my campaign?” and “Is my cost data aligned against the right conversion revenue or is there a mismatch?” Also, if you’re filtering results using metadata you might be missing critical data for your analysis — if certain metadata is not one 100-percent available or consistent. Look carefully the next time you filter products on a retail site like Amazon and compare the number of results available for a given metadata category and the overall set of product results. When metadata is not available for every item you may be removing instances you assumed were part of your analysis.

Hierarchies and classifications appear throughout most complex databases. These structures can improve your understanding of data and even help you make better comparisons in your analyses. However, hierarchies and classifications can also cause confusion when misunderstood or misapplied. Suppose your company reports sales figures for its western sales territory. You might compare your company’s performance against general economic data for the ‘US West’ region; but if your source of economic data uses a different combination of states in its definition of “US West” you might draw all the wrong conclusions. You should always understand whether your data classifications are mutually exclusive or overlap in some areas, and whether the classifications used by different data sources are the same.

Poor data hygiene is another cause of data mismatch. If your company follows a standard convention for object names (like campaigns) but practitioners don’t follow those standards when creating campaigns it’s easy to lose some of your results within the large expanse of enterprise data. Your diligence in properly naming and classifying data often makes the difference between working with complete or incomplete data for your analyses.

There’s a lot to think about when it comes to using data effectively. One simple piece of advice that will always help: seek to really understand what your data represents.

Where Data Meets Reality

Data can seem extraordinarily complex. It looks cryptic and often comes in large quantities. It can be organized many different ways and is misleading when used incorrectly. These challenges are compounded when you work with data that lies outside your domain of expertise. However, data also has common attributes that can enhance your understanding of new or unfamiliar data types. Whether you produce or consume data, learning the following characteristics will improve your ability to work with data.

1. Data describes the physical world

Data of all kinds describes the physical world. It may appear unrecognizable or define unfamiliar things, but data always describes real times, places, people, and events. Your first question when faced with complex or unfamiliar data should be “what real-world things is the data describing?” Sometimes this requires you to learn new aspects of your business—including new processes, technologies, and solutions. Once you understand these real-world activities, it is much easier to analyze the data in meaningful ways.

2. Data is symbolic

The ability to use symbols to represent real-world things is a characteristic that distinguishes humans. This is especially true with data, which is a symbolic representation of the world around us. Data is similar to language in its ability to communicate; like a narrative form of history, data also describes “what happened” using words, numbers, images, and other symbols. The more ambiguous the symbols are, the more likely you are to misinterpret what the data really means.

3. Data is shorthand

Data can represent complex things like people in brief format. For example, the name “Abraham Lincoln” is probably enough to tell you I’m referring to the 16th President of the United States. Add a home address or birthdate and you can also refer to less famous people. The same can be said for descriptions of people; it takes a lot of information to describe someone’s behavior over the past 10 years, but it’s possible with data to capture a meaningful portion of that history with a simple attribute like “trustworthy.”

Data is valuable for its ability to communicate facts concisely. This is especially true when describing things that happen repetitively or in large quantities. A baseball box score is a good example of how to briefly summarize essential facts from a lengthy game. Using the same format allows people to quickly digest highlights from any game, whether it was played yesterday or a century ago.

4. Data summarizes more detailed information about the world, but can still capture the essential facts

Information is lost when reducing complex, real-world things into representative data. For example, the term “iPhone 6” may be all that is needed to communicate an important fact, like how someone accessed my mobile application. But by itself, this word is missing a lot of information about the phone. It doesn’t tell me what individual parts the phone is made of, what condition it is in, or anything about the phone’s history. Each word that is added to the phone’s description contributes more information, but each one is itself a reduction of more complex information.

If you’re feeling adventurous, take this idea to an extreme and imagine what a “loss-less” description of the world would look like using data—one in which all available information is captured. A good example is the way Star Trek characters are beamed from the Starship Enterprise to remote locations. All of the very granular information about each person must be collected, organized, and transferred in order to reconstruct the characters somewhere else. This would require an enormous amount of data whereas calling one of them “Spock” is enough detail for most purposes.

5. Data is art as much as science.

There are many ways to describe the world using data, and there’s no right or wrong way. However, some ways are better than others. An important thing to consider is how each data element maps to its real-world equivalent. Objects, categories, hierarchies, and relationships that do not associate data with their worldly counterparts can add confusion. For example, mistaking individuals for web browser cookies will lead to poor decision making downstream. So can broad definitions of events like “ad impressions” when the data contains a variety of these events measured in different ways, such as “viewable impressions,” “eligible impressions” and “measurable impressions.”

Another common mistake is confusing similar but different events. For example, ad clicks may be measured by an advertiser’s ad server when it delivers an ad, and again by tracking code on the advertiser’s landing page. Each of these events is measured at a different time and place, and by different technologies. The two events often differ by a considerable margin and could lead to different conclusions.

Understanding these five data characteristics—and the relationship between data and the real world—will help you avoid these and many other mistakes when working with data. And knowing what the data really means is an essential precursor to drawing accurate conclusions.

Customer Data Strategies for Marketers

Data is driving marketing effectiveness to new heights. Using data, sophisticated marketers are breaking acquisition funnels into smaller, more personalized paths that convert prospects into customers at increasing rates. Marketers are speaking with customers more frequently at an individual level, even through mediums that historically accommodated only one-to-many communications. These conversations are driven by a wealth of customer data. But what is “customer data” and how should marketers use it?

A good way to think about customer data is to separate it into two categories. The first includes the individual actions or behaviors you observe in your interactions with the customer, or that you acquire through third parties. These historical “events” are descriptions of what the customer was doing at a specific moment and usually involve an interaction with your promotional messages and branded experiences. Examples of customer event data include viewing a television ad, clicking a product description in your mobile app, engaging with one of your sales associates, or subscribing to your services.

The second category emerges from the first and includes customer characteristics, preferences, and motivations summarized from past events. These customer “attributes” include psychographic descriptions of what the customer thinks or feels, including his or her interests and motivations; demographic attributes such as age, gender, and income; and summarized events such as lifetime value and loyalty status. These attributes do not occur at a single moment, but are interpretations of historical events that define important characteristics of the customer. Attributes are created from one or more events.

Understanding the relationship between these two customer data categories helps marketers establish a sound data strategy. A few things to keep in mind:

• Customer attributes are interpretive and continually evolve, whereas events are historical records that never change.
• Collecting a comprehensive record of customer interactions throughout the acquisition, conversion, and retention life cycles affords the greatest flexibility for future marketing optimization. These events are valuable information you should continually mine to drive business performance.
• Customer attributes are generally sufficient for indirect customer interactions. The quantity of events occurring outside your branded interactions is enormous and most of this data has limited value to your business.
• Your effectiveness in communicating with customers depends on how well you interpret the context and meaning of historical events and turn them into actionable data. Processes that drive customer experiences should update in real time whenever possible, and algorithms should be continually evaluated and improved.