📚 Rethinking ‘Big Data’ — and the rift between business and data ops

When it broke onto the IT scene, Big Data was a big deal. Now, you’d be hard-pressed to find many IT leaders willing to be caught using the term anymore.

Still, CIOs should not be too quick to consign the technologies and techniques touted during the honeymoon period (circa 2005-2015) of the Big Data Era to the dust bin of history. There remains an enormous amount of value to be harvested from basic data blocking and tackling.

Part of the issue is that futurists, who have been justly accused of being kind of “listy,” can accelerate the trendiness of trends, with penchant for the dramatic over the practical.

For example, we love enumerating things that have become obsolete — BlackBerries, cassette tape players, DVDs, floppy disks, overhead projectors, paper phone books, you name it. And being right early in pronouncing demise brings even more cachet.

We also love titillating audiences by listing jobs, roles, and tasks that have lost relevance — telephone switchboard operators, bowling pin setters, video store checkout clerks, milkmen, elevator operators, anything so long as it seems to be going away. Relevance, after all, is our forte.

On the flipside, we love exciting executives with hot new technologies. AI is having its moment for us, but so too can quantum computing and even drones.

But what futurists in particular, and all executives in general, should really be doing is working the great mystery of the modern era: how do we create stakeholder value.

Data is the cement that paves the AI value road. “Big Data” is a critical area that runs the risk of being miscategorized as being either irrelevant — a thing of the past or lacking a worth-the-trouble upside. CIOs and digital executives need to ensure that Big Data is getting the attention it deserves.

Rethinking our thoughts on data

When confronting a long-lived substantive problem — for example, why aren’t we getting full value from data — that is not immediately life threatening, I suggest conducting a mental model audit. How are we thinking about this problem?

I asked a group of senior executives to think about how they think about data. I asked whether they would classify themselves as Swallows, with big ideas and soaring ambitions; Hedgehogs, execs who view the world through the lens of a single defining idea; or Moles, focused on short-term projects with specified deliverables.

I was not surprised to learn that very few execs self-identified as Hedgehogs. This is probably because a universally accepted and deployed framework for creating value with data has not emerged. The consensus was that value was most frequently harvested when stretch data ambitions were clearly articulated with linkages to doable-projects with achievable milestones.

Evidently there is value associated with merely inserting data and analytic ambitions into the multiple strategy-making processes at work in any given enterprise. At the Gartner Data and Analytics Summit 2024, VP Analyst Ehtisham Zaidi shared, “Using frequency counts in earnings calls, companies in the global S&P 1200 that talked about data, analytics, and AI being strategic outperformed their peers 80% of the time in the last nine years.”

When I asked a group of seasoned data practitioners — data scientists and statistics Phd’s — to self-identify using the Swallows, Hedgehogs, and Moles schema, there were not so many Swallows, a few more Moles with most seeing themselves as little “h” Hedgehogs. It appears that the muggles (business execs) and wizards (data professionals) see the world through different eyes. We need to work on this.

Blending forces

Just as data scientists need to think more like businesspeople, so too must businesspeople think more like data scientists. This goes to the issue of occupational identity. Executives need to expand their professional identities to include data. Data professionals need to recognize that DI (changes in information) do not necessarily equate to DB (changes in behavior). Going forward data professionals are not just in the information/insight delivery business, they are in the “create insight that drives value creating behavior” business.

The portfolio of tools available now have democratized the practice of data science. One no longer needs to be a math genius or coding phenom to extract value from data — see Becoming a Data Head: How to Think, Speak, and Understand Data Science, Statistics, and Machine Learning by Alex J. Gutman, Jordan Goldmeier.

But many execs suffer from “data defeatism,” erroneously thinking that data value is dependent on having degrees in math, statistics, or machine learning. Executives need ready access to data professionals to guide their use of data power tools. Data professionals need to be embedded in the business rather than quarantined in specialized data gulags.

Executives spend too much time overthinking sibylline distinctions between sic “traditional analytics,” Big Data, and now artificial intelligence. You can’t do Big Data without traditional analytics, and you can’t do AI without Big Data. Data is data. Stop splitting hairs. Get out there and create value with data.