Goodbye Mr Big-Data

By Uncrowd CEO and Founder Richard Hammond. This article was originally published April 18th, 2019

The era of Big Data is already on the wane. I could pull out a few choice supporting evidences here but like so much analysis of that same data, I would be using powerful confirmation bias to attempt to activate the limbic brain’s desperate desire for the creation of order from the reality of chaos.

We shaved apes like to believe that the organisational rigor that took us to the moon, the civilisation that finally taught us that Dr King and Gandhi were right, the single-minded focus of Steve Jobs’ on black polo necks and Levis; that all these things are evidence of our upwards thrust away from chaos.

Big Data feels like it is part of that thrust; our digital ghosts, habitual contrails landing ephemerally on touchpoints eliciting particle effect splashes of data like a discordant tinkling of the ivories on an infinite concert grand.

For every basket analysis that discovers, as Dunn Humby famously did, that nappies and beer often appear in late night baskets because Dads are sometimes sent out to fetch supplies (and just how old fashioned does that insight feel now?) there is an infinitely long tail of marginally meaningful patterns. Those are still worth exploiting, effectively planning adjacencies, for example, is possibly the rarest of skills in all retail but having run such analysis once, it never again yields the big bang promise.

The story of Big Data often follows that same curve; explosive initial discovery (Stitch Fix and their data-derived customer feedback on garment performance is a great exemplar of Big Data used right), followed by a precipitous drop-off in discovery. That’s a natural curve, but I wonder if those much vaunted initial Big Data Big Bang events are so marketable and so attractive as ideas, that they lead retailers to make disproportionate investments in their own Big Data capabilities?

Big Data promises order, promises fluid streaming from the bouncing chaos; all the blue marbles into the blue hopper, the red ones into the red hopper and everything in place. Our world, however, is absolutely not structured that way. After the desperate and sad fire that killed Apollo I’s crew on the ground, investigators found a wrench socket pressed up in the capsule’s wiring. It would never have been discovered if the mission had succeeded. Another node of chaos in service of the illusion of organisation.

I’ve often heard pilots talk about the physicality of flying 747s versus the light-touch joystick control of Airbus alternatives “…it’s quite common for pilots to strongly prefer older aircraft […] in which fewer tasks are automated or computerized, many pilots feel closer to the simplest mechanics of flying” (source: Skyfaring: A Journey with a Pilot, Mark Vanhoenacker, 2015). Are we doing something similar in our use of data? Moving away from the base skills that lead to really understanding what customers think and do? ‘We have the numbers, so we must apply significance to them’ versus the older skill of asking the right questions.

We’ve built the Uncrowd platform to be capable of pulling insight from huge live datasets down to hand-written one-off notes scribbled by the Chairman’s partner. We have the big data capability but our experience of the outcomes from various data sources has thrown up one very big insight of our own: that practical, actionable retail improvement comes from weighting all data sources against each other, listening to the small data as well as being streamlined by the big.

It’s led us to an important conclusion; that the era of Big Data dominance must end, instead we should Right-Size Data based on asking better initial questions. The implications are that the importance of those huge datasets is downgraded versus the richness of human-scale descriptive data.