Human Beings and Little Data

The promise of big data seems to be that it does away with needing to understand causation and intentionality. We don't need to know why, only what.

In my experience this is true in a limited number of cases where the phenomenon under discussion is narrow, and there are no vested interests looking to push an agenda. So contra Peter Norvig, "All models are wrong, and increasingly you can succeed without them" but not in most of the domains where we're trying to make decisions with limited data, which include some important ones.

I'm not trying to setup or knock down a straw man here. I'm sure that Norvig, when pushed, will agree that theory, intentionality, causality, remains important and that the strong claim was more for rhetorical effect than an axiom. But I do want to point out a dangerous arrogance behind these statements, and this is beyond the conceit that you can build a perfect dashboard, being fed information, with you omniscient in the center controlling everything.

The arrogance I'm talking about is to believe that the world has nothing more to tell you. This generates a particularly awful sort of blindness because now, not only can the person not see, they also cannot get any better. Serious business.

Anyone who has seriously done this sort of work for any length of time will probably have realized that true learning comes not from accumulating facts, but from developing new ways to see. It's a cognitive flip that brings structure to facts so lets you learn them very quickly, but the initial comprehension may take an age to get to, but once it comes it's fast. With Big Data, you can only see what you have instrumented, and what you instrument is a function of your comprehension of the situation at that time. So not only are you limited in what you can see, you are similarly limited in how you can see.

Two related articles that made me think of this.

The first is from Jason Friend about all the "inefficiency" they've purposefully instrumented into their new product.

But automation can also lead to myopia. And premature-automation can lead to blindness. When you take human interaction out of a system, you’re removing key opportunities to see what really happens along the way. You miss stories, experiences, and struggles – and that’s often where the real insights are hiding...

So with Know Your Company we wanted to reset our assumptions and eliminate a lot of the automation we usually lean on. Rather than separate ourselves from the customer, we wanted to bump into the customer as often as we possibly could.

He lists a number of crazy inefficient processes, like insisting on live demos, manual, one-at-a-time record entry, etc. These aren't things to hang onto forever, but they will give them opportunities to see things in new ways.

The second is from Venkat Rao about how, when you're going through a change, it's the miscellaneous folder that explodes and how exception handling becomes the core organizational driver.

When the flow changes, the way we usually handle it is by consigning more of it to the miscellaneous folder. This follows obviously from the fact that the new elements are by definition the ones that are not comprehended by the existing organization scheme.

Depending on urgency and importance, we interrupt normal functioning to cannibalize resources to handle it there, using ad hoc schemes, on a case-by-case basis. This is an opportunistic process and we call people who are good at stealing resources this way resourceful.

You know you have a problem when the miscellaneous folder swells from handling 20% of the flow to 80%. At this point, your organization scheme is adding no value at all, since 20% of the scheme is handling 20% of the flow and 80% is handling 80%. There is no leverage. You might as well dismantle the scheme and let the anarchy of the miscellaneous folder reign everywhere.

Lived this one many times. Again, the miscellaneous folder is the right place to look for where the actual flow of actual work is going, as it's where the stuff which cannot be handled elsewhere in the organization for structural reasons ends up living.