The Walls Were Doing the Counting

Every organisation that says it takes learning seriously eventually publishes a number for it. Hours per employee per year. Completion rates. Percentage of time invested in development. The number gets into a deck, the deck gets into a strategy, and nobody asks where the number came from.

I have produced one of those numbers. It took me about three seconds.

The question I answered from feel

During a session on employee development the class ran four live polls. Three asked for opinions — what inspires you, whether your organisation’s development spend is aligned to its objectives, how adequate your learning resources are. Opinions are the right unit for those. Your judgment is the instrument, and there is nothing wrong with that.

The second poll asked how many hours a week I use for learning. That one has a unit. It is the sort of claim that could be checked against a record.

I answered it in the same three seconds I gave the other three, from a general sense of myself as someone who learns quite a lot. There was nothing to check it against, because I have never kept a record of it, and I have never kept a record of it because I have never needed to produce the number for anyone who could audit it.

Then the bar chart filled in on screen, and everyone’s guesses looked exactly like data.

I assumed this was a personal gap. Then I read the chapter, and found the whole discipline in the same position.

What the room was providing for free

Classroom training is usually described as a delivery format that technology has improved on. Bound by time and place, as the week’s framing put it, versus available on demand and around the clock.

That is true. It is also, read as a measurement statement, a list of everything that has been removed.

Nobody in the history of corporate training built those five things in order to evaluate anything. They fell out of the physical problem of getting people into one place at one time. The measurement rode along on the logistics for free, for about a century, and the field mistook that for having solved the problem.

The fifth one needs a correction, because I had it wrong when I started writing this. Everyone not in the room is not a control group. A control group is a set of people equivalent to the treated set in every respect except the treatment, and the people left out of a training cohort are nothing of the kind: managers send their high performers first, or their problem cases first, and either way the exclusion is the opposite of random. What the room supplied was an untreated population that was visible. Not a valid comparison — an available one, which is a far lower bar, and which turns out to be the bar that decides whether anyone attempts the analysis at all.

That distinction is the difference between a bad experiment and no experiment. The field has moved from the first to the second, and the reported figure did not change on either occasion.

Four levels, all pointing at the same object

The chapter’s evaluation section opens with a warning: after $560 billion of training spend in one recent year, two experts contend that employers are getting a poor return, because it “doesn’t lead to better organizational performance.” So the standard tools get laid out. The widely used one is the Kirkpatrick model, four effects an employer can measure.

Read them for what each one requires to exist.

Every level takes the program as its unit. Not the person, not the skill, not the week — the program. And the chapter is explicit that the gold standard above all four is a controlled experiment: a training group and a control group, measured before and after, so you can tell whether the improvement came from the training or from the pay rise everybody got in the same quarter.

When the thing being evaluated is a two-day course in a building, that comparison set is at least identifiable, even where it is badly chosen. When the thing is learning on demand, it does not exist in any form, because there is no treatment anyone could have been excluded from. Everyone is in the treatment group, permanently, at an unknown dose.

The obvious objection here is that the four levels are sixty years old and every practitioner knows their limits. That is fair, and the chapter does what the field has generally done about it: it adds a stage. After the four levels comes a careful comparison of the training program’s costs and benefits, so the team can compute the program’s return on investment. Read that sentence for its unit. The extension inherits the dependency exactly — it is a fifth question about the same object, and it needs that object more urgently than the first four do, because you cannot divide by a cost you cannot bound.

Read the good news again

Here is the sentence that opened the week: traditional training and development were often bound by time and place, but in a digital, knowledge-driven economy, opportunities for learning are available on demand and around the clock.

I read it the first time as a story about access, which it is. Read it a second time and it is also an inventory of what the field’s entire evaluation apparatus was resting on.

My own note from that week defines social learning as learning from other people — observing, asking questions, sharing experiences — the kind that happens naturally through conversations and working together. I wrote that down as a tidy summary. The load-bearing word turns out to be “naturally.” Naturally means unscheduled. Unscheduled means unrostered. Unrostered means, for every instrument in the chapter, invisible.

So where does the published number come from

There are only two places it can come from, and organisations use both, usually added together in the same chart.

This is the shape of measurement failure I keep running into: a quantity that is genuinely hard to observe, standing next to a quantity that is easy to observe, until the easy one quietly takes the hard one’s name. Attendance is not learning. It was never learning. It correlated with learning for as long as learning required attendance, and that era ended without the metric being retired.

Better instrumentation does not help with this, and it is worth being clear about why. Tracking every search and every click makes the easy quantity more precise. It does not turn it into the other quantity. A system that knows I opened nine documents knows nine things about my afternoon and nothing about my competence.

What would have to be true instead

I do not have a working instrument for this, and I want to be careful not to present a form I designed on a Tuesday as though it had produced findings. What I can state is what it would need, because the chapter makes the requirements legible even where it cannot meet them.

Forty minutes that changed how a whole test suite gets written beats twelve hours that changed nothing, and an hours metric ranks them the other way round. That is not a subtle failure. It is the metric preferring the wrong answer, every time, by construction.

What I would answer now

I still do not have a number, and I have stopped treating that as something to apologise for. “I don’t measure it” is checkable. A confident estimate is not, and a number is not a verdict just because it has a decimal point.

The thing I got wrong was smaller and more interesting than not keeping a log. I assumed that a measurement this old must be well founded — that an industry spending $560 billion a year would not be reporting a figure with nothing underneath it. But the field never had a measurement of learning. It had a measurement of attendance, which is a completely different quantity that happened to travel alongside learning for as long as learning required a room. When the room went away, the number stayed on the slide.

Thanks to a chapter that laid out its four evaluation levels plainly enough that you can see, in the wording of each one, the thing it cannot do without.

If your organisation publishes a learning figure: which of the two sources is it, and does the chart say so? I would be interested in how many can be traced back to either.