Quantifying Human-Technical Stability: My Final Survey Deck & Its Weakest Slide

Every research design has one load-bearing assumption. Pull it out and the whole thing comes down. It is almost never the assumption you spend the most time defending — the load-bearing one usually sits in a bullet you wrote quickly, near the end, because by then you were tired.

Mine was on slide four.

The assignment

The course ran as ten progressive assignments, each building on the last, ending here:

This is a short presentation of no more than five minutes, using slides or visual aids of some sort.

We will schedule a live session for students to provide short overviews of their progressive survey designs.

Five minutes for ten weeks of work is a good constraint and a slightly cruel one. It forces you to decide what the study is, because there is no room to show what it also happens to contain.

What I was trying to measure

The setting: a distributed DevOps team that runs continuous deployment. The official health metric for that kind of team is change-failure rate — what proportion of deployments cause a problem.

My claim was that this number is subject to a specification error. When engineers suppress technical doubt to avoid blame, the metrics record declared readiness rather than actual risk. Nothing lies. The number just answers a question adjacent to the one you asked.

So the study proposed two constructs and a relationship:

• Techno-emotional authenticity — whether an engineer can say “I am not sure this is ready” out loud, on the record, before the deploy.
• Blame-free leadership inquiry style — whether the person asking questions after an incident is read as learning or as assigning fault.

The hypothesis: higher authenticity predicts fewer deployment failures, with the leadership inquiry style acting as the catalyst that makes the honesty possible.

The part that worked: cognitive testing

The single most useful thing I did in the whole ten weeks was sit with people while they read my questions aloud, and two things fell out of it.

I was using my vocabulary, not theirs. I had written items about “near-misses” — a phrase I like, from safety engineering. Nobody on the team used it. They said “deployment issues.” An item written in a word the respondent has to translate is measuring their translation, not their experience, so I changed it.

I nearly walked into the jingle fallacy. Two groups were using the same phrase to mean incompatible things. Leadership said root cause analysis and meant learning. Engineers heard root cause analysis and understood it as assigning blame. Same three words, two constructs. Had I left that on a single item, every response would have been a coin flip on which meaning the reader held — and the item would have quietly averaged them into a number that described neither group.

Splitting it into separate items is a small edit. It is also the entire reason the instrument has a chance of meaning anything, and I would not have found it by reading my own draft again.

The weakest slide

Slide four was Next Steps & Bias Strategy, and it is where I put the answer to the question every methods reviewer asks: how will you know your respondents told you the truth?

My answer, in the speaker notes: compare the self-reported deployment failures in the survey directly against the official internal change-management logs. If engineers who experience failures are hiding from the survey, the comparison will show it.

That sounds rigorous. It is circular.

The entire premise of the study is that the official logs understate risk, because engineers suppress doubt and the metrics record declared readiness. I built a research question on the claim that the administrative record is biased. Then I proposed using that same administrative record as the benchmark to validate my instrument.

Work through the two outcomes:

• The survey and the logs agree. I would report this as evidence the survey is accurate. But agreement is equally consistent with the survey inheriting the same suppression as the logs — which is exactly what the theory predicts should happen.
• The survey and the logs disagree. I would report this as evidence the logs understate risk, confirming the hypothesis.

Every result confirms me. There is no observation the comparison can produce that would tell me my instrument is wrong. That is not a bias strategy; it is a slide that looks like one.

Two more things I would fix

The census is too small for what I promised it. A finite population of about forty, with an expected response somewhere below that, against fifteen items and three constructs. The usual working rule for item analysis is something like five to ten respondents per item — call it seventy-five to a hundred and fifty. I will not have it. The same slide mentions propensity weighting as a possibility, which needs enough respondents to model a response propensity at all. On this population, that word is aspiration wearing a method’s clothes.

Phase one contradicts the thesis. The design opens with group interviews over video, to capture richness before the private questionnaire. But the study’s own claim is that people suppress doubt in front of colleagues when blame is in the air. A group call is the exact condition under which the thing I want to observe stops happening. And I am not a neutral researcher here — I work with these people. The right sequencing is private first, group second, and I had it backwards for a reason I recognise: group calls were easier to schedule.

What I actually got right, and why it isn’t enough

I want to be fair to the design, because the diagnosis in it is sound and I still believe it.

Engineers do bundle memory. The methods teaching that stuck hardest was that human recall is not a hard drive — we store information in bundles, so a stressful rollback is filed together with the late night and the panic. If you ask about the technical event in a context that reactivates the blame, you get the emotional bundle back, not the fact. Designing a private, unhurried instrument is a response to how memory actually works, not a courtesy.

That is a good reason to build the survey. It is not evidence the survey works. And the difference between those two sentences is the entire content of slide four, which is why it is the slide I should have spent the most time on and spent the least.

What I took from it

The pattern I keep finding in my own work: I am rigorous about the parts that are interesting and casual about the parts that are load-bearing. Cognitive testing was interesting, so I did it properly and it paid. The bias strategy was a box to fill before the deadline, so I filled it with the most available comparison — and the most available comparison was the very artifact my thesis had already declared untrustworthy.

There’s a cheap check for this, and I now run it: for each validity claim, write down the result that would embarrass you. If you can’t produce one, the claim is decorative. It took me thirty seconds to apply that to slide four, and I applied it after the presentation rather than before.

Thanks to the classmates who sat through their own five minutes and to whoever designed a course where the questionnaire gets tested on real readers rather than admired on a screen. If you’ve built an instrument recently: what’s the observation that would tell you it’s wrong? I’d like to hear one that isn’t circular.