Nonresponse Bias Strategy: Planning for the People Who Won't Answer

Planning for nonresponse is the least glamorous part of survey design and the part that decides whether any of the rest was worth doing. It is also, I discovered, the part where fieldwork pragmatism can walk straight into your research question without anybody noticing.

Here is what I submitted, and the four things I’d correct in it.

The four strategies

The denominator item in strategy three is a genuinely good catch. The survey already asked how many deployments failed, and without a total you cannot compute a rate, so the benchmark comparison would have been impossible. Finding that gap before fielding is the whole point of writing the plan.

Then there is everything else.

The sentence that should have stopped me

Under responsive field design, describing what to do if operations engineers lag behind developers, I wrote:

Leveraging positional power and sending customized reminder emails tailored to the specific concerns of the lagging group can increase their cooperation.

Read that against the study’s own hypothesis.

An engineer who answers because a manager leaned on them is not in the same psychological position as one who answered unprompted. If the construct is how safely can you disagree with power, and my recruitment method is power asks you to participate, I have contaminated the measurement with the exact variable under study — and contaminated it worst in the subgroup whose silence is the most informative data I have.

Worse: their non-response was a finding. Operations engineers, disproportionately offshore, declining to answer a survey about whether it’s safe to speak up is not a fieldwork problem to be solved. It’s a partial answer to the research question. Converting it into responses destroys the signal and replaces it with a compliant one.

I wrote that line because I was thinking like someone trying to hit a target rate. It reads as ordinary survey pragmatism. It is ordinary survey pragmatism — in a study where it happens to be fatal.

The second problem: forty people and a tier number

The other new item asks respondents for their engineering tier, on a seven-point scale from Software Developer 1 to Principal Developer 7, so it can be matched to the roster for weighting.

Methodologically that’s correct — categories must match auxiliary data exactly or the comparison doesn’t work. But do the arithmetic on the population.

And the respondents will work this out. These are engineers; they know the org chart and they can count. A survey asking do you feel safe admitting doubt while collecting three fields that jointly identify them is going to get careful answers, and the carefulness will look like data.

So the item added to measure nonresponse bias is a plausible cause of it. The fix looks simple — collapse seven tiers into two or three bands, drop location or role, accept coarser weights.

It isn’t simple, and the reason is the strategy sitting above it.

Why the tier item was there at all

Strategy two says the roster is rich enough to compare responders against nonresponders on role, seniority and location. The submitted sentence claims this works without relying on the survey instrument itself, and that is exactly what makes it attractive: the same variable, measured the same way, for the people who answered and the people who didn’t.

That sentence requires knowing who answered.

So the tier item is not a lapse in judgment sitting next to strategy two. It is the workaround strategy two needs. If I can’t link individuals, I can still compare the distribution of respondents against the known distribution of the roster — but only for variables the questionnaire itself collects. The item exists because the survey is anonymous.

Which makes strategy two weaker than the way I wrote it. A distribution comparison tells me which tiers are underrepresented, and that is worth having. What it cannot do is the thing I claimed for it: separate nonresponse bias from measurement error. The moment the variable arrives through the instrument, it can be mistyped, rounded up, or quietly inflated by someone who has been waiting eighteen months for the promotion. Frame data’s whole advantage is that it doesn’t pass through the respondent. Mine does.

It also changes the price of banding the tiers. That doesn’t just cost precision in the weights; it costs resolution in the only nonresponse detection I have left. I’d still band them — a census of forty with any realistic response rate leaves a respondent pool in the teens or twenties, and weights built across seven tiers on that many people are mostly noise with a decimal point. At this size the honest instrument is a description of who is missing, not an adjustment that behaves as though they aren’t.

The one I’d defend, and how far

I’ve been hard on the administrative benchmark elsewhere, and I want to be more precise than I was.

The circularity objection is real if you treat the comparison as a general validity check: the study’s premise is that official logs understate risk, so the logs can’t certify the survey. Agreed.

But that isn’t quite what strategy one does. Comparing the survey’s aggregate self-reported failure rate against the logged rate aims at one specific thing — whether people who experienced failures were less likely to respond. If the survey aggregate comes in below the official rate, the people who had failures look to be missing from my sample: a differential response propensity on the dependent variable, which is exactly what a nonresponse strategy should detect.

That much holds. Here is where it stops.

A survey aggregate below the logged rate has a second explanation, and my own study supplies it. The engineers with failures may have answered and understated them. That is not an exotic possibility I’m raising for completeness — it is the definition of my dependent variable. I wrote that true delivery stability is official change failures plus the near-misses people resolve without logging anything. I built an entire variable on the claim that engineers under-report technical trouble, and then designed a bias check that only returns a clean reading if they don’t.

The name for it is social desirability, and I had already used the term — a few assignments earlier, worrying about the group interviews. I listed mode of participation as a control specifically to catch it there, and treated the anonymous questionnaire as the high-privacy end, the place where people would finally tell me the truth. So I named the bias, built a control for it in one phase, and then quietly assumed it away in the other.

Two failure modes, one number, and no way to tell them apart without individual linkage — which, per the section above, I don’t have.

What the comparison also cannot do is tell me whether the logs themselves undercount, which is the study’s whole thesis. Three questions, one number, and I ran them together in my head.

What I took from it

The general pattern, and the reason this one bothers me more than an ordinary mistake:

A method choice made for operational reasons can be a treatment applied to your subjects. Reminder emails, who sends them, when, in what tone, to whom — I filed all of that under logistics. In a study about power and candour, the logistics are the experiment. There is no neutral way to contact people about how safe they feel; the contact itself is a data point about how safe they are.

The second pattern is duller and probably more common. A promise I made in week four constrained a method I chose in week thirteen, and I never went back to check. Each document was defensible on its own; the study was only incoherent when you read them in sequence, which nobody does, including the person who wrote them.

I’d add two checks to any future plan. For each recruitment and follow-up tactic, ask whether it would move the dependent variable — if it would, it isn’t fieldwork, it’s an uncontrolled manipulation, and it has to be designed in advance rather than improvised in week two when a subgroup goes quiet. And before adding any analysis strategy, reread what the respondent was promised, because the promise is a constraint on the method and it was made first.

Thanks to a course that made nonresponse its own assignment instead of a paragraph at the end of the methods section — it’s the only reason I have a document to reread.

If you’ve run an internal survey: how many identifying fields did you collect, and how big was the population? Multiply them and see whether your promise of anonymity survives.