Letter to Reviewers: Answering Every Critique of My Scale in Writing

Somebody has taken apart something you made and handed you the pieces. Everyone knows the two bad responses — concede everything, or defend everything. Almost nobody talks about the third failure, which is the one I committed: respond to each point on its merits, honestly, one at a time, and end up with a pattern you never chose.

You can only see it by laying the answers side by side afterwards. So here they are.

The structure the letter set for itself

We opened by sorting our responses into two categories, which I still think was the right move:

Generally, our responses fall into two categories: 1) reviewer feedback which we have sought to implement into our measure, and 2) reviewer feedback which we felt needed more of an explanation to understand our intentions.

That framing does something useful and something slippery at once. Useful, because it forces you to say for each point whether you changed anything. Slippery, because “needed more of an explanation to understand our intentions” is a very comfortable way to say we disagree — it locates the problem in the reviewer’s comprehension rather than in our design.

I didn’t notice the tilt in that sentence until I sorted the replies.

Every response, sorted

The reviewers raised ten distinct points across their eight sections. Here is every one.

Every clean agreed is a wording or mechanics point. Every flat disagreed touches the construct or a design decision we’d argued about internally. The partials sit in between — four of the ten, and they are where this gets complicated rather than clean.

I did not plan that. Each reply was written on its own, and each is individually defensible. The sorting emerged.

Is the pattern wrong, though?

Here’s where I want to resist the tidy confession, because there is a real argument on the other side.

We defined the construct. We had spent eight steps on it. When a reviewer says an item is measuring social anxiety rather than catastrophizing, they are making a claim about our definition — and we are better placed than they are to say what we meant. Deferring on construct points would be its own failure: a measure that accommodates every reader’s intuition measures nothing.

And some of the disagreements are properly argued. On the double-barrelled charge for items 1 and 3, we said the sentences have multiple clauses but each connects a specific situation to a catastrophized outcome — which is what the construct requires, since catastrophizing is by definition a link between a trigger and a projected disaster. An item with no situation in it isn’t measuring the pattern. That’s a real answer, not a dodge. It isn’t a free pass either: if a respondent rejects the situation in the item, their answer about the catastrophe half is confounded, and we never addressed that.

On reverse scoring we did the work in public. We started with the position that items should all run the same direction to maximise unidimensionality, because reverse-coded items tend to have smaller discrimination and introduce method factors. Then we went further into the literature, weighed it, and landed on one reverse-scored item — while writing down that with a new measure and a small test population, reverse scoring can introduce bias and create reliability problems. The letter also says plainly that the group had differing opinions and resolved them after reading the feedback.

Saying “we disagreed among ourselves and here is how we settled it” is the most credible sentence in the document.

Five of the eight sections carried the actual weight:

The concession that mattered, and where it sat

Under construct validity, we conceded this: item 1 was perhaps more related to social anxiety than catastrophizing, given that it was narrowly focused on a social situation rather than a distortion of what would happen because of that social situation.

That is the most valuable line in the letter. It’s the only place we accepted that an item might be measuring the wrong thing — a construct-level concession, the expensive kind. And the distinction it draws is genuinely sharp: the item named a situation without naming the catastrophic inference, so it captured the trigger and missed the mechanism.

It arrives in section six of eight, in a paragraph that opens by defending the social dimension as intentional.

Placement isn’t nothing. The strongest self-criticism in the document is buried behind a defence, three-quarters of the way down, in a reply that has already established a rhythm of pushing back. A reader skimming for whether we took the review seriously would find four wording fixes and a lot of citations before reaching the one thing we got wrong that mattered.

I don’t think we hid it. I think we ordered the letter by the reviewers’ section numbers, which is the polite thing to do, and the polite ordering buried the lede.

Two smaller notes I’d keep

We found a defect they missed. While arguing that items 1 and 3 were not double-barrelled, we noticed they were too similar to each other and adjusted both. Redundancy is a real scale problem and it wasn’t in the review. Arguing carefully against a wrong criticism made us reread the items closely enough to find a right one — which is an underrated reason to take even the critiques you’ll reject seriously.

We cautioned against our own measure. On whether it could support assessments of depression or anxiety, we said it shouldn’t be used that way because it still needs validating, while acknowledging the likely overlap between catastrophizing and anxiety. Limiting the claims for something you built, in a document whose purpose is defending it, is the right instinct.

What I took from it

The lesson isn’t “concede more.” It’s that an author’s agreement and disagreement will sort themselves by cost unless something forces them not to. Left alone, honest point-by-point replies produce a letter that is generous about typography and immovable about everything structural — and the author experiences the whole thing as fairness.

The fix is mechanical and takes ten minutes: tabulate your own responses before sending, then check whether everything you accepted was cheap.

I ran that table for this article, two years late. It would have changed one reply and the order of two paragraphs — and the reviewers would have got a letter that was better at admitting the thing we already knew.

Thanks to the reviewers, who wrote pages of specific, unhedged criticism of something three people had worked on for a term. That is a favour and it does not feel like one at the time.

If you’ve answered a review recently: sort your own replies by how much each concession cost you. Mine came out cleanly, and not in my favour.