Here’s what Columbo does. He hears the killer’s story and he takes it very seriously (it’s murder, and Columbo never jokes about murder), examines all its implications, and finds where it doesn’t fit the data. Then Columbo carefully examines the discrepancies, tries some model expansion, and eventually concludes that he’s proved there’s a problem.
OK, now you’re saying: Yeah, yeah, sure, but how does that differ from any other fictional detective? The difference, I think, is that the tradition is for the detective to find clues and use these to come up with hypotheses, or to trap the killer via internal contradictions in his or her statement. I see Columbo is different—and more in keeping with chapter 6 of Bayesian Data Analysis—in that he is taking the killer’s story seriously and exploring all its implications. That’s the essence of predictive model checking: you take advantage of the fact that you’re working with a generative model, and you generate anything and everything you can.
I came up with the LARPing excercise in the context of Bob Seidensticker’s participation in the atheist prayer experiment, so I was recommending it as a way to make it easier to notice when you’re wrong. It lowers your emotional resistance to possibly having to admit error and gives you a way to think deeply and curiously about the implications of a rival theory. Like Columbo, you may find your way from the inside to a contradiction, and then have more reason to think you’ve done your due diligence to be able to write it off.
I think the kind of model expansion that Gelman is talking about is also relevant to another problem Bob’s been talking about: how to convince someone to come over to your side. You can present the arguments that convinced you, but if your interlocutor doesn’t share enough of your first principles, they may be unmoved. (Bob and I have been talking off-site about whether morality is objective, which is the linchpin of my conversion and isn’t a premise that he grants).
If Bob wants to try any devotions to a saint while the prayer experiment is running, I think he might like Blessed Raymond Llull. I happen to like him a lot because he discovered the Condorcet method of voting before Condorcet. But Bob might be more interested in Llull’s Ars generalis ultima. Llull was trying to write an exhaustive apologetic, such that, wherever you started metaphysically, he would be able to lead you to Christianity. This is the task Bob set for converts, so maybe, in his two minutes of prayer a day, he can ask Llull to give us a hand!