Here’s what Columbo does. He hears the killer’s story and he takes it very seriously (it’s murder, and Columbo never jokes about murder), examines all its implications, and finds where it doesn’t fit the data. Then Columbo carefully examines the discrepancies, tries some model expansion, and eventually concludes that he’s proved there’s a problem.
OK, now you’re saying: Yeah, yeah, sure, but how does that differ from any other fictional detective? The difference, I think, is that the tradition is for the detective to find clues and use these to come up with hypotheses, or to trap the killer via internal contradictions in his or her statement. I see Columbo is different—and more in keeping with chapter 6 of Bayesian Data Analysis—in that he is taking the killer’s story seriously and exploring all its implications. That’s the essence of predictive model checking: you take advantage of the fact that you’re working with a generative model, and you generate anything and everything you can.
I came up with the LARPing excercise in the context of Bob Seidensticker’s participation in the atheist prayer experiment, so I was recommending it as a way to make it easier to notice when you’re wrong. It lowers your emotional resistance to possibly having to admit error and gives you a way to think deeply and curiously about the implications of a rival theory. Like Columbo, you may find your way from the inside to a contradiction, and then have more reason to think you’ve done your due diligence to be able to write it off.
I think the kind of model expansion that Gelman is talking about is also relevant to another problem Bob’s been talking about: how to convince someone to come over to your side. You can present the arguments that convinced you, but if your interlocutor doesn’t share enough of your first principles, they may be unmoved. (Bob and I have been talking off-site about whether morality is objective, which is the linchpin of my conversion and isn’t a premise that he grants).
If the truth is out there, then LARPing as your opponent might give you a chance to see which parts of their worldview are the source of their predictions that some metaphysical point in your argument is false. Then you can tinker, expand, and see if that bit makes other more obviously false predictions that might cause your opponent to strike it from their model. You can try to slot it into your own worldview to try to see which gears it jams and how you know that those gears are necessary. Working from both ends, you start trying to build a chain of logic from your opponent’s position to yours.
If Bob wants to try any devotions to a saint while the prayer experiment is running, I think he might like Blessed Raymond Llull. I happen to like him a lot because he discovered the Condorcet method of voting before Condorcet. But Bob might be more interested in Llull’s Ars generalis ultima. Llull was trying to write an exhaustive apologetic, such that, wherever you started metaphysically, he would be able to lead you to Christianity. This is the task Bob set for converts, so maybe, in his two minutes of prayer a day, he can ask Llull to give us a hand!