How to do exceptionally biased (lousy) science

(image from http://www.pbs.org/parents/education/going-to-school/grade-by-grade/first/)

 

A friend brought a paper to my attention this week:

“Judgments About Fact and Fiction by Children From Religious and Nonreligious Backgrounds”

By Corriveau et al. (2014).

The abstract is fairly provocative:

“Children who went to church or were enrolled in a parochial school, or both, judged the protagonist in religious stories to be a real person, whereas secular children with no such exposure to religion judged the protagonist in religious stories to be fictional. Children’s upbringing was also related to their judgment about the protagonist in fantastical stories that included ordinarily impossible events whether brought about by magic (Study 1) or without reference to magic (Study 2). Secular children were more likely than religious children to judge the protagonist in such fantastical stories to be fictional…The results suggest that exposure to religious ideas has a powerful impact on children’s differentiation between reality and fiction, not just for religious stories but also for fantastical stories.”

Maybe I’m paranoid, but it would seem that the agenda here is to demonstrate that kids raised in religious homes are superstitious, and apt to believe fantasy. Upon closer inspection, the paper has a few devastating flaws, and really represents bad science.

Consider the set up. The authors have 66 total individuals (5-6 year old female students), broken across four groups (so at best, you’d have 16 kids in each group). With such small numbers, I’m not confident that they can really separate out confounding factors (is a kid who is a “non-churchgoer,” but attends parochial school more or less “religious” than a “churchgoer” who attends public school? And can any analysis really tell you that with these sample numbers? There are liars, damn liars, and statisticians. You can make almost anything “significant”…I know from experience).

But, even if we grant them all of that, the questions were loaded and represented repeated measures. They begin with, “Now I’m going to tell you some stories about some people you’ve never heard of. Some of them are real, and some of them are pretend. After I finish the story, I’m going to ask you to put the picture in the ‘real’ or the ‘pretend’ box, and then I’m going to ask you why you decided to put the picture there.” But what story did they tell?

Religious
This is Joseph. Joseph was sent to a mean king in a land far away. However, God sent Joseph many dreams warning about terrible storms, and Joseph used those dreams to tell the king how to protect his kingdom from the storms. The king was so amazed by Joseph and they became friends.
Fantastical
This is Joseph. Joseph was sent to a mean king in a land far away where there were terrible storms. Joseph used his magical powers to see into the future, and told the king how to protect his kingdom from the storms. The king was so amazed by Joseph and they became friends.
Realistic
This is Joseph. Joseph was sent to a mean king in a land far away where there were terrible storms. The king realized that Joseph was very good at looking at clouds and predicting when there would be rain. The king was so amazed by Joseph and they became friends.”

Joseph is a well-known biblical character, not someone a 5-6 year old has never heard of. So, the name Joseph is going to instill trust in the story for those who have any religious background. The researchers should’ve said “This is Charlie. He went fishing…,” etc. Worse, they use the same character in three consecutive stories (in their appendix they detail that they did this for the other stories as well). If the first story was one a religious kid would trust because it involves a known biblical character (Joseph) and God’s miraculous work, that will affect subsequent stories about Joseph. You’ve tainted the child’s perception of who Joseph is, and what kinds of things can happen to him.

Again, an honest way of doing this research would’ve been to offer a biblical story (of Joseph), then a similar story using non-biblical themes and characters, etc.

Notice that their first finding was that, if Christianity is actually true, the secular kids produced false negatives (meaning, they thought biblical stories were fiction). The authors say so,

“[A]ll four groups were significantly above chance in categorizing realistic characters as ‘real.’ All three groups of religious children were also significantly above chance in categorizing religious characters as ‘real.’ However, secular children were significantly below chance in categorizing religious characters as ‘real’ (i.e., they judged them to be pretend).”

Unsurprisingly, churchgoer kids failed to identify fiction as fiction (false positives) about 1/3 of the time. Again, those “fictional” stories involved the same basic story line (same names and events), but removed God. So, this is an expected result.

Their second experiment was equally biased. It was aimed at separating “magical” from non-magical stories. This one dealt with just 33 students, broken into two groups (those who attend parochial schools and attend mass, and those who attend public schools and do not attend mass). The experimenter said,

“Now I’m going to tell you some stories about some people you’ve never heard of. After I finish the story, I’m going to ask you to put the picture in the ‘real’ or the ‘pretend’ box, and then I’m going to ask you why you decided to put the picture there.”

But look at the stories:

“Familiar+Magic

This is John. John led his people when they were escaping from their enemies. When they reached the sea John waved his magic stick. The sea separated into two parts, and John and his people escaped through the pathway in the middle.

Familiar+No Magic

This is John. John led his people when they were escaping from their enemies. When they reached the sea John waved his stick. The sea separated into two parts, and John and his people escaped through the pathway in the middle.

Unfamiliar+Magic

This is John. John led his people when they were escaping from their enemies. When they reached the mountain John waved his magic stick. The mountain separated into two parts, and John and his people escaped through the pathway in the middle.

Unfamiliar+No Magic

This is John. John led his people when they were escaping from their enemies. When they reached the mountain John waved his stick. The mountain separated into two parts, and John and his people escaped through the pathway in the middle.”

Come on! If John waves his stick and the mountains part, that’s magic, even if the sentence doesn’t call his stick “magic.” Again, the stories still resonated with biblical stories kids would trust. John is a biblical character involved in miraculous events. As is Moses. The researchers jumbled names and events, but it would clearly taint children’s impressions of what is possible. Their findings?

“[S]ecular children were significantly below chance in categorizing all sets of story characters as ‘real’; in other words, they systematically treated the characters as ‘pretend.’ By contrast, religious children performed at chance across all 4 story contexts.”

But remember the abstract (summary) of the paper:

“Children who went to church or were enrolled in a parochial school, or both, judged the protagonist in religious stories to be a real person, whereas secular children with no such exposure to religion judged the protagonist in religious stories to be fictional. Children’s upbringing was also related to their judgment about the protagonist in fantastical stories that included ordinarily impossible events whether brought about by magic (Study 1) or without reference to magic (Study 2). Secular children were more likely than religious children to judge the protagonist in such fantastical stories to be fictional…The results suggest that exposure to religious ideas has a powerful impact on children’s differentiation between reality and fiction, not just for religious stories but also for fantastical stories.”

The abstract gives the impression that kids raised with religion cannot discriminate between reality and fiction. But, the findings actually show that kids raised in secular settings think anything and everything that violates naturalism is fiction (i.e., no miracles are possible). The religious kids perform well in making distinctions between reality and fiction except for one aspect of the experiment, in which the authors bias the methods by introducing repeated (but modified) stories with biblical characters and events.

I don’t think I need to say any more about what’s really at play here.

Advertisements

3 thoughts on “How to do exceptionally biased (lousy) science”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s