Alex Morgan was halfway through a lukewarm coffee when Professor Lucas Janssen arrived for his appointment, waving a USB stick like a talisman. “Morgan,” he began triumphantly, “I need your help. I’m trying to do a repeated-measures ANOVA in JASP, and I want to get those little asterisks next to the significant comparisons in the post hoc tests. You know, the stars. Very important.” Alex nodded. “That’s just under the post hoc options—tick ‘Flag significant comparisons’ and it’ll add the stars automatically.” Janssen blinked. “Yes, yes, but can you show me? These programs are designed to be needlessly obscure.” With a resigned sigh, Alex opened JASP and motioned for the USB. As they started navigating the menus, Janssen leaned back and said, “Actually, I should tell you about this amazing experiment we’ve been running in the psych lab—total breakthrough, really.”
Alex hesitated, cursor hovering mid-click. “It’s a manipulation of ambient scent and interpersonal trust,” Janssen continued, practically vibrating with enthusiasm. “We’ve done it nearly 30 times now—sandalwood, lavender, even a pine-vanilla hybrid. But only a few runs actually worked. Most fail, thanks to student assistants who can’t follow basic protocol. One time they diffused the scent after the trust game. I mean, come on. But the successes—those are something. Scent increases trust, Morgan. It’s there.” Alex, already feeling a statistical migraine brewing, asked carefully, “And by ‘successes,’ you mean… significant p-values?” Janssen nodded proudly. “Yep. Like last week—p = .048 for sandalwood. That’s the third success. The others? Garbage data. Assistant interference.”
Alex stared. “Lucas… do you see how running thirty near-identical experiments and only keeping the ones with p < .05 might not be showing a real effect?” Janssen frowned. “No, that’s the point of replication. You throw out the noisy ones. You don’t publish the failures. What would that achieve?” Alex opened their mouth, then closed it again. “That’s not replication. That’s just stacking Type I errors and calling it aromatherapy.” Janssen looked genuinely offended. “You think three significant results for sandalwood are a coincidence? Come on, Morgan.” He stood abruptly. “Anyway, now I know where the stars are, so—great talk.” And with that, he swept out of the office.
Alex leaned out into the hallway and shouted after him, “Please just report all the experiments in the final paper!”
Janssen called back cheerfully, “Of course! I always appreciate your thinking along, Morgan—so helpful!”
Write a short story where Lucas visits Alex for a consultation on a very simple question on how to do a very specific option for an analysis in JASP or R or SPSS (your choice). Then at some point have the consultation derail in Lucas Janssen boasting about this cool experiment he has been running, but how "many" of the experiments fail as a result of the incompetence of his student assistants that run the experiments in the psychology lab. Have Alex react uncomfortable where they slowly ask clarifying questions until it becomes quite clear that all the 'successful' experiments are type I errors. Make it about 4 paragraphs long.
Instead of Janssen be satisfied and say 'that's it' in the first paragraph, have him insist that Alex show them because software is way to confusing and in the middle of Alex starting JASP in order to do this he just starts telling about this study. Rather than him not leaving and announcing that he wants to talk about that in the start of the second paragraph.
Make 'As long as it smells like science' the title of the story. However, instead of having Janssen understand the point of Alex in the end and say 'Well, as long as ...." have him become a bit annoyed and disagree with Alex about the coincidence part, and then leave because he now knows how to get the stars. Have Alex shout after him through the hall to please just report all of the experiments in the final paper. Then wrap it up with one sentence.
In the final sentence have Janssen responds happily with a remark about how he loves/appreciates Alex' candour and helpfulness (in a passive agressive way, but make that hardly noticable)
change brutal honesty in something more innocent
change straightforwardness into 'thinking along'
[I added *for sandelwood* in the story with direct editing].