Scientists get jobs based on the judgments of their colleagues through the process of peer review (see last week’s post for an intro). The review part can be helpful, because it means you get expert eyes on your work, which can point out limitations and make requests on ways to strengthen it. The peer part can be problematic, because politics and biases—which shouldn’t influence a judgment of the quality of science—end up doing so.
In particular, the peer review system gives extra weight to scientists who are already famous and well-connected, at the expense of scientists who aren’t. This is because scientists only get professional credit for publishing their work in journals, especially highly regarded journals, and what gets published in them is decided by a very small number of science insiders. The editors pick reviewers who will read the work, the reviewers share comments with only the editor and the author, and the editor ultimately can decide whether to accept or over-ride their decisions.
Science reformer Simine Vazire is flipping herself, and wants other scientists to flip, too.
Source: By A. Danvers
The outgoing editor of a well-known social and personality psychology journal, Simine Vazire, recently reflected on this process in her blog. Weeks ago, she had incredible influence over the kind of science that got promoted. To people like me, she was a hero because she would hold scientists to a high standard of evidence, promoting openness and sound methods above prestige and the “well, we always did it that way” inertia in reviewing. But Vazire asks a broader question: should she, or anyone, have that much influence over who gets to stay in and get rewarded in science? What’s so special about an editor’s judgment, anyway?
If Vazire’s opinion was worth a lot before, when she was editor, that should be because she was able to give thoughtful critiques and helpful suggestions regarding the quality of scientific research. She should be able to still provide this value to the field, and it shouldn’t have to come at the invitation of an editorial board. So Vazire is “flipping” herself, a term she is using to refer to the practice of performing her editorial work openly, without waiting for requests to review. She will read manuscripts—especially those posted as “preprints,” or drafts that might not have been yet submitted to journals—and provide feedback on them. She will particularly focus on reviewing the research of lesser known scholars, who might not be getting attention from “Big Journals.” This lets Vazire do valuable work she loves without needing the official stamp of approval from a small network of science insiders.
Vazire is also encouraging people to flip themselves. Any scientist who is already doing volunteer reviewing for academic journals—which is most scientists who have stable jobs—could do the same thing she’s proposing for herself. We could all go out and look at the work of colleagues, choosing to participate in peer review on our own terms. Essentially we would be doing the same thing that scientific reviewers already do, but cutting out the politics of who gets to invite who to review and for which fancy scientific journal.
Flipped reviews have the potential to use expertise & ideas from people who might not otherwise get tapped by editors.
Source: By A. Danvers
If enough people participated in this process, then ultimately we could have a whole set of “journal articles” that have been made freely available and have been reviewed for quality (the same processes in the current system), but that aren’t necessarily in a journal at all. Anyone could participate, meaning that the voices of junior scholars that might not get taken seriously when submitting to a fancy journal might have a better chance of getting noticed. Additionally, people who have very specific areas of expertise—like knowledge of how heart rates tend to change over time—could post comments on that specific aspect of many manuscripts, while making it clear that they’re not giving a comprehensive review. Less established voices might also have more of a chance to provide feedback on the science they would otherwise just be reading. If anyone can review, that means a well-informed but unknown graduate student would be able to jump in and post a thoughtful comment—even though no journal editor even knows they exist.
Some people might argue that this would make it hard to find good new research. What about the benefits journals provide in curating new research so people can stay abreast of the best new work? Well, if there’s one thing the internet is good at it, it’s creating Best Of lists. In a flipped world, anyone could create a list of the best new articles on a topic. Or a group of people could come together to put out a list every month. This would be a lot like a journal—but everything in it would be open for scrutiny and discussion, without the lurking suspicion that “eminent scientists” were privately pressuring editorial boards to get special favors. (In a recent episode of her podcast, Vazire describes just such an episode.)
A flipped world has the potential to allow more of the people who use science to contribute to & evaluate it.
Source: By A. Danvers
At its best, a flipped world takes the power to bestow the title of “Good Science” out of the hands of a few well-networked people and puts it in the hands of anyone who can thoughtfully and persuasively articulate their preferences. Ultimately, the hope is that the norms of good science—including things like being able to verify claims with observations, valuing ideas regardless of who said them, and honestly and openly reporting what was done—will overwhelm comments from people who de-emphasize those things.
But this is definitely a high risk play. The internet has been a great equalizer, allowing incredible new voices to be heard, but it’s also created echo chambers and fostered the rise of “conspiracy thinking.” A flipped system may be vulnerable to people posting in harmful ways—flaming rivals’ papers, getting friends to submit positive reviews, or posting disingenuous criticisms that sound persuasive—but don’t follow logically from the argument in the paper. They might also be gamed by having less prestigious people trying to curry favor with the “star scientists” by posting mindlessly praising reviews of their work. A lot depends on the culture that grows up around the practice.
The thing to remember with critiques, though, is that all of this is already happening to at least some extent in the current system—but since it’s opaque, we just don’t know how much. Our flipped world doesn’t need to be a utopia to be an improvement on the one we live in. Peer review horror stories abound, and as I covered last week, there is experimental research demonstrating instances of bias. Vazire’s proposal for scientific reviewers to flip themselves has the potential to radically transform—and potentially “disrupt” (to use a term from tech)—a system that needs reform. Scientists should consider whether this new proposal can support their values better than the system we work in now.