Speaker: Maurice Rice (AT&T Bell Labs/ETH, Zurich)

Panel Discussion: Peer-Review

I guess Ben Bederson invited me here as the token non-high energy physicist in this community. I'm a condensed matter physicist and in the last number of years, like lots of others, I've been working in the High TC field. The High TC field has, I guess, colored my view of the question of peer review and made me perhaps even more of an agnostic. I know peer review is regarded as Holy generally in the scientific community. But having watched the field of High TC, which has been a very contentious field and a very active field in the last while, I've become somewhat of an agnostic on peer review, on the uses and abuses of peer review. There are three reasons that are generally given for why we should do peer review.

The first one is validation. This says that refereeing somehow assures the scientific soundness of work. Of course this only occurs to a certain extent. Certain works get through which are not scientifically sound. We all know that. And also, as it was pointed out earlier by Michael, the number of papers that are actually rejected is relatively small. 20% maybe are rejected out of the Physical Review but then a lot of them will appear somewhere else. So actually it doesn't end up rejecting a lot of papers. I think the point is that the real validation of work occurs in a different way. Mainly, if an important result is claimed by somebody, people try to reproduce it, to repeat it, to check the calculations, and that is how work is validated, and that's the real validation process. What really makes a scientific result important and validated is when it has been reproduced and checked, and not really whether it has appeared or not appeared in Phys Rev or another refereed journal. It's not the refereeing process really, it's process of repeating it and checking it that shows it is a valid result.

The second thing that's also mentioned as a virtue of peer review is that it improves the publication. Well, we again had a presentation from Michael who presented numerical evidence that, in fact, the improvements are generally small and relatively minor. And that tallies with my own experience as a referee mainly, but also as an author. When some referee tells me to change something, I usually do the minimum that will pacify this referee and I think I'm not alone in that. So in many cases people are minimalists. They want to do the minimum to get around the referee and publish. You know, that's the main motivation. So, the number of people who really seriously rewrite their paper, a major rewrite, is relatively rare. And that, as often as not, comes in another way. It comes from informal contact from somebody else or some of your colleagues might tell you "what about this, you've forgotten about this effect, or that paper." You'll get another preprint in the mail in the meantime, which then causes you to modify. So that other method I think is more important and more common as a way of improvement to publications.

The third thing that peer review is used for is to give a rating to publications. We have a hierarchical system, particularly in condensed matter physics. We have PRL. We have PRB Rapid Communications, PRB and lots of other journals. We have a hierarchy and this, of course, has led to the fact that people want very much to get into say PRL and are willing to try very hard. This gives rise to a lot of tension, a lot of aggravation, ill-will etcetera in the community over this issue of whether or not a particular paper can go into Physical Review Letters. Because there is a criterion there, which is a subjective criterion. It's not necessarily scientific soundness. It's something called general interest. So often now-a-days referees can write three line reports and just say this paper is not of general interest, where as every week we get up to thirty papers in condensed matter physics in PRL! If you get such a rejection then you open your PRL copy the next week and you look through and you see these thirty papers and you look at your manuscript and you get angry. This happens to a lot of people and this causes a lot of ill-will. I'm sure Gene Wells is well aware of this issue. But still people fight to get into PRL because of the prestige factor. I think that's an important thing that journals do. They provide prestige to particular scientific reports and I think for that reason journals will continue to have an existence in some form in the future as sources giving prestige or grades in effect, although Geoffrey didn't like that term. The question is how one should organize that. I think personally it should be somewhat more democratic. Because at the present time, as I said if we take again PRL as the example, one has two referees. The reports can often be very random. It's really sometimes like a crap game. You pay your money and you take your chances. Well, yes, you get back some reports. And it is often, I believe, a very random process. It's an attempt to assess the importance of a paper before it is actually published. Now with this E-print archives that is becoming more and more important in condensed matter physics, I believe we should have other ways of doing this - more democratic ways. Because in many cases now, and I think more and more in the future, the papers will appear on these bulletin boards and these archives at the same time as they're submitted. And so when one is assessing a paper and, again if I take PRL as an example, it takes about six months or nine months, or in many cases even a year to decide whether it should be accepted into PRL; by that time, the paper has already been circulating in the community for some time and one could try to make some estimate of the impact this paper had and use this, at least to my mind, as an additional criterion for deciding whether one should put this paper into the prestigious category that we call "PRL."

How could one do this? Well, one suggestion has been that one could allow or set up a forum for public comments on papers. I tend to be somewhat leery about this, as I said earlier this morning. Maybe it's because I come from this background of High TC where there's a lot of conflicting points of view and strong personalities that I feel that this could easily become a forum which created more ill-will; more harm than good to the field. So I would think maybe we should try to have a more neutral criterion. Can we have a more neutral criterion such as the number of citations that paper gets, or the number of genuine requests to read it. Obviously, one has to somehow stop misuse of such a system otherwise people would just get all their friends to order copies and inflate the statistics. But I believe it's possible to get around that in various ways. But I think we need some kind of more neutral criteria where by if one tried to assess a paper after a period of time, say six months, nine months, or a year on the bulletin board, How much impact did this paper have? How much influence has it had in the community? I agree that maybe one shouldn't rely totally on the numerical criterion. Some editors and editorial boards to look at the papers and make sure that there was no abuse going on. But I think new systems with different forms of peer review are open to us now with the new forms of publications, and I think we should try to take advantage of these.

So, I do see a future for journals. I think they will be a future as compendia for more important papers in various forms so that they're, in that sense, adding prestige. I think people would be interested in having collections (something like the Physical Review Letters) of papers that are considered the most prestigious, to browse through at a later stage. So, I think there will be a continuing role for journals but I do think we should try to take advantage of the new methods of assessing papers that the electronic E-print revolution offers to us. Thank you.