Speaker: Jack Sandweiss (Yale, PRD)

Discussion Panel: Peer Review

Thank you, Ben. I should say that I really look forward to coming here mainly to learn. Not so much to tell people things. As it happens, on this subject, as it has been pointed out, I do have strong opinions. But they are rather general and I'll try to make the points fairly simply. I've really enjoyed meeting the people and learning who's doing what. The range of things has been very impressive and interesting. I'd like to just make two statements, flat statements, and then I will briefly talk about those statements and that will be it.

The first one is that I believe peer review journals will have an increasing importance for the foreseeable future. And the second statement is that the whole body of technology that we loosely talk about as electronic publishing will have a major impact on the operation, character, style and interconnection of peer review journals, but they won't change the fact that they still will be peer reviewed and they will still be fundamentally organized along field lines or journal lines by publishers. Now I should explain why I believe those two things to be true, even though I certainly don't know the answers to many related questions.

The first one, why do I think that we will have to maintain peer review journals, is something that has been touched upon by quite a few other speakers today; it's not original. The basic reason is that there's an increasing volume of literature. Scientific literature is growing steadily and the value of quality control, as to what is published, is ever more important. Our journals, which I know best of course, have a rejection rate of between 20% and 60%. Those numbers are not so very large. Nevertheless, reducing the volume by even that amount is quite significant. You would change five hours a day of reading journals to 10 hours a day which makes quite a difference. Probably many people can't spend 10 hours a day, and other things like that. Of course, this is a grossly exaggerated example, but the general idea is valid.

The other point is that even if you have a relatively small number of erroneous papers they have the potential to be quite wasteful. Bob Austin referred to that in his talk. Eliminating even a few wrong papers that are not perhaps so obvious could be quite useful. And I should say at this point that this process, as everybody knows, and certainly I know, is far from perfect. And it probably never will be perfect. And we have to always work the improve it. We have to figure out better ways, new ways, clever ways, etc., to improve the peer review process. But I do believe it's much better than just leaving it to the authors. In some fields it would work reasonably well, as Ben pointed out, and in other fields it might not work so well. And every once in a while you would have a disaster if you just left it to the authors. The peer review system uses something in the order of two, three, four referees on a given paper and that paper might be accessed in its lifetime by a thousand readers. I'll talk a little bit more about the nature of those readers but that ratio is very high. And simply the average reader cannot study the average paper, just cannot have the care which it will have received with this peer review.

Now the valuation of this quality control is different for different segments of the physics community. If you have a small group of researchers working in an area which is very interconnected, that group tends to value the peer review the least. They all know each other, they all know the work, they're quite capable of evaluating what each paper is. They actually would like to know what everybody in the group does. In that case it has the least value. On the other hand, if you look at a somewhat larger area where a physicist should know what happens in parts of his or her field, but which they are not expert in, there they will find it, and do find it, to be very useful. For example, many theorists can't tell if experimental data should be believed. Similarly many experimenters can't tell if an interesting theory, for example, has mathematical inconsistencies. If you define things narrowly enough, those won't matter, but if you have a little broader view it will matter. People mention physicists who change or modify their research fields. They are, by definition, not experts in the field they're moving into and they will find and do find refereed journals important and efficient for them. Again, that doesn't mean it's perfect, or it shouldn't be improved. Of course, it needs to be and should be. Incidentally a special subset of people who are changing their fields are known as graduate students. They're starting a research career and they also find it very useful to know that particular articles have at least had some thought and care in their selections; some peer review.

Finally, institutions use publications in a peer review journal as a measure of the correctness of the work of a candidate for a position. Institutions also tend to use Physical Review Letters as an indication of the importance of the candidate's work. I could tell you, and I"m sure you know it too, that if a body of work by a researcher was never to appear in a peer review journal people would wonder as to why it was correct. And, if it never appeared in PRL there would be questions. Perhaps there shouldn't be, but there certainly would be, as to whether the work was important. Now, some would say that with giant author lists as we have in some fields, I could think of at least one, where these conditions don't apply anymore. But I know, and you people who are in it will also know, that's not true. What really happens is the job of getting the candidate splits into two parts. Part one: Is the work that the group did important, correct, valuable?. That uses the peer review literature just the way it did before. The second part of the problem is: What did that individual do for that research?. That you can only know by personal experience or by letters of recommendation. But the first part is very important. If you had a big group who never published in the peer review journal you have a much harder time getting that person who worked in that group promoted even if people said "Yes he did all the work." He had all the good ideas but none of them ever made it into the peer review journal. So whether that's good or bad, that is the way it is and, probably, it isn't all bad as it does have a way of using the standards which the entire community applies.

And finally, someone else mentioned this. I think it is also true that if you have a peer review standard, at least for the desired publications, what will happen is that the standards of the authors themselves will be caused to be higher. The very fact that there is going to be a review and there is a possibility of rejection, nobody would like to have their research or their manuscript rejected, raises the standards. If you teach it is a little bit like the fact that you announce there will be a final exam. You don't actually have to give the final exam; if you don't give it for a few years in a row the students will get to know of course. But the first year really won't matter so much. They will study for the exam because they think there will be an exam. I realize that this "effect" is anecdotal but I believe it's really true. Okay, there are many other aspects, but those are, I think, the main reasons why, broadly, even though much will change, the essence of peer review will remain as an important part.

The second point that I mentioned is that the journals will change a great deal because of the impact of electronic technology. They will still remain essentially journals. Now the reason for that is obvious. As everybody has heard today there are wonderful new aspects that are going to be able to be applied in the journals because of the technology. Just to repeat some of the things that other people have said, it's very clear that the whole notion of linkage and reference will become far more useful than it currently is. The whole idea of linking references, either to the same journal, to other journals, to preprints, e-prints, to design reports (that was mentioned earlier by Pat) that will be extremely useful. There's a tremendous amount of information, at least in high energy physics that I know about, in the design reports. In fact, I've personally gone and gotten some of them and used them in various research efforts. Those will be very nice and they will make the subject much more useful. Publications will be much more useful.

Obviously, the fact that the distribution is no longer volume dependent will be very important. Libraries will be able, and people will be able to have much more widespread distribution. The base cost will not be so different. But the distribution cost, of course, could be very much less and so there will be a much broader distribution which will be excellent. Data and calculations could be presented in much richer varieties than is present now. Mike Peskin commented about the live programs, about being able to point to a computer program which you do then run yourself. And there could be other things of that character. So it's clear there will be a much richer nature of the report than is now a possibility. And it's also true that preprints could be distributed electronically, which I think is a good idea. I think that preprints used to be commonly sent. We used to send them all the time until we couldn't afford to send them because, unfortunately, you have to put a stamp on each one so that stopped it a long time ago. With the electronic distribution they have all the advantages of interconnection, but also the fact that they will now become much more democratic and much more available. Those things, I think, will happen and they will evolve and they will interact with the peer review journals. But I believe, for the reasons I've said, they won't change the basic character. They will, of course, raise important questions as we've heard about scientific priority and things such as intellectual property rights in who discovered what and so on. There could be a lot of interesting questions there. But those are the everyday life problems that we have all the time. So we'll live with them.

Okay. Many other aspects will evolve and I think others have been presented here. The fundamental organization, however, I think will want to remain, and will remain, the same. It'll be so because an organizing structure is needed. A brief comment was made by Michael Keller about the archival integrity. Electronic media has a very risky nature to electronic integrity. Fundamentally all that any group can sell is its promise to maintain that integrity. You can't sell anything which the library can hold and say yes that's it, that's the thing. If its a tape, for all you know after a while the program to read it will no longer be there. The guy who wrote it will be dead, or he will have quit, or something. I mean the problems that Michael mentioned are extremely true and yet we must have archival integrity for these journals. And so, what you need is an organization which will, in fact, maintain that. Will feel its responsibility. And that will be an important thing and I think that's another reason why journal organization will remain, although very much modified and enhanced, it will remain such an organization. And finally our field of physics really does want its research to be [ported] along broadly defined field lines. You don't want to read a paper in a journal primarily about elementary particle physics and suddenly come across something that has to do with band structure and semiconductors. I mean, if you wanted that you would find it somewhere else. Phys Rev Letters tries to cover everything by being very selective and very restrictive and it has many problems in doing that. I mean, as you all know, I'm sure those are not strictly electronic publication problems, but they're problems of size, clarity, understandability, unity of physics and so on. In any case though, the journals will still remain, I think, as organizing principles. Even though their character will change a great deal. Well, as the volume of literature grows, and the diversity of dissemination means grow, and as electronic publication technology develops, there are possible enormous gains in the productivity of all of this and making good use of it, of course, is going to be the challenge for all of us. Thank you.