Sept 07, 2007
Originally posted on Half an Hour, September 7, 2007.
For some time I have been thinking of launching a journal. Because I think that when people talk about 'peer reviewed publications' they have a point, and that point is, that a piece of writing is not merely popular, but also, respected and recognized by a particular academic community.
We need such mechanisms because there is too much to read, too much even in narrowly defined disciplines. And there is no particular mechanism for identifying that which is important within a particular discipline. The popularity-based systems, like Slashdot and Digg, cater to certain communities, sure, but tend, eventually, to what we might call a scholarship of the middle - no particular discipline, no particular level of quality, no particular virtue.
That is not to discount the systems whereby content is selected and reified by the masses. I am a regular reader of such lists and they are a constant source of amazement and amusement. High quality content does get selected by the crowd, but not all of it, and not reliably within a certain discipline.
Historically, as I mentioned, content selection for academic materials has been by means of 'peer review'. The process varies across journals, but in its most typical instantiation, proceeds as follows: a writer submits a manuscript to an editor, who reads it. The editor, at his or her discretion, sends the manuscript to a small committee of reviewers. The reviewers rate the submission for appropriateness for publication. They will often recommend changes and improvements. A final version is drafted, and it is typeset and published.
There are, in my view, two major weaknesses of this approach:
First, it proceeds in secret. The manuscript is not viewed by the reading public until after it has been selected. That it is being considered for publication is not known. Thus, if a manuscript is submitted and rejected, it may never see the light of day. This is wasteful. And it results in the very real possibility that highe quality works might never be seen because they did not pass the scrutiny of a few people.
Second, the decision is made by only a small number of people. Very few people actually review manuscripts; typically three or four. If these people are not attentive to the material they are reviewing, they may accept substandard material, or reject high quality material, simply because they were not paying attention. Having material reviewed by a wider number of people reduces this likelihood. It creates the possibility for buzz around a selection, a conversation that will result in its being not only improved but also brought to the attention of reviewers.
So how do we fix these things? The approach is to decouple access from review. Specifically, authors' manuscripts ought to be easily and widely accessible prior to publication. In this way they can be read by a large number of people. This does not mean that all people read all articles; there is no need for that. But it does mean that the typical article would be read by well more than three people.
Once access has been enabled, then we need to develop a review process. The problem (in my view) with sites like Slashdot and Digg is that a resource rockets from access to acceptability with virtually no restraints. Insofar as there is a community, it is like they are a pack, jumping from one popular thing to the next, with no sense of direction or consistency.
Also, there is no sense of 'peers' in this process. There's no sense that the submissions have been evaluated by people who have demonstrated their commitment to a certain subject area or background or expertise in the field. It's one thing to say "The Golden Age of Paleontology" was popular; it is quite another to say that it was popular among paleontologists.
But what constitutes 'being a paleontologist'? Traditionally, we have required some sort of certification. A person needs to become a PhD in paleontology. Then they need to be selected by an editor of a journal to sit on a review board. This qualifies them to review publications in paleontology.
There is merit to this approach. We have it on good grounds that the person is very likely an expert in the field. They have passed a rigorous and formal course of study. They have been subjected to examination, and have the academic credentials to prove it. Very often, they will also hold a position at a university or a research institute. By the time they are selected to become a member of a review committee (and eventually, an editorial board) they will have successfully published a number of articles, establishing their importance in the field.
This approach has served us well historically, however, there are signs of strains. It takes a long time to earn a PhD, under fairly restrictive circumstances. New disciplines and technologies are being developed so rapidly that by the time a person becomes an expert in one thing, it has been replaced by another, which didn't even exist at the time she began her studies. The membership of peer review committees adds to this ossification; their expertise may be of disciplines that have long since come and gone.
Moreover, even if the academic route is a reliable means of establishing expertise, it is no longer the only one. We are seeing with increasing frequency people establish expertise outside their domains or disciplines. We are seeing people, through a process of self-education and public practice, become well established and well respected even in academic fields.
In some cases this is by necessity; it costs a great deal to earn a PhD, much more than the cost of a computer and internet access, and so the informal route is the only means available. This is the case for the majority of people in the world.
And in other cases it is by choice, as no PhD programs exist in a new area of study or invention. This was the case, for example, in internet technology. It had to be built, first, before people could become experts in it, while the people who built it became experts by building it.
So we need to allow for the likelihood that there is a great deal of expertise in the world that exists outside the domains of the traditional academic community. That the path of obtaining a PhD is one way to establish one's expertise, but not the only way. And that there will exist people who can quite genuinely be called experts in practitioner communities, self-selected or intentional communities, communities of practice, and elsewhere.
It is with these thoughts in mind that I have, over time, be thinking of the appropriate sorts of mechanisms for the management of academic journals. And so it seems a good time now to suggest how I think what I'll call 'the open journal format' should proceed.
I call the system 'open journals' not to confuse them with 'open access journals' but to stress that much the same principles are being applied. An open journal will be at heart an open access journal, but in addition, the process of selecting and reviewing articles for submission will also be open.
Here, then, is the process:
The idea of such a system is that there are things that balance the journal between popularity and rigor.
It is possible for a journal to become too much of a clique, for the members to select only each others' papers. If so, then the people who are being left out can found their own journal. Because nominations are public, it will be easily evident which journal is the most difficult to get into because of quality, and which are the most difficult to get into because of exclusivity.
Will this work? I think it will. It might not work for any particular journal - some journals may simply not attract readership because the writers admitted were not of a high quality, or because the members make poor choices, or because the subject area is simply not useful or inappropriate. It will take a certain amount of momentum to launch a journal, a momentum that can be gained only by having qualified people and quality ideas to begin with.
I realize that similar projects have been tried by others. I am especially aware of David Wiley's attempt - and this is very similar to that. With one exception - that it draws from content that people have already posted. There is no particular danger of 'rejection', no chance that your submission won't see the light of day, because you don't even submit it. The process of recognition and nomination is undertaken entirely by your readers.
I am seriously considering this. I invite your comments.
For some time I have been thinking of launching a journal. Because I think that when people talk about 'peer reviewed publications' they have a point, and that point is, that a piece of writing is not merely popular, but also, respected and recognized by a particular academic community.
We need such mechanisms because there is too much to read, too much even in narrowly defined disciplines. And there is no particular mechanism for identifying that which is important within a particular discipline. The popularity-based systems, like Slashdot and Digg, cater to certain communities, sure, but tend, eventually, to what we might call a scholarship of the middle - no particular discipline, no particular level of quality, no particular virtue.
That is not to discount the systems whereby content is selected and reified by the masses. I am a regular reader of such lists and they are a constant source of amazement and amusement. High quality content does get selected by the crowd, but not all of it, and not reliably within a certain discipline.
Historically, as I mentioned, content selection for academic materials has been by means of 'peer review'. The process varies across journals, but in its most typical instantiation, proceeds as follows: a writer submits a manuscript to an editor, who reads it. The editor, at his or her discretion, sends the manuscript to a small committee of reviewers. The reviewers rate the submission for appropriateness for publication. They will often recommend changes and improvements. A final version is drafted, and it is typeset and published.
There are, in my view, two major weaknesses of this approach:
First, it proceeds in secret. The manuscript is not viewed by the reading public until after it has been selected. That it is being considered for publication is not known. Thus, if a manuscript is submitted and rejected, it may never see the light of day. This is wasteful. And it results in the very real possibility that highe quality works might never be seen because they did not pass the scrutiny of a few people.
Second, the decision is made by only a small number of people. Very few people actually review manuscripts; typically three or four. If these people are not attentive to the material they are reviewing, they may accept substandard material, or reject high quality material, simply because they were not paying attention. Having material reviewed by a wider number of people reduces this likelihood. It creates the possibility for buzz around a selection, a conversation that will result in its being not only improved but also brought to the attention of reviewers.
So how do we fix these things? The approach is to decouple access from review. Specifically, authors' manuscripts ought to be easily and widely accessible prior to publication. In this way they can be read by a large number of people. This does not mean that all people read all articles; there is no need for that. But it does mean that the typical article would be read by well more than three people.
Once access has been enabled, then we need to develop a review process. The problem (in my view) with sites like Slashdot and Digg is that a resource rockets from access to acceptability with virtually no restraints. Insofar as there is a community, it is like they are a pack, jumping from one popular thing to the next, with no sense of direction or consistency.
Also, there is no sense of 'peers' in this process. There's no sense that the submissions have been evaluated by people who have demonstrated their commitment to a certain subject area or background or expertise in the field. It's one thing to say "The Golden Age of Paleontology" was popular; it is quite another to say that it was popular among paleontologists.
But what constitutes 'being a paleontologist'? Traditionally, we have required some sort of certification. A person needs to become a PhD in paleontology. Then they need to be selected by an editor of a journal to sit on a review board. This qualifies them to review publications in paleontology.
There is merit to this approach. We have it on good grounds that the person is very likely an expert in the field. They have passed a rigorous and formal course of study. They have been subjected to examination, and have the academic credentials to prove it. Very often, they will also hold a position at a university or a research institute. By the time they are selected to become a member of a review committee (and eventually, an editorial board) they will have successfully published a number of articles, establishing their importance in the field.
This approach has served us well historically, however, there are signs of strains. It takes a long time to earn a PhD, under fairly restrictive circumstances. New disciplines and technologies are being developed so rapidly that by the time a person becomes an expert in one thing, it has been replaced by another, which didn't even exist at the time she began her studies. The membership of peer review committees adds to this ossification; their expertise may be of disciplines that have long since come and gone.
Moreover, even if the academic route is a reliable means of establishing expertise, it is no longer the only one. We are seeing with increasing frequency people establish expertise outside their domains or disciplines. We are seeing people, through a process of self-education and public practice, become well established and well respected even in academic fields.
In some cases this is by necessity; it costs a great deal to earn a PhD, much more than the cost of a computer and internet access, and so the informal route is the only means available. This is the case for the majority of people in the world.
And in other cases it is by choice, as no PhD programs exist in a new area of study or invention. This was the case, for example, in internet technology. It had to be built, first, before people could become experts in it, while the people who built it became experts by building it.
So we need to allow for the likelihood that there is a great deal of expertise in the world that exists outside the domains of the traditional academic community. That the path of obtaining a PhD is one way to establish one's expertise, but not the only way. And that there will exist people who can quite genuinely be called experts in practitioner communities, self-selected or intentional communities, communities of practice, and elsewhere.
It is with these thoughts in mind that I have, over time, be thinking of the appropriate sorts of mechanisms for the management of academic journals. And so it seems a good time now to suggest how I think what I'll call 'the open journal format' should proceed.
I call the system 'open journals' not to confuse them with 'open access journals' but to stress that much the same principles are being applied. An open journal will be at heart an open access journal, but in addition, the process of selecting and reviewing articles for submission will also be open.
Here, then, is the process:
- People write articles and post them online. They may be blog posts. They may be contributions to discussion lists. They may be comments or web pages. It doesn't matter how the content has been published online, simply that it be published online, be licensed in such a way that would allow publication in the journal, and be accessible to whomever wants to read it.
Typically a journal would have a 'subscription list' consisting of a set of RSS feeds recommended by its members. The list, available as an OPML file, would allow readers to subscribe RSS feeds from the larger writing community that produces content relevant to journal readers. The list could be subscribed as a single feed. An example of this is the Edu_RSS feed. Readers can consult a single source, selecting from the hundreds of posts created every month, to find the few that ought to be included in the journal. - A journal's 'readers' nominate an article for submission. To 'nominate' an article is to bring it to the attention of the editor and other readers. People may become journal 'readers' by registering at the journal (other journals may allow at their discretion for anonymous 'readers').
There are various ways to manage nomination. It is important to keep it fairly simple, and at the same time, to allow for buzz and community to develop around an article and around a readership in general. Typical process would resemble a 'del.icio.us' or 'Digg' style mechanism, where readers could '+1' an article, and participate or discuss the article. Another mechanism may be to read readers' blogs and count the number of members that link to the article. - The journal's 'members' select from the nominated articles those articles that they think should be published.
What is a 'member' of the journal? Simply - the founder of the journal plus any person who has had an article previously published in the journal.
To be a member of a journal (I will capitalize this from now on) is not only to have had something published in the journal, but also to have been recognized as a 'peer' by the other authors who have had something published.
The selection process is therefore two-fold: members are selecting not only a submission, but also the person. This means that to a degree, the candidate's previous body of work will be assessed as well as the actual submission. The role is not of 'gatekeeping' but of recognition.
Members' votes are public, and they would typically comment on the accepted submissions, perhaps suggesting improvements. The number of articles published each month would vary, depending on the members' selections, but would typically be small. The top five vote-getters, say. - The submission is prepared for publication. It is submitted into the journal editing system, spelling and grammar are checked, links to references confirmed, and the like. Galleys are created for the print edition (which will be published at a print-on-demand service such as Lulu). The author, in consultation with the editor and members, may make changes to the original submission at this point. The issue appears as an open access publication on the web.
The idea of such a system is that there are things that balance the journal between popularity and rigor.
It is possible for a journal to become too much of a clique, for the members to select only each others' papers. If so, then the people who are being left out can found their own journal. Because nominations are public, it will be easily evident which journal is the most difficult to get into because of quality, and which are the most difficult to get into because of exclusivity.
Will this work? I think it will. It might not work for any particular journal - some journals may simply not attract readership because the writers admitted were not of a high quality, or because the members make poor choices, or because the subject area is simply not useful or inappropriate. It will take a certain amount of momentum to launch a journal, a momentum that can be gained only by having qualified people and quality ideas to begin with.
I realize that similar projects have been tried by others. I am especially aware of David Wiley's attempt - and this is very similar to that. With one exception - that it draws from content that people have already posted. There is no particular danger of 'rejection', no chance that your submission won't see the light of day, because you don't even submit it. The process of recognition and nomination is undertaken entirely by your readers.
I am seriously considering this. I invite your comments.
