Thursday, December 3, 2009

New Protests against the ridiculus WESSEX INSTITUTE OF TECHNOLOGY

Journals, Conferences, and Referees, or why "modern" "professional"
Science is organized in a completely screwed up Victorian amateur way
----------------------------------------------------------------------------------------------------------

All science is organized around "journals"
and "conferences". In principle, the (laudable) goal
is to generate and disseminate scientific information to us all.
The decision about which articles and/or talks
get into which journals and conferences is made by
unbiased expert "referees". Sounds great, but let's examine how
this actually works in practice.

REFEREES
First of all, 80% of all referees are idiots and 100% are amateurs.
By "idiot" I mean, either way too lazy to do a good job, or incompetent,
or too biased due to some goofy agenda to judge the work unbiasedly
(in roughly descending order of frequency),
and in any of these cases, their effect is essentially that of tossing
a coin. In the remaining 20% of cases, let us say that the referee
is good and makes the right decision. In that case, the effect of any
one referee is a 60-40 biased coin toss. In that case, if you have
3 independent referees (which is common, although in some cases they
use fewer) the probability is .4^3 = .064 > 1/16 that a unanimous wrong
decision is made. Thus, you can publish essentially ANY garbage if you simply
keep re-submitting and have enough patience (and political skills
probably help, e.g. being friendly with editors).
Note that on average 1 out of every 16 garbage submissions
leads to a unanimous acceptance decision, so if you submit and re-submit
a total of up to 8 times, the probability is >50% you'll get it published.
It is considered unethical to keep resubmitting, and especially
unethical to submit simultaneously to more than 1 journal,
since these tactics overload the (already busted) system. This
ethical notion makes sense to me. Unfortunately there is essentially no way to
detect or (especially) enforce these ethics and I believe (and have
heard several scientists simply tell they do it and I should) these
unethical methods are simply standard methodology for many scientists.
Indeed I have encountered several cases where apparently somebody slipped up
in their tricks so that almost-verbatim copies of the SAME article
got published in more than 1 journal roughly at the same publication
time (which definitely is unethical; every published article is supposed to be
NEW original research, aside from articles that clearly are reviews).

On the other hand, if you wrote an excellent paper, then the probability
is still quite considerable that it will be rejected. Each
rejection or acceptance can easily require years due to the
presence of super-slow referees (remember, all referees
are amateurs, i.e. unpaid volunteer do-gooders with
zero accountability, indeed whose identities are kept secret!).
Indeed in maybe 50% of all submissions, the journal simply apparently
loses the paper (or something) and no rejection or acceptance ever comes
back to the author. Ever. You get a slip acknowledging receipt of the
article, but that is all you ever get.
Note the expected number of submissions for an excellent
paper before it gets accepted unanimously is 1/(.6^3) = 4.6. Thus
if you are ethical you could easily wait 10 years even before even being
able to publish an excellent paper. If you do not have the luck, patience,
organization (and political skills?) necessary for this - merely the
ethics - then you aren't going to be very successful at publishing
even if your science is superb.

The entire referee system is a non-working anachronism from the Victorian Age
of Noble Rich Dilettante Do-Gooders. It is sort of like the asinine
notions of "Amateurism" in athletics which are FINALLY being scrapped
(e.g. the Olympics) after over 100 years. As science became bigger and became
an enterprise of numerous professionals rather than few amateurs, this system
became incapable of doing the job, but everybody refused to admit it.
The resulting cost to society has been, and continues to be,
absolutely immense.

MERE ANECDOTES?
All the above numerical figures are based on my experience
as a professional scientist. However, my opponents could
attack them as merely my biased special anecdotal claims.
To respond to that, consider the following study [PLACE CITE HERE LATER]:
The authors of the study took 18 random already-published scientific
papers, changed the authors and titles, and sent them in to the
same journal as submissions, in all cases within a few years of
their publication date (thus the articles were still reasonably up to date).
Result: 16 of the 18 got rejected. (In 0 cases was the plagiarism detected.)
I think this study totally supports my "anecdotal" conclusions from my
personal experience, don't you? Case closed.

JOURNALS
Many are exceedingly overpriced. Often they adopt two-tier pricing systems
where "libraries" have to pay an enormous rate (such as $20000 per year
per journal!?) while certain individuals pay a far smaller rate
(otherwise none would buy it). This combined with the vast proliferation
of journals means few libraries can afford the cost. The result is the
failure of the whole mission of journals in the first place. The outrageous
prices (which also vary by vast amounts from journal to journal; Elsevier
and Kluwer journals have particularly outrageous prices) seem particularly
hard to understand since many scientist-authors now do their own
computer typesetting, and the editing and refereeing costs are usually
free (done by volunteer amateurs).

Many journals have reams of "for show" editors who are selected because
putting their names on the masthead makes the journal "look prestigious" -
but those "editors" actually do no work. Example: Turing Award Winner
and famous scientist Robert Tarjan told me he was a for-show editor
on numerous journals and he'd finally decided to tell them all to remove
his name - although he was primarily concentrating on getting the
journals which actually made him do work, to remove his name first!
If you send your submission to some of those prestigious "editors" you
can be in for an especially high probability your submission ends up in
limbo land - never accepted or rejected! (In my experience, this happened
100% of the time I submitted to a non-chief editor, since at that
young and naive stage in my life I had not realized that only the CHIEF
editor is for real and should actually be submitted to, despite
what the journal submission policy and masthead may say.)

CONFERENCES
Since journals are so screwed up (and they seem especially screwed up in
fast-growing fields such as Computer Science, as opposed to say, physics,
perhaps because everybody has been too rushed to get a semblance
of professionalism going) many subfields of science (especially
computer science) have instead become reliant on conferences. Conferences
usually have far shakier standards of "refereeing" than journals
(which already are pretty shaky!) and often lie by pretending to
have more refereeing than they in fact do have (in an effort to
appear more "prestigious").

Example: A too-typical "scientific" conference was VIDEA'95 organised by the
Wessex Institute of Technology. (VIDEA = "Visualization and
Intelligent Design in Engineering and Architecture", '95 is the
1995 incarnation.) The fact that in this case the emperor had no clothes was
exposed by Werner Purgathofer, Eduard Groeller, and Martin Feda
(Institute of Computer Graphics, Technical University of Vienna).
Previously, one of them had accepted the role of "member of the program
committee" for VIDEA'93 and noted that (a) he received exactly zero
abstracts and zero papers to review, and (b) was never informed about any
program committee meetings nor of any reviewing results.
So for VIDEA'95, these 3 generated and submitted 4 bogus papers with
varying degrees of hilarity (in one case the "call for papers"
was submitted AS a paper, in another they
"took a dictionary of information processing words and selected
randomly some 40 phrases from there and joined them together
to a fantastically technical sounding text."
All four were then "reviewed and provisionally accepted"!
Their conclusion was: "VIDEA accepts EVERYTHING!"
At that point the only remaining requirement for their 4 nonsense
papers to be published in the Elsevier proceedings ("high-quality books") was
that the authors pay a "registration fee" (these fees often are $600
per paper). Oho, now we see the motivation...

Wessex Institute of Technology also does 25 other conferences which
Purgathofer, Groeller, and Feda now suspect are also a joke.
Personally I think this kind of thing is quite typical of many of
the high-volume low-quality conferences out there.

At the other end of the scale, let us consider the
allegedly highest quality conferences out there, which in
Theoretical Computer Science means the SFOCS "Symposium
on Foundations of Computer Science" and STOC "Symposium on
Theory of Computing" annual conferences. I've heard a professional
computer scientist say "Journal publications do not matter.
Only FOCS and STOC papers matter." Careers can rest on publications
in these conferences, which supposedly have a high rejection rate, and
are refereed, thus assuring high quality. Oh? Really? Actually, for many
years, the only "refereeing" you'd get back was a number from 1 to 5,
as opposed to a referee's report. Indeed often this was condensed into 1 bit,
rather than 1 number! Oh yeah. We can be sure a lot of consideration went
into that. And indeed various well known completely wrong FOCS and STOC
papers were published, although it was much rarer for anybody to ADMIT that
(the first ever example I saw where it actually was admitted was again a
paper by the admirable Tarjan, who in the next conference published a
retraction [R.E. Tarjan, C.J. Van Wyk: Correction to ``A Linear-Time
Algorithm for Triangulating Simple Polygons''. SFOCS 1987 page 486]
perhaps in a failed effort to convince the rest of the conference
authors to be equally responsible - since apparently this was the first
such retraction in about 20 years of these conferences).

In fact, once when I was a referee and wrote a long referee report for a paper
I cared about, the editor (Valerie King) REFUSED even to GIVE
my report to the paper authors, saying "I don't want a report. I want a
number from 1 to 5." I then had to bypass her and send it to
them directly. I complained to the conference's editor-in-chief about this but
nothing came of it. Also, once, after my paper was rejected (this was
back when I was a grad student) with of course zero explanation, I
luckily managed to get some verbal explanation of the reason by
talking to a Committee Member. It was:
"Oh, well, somebody said this was all basically done before by Lovasz,
so we rejected it." So then I said "But - Lovasz was the professor who advised
me to submit this paper!" A typical scenario - the judgements
are based largely on vague rumors. Another amusing case demonstrating
the importance of being in the politically "in group" and the shallowness
of the acceptance considerations was by my friend Bruce Maggs.
He was in his advisor's (Leiserson, MIT) office one day.
Advisor: "Well Bruce, I just was at the Program Committee Meeting for SPAA and
there was considerable debate about whether to accept your paper. The going
was tough, but finally, I managed to ram it through. You are In."
Bruce: "That's great doc, and I appreciate it a lot. But... I never submitted
a paper."
Nowadays these 2 conferences actually do send back actual reports (sometimes)
although sometimes you still get nothing back, and more often than not
the reports are just 1 useless hastily written paragraph. Indeed
I recently submitted 3 papers to one of these conferences, result:
2 rejects and 1 accepted. I wrote to the editor saying in
all 3 cases the judgement should have been reversed(!)
documenting my reasons. I concluded the reasons for the errors were
a combination of stupidity, low quality, zero referee accountability, and
political biases both pro and con (e.g. in one case the single referee, whose
identity was obvious, found it very convenient for his career if my paper
criticizing a certain approach, would just vanish. In the case where
my paper which contained numerous errors and unsupportable hype,
was accepted, it was politically inconvenient for it to be rejected
for reasons of a similar but opposite kind.).

JUNKETS
As far as I can see, conferences are not good ways to disseminate
information. Often authors are forced to talk for 20 minutes at most -
and that is in the GOOD conferences like FOCS & STOC; in some
conferences I have seen time limits below 10 minutes. This is a joke.
In some conferences there are hundreds of such micro-talks. What percentage
are you going to get anything from?
One is supposed to fly 1000s of miles, consuming 100s of gallons
of irreplaceable petroleum (more than the annual per capita consumption
of average people in the world - each such flight is thus costing
some average world inhabitant over a year worth of the economic
benefits of gasoline) and the benefit that results is: a 10-minute talk??!
You get far more information by reading the paper in the proceedings
rather than seeing the talk.

This is the communication age: The telephone; internet; Email; computerized
search tools to find just the paper you need to find. In such an age,
it is scientifically and morally unsupportable to have most of
these conferences. The real reasons many of them exist is NOT their
scientific value, but
(a) insane holdovers from past ages,
(b) bean counting based promotion decisions causing pressure to "publish",
(c) opportunity by scientists to abuse their funders or employers to get
free junkets to faraway places in the guise of "working".
(d) The conference organizers pocket hefty fees (over $600 per participant
easily). The airlines and hoteliers are also happy of course...
Only people who should be unhappy are the granting agencies and employers
(and taxpayers), but they are manned by scientists who are in on the junkets
too, so they don't complain and the corrupt system cruises on.
There have been cases [e.g. the "Winter brain conference"]
where conferences held at ski resort areas have been
cancelled due to lack of snow. My colleague Kevin Lang actually witnessed
a medical conference at such a ski resort. Rather than giving their talks,
the MDs had pre-recorded the talks on videotapes (allegedly this allowed
them to present a better quality talk in less time, plus it got
recorded that way - sounds like a good idea I guess), and they then
played the videos at the lecturns. Lang witnessed a case where an
absent lecturer was lecturing via videotape-player to an empty room - while
all the MDs involved presumably cavorted on the ski slopes.

SCIENTIFIC CAREERS
Are often advanced or stalled by boards who mainly utilize
"bean counts" (publication counts) as their decision making tool,
as opposed to actually trying to READ one's papers (that would require
work). Thus the pressure to publish can be immense, leading
to a giant proliferation of garbage publications,
a trend toward more papers with fewer ideas, overloading of the busted system,
and encouraging non-ethics. I believe bean-count is anti-correlated to ethics.

The fact that most of these boards are mainly populated by coprolites
causes any consideration of the idea that maybe, just maybe, something
is screwed up about the current system of Journals, Conferences, Referees,
bean counts, Zero Accountability, and 100% Amateurism, to be dismissed. That
in turn prevents any reform of the system.

HOW TO FIX THE SYSTEM:
1. Referee anonymity and consequent lack of
accountability and responsibility has to be got rid of. Papers should be
placed on a web site like the xxx.lanl.gov site BUT submissions, and
comments, ref reports, and author responses to said reports,
and rejections, should non-eraseably accumulate
as attachments to it on the site. This also will make the tactic of
multiple unethical submissions, visible.
2. I think if anything the anonymity should work in the
other direction: refs should be named, author-identities should be
hidden from them.
3. Boards should actually (gasp!) read the scientist's work,
which I believe they do not do, in general. (There even have been cases
of scientists who enjoyed numerous promotions, including being granted
tenure, who in fact had zero publications, but listed a lot of fake
publications on their resumes. Evidently nobody actually went to the
journal "cites" in the resumes to actually read those "articles.")
If Boards plan to rely on bean counting, they could be replaced by
my (non-scientist) sister, for a lot cheaper! This could be
forced (or at least we could try) by making them make a report
public reviewing that work.
4. Boards generally should be forced to operate in a non-secret mode
(and all votes by them should be revealed vote-by-vote) and
the evaluations and salaries of those they judge have to be made public
so that info exists so that feedback and restoration-to-sanity
forces become a possibility to try to at least have the possibility to
get rid of incompetent board members, etc.

All my attempts to make 1-4 happen have been failures, however.
The whole way refereeing is run (amateur volunteerism with zero
responsibility) is a ludicrous way to run science.

The whole journal system is obsolete and massively overpriced
and fortunately may FINALLY be BEGINNING to collapse under its own weight,
being replaced by web sites. Note that web sites featuring a plethora
of unrefereed scientific papers actually in many ways have papers
of HIGHER quality than do refereed journals:
(a) the papers can be searched by computer to find what you want,
(b) they can include data and programs no journal would publish
(for space reasons)
(c) they could in future include actual software calculators, e.g.
the reader fills in the blank with a number and out pop more numbers...
(d) the cites could be implemented as hyperlinks (including cites
OF this paper by future papers, could be traversed both directions)
(e) accumulating positive and negative Comments and author Responses could
be attached, ultimately having a referee-like effect far more
accurate and severe than most actual refereeing ever is
(f) If the paper is bogus, this could eventually be revealed by an attached
Comment, preventing the paper from sitting around permanently as a trap
waiting to destroy the work of some poor gullible graduate student
or waste the time of some other reader.
Meanwhile conferences show no sign of slackening - to the contrary.

1 comment:

  1. It is good to have "easy" conferences of Wessex Institute of Technology.
    "Easy" conferences that publish everything is like the "Easy" Women.
    We can publish every junk paper in WIT (Wessex Institute of Technology) and justify our summer or winter vacation to our Univeristies.
    Why is this bad?

    Mark R. Wiesner

    ReplyDelete