On bad statistical pratice
I have just published in GigaScience a short note entitled The signed Kolmogorov-Smirnov test: why it should not be used. I had discussed this issue previously on The Grand Locus, and I have refined the arguments through the publication process.
The “signed Kolmogorov-Smirnov test” is a non standard statistical practice on the rise in the field of genomics. This is unfortunate, as there are several reasons why this test should be avoided. First and most importantly, is that it does not test whether two samples have the same mean. Second, it is less powerful than the standard t-test.
Why is it used at all then? It is tempting to speculate that the reason may have something to do with p-hacking, which is the practice of changing statistical test until you find one that gives the p-value you expect. This of course has to be discouraged, and one way to do this is to highlight poor statistical practice. So my aim with this paper was to give a peer-reviewed reference that can be cited in order to argue against the use of this practice.
The editor of GigaScience and the reviewers have been very supportive and constructive, so overall it was a great experience (after some frustration with other journals). It was also the first time I published with open peer-review, which has been very useful to improve the quality of the paper. Garrett Jenkinson and Desmond Campbell have highlighted some of my misunderstandings and I have learned a lot from their comments. In the pre-publication history of the paper, you can even find a power analysis carried out by Garrett Jenkinson to complement my arguments.
Overall it was a great experience. After publishing the initial idea on my blog, several readers suggested to publish the argument with more visibility for the target audience. This gave me the motivation to find the right journal for this and to make the text more accurate and more accessible for biologists. This is my first community-driven publication, and for this reason, I am very proud of it.
comments powered by Disqus