Having read my fair share of papers, I've noticed a few large areas which call into question the reproducibility of a worrying percentage of papers, to the point that I view CS as not much better than the social sciences:
- Papers that can't publish their code due to NDAs (thankfully, reputable journals aren't accepting this with the same frequency they used to, but there's still a large body of existing papers published in reputable journals that fall into this category)
- Papers whose datasets are not disclosed despite being essential since you can't just go and generate your own data unless you have ultra-specialized equipment laying around (typically this is also due to NDAs)
- Papers that rely on specific software versions in specific configurations and will break after the paper is published and the authors stop maintaining it ("oh btw this only runs on kernel 3.6" - security conferences are still rife with this stuff)
- Papers that rely on not just specific software versions but things like environment size and link order for their benchmarks (this was called out specifically in [0])
In theory, this should all be easy to reproduce, but in practice, it rarely is. Especially since many papers that do have code and datasets available upon request will have said code and data lost to time as professors move around, files get routinely purged, and the project is forgotten about.
It sucks because there have been some interesting papers I've read that I really wanted to check out the code for, only to find out that everything was lost when the fileserver went belly-up (and nobody had been testing the backups) or when the first author left and IT automatically purged the home folder it was stored in.
- Papers that can't publish their code due to NDAs (thankfully, reputable journals aren't accepting this with the same frequency they used to, but there's still a large body of existing papers published in reputable journals that fall into this category)
- Papers whose datasets are not disclosed despite being essential since you can't just go and generate your own data unless you have ultra-specialized equipment laying around (typically this is also due to NDAs)
- Papers that rely on specific software versions in specific configurations and will break after the paper is published and the authors stop maintaining it ("oh btw this only runs on kernel 3.6" - security conferences are still rife with this stuff)
- Papers that rely on not just specific software versions but things like environment size and link order for their benchmarks (this was called out specifically in [0])
In theory, this should all be easy to reproduce, but in practice, it rarely is. Especially since many papers that do have code and datasets available upon request will have said code and data lost to time as professors move around, files get routinely purged, and the project is forgotten about.
It sucks because there have been some interesting papers I've read that I really wanted to check out the code for, only to find out that everything was lost when the fileserver went belly-up (and nobody had been testing the backups) or when the first author left and IT automatically purged the home folder it was stored in.
[0]: https://users.cs.northwestern.edu/~robby/courses/322-2013-sp...