The article is very interesting/good, but I do want to mention one issue I have with it:
“To say that there’s a large amount of literature on the benefits of this approach would be an understatement.”
I've brought this up before, but “literature” carries the connotation of “scientific literature”, and I actually haven't heard of many rigorous, well-constructed scientific experiments that have produced conclusive scientific literature that indicates the benefits of TDD. There's certainly a lot of anecdotal posts and writings, but there seems to be relatively little actual “literature” in the most commonly used sense.
Not saying TDD is bad, of course… Just wondering if this is in fact an overstatement rather than an understatement. If we're going to appeal to authority, we should make sure that authority is valid :)
"Making Software" [1] has a chapter called "How Effective is Test-Driven Development?" where they do a meta review of 32 clinical studies on TDD pulled from 325 reports. They filtered out reports based on scientific rigour, completeness, overlap, subjectivity etc, and ended up with 22 reports based on 32 unique trials. "TDD effectiveness" was defined as improvement across four dimensions - internal code quality, external system quality, team productivity, and test quality - and each dimension is strictly defined in sufficient detail.
They report that high-rigour studies show TDD has no clear effect on internal code quality, external system quality, or team productivity. While some studies report TDD has a positive impact, just as many report it has a negative impact or makes difference at all. The only dimension that seems to be improved is "test quality" - that is, test density and test coverage - and the researches note that the difference was not as great as they had expected.
The chapter concludes that despite mixed results, they still recommend trialling TDD for your team as it may solve some problems. However it is important to be mindful that there is yet no conclusive evidence that it works as consistently or as effectively as anecdotes from happy practitioners would suggest.
The meta-study is relatively short and can be read online [2].
The interesting thing to me about TDD is it almost feels like science: make a test, specifically a test of your assumptions (you assume the test will fail and you assume that the code you will create will fix it), test it, then modify from there.
> I actually haven't heard of many rigorous, well-constructed scientific experiments that have produced conclusive scientific literature that indicates the benefits of TDD.
The whole of the industry suffers from this. I know they are vastly dissimilar as software doesn't have the issues that tangible real-world stuff has, but you'd never see someone proposing a bridge being built on the principals read in some book or a paper that doesn't compare any other theory. I dread the day we have rigorous IT standards as much as I can't wait for it. I've heard it is possible with Ada and lots of concern, money, and talent, but I wonder if we'll ever see that elsewhere. I guess as a whole the tools we rely upon are getting better, but I'm not convinced the methodologies haven't been just more perpetual motion machines.
> “literature” carries the connotation of “scientific literature”
> there seems to be relatively little actual “literature” in the most commonly used sense.
Literature: books, articles, etc., about a particular subject
I don't think you should add words to what people have said to make a point unless you get some clarification from the author that the additions actually clarify what they said. There have been quite a few books and articles about TDD.
In industries such as rail and transport, industrial systems software - developed under Waterfall principles - tend towards the TDD approach, and while there may not be much academic/scientific literature on the subject, TDD is fairly well observed and applied in software industry. Because its also a 'natural act' in many other industries; you plan to fail once, improve, then pass the process, no matter if you're booting up an OS, or indeed a furnace, an accelerator, etc. Test before use, is TDD in a tl;dr, where 'use' to a developer means 'give to customer'.
Just try it for a while. If it suits you, then keep it in your bag of tools. If you wait for scientific researches to tell you what to do to get better, you will miss on a lot of learning opportunities.
- My two cents
I don't understand how anyone can develop software with myriad moving pieces without isolating and testing each component as it is written. It's like trying to figure out why your car won't start by sitting in the driver's seat and pushing buttons. I wouldn't ever presume to tell someone else how to do their job, but I can't imagine doing it any other way. (I prefer to write out my spec in BDD style, and unit test when the logic gets hairy).
I work at a development firm that mostly services local corporations, and the average time we have to maintain the software we build is fairly low. My boss and I both appreciate the TDD philosophy, but when client budget constraints are what they are, TDD is one of the first things to go out the window. Our CEO has (understandably) a hard time selling a project for 20% more money when the client won't get 20% more features. Additionally, if/when bugs come up, we're able to sell support and enhancement sprints. So really, TDD would reduce the (already small) revenue stream we get from support, at the benefit of being "one of the cool kids".
>I've brought this up before, but “literature” carries the connotation of “scientific literature”
to you. That connotation didn't even cross my mind. Regardless. Are you going to wait until there is a peer-reviewed scientific paper telling you TDD is good before you'll believe it? Do you have personal experience that TDD seems to make your designs better? If so, is that less true to you because we haven't scientifically "proven" it?
The trend "Jabberwocky" comes at a very considerable startup cost, a significant learning curve, and a negative impact productivity (at least during the early days). In return for strict adherence to its philosophy, it promises to possibly help you create a better designed and more reliable system over the long run. Some cool cats are using Jabberwocky, many more denounce it. Some studies claim it's at least not harmful, others say it's a bureaucratic add-on.
Meanwhile, you mostly deal with small, simple, or short-lived systems and have little expectation in building skyscraper enterprise apps, so the promised benefits fall in the "oh that's nice" bucket. Or perhaps you lead a team that is deeply invested in a three-year codebase not using Jabberwocky, and the cost of getting everyone to switch has a serious financial implication.
The stance of "Jabberwocky sounds nice enough, but I do wonder if anyone systematically proved it is worthwhile" is a perfectly valid position. It is not about what you should do, it is about what you can do, and you can't just follow every trend (also see: Agile+XP) or even trial every alternative. For many non-engineering firms, there is "real work" to be done, and the never-ending increasingly expensive pursuit of operational excellence can be a hard thing to sell internally.
This is a huge benefit of science of the rest of us - a few million people can experiment on the edge of what is known, most of them will get nowhere, but a few thousand will determine that "X448YAB" tends to be superior to "X28YNB". Over time, the rest 7 Billion of us will slowly migrate to the more effective way of doing things and everyone benefits. We cannot denounce someone for not using stuff from the cutting edge until that thing is proven effective.
It is indeed less true, until the possibility that we could be committing a type I error can be ruled out.
It may be that TDD has little negative effect upon development and results in a FeelGood™ response that could be present within any project with non-zero productivity.
“To say that there’s a large amount of literature on the benefits of this approach would be an understatement.”
I've brought this up before, but “literature” carries the connotation of “scientific literature”, and I actually haven't heard of many rigorous, well-constructed scientific experiments that have produced conclusive scientific literature that indicates the benefits of TDD. There's certainly a lot of anecdotal posts and writings, but there seems to be relatively little actual “literature” in the most commonly used sense.
Not saying TDD is bad, of course… Just wondering if this is in fact an overstatement rather than an understatement. If we're going to appeal to authority, we should make sure that authority is valid :)