I'm a big fan of TDD, but I quit reading this article the second my eyes came across "IBM System Sciences Institute" and the chart that accompanies it.
I can't help but notice that this author, seemingly concerned with correctness, also wrote a post about static typing's benefits being unclear, with his argument there again based on an arbitrary and completely unscientific graph pulled from another Medium blogger.
He seems to base his views on the hearsay from his Javascript monoculture, while ignoring the more relevant literature like Code Complete or (when it comes to TDD) Making Software: What Really Works, and Why We Believe It, both books basing their claims on empirical research. Making Software's conclusion is that TDD can help, but does push complexity upward (i.e. programs end up with simpler components so as to make TDD easier, but with a higher numbers of them and therefore more dependencies between them) so resulting bugs will be harder to track.
But it's hardly surprising this article is poorly researched: it's just rehashing TDD pablum for self-promotion, there is not a single original thought in there.
I hung on a bit longer, but lost any hope of seeing something relevant when he launched into an imaginary case study. Later on, we find him mentioning an actual study (of code reviews), only to dismiss its results as being implausible to him.
The author appears to have drunk so deeply of the dogma that he has no idea of how to make an objective case.
I thought it was funny he waved off the code review study as well. If the studies don't actually mean anything then why can't we similarly disregard the one that says TDD is good?
I briefly glanced over the paper, and in references to the 171 projects studied, the following stood out:
> The projects were mostly of short duration and small
to medium size. The median project began in 2012 lasted 61 days, produced 4,200 Lines
of Code, 49 components (modules or features). Duration ranged from 7 to 1,918 days. Size
ranged from minimal (this may represent a short maintenance project) to 88,394. The earliest
project was in 2006 and the most recent in 2014.
And from Fig. 11:
> Team size (median): 6
I suspect that a sample of projects involving larger teams, or a larger codebase, would have produced a somewhat more pronounced effect.
Furthermore, it is not clear to me what domains these projects where from (but I admittedly only skimmed the article). In the financial industry, for example, the defect cost increase is certainly more pronounced.
Hm, I thought the leading person around the concept of cost increases later in the cycle was actually Capers Jones, and his text about it and cleanroom: The Economics of Software Quality.
Obviously this is NOT about TDD, but about Cleanroom, but it seems like you've done a lot of research in the area.
Actually I'd be curious if you've done an in-depth analysis of the available TDD studies. What are the benefits compared to simply writing unit tests that made you a big fan?
Not in-depth across the whole set; I picked apart one or two before coming to the conclusion that there was something inherently flawed in the whole approach of modeling study of the phenomenon after the protocol for medical intervention. That won't work; you can't do a double-blind study of TDD vs placebo. It's a conceptual tool, and as such requires knowledge and skill.
"Treating" a convenience sample of students with a 2-hour training session and trying to measure "performance" afterwards isn't going to yield much insight into what goes on in the mind of a seasoned programmer who's used TDD exclusively for a while. In my experience, the effects of such practice most definitely carry over to programming where you don't even try to apply TDD.
What convinced me to try TDD was a combination of naivete and desperation. What kept me at it was the undeniable, if entirely anecdotal, feeling of wrestling back control over the code I wrote. It's a more complex set of techniques than just "writing the test first", but it's hard to tease them apart and introduce them into your coding independently of each other.
Part of it is a commitment to evolving a program in very small steps, at every step having a whole program that works. Part of it is a set of instincts for refactoring, so that this program also has just as much "design" (if such a thing could be quantified) as is strictly necessary for the time being. Part of it is a habit of framing the next capability as a tiny experiment, with the test being the experimental protocol, and focusing your mind on just those interactions within the code that are relevant. Part of it is a way of "slow debugging", of not jumping to conclusions when you encounter unexpected behaviour but drilling down into what made the behaviour surprising, and revising your mental model of the program.
Put like that it's clear that to "simply write unit tests", for instance to check a box in a process model, cannot possibly give you the same benefits as practicing the above interrelated set of skills. But it's also clear that it requires more involved guidance than "just write the test first". You can pick up a lot of it on your own, to be sure, just like many people can learn (say) guitar on their own and get all sorts of nuance in technique from focused practice.
But you have to love doing it, and that's not something easily picked up in academic studies.
It's normal to experience emotions related to what happens at work. It's called "being human". And dealing with emotions - putting words on them, accepting them and moving on - is a key skill and a part of growing up.
What varies, in my experience, is whether and how much it is acceptable to discuss (and thus process) these emotions in the workplace itself. Being able to say "I'm sad / angry / joyful about X" makes a world of difference.
Once I became aware of that I started being able to remedy it. To start with I actively sought and encouraged discussions of the emotional components of whatever work I was a part of. Retrospectives were a great way of having a structured framework for these discussions (as opposed to giving the impression that I wanted to psychoanalyze my colleagues or vice versa).
After a while I noticed, too, that management in some places actively preferred the dehumanizing effect of making emotions undiscussable, because it afforded easier control over people. I started avoiding these places and selecting jobs that accepted and expected me to behave as a human adult.
I'm disappointed to see a book aimed at "professional" developers continue to spread outdated and debunked information, such as the NIST "study" adduced as evidence for the imperative necessity of "defect cost containment". See my post on the topic: https://plus.google.com/+LaurentBossavit/posts/8QLBPXA9miZ
I'll add my voice to those that have already stated such a book shouldn't start by assuming the SDLC as a reference model: it embodies too many of those outdated assumptions. More in that vein in my own book http://leanpub.com/leprechauns
I haven't read all of it yet but it also seems to miss the vein of development that has led to movements such as Correct by Construction et al. The idea that we should model our computations in a high-level specification and check that those designs meet our goals and hold the invariants we press upon them.
Instead there's Agile and the idea that we can throw together something that roughly works and iterate until our confidence is enough such that we can release it. The so-called, beta-driven-development. (Perhaps a vestigial remnant of the unix philosophy?).
I'm not arguing that formal methods should be used for every software project. I think Carmack was right to point out that if all software was written like the software at JPL we'd be decades behind where we are now. However I do think that it should be a part of the experience of becoming a programmer so that when we encounter hard problems we have the correct instincts to rely on mathematics to help us.
This is a beginners book, and keeping in mind we can't teach them everything in the first "semester," do you still think this topic should be included?
Definitely think the groundwork can be laid out without covering the entire breadth of formal methods. Even a history lesson with Djiskstra, Lamport, Mizra, et al can be illuminating. It's good to lay the foundation so that when a beginner encounters a problem they have the instincts to know that they should look up these tools in order to help them.
I found it a shame that I didn't encounter these tools only until very recently. It's a well kept secret in academia that I think should be shared.
The NIST study graphic matched another from the book Code Complete, to illustrate the cost of defects. I did not have access to the CC graph so I used it instead.
Do you believe the larger point (about cost of defects) is incorrect? Your g+ critique seems to take issue with the details of the study, but I missed an assertion it is wrong.
As to the wikipedia timeline, I cherry picked from it. The point being to pick the important things a student should know, not list everything possible. Seems your WP edits didn't make it through?
I'd put the cost of defects claim in the category "not even wrong".
If someone made quantified claims about "the number of minutes of life lost to smoking one cigarette" I would refuse to take them seriously: I would argue that the health risk from smoking is more complex than that and can't be reduced to such a linear calculation.
This talk about "the cost of a defect" has the same characteristics. I don't mean the above argument by analogy to be convincing in and of itself, and I've written more extensively about my thinking e.g. here: https://plus.google.com/u/1/+LaurentBossavit/posts/8tB2RQoHQ...
But it's a large topic that quite possibly deserve a book of its own.
As for the history of software engineering, it's pretty much the same - to do it properly would entail writing a book, pretty much, and I didn't want to do it on WP unless I could do it properly.
Table and Figure 3.1 in Code Complete support the conclusion and is well cited:
Source: Adapted from “Design and Code Inspections to Reduce Errors in Program Development” (Fagan 1976), Software Defect Removal
(Dunn 1984), “Software Process Improvement at Hughes Aircraft” (Humphrey, Snyder, and Willis 1991), “Calculating the Return on
Investment from More Effective Requirements Management” (Leffingwell 1997), “Hughes Aircraft’s Widespread Deployment of a
Continuously Improving Software Process” (Willis et al. 1998), “An Economic Release Decision Model: Insights into Software Project
Management” (Grady 1999), “What We Have Learned About Fighting Defects” (Shull et al. 2002), and Balancing Agility and Discipline:
A Guide for the Perplexed (Boehm and Turner 2004).
Joel on Software also has a convincing narrative on the subject. Therefore I'm not in a big hurry to replace the image, though I will put it on my todo-list.
I'm all too aware of these many citations. A few years ago, I went to the trouble of chasing down most of the papers and books, and evaluating how well each of them supported the claim. To put it mildly, I was underwhelmed.
However, I haven't (entirely) changed my mind about TDD and similar practices. I do still believe it pays to strive to write only excellent code that is easy to reason about. I like to think that I now have stronger and better thought out reasons to believe that.
Looks like you've done your homework. Yet in my experience it has been true that the farther in space and time I've been from a bug, the harder it was to solve.
"Though it promises robot carers for an ageing population, it also forecasts huge numbers of jobs being wiped out: up to 35% of all workers in the UK and 47% of those in the US"
...not only are those forecasts not from the cited source (they're from an Oxford study), they're also not credible:
I think the time-frames of when this will happen are debatable, but not the general arc of events.
Machines and AI continue to get incrementally better. There is no reason to think that this incremental progress will magically stop at some threshold which leaves a large swath of employment for humans to perform.
