I'm totally not getting why you're saying agile approaches are waterfall in disguise.
The essential difference to me is in length of feedback loops. Microsoft releases a new OS every few years; some place like Etsy releases new code many times a day. A developer or product manager at Etsy can know within a few hours whether some change is really a good idea; Microsoft not so much.
You seem to be saying that because there are the same sorts of people involved, and because one has a giant spec document while another has a stack of index cards that might be called a spec, then they're the same. Which isn't making any sense to me.
Nope, it has nothing to do with the commonalities in people or roles, sorry I wasn't clear. Let me try again.
It's a knock on all rigid processes and dogma, not agile in particular. I trimmed the rant down by taking out the sections that rip into UX firms, with their Big Upfront Design and cocky rockstar designer personas, but it could just as easily been about them. I only cut it because I thought the parallels were obvious with the diagram on the Epic Consulting website.
What I'm saying is that if waterfall means people are grouped together by specialty - programmers with programmers, designers with designers - each group is run by one dominant process suited to only one of the specializations, and each group tends to work in different stages, then both traditional UX design and agile development are run as de facto waterfall teams, whether by explicit hierarchies or simple stakeholder buy-in.
Consider this - what's the difference between:
1) a PM spec'ing things out in advance and basically telling everybody what to do and in which order.
2) an agile shop coding things up on the fly and telling the designers to run after them to clean up the visuals without making major changes, so they might as well be building to a spec.
3) a UX firm mapping out how the product works before handing off to peon engineers that can't make major changes and might as well be building to a spec.
To end on a positive note, there is one other solution to expertise, by the way. It's grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty. That way you're picking good ideas from across different processes and adapting to the company culture to boot. For example, TypeKit does daily stand ups at 10:05am every day and that's an aspect of agile that they've liked enough to use for the whole company, but I'd be surprised if their designers were checking their art assets into Pivotal Tracker or the programmers were writing up UX journey maps of each user persona.
Steve Jobs always liked to describe Apple as the biggest startup he knew and a more accurate description would be that it's a collection of startups, with each one focused on a single product. Seems to work pretty well for them.
This is the paper that was the basis of Scrum: The New New Product Development Game, Harvard Business Review, 1986. [1].
The crux of the paper is that you should be "grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty".
If the word "Agile" freaks you out, then by all means don't call it that. And certainly there are a lot of bullshit artists plying their trade out there. Indeed, if someone tells me they are "Agile", by default I don't believe them. But as this 1986 paper shows, there is a lot of good data and good theory out there if you but look.
I agree that "agile" teams that group people by specialty are clueless. Done right, the teams are, as you suggest, cross-functional. That was the original intent for the people who coined the term Agile, but much of that has fallen by the wayside as larger organizations decided Agile was fashionable and consultants rushed in to sell them 3% dilutions of XP or Scrum.
That's a great writeup! I know exactly how you feel except my disappointment was doubled, because I was a hybrid developer/designer for years before completing the switch from developer to designer and I lost the faith in both agile and UX at about the same time.
I wish I had the courage to write up my disappointment like you did, but I'm happy that I've moved on to a meta-process of borrowing from different techniques as needed.
Agilistas like to pretend that if you're doing waterfall you can't ever go back to the previous step.
Whereas in practice, if you find something wrong or confusing with the specification, you just hunt down the business analysts and ask them what drugs they were on when they pooped out the spec. And then after walking them through their mistake, you get a new and better spec.
If you do it often enough, they will start to catch on and actually do their job better. Don't think of that as a poke at BA's either, everybody puts more effort in if they know that someone gives a crap about what they do. They may not like it at first, but you're actually empowering them.
"Agilistas like to pretend that if you're doing waterfall you can't ever go back to the previous step."
The Agilistas are correct.
The reason for this is that Waterfall was constructed deliberately as a strawman. The very point of constructing Waterfall was to draw a distinction between the process as it was being presented to the VP, or in this case as it is presented to the customer, and the fact that in practice it doesn't work at all and therefore it could not have been what actually happened. Therefore, given that it isn't a possible process anyhow, we should embrace an iterative approach, because ultimately there's no alternative. We should face up to the truth and thereby learn to take proper advantage of it, instead of sneaking it in the back door, hiding it in shame, and deliberately doing stupid things that can't work because management thinks it makes them happy. And in this case, there's no possible way that that is an accurate description of what happens in any significant project that company undertakes.
If you don't understand that Waterfall is not and never was a "real" methodology and don't grasp the history, which it seems still almost nobody does, then you don't really get what's going on. And attempting to rehabilitate Waterfall into a real or viable methodology is metacontrarianism [1] run amok. There's no point in performing CPR on a scarecrow.
I thought waterfall was a strawman too, until I read The Checklist Manifesto and started learning more about other industries.
For example, large-scale construction projects like skyscrapers are run by gantt charts and committees, which sounds unbelievable and like a disaster waiting to happen. It's like the opposite of every startup I've worked at in the last 4 years. Except it actually works and they keep the everything running smoothly through:
* (at the top level) real time adjustments to huge room-filling charts that shown all the dependencies.
* (at the grunt level in the trenches) an abundant use of checklists to keep work mistake-free and to avoid groups stepping on each other.
* (at all levels) constant and real time communication. Lots of it.
So yeah, I think it's hard to write off waterfall as both ineffective and non-existent if a large industry uses it to such a degree.
Large-scale construction projects are also notorious for delays and cost overruns, so I wouldn't make too much of that.
Even granting your point, though, that doesn't tell us much about software. Nobody expects to start a building as a small house and work it up to something the size of the Pentagon. You can't automatically replace every screw in a building in minutes. Buildings aren't made out of words, and they don't have requirements that change continuously. Buildings aren't expected to respond quickly to competitor's new amenities. The construction of buildings is not 100% automatable with modern technology. So I wouldn't be too hasty in suggesting that the Waterfall approach is made more plausible by a similar-sounding process being used in an entirely different industry.
And honestly, real buildings aren't even much like what you're imagining. Stewart Brand's book "How Buildings Learn" examines actual buildings and how they adapt and change in ways very different than what people with GANTT charts think should happen.
Waterfall has no such history in the software domain. Analogizing software to construction has a long and worthless history, and it has failed to produce one actionable insight that I'm aware of. (It has produced some really dangerous pronouncements about how software engineering should be done, of the kind that you can crash and wreck a company on, though, so there's that.)
The problem is that construction has a completely, radically different cost model than software. If they could possible throw up one building today, tweak it, and throw up another tomorrow, they'd go iterative too. Instead, they have horrifying implementation costs that have no analogy to software; if Waterfall "works" for them it is only because they have no other choices. That's hardly a ringing endorsement, as wpietri observes.
The essential difference to me is in length of feedback loops. Microsoft releases a new OS every few years; some place like Etsy releases new code many times a day. A developer or product manager at Etsy can know within a few hours whether some change is really a good idea; Microsoft not so much.
You seem to be saying that because there are the same sorts of people involved, and because one has a giant spec document while another has a stack of index cards that might be called a spec, then they're the same. Which isn't making any sense to me.