Yeah. When that first report said 'experts' said how hard it would be to have predicted and prevented this, I choked. Amazingly lame.

Citi has an internal web application pen-testing group. My guess is they were only ever attacking the outward facing apps and not the ones after successful authentication. Even if so, they may have hundreds of apps to test which constantly change, and sometimes attackers just get lucky and find a hole as soon as it pops up.

My understanding, just to clarify: Citi has a web app pentesting operation, which engages and is among the largest customers for several well-known pentesting firms. Just in case the impression was that Citi has a couple guys in a room doing this stuff.

As for constantly-changing apps, let me speak against my own direct financial interests here (we have a product coming out that addresses that problem, so I'd like it to be a big one). Many, if not most, of the large financials we work with or have talked to have a fairly strict process for deploying new code, and the process gates on security review. Not deploying unreviewed code comes pretty close to being part of the due care standard at large modern banks. If I had to gamble on this, I'd bet that this specific code did get reviewed.

I can only think of three scenarios which would result in this breach (but I could just be lacking imagination):

1.) This app has been sitting around in production and was never tested.

2.) This app was part of the normal testing procedures (which usually means it's tested annually) and somehow this vulnerability was missed in every test.

3.) This vulnerability was not present the last time the application was tested, and somehow this version was deployed before it was signed off on.

I've been around too long in this industry to claim that scenario 1 or 2 are impossible, but knowing the particulars, they seem exceedingly unlikely.

That leaves me to think it was the third scenario, which is still abberant behavior on their part.

I feel bad when I hear about situations like this. As you mentioned in another comment, this is pretty much what we fear the most.

I don't know Citi at all, but at our fisrv customers I think (2) is more likely than (3) (neither is a mortal lock). I also think that this is a hazard of working with high-volume Big-4 type firms... but I want to tread lightly with that thought for obvious reasons.

No no, I absolutely agree with you (about the hazard). I worry about any company that puts all it's app test eggs into one large contract with a big firm (a statement which I'm sure would make one of my salespeople cringe). I find that the places who use a combination of multiple app testing companies in combination with their internal teams seem to fare much better.

For this specific vulnerability, I find it shocking that even the most rudimentary assessment wouldn't have caught it; but my own personal befuddlement might be biasing me against thinking that (2) is likely.

Ah. The guys I knew at Citi had a small team but i'm not sure if they were direct hire or via contract, but they basically dealt with the low-hanging web-app-pentest-fruit.

I don't work for a financial company but I work for one that works for/with them. Their code review is more like: "Hey, did you even test this code in QC? I can see a syntax error." I don't think anyone here actively looks for security problems during review. If it compiles and it's sat in dev/qc for a month (we just assume it's been tested), it's pushed out. I don't think anybody here would recognize XSS if it hit them in the face, and this particular bug ("allow any authenticated user to view any URI matching a given string") sounds suspiciously like a bad ACL rule in their identity/access management servers.

Edit: and the web app would have to not be rejecting access by an invalid user. I can see a single line's test being formatted in a weird way and this getting missed when somebody committed it - after all, if it doesn't cause failure, is there a bug?

From the article:

"this is a dead simple and common hack and Citi should have seen it and prevented against it. Seriously, this is kindergarten level stuff. Really, really stupid."