I don't know about bounties, because I'm not personally in favour of vigilantism, but I do take your point.
Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.
At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.
I would honestly like to know the upside to posting vs. the downside. I think there is also a saying for this, something like "don't spit into the wind".
Generally, we talk about what we're doing because we're all excited about what we do. It's always been that way for us, and our customers really appreciate the honesty and transparency.
On this particular one, we really did learn a lot and we were keen to share some of that. It's really difficult to run an internet service and we feel we have a duty to try and make this easier, or at less better-understood, where we can.
Our business can't exist with a large diverse network that anyone can get involved on, and we couldn't have got to this point without the knowledge of others, whether that's embedded in the open-source software we run or in the blog posts and emails of other people that figured out hard stuff. It wouldn't be right for us to take and not give something back in return.
There's also an element of defiance in this post too. We got punched in the face. We're not going to respond by hiding in a corner. We're going to say "you know what, fuck you" and we're going to help (and have helped) others to do the same in whatever way we can.
Indeed. As I note in the post, this wasn't something we _needed_ to do at that time, we were just curious.
Replacing a function with a faster implementation is trivial; its just another deployment and we do several of those each week. Changing data format adds operational complexity - two codepaths need to be run and maintained and we increase system load and lower response times for the duration. (The data format is versioned, so that's easy - no architectural problem there).
Its totally worth doing if you need to, and we're not afraid of that, but you don't do it on a whim - it takes proper planning and testing and needs multiple people involved.
The results in the tests are all with optimisations (-O3 -march=sandybridge -mtune=intel), as mentioned in the post.
The exception is the Debian packaged version of zlib, because we don't control that. That's the reason I include stock zlib in the tests - if it had been wildly different from the system zlib, I would have looked into recompiling the Debian package with more optimisations. There were no major differences and indeed, inspecting the package further shows that it is compiled with optimisations.
On the Cyrus side, we used to link to the Debian zlib, so we get those optimisations. For the new code bundled in Cyrus itself, we have to enable optimisations ourselves.
I think that mostly, your benchmarks have to match your workloads. Most of the CRC32 benchmarks I've seen are looking at larger buffers. The xxhash function mentioned elsewhere in this thread was claimed to be "an order of magnitude" faster, but again, large buffers - the gain over CRC32 on the same tests were rather more modest (though not at all insignificant).
In this case, I think (but am curious, will investigate further at some point) our Cyrus servers are doing enough checksumming work to keep any tables hot in the cache. So the tests are hopefully a useful indicator of where improvements can be made.
According to the Stephan Brumme website you linked to, the slice-by-8 lookup table is 8K and the slice-by-16 table is 16K, so your combo version of crc32 needs 24K of L1 cache to run at full speed. Modern server class CPUs typically have 32K of L1 dcache so that doesn't leave much room for the rest of your work. Maybe that's reasonable (I don't really know what Cyrus does), but I thought it was worth thinking about.
Most of the time we're iterating through a cyrus.index, where there's 104 bytes per record, and we're doing a CRC32 over 100 of them, or we're reading through a twoskip database where we're CRCing the header (24+bytes, average 32) and then doing a couple of mmap lookup and memcmp operations before jumping to the next header, which is normally only within a hundred bytes forwards on a bit and mostly sorted database. The mmap lookahead will also have been in that close-by range.
Also, our oldest CPU on production servers seems to be the E5520 right now, which has 128kb of L1 data cache.
- Instruction Cache = 32kB, per core
- Data Cache = 32kB, per core
- 256kB Mid-Level cache, per core
- 8MB shared among cores (up to 4)
So I guess the confusion is that Intel moved the L2 cache onto each core (from Nehalem onwards, I think?) and used that opportunity to substantially lower latency for it.
My quick tests here (using the same methods outlined in the post) suggests its around 30% faster on 64-byte buffers. If we're ever shopping for a new hash function entirely, I'll make sure its considered. Right now its not worth it because as noted in the post, we're not actually under any particular stress and changing hash functions is a big job.
Interesting, I'll have to try again. When I first used the beta I imported a VCF exported from ownCloud (which I think stored the images as data URIs). I'll try to figure out how to upload them via CardDAV now, thanks.
Thanks for taking this stance. I'm a customer (I think we actually talked together recently about Apple/Google vCard implementation) and I appreciate you guys making a stand against this criminal behaviour.
It could seem to some people that such a stance is easy, but no matter the strength of principle, when you see your business go offline and customers start banging at the doors for you to sort it out then the situation becomes more complex.
So from this customer - keep up this stance, keep being transparent and I for one will stick by you guys without hesitation!
Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.
At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.