Kafka is represented as a very unix-friendly message tool, when it is no such thing. Kafka is (one of) the worst of the collection of message bus daemons. It imposes small(3-6mb) message size limits, or else performance goes to hell. Kafka also fails to "do one thing well", and instead tries to provide everything-and-the-kitchen-sink (clustering, compression, etc) built-in, but poorly.
For some contrast, take a language like python, which will allow you to create example programs to test out Kafka VS RabbitMQ VS zeromq. You will quickly find that one of the message bus daemons constantly gets in your way, fails to reliably deliver messages, and consumes a ton of system resources compared to the other two. Hint: It's kafka!
I really can't say enough bad things about kafka. Having used it and unfortunately been forced to implement it at a number of "big"(juniper, cisco, vmware) companies, it has been a horrible and disappointing experience for both the end-users and the developers, every single time.
Can anyone provide a legitimate criticism of Kafka? "It sucks, doesn't use it" isn't very helpful or productive. The closest thing I can find is http://engineering.onlive.com/2013/12/12/didnt-use-kafka/ which is more critical of zookeeper then anything else.
I'm evaluating Kafka for a new project and it seems to be a perfect fit. I've contemplated building something from scratch in python, as my reliability and performance demands are pretty minimal. However, it seems that a lot of thought went into Kafka's design and it's feature set is perfect match for my problem. Specifically the unlimited buffering, log compaction and the ability to replay logs from arbitrary offsets.
If there are any viable alternatives to Kafka what are they? Bonus points if the JVM isn't involved.
For some contrast, take a language like python, which will allow you to create example programs to test out Kafka VS RabbitMQ VS zeromq. You will quickly find that one of the message bus daemons constantly gets in your way, fails to reliably deliver messages, and consumes a ton of system resources compared to the other two. Hint: It's kafka!
I really can't say enough bad things about kafka. Having used it and unfortunately been forced to implement it at a number of "big"(juniper, cisco, vmware) companies, it has been a horrible and disappointing experience for both the end-users and the developers, every single time.
tldr; don't use Kafka, use a real messaging bus.