I am also pondering this problem literally right now, and working on it today. It's an issue with several of the new 'push' aka 'webhook' systems: user/key management is slim to none. Riemann server and dash setup is also cumbersome. Isn't the point of a monitoring server that I have to do less manual configuration and management?
Riemann is also a rather heavy server load just to push webhook events to a database. Requires some effort to scale. Godot (Node app that aims to be Riemann API-compatible) seems to be a better option for smaller scale operations.
Right now the solution for many CI and webhook servers is to push events to Hipchat. That does not seem to be the most reliable or efficient way to triage issues.
How are other small to midsize businesses (i.e. those without dedicated devops) managing this architecture?
It's not clear from your comment exactly what you're looking to do, but difficulty of configuration was more or less why we built Cabot (https://github.com/arachnys/cabot) internally for monitoring our infrastructure. However very possibly you're trying to do something more complicated.
Thanks! I skimmed the docs, but I'm not sure I see the strong differentiating qualities. Cabot seems to be a polling (not push) solution? Meaning that each service has to be manually set from within Cabot. I also see several settings are hand-set in the foreman environment.
With Riemann/Godot, I can simply POST JSON (with any metrics) to the Riemann webhook from any number and type of services and servers. Riemann/Godot also make it easy to push data back out, to further analyze or aggregate.
In a way, I'm looking for something less complicated -- a 1GB VPS should be overkill for a small instance. But I'd also like it to accept JSON POST from anything with a matching API key. And store it in a database not run on localhost. With this configuration it should be able to scale as the services scale.
Cabot does make some good points about distributed infrastructure and external testing. Best wishes with the project!
Edit: What is intresting to me is that he says this is single box but he wished it was distributed but its to hard. Seams to me that Strom would have been perfect for this, the auther even knows Clojure so that works out well.
Granted I dont know much about storm but it sound like a perfect match.
I've been looking at a couple different 'push' style monitoring solutions, including Sensu. Does anyone have any experience between Reimann and Sensu, or any of the other similar alternatives out there which they'd be willing to share with the community?
I have no experience with Sensu, but I can tell you why we went with Reimann: we are processing non-server events.
We have instrumented our iOS app very heavily, and we publish those updates (encrypted via libsodium) to our backend over UDP, which then pushes them into Reimann.
We really like it so far, although I agree that setup could be easier. It's just so incredibly flexible. That said, the machine it runs on is 4x the cost of our main server, thanks mostly to the Java tax, but also the extremely high number of events we push through it.
tl;dr Reimann is very flexible when you want to move beyond monitoring servers to many different kinds of events.
UPDATE: Forgot to mention, docs are reasonably good and the developer is very passionate about the project. I have high confidence in its future.
Neustar's Monitoring and Intelligent alerting provides a push interface to process events. The events are processed by a small javascript program which has a restricted set of APIs.
Riemann is also a rather heavy server load just to push webhook events to a database. Requires some effort to scale. Godot (Node app that aims to be Riemann API-compatible) seems to be a better option for smaller scale operations.
Right now the solution for many CI and webhook servers is to push events to Hipchat. That does not seem to be the most reliable or efficient way to triage issues.
How are other small to midsize businesses (i.e. those without dedicated devops) managing this architecture?