Icinga 2 API and Request Tracker
In the process of switching to Icinga 2 for our monitoring, we needed a new solution to automatically create Request Tracker tickets for monitoring events. While our old solution worked well with Icinga (1), it was incompatible with the new Icinga release.
Meet the API
One of the new features of Icinga 2 is a HTTP API which enables you to receive various types of events, be it check results, notifications, acknowledgements and more. This made the descision for an approach to build the new tool easy: It should be an Icinga 2 API consumer which in turn creates and updates tickets via the RT API.
Too many tickets!
Of the various event streams one can select to receive from Icinga 2 choosing the right one can be a bit tricky. The first attempt was using the
StateChange event type.
While working, this led to many unwanted tickets being created. It turned out that the best choice for this problem would be the
Notification type, the only events
you'll receive there are about sent notifications. This allows to use the notification settings in Icinga 2 so that only the relevant events are emitted.
The information about what caused the notification (event) to be send was a bit hidden in the JSON objects received from this stream, but after discovering it everything required was at hand.
As the Icinga 2 API could be useful for other things, a go package can be found on GitHub. Currently only event streams are implemented, but that could change in the future. Pull requests are welcome!
What prevented us from using our old solution was that Icinga 2 doesn't give you any information about what happened before the current event. There was no way to close a ticket if the problem disappeared. To solve this, a small caching layer was built using Bolt. Here the mapping from Icinga 2 host and service (which should be unique) to a rt ticket number is saved so tickets can be updated on new states or closed if the service goes back to OK.
This new tool is running for a few weeks now and works fine (with the occasional tweak needed). If you have a similar setup and want to use it, icinga2rt is on GitHub too. Again, enhancements are welcome!