Tag Archives: Elasticsearch

Centralising Logs with Logstash and Kibana

Image

We have recently centralised our logs (IIS, CRM, our application of about 5 components) into Elasticsearch on Windows Server using Logstash as the data transformation pipeline (over RabbitMQ) and Kibana as the UI.   It allows us to see all our logs in one place (and if needed in a single timeline), developers can access live logs in a way that they can easily slice and dice the information with out requiring server access. And the front end (pictured) Kibana, is damn sexy! Its dead easy as well. All in all it took about a day to setup.

Architecture

Log Producers

All servers that produce file logs have Logstash installed as a service. Logstash monitors the log file and puts new entries onto a local RabbitMQ exchange . There are much lighter weight shippers out there, however they write directly to Elasticsearch. We wanted something a little more fault tolerant.

Log producers which we control (i.e. our custom components) write directly to RabbitMQ. We use NLog and a modified version (I’ll post more about that later) of the NLog.RabbitMq Target to write our log messages directly (async) to the local RabbitMQ exchange.

Log Server

Our centralized log server has Elasticsearch (the datastore) and Kibana (the UI) running. It also has another logstash agent that reads the messages off RabbitMQ, transforms them into more interesting events (extracting fields for search, GeoLocating IP addresses etc), and then dumps them into Elasticsearch.

Tagged , , ,