Centralising Logs with Logstash and Kibana

Image

We have recently centralised our logs (IIS, CRM, our application of about 5 components) into Elasticsearch on Windows Server using Logstash as the data transformation pipeline (over RabbitMQ) and Kibana as the UI.   It allows us to see all our logs in one place (and if needed in a single timeline), developers can access live logs in a way that they can easily slice and dice the information with out requiring server access. And the front end (pictured) Kibana, is damn sexy! Its dead easy as well. All in all it took about a day to setup.

Architecture

Log Producers

All servers that produce file logs have Logstash installed as a service. Logstash monitors the log file and puts new entries onto a local RabbitMQ exchange . There are much lighter weight shippers out there, however they write directly to Elasticsearch. We wanted something a little more fault tolerant.

Log producers which we control (i.e. our custom components) write directly to RabbitMQ. We use NLog and a modified version (I’ll post more about that later) of the NLog.RabbitMq Target to write our log messages directly (async) to the local RabbitMQ exchange.

Log Server

Our centralized log server has Elasticsearch (the datastore) and Kibana (the UI) running. It also has another logstash agent that reads the messages off RabbitMQ, transforms them into more interesting events (extracting fields for search, GeoLocating IP addresses etc), and then dumps them into Elasticsearch.

Advertisements
Tagged , , ,

5 thoughts on “Centralising Logs with Logstash and Kibana

  1. Mark Walkom says:

    Any chance you could share this kibana dashboard for a community repo I’m kickstarting here – https://github.com/markwalkom/kibana-dashboards

  2. You mentioned a modified version of haf’s NLog.RabbitMQ target, why did you modify the existing target? I am noticing some strange behaviour from his target including multiple connections to the RabbitMQ server and hangs when disposing target.

    Would be interesting to know if you had similar issues.

  3. adammills says:

    I don’t remember anything of that nature.
    These are the changes, https://github.com/haf/NLog.RabbitMQ/pull/10

  4. Eric Brown says:

    Thanks for this. In your example where you write directly to RabbitMQ via NLog, are you using Logstash at all to get it to ElasticSearch or does it just go straight there? So is it NLog -> RabbitMQ -> Logstash -> ElasticSearch or without the Logstash step? Thank you!

  5. adammills says:

    It was a long time ago! But we were using Logstash with a RabbitMQ input and an Elasticsearch output

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: