As an extension of writing the app, I also wanted to take a stab at getting the app to run in Azure. I didn't go into the setup expecting anything to work right out of the box. Getting the app deployed from my local Git repository was easy and painless. But upon trying to browse to the app I was greeted by a 502 error page and the following error in the Application's Event Log:

Application 'MACHINE/WEBROOT/APPHOST/IMAGETAGGERTEST' with physical root 'D:\home\site\wwwroot\' failed to start process with commandline '"%LAUNCHER_PATH%" %LAUNCHER_ARGS%', ErrorCode = '0x80070002 : 0.

After a fair amount of Google-ing I managed to find an answer on StackOverflow (where else?) that points to the two environment variables, %LAUNCHER_PATH% and %LAUNCHER_ARGS%, that are used by Visual Studio to fire up IISExpress. But that's it.

So all it took was replacing those two variables in the app's web.config file. Replacing %LAUNCHER_PATH% with dotnet and %LAUNCHER_ARGS% with the relative path to my application's DLL file.

But this week I ran into the problem where adding the Source parameter didn't work, but I was lucky enough to stumble on a solution. Windows Update KB2966828 has something to do with an ASLR vulnerability in .NET, and it also seems to break the installation of .NET 3.5.

So to get the install working again, I just uninstalled KB2966828 and adding the feature worked as expected.

Source: https://social.technet.microsoft.com/Forums/systemcenter/en-US/5c16b88a-0f19-4aea-ad65-38f0bdb59b9c/install-net-framework-35-on-windows-server-2012-behind-the-firewall-does-not-recognize-sources?forum=winserver8gen

Not using Logstash for centralized log collection? Check out my post on getting it set up.

The way I'm going to structure this post is to start with the most basic configuration possible, and build up from there explaining my rationale along the way. Ready? Let's get started!

Logstash Configuration Basics

First, give the Logstash documentation a once-over, if you haven't already. It's pretty good.

Logstash configurations are separated into three different sections: input, filter, and output. Our config is going to start with these three sections, each empty for now:

# Comments look like this
input {

}

filter {

}

output {

}

The input section, as you might have guessed, is where we tell Logstash how to listen for logs from your sources. The filter section is where all of the work happens. Mutating and massaging logs into useful data. And in the output section section, we tell Logstash where to send the data once it's done with it. For a list of all of the inputs, filters, and outputs check out the Logstash documentation (but you did that already, right?).

Inputs

The environment I manage (at the time of this writing) and will be collecting logs from includes five Windows Server 2008 machines (2 DC's, IIS+MSSQL server) and a Sophos UTM firewall. To get Event Logs from the Windows boxes, I'm using nxlog-ce and (the first half of) a config that I found on the 'net somewhere.

Event Logs

The nxlog config is pretty straightforward, I think, so I won't cover it in-depth. But what it does, basically, is hook into the Windows Event Log stream, roughly format the logs in JSON, and send them via TCP to whatever host and port you desire. Since nxlog is going to be sending its data via TCP, we'll use a TCP input in Logstash. That'll look like this:

tcp {  
    type => "eventlog"
    port => 3515
    codec => json_lines
}

Let's break it down:

• tcp is the type of the input
• The type parameter sets the type field for all of the logs received through this input. This will come in handy in the filter section.
• port is a little self-explanatory, I hope. Port 3515 doesn't have any significance, it's just what I chose when first setting everything up. Something to note: Logstash will require elevated privileges if you want to use a port in the 1-1024 range.
• codec tells Logstash what kind of data to expect on this input. From the docs: "[json_lines] will decode streamed JSON that is newline delimited."

IIS Access Logs

To get Access Logs from IIS we'll need an nxlog config which is a little bit different. Something to note is that you might need to change the log-file path (line 39) and you'll need to configure your IIS site to use the W3C log format, and enable all of the fields. Check out this MSDN page for instructions.

I also use a separate input for the IIS logs so that they're easy to separate out. Here's what that looks like:

tcp {  
    type => "iislog"
    port => 3516
    codec => json_lines
}

Sophos UTM Logs

The Sophos UTM (formerly Astaro) sends logs in a mostly-syslog format. Though, like most things I imagine, it doesn't follow the syslog format very strictly which will cause some extra work later on. The most immediate consequence of this is that we can't use the "syslog" input which only supports strictly RFC3164 or ISO8601 formatted logs. So we'll employ another "tcp" input:

tcp {  
    type => "syslog"
    port => 5000
}

I didn't specify a codec here because the default of "plain" works well enough.

Putting it all together

These three inputs are enough for my environment, so my entire input section looks like this:

input {  
    tcp {
        port => 5000
        type => "syslog"
    }
    tcp {
        type   => "eventlog"
        port   => 3515
        codec => json_lines
    }
    tcp {
        type => "iislog"
        port => 3516
        codec => json_lines
    }
}

Filters

Here's where all of the heavy-lifting and usefulness of Logstash actually comes into play. First we need a way to separate out the logs from the different sources, and we'll use the type field (which we set earlier) to do that. Applying certain filters to specific events is done using conditionals. The basic setup for the three types we've set up will look like this:

filter {  
    if [type] == "syslog" {

    }
    if [type] == "eventlog" {

    }
    if [type] == "iislog" {

    }
}

Syslog

Note: Some of this section might be pretty specific to my particular firewall
Here's an example of one of the more well-formatted syslog messages from my firewall:

<30>2014:05:27-22:23:03 nw-asg-fw0 ulogd[4520]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60001" initf="eth3" srcmac="aa:aa:aa:aa:aa:aa" dstmac="bb:bb:bb:bb:bb:bb" srcip="a.a.a.a" dstip="b.b.b.b" proto="6" length="109" tos="0x00" prec="0x00" ttl="46" srcport="443" dstport="64091" tcpflags="ACK PSH"  

To parse this (and the other logs), we'll start with a grok filter:

grok {  
    match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
    add_field => [ "received_at", "%{@timestamp}" ]
}

What this does is looks for the various patterns, like POSINT and put the value in a field like syslog_pri.
So for my example log:

• syslog_pri = 30
• syslog_timestamp = "2014:05:27-22:23:03"
• syslog_program = "ulogd"
• syslog_pid = 4520
• syslog_message = "id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60001" initf="eth3" srcmac="aa:aa:aa:aa:aa:aa" dstmac="bb:bb:bb:bb:bb:bb" srcip="a.a.a.a" dstip="b.b.b.b" proto="6" length="109" tos="0x00" prec="0x00" ttl="46" srcport="443" dstport="64091" tcpflags="ACK PSH""

Next, we want to turn the Syslog PRI number into some useful data. This is done using the syslog_pri filter, which converts the numeric value into a "severity" and a "facility". It looks like:

syslog_pri { }  

By default, the syslog_pri filter will look in the syslog_pri field, which is why I put it after the initial grok filter.

Next, we use the string timestamp from the syslog message as the actual timestamp for the Logstash event. This is done with a date filter:

date {  
    match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}

Next up is something that I do, which is just a personal preference. The firewall doesn't include its FQDN in the syslog message, but I want to have the FQDN stored in the @source_host field (the default place for storing the source of the log). Also, I want to copy the syslog_message field into the @message field which is the default field displayed in Kibana. I do that using a mutate filter:

mutate {  
    replace => [ "@source_host", "firewall-name.ad.company.com" ]
    replace => [ "@message", "%{syslog_message}" ]
    remove  => [ "syslog_message", "syslog_timestamp" ]
}

"%{syslog_message}" is the syntax used to reference a field from within a string. See sprintf format and field references in the Logstash docs.
I remove the syslog_message and syslog_timestamp fields, using a mutate filter, because they now duplicate other fields.

Finally, I use the kv filter to make individual fields out of the key-value pairs that exist in most of the messages (and especially those packet filter violations). I simply point the kv filter at the @message field and it does all of the hard work:

kv {  
    source => "@message"
}

In this example, fields like srcip, dstip, srcport, and dstport are added with their corresponding values.

So, our final syslog section looks like this:

if [type] == "syslog" {  
    grok {
        match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
        add_field => [ "received_at", "%{@timestamp}" ]
    }
    syslog_pri { }
    date {
        match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
    }
    mutate {
        replace => [ "@source_host", "firewall-name.ad.company.com" ]
        replace => [ "@message", "%{syslog_message}" ]
        remove  => [ "syslog_message", "syslog_timestamp" ]
    }
    kv {
        source => "@message"
    }
}

Event Logs

No example this time, just going to dive right in (and move a little bit quicker).

First, a matter of personal preference:

mutate {  
    # Lowercase some values that are always in uppercase
    lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
}

This uses the mutate filter to convert the values of the listed fields to lowercase. The less my computers yell at me, the better.

Next, converting the hostname field to the Logstash standard and setting the timestamp:

mutate {  
    # Set source to what the message says
    rename => [ "Hostname", "@source_host" ]
}
date {  
    # Convert timestamp from integer in UTC
    match => [ "EventReceivedTime", "UNIX" ]
}

If you take a look back at the nxlog config, you can see where we set EventReceivedTime to a UNIX timestamp.

Some more renaming and redundancy removal:

mutate {  
    # Rename some fields into something more useful
    rename => [ "Message", "@message" ]
    rename => [ "Severity", "eventlog_severity" ]
    rename => [ "SeverityValue", "eventlog_severity_code" ]
    rename => [ "Channel", "eventlog_channel" ]
    rename => [ "SourceName", "eventlog_program" ]
    rename => [ "SourceModuleName", "nxlog_input" ]
    rename => [ "Category", "eventlog_category" ]
    rename => [ "EventID", "eventlog_id" ]
    rename => [ "RecordNumber", "eventlog_record_number" ]
    rename => [ "ProcessID", "eventlog_pid" ]
}
mutate {  
    # Remove redundant fields
    remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
}

Now for one of my favorite things to do with Logstash. I use the mutate filter add some tags to specific Event Log messages because they're ones I care about. In this case, events related to Active Directory logins.

if [eventlog_id] == 4624 {  
    mutate {
        add_tag => [ "ad-logon-success" ]
    }
}
if [eventlog_id] == 4634 {  
    mutate {
        add_tag => [ "ad-logoff-success" ]
    }
}
if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {  
    mutate {
        add_tag => [ "ad-logon-failure" ]
    }
}
if [eventlog_id] == 4723 {  
    mutate {
        add_tag => [ "ad-password-change" ]
    }
}
if [eventlog_id] == 4724 {  
    mutate {
        add_tag => [ "ad-password-reset" ]
    }
}

Another cool feature, metrics:

if "ad-logon-success" in [tags] {  
    metrics {
        add_tag => [ "drop", "metric", "ad-logon-success" ]
        meter => "ad-logon-success-metric"
    }
}
if "ad-logon-failure" in [tags] {  
    metrics {
        add_tag => [ "drop", "metric", "ad-logon-failure" ]
        meter => "ad-logon-failure-metric"
    }
}

The metrics filter keeps a count of each time the filter is invoked, and periodically generates events containing one minute, five minute, and 15 minute averages of the rate of occurence of the relevant event. I add the "drop" tag because I don't want the generated events to be stored in elasticsearch (more on that later), the "metric" tag because, well, it's a metric, and the "ad-logon-" tag to indicate which events are counted in the metric. Metrics are one of the more buggy filters at the moment and should be placed as close to the bottom of the filter section as possible.

Now, here's our completed "eventlog" section:

if [type] == "eventlog" {  
    # Incoming Windows Event logs from nxlog
    mutate {
        # Lowercase some values that are always in uppercase
        lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
    }
    mutate {
        # Set source to what the message says
        rename => [ "Hostname", "@source_host" ]
    }
    date {
        # Convert timestamp from integer in UTC
        match => [ "EventReceivedTime", "UNIX" ]
    }
    mutate {
        # Rename some fields into something more useful
        rename => [ "Message", "@message" ]
        rename => [ "Severity", "eventlog_severity" ]
        rename => [ "SeverityValue", "eventlog_severity_code" ]
        rename => [ "Channel", "eventlog_channel" ]
        rename => [ "SourceName", "eventlog_program" ]
        rename => [ "SourceModuleName", "nxlog_input" ]
        rename => [ "Category", "eventlog_category" ]
        rename => [ "EventID", "eventlog_id" ]
        rename => [ "RecordNumber", "eventlog_record_number" ]
        rename => [ "ProcessID", "eventlog_pid" ]
    }
    mutate {
        # Remove redundant fields
        remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
    }
    if [eventlog_id] == 4624 {
        mutate {
            add_tag => [ "ad-logon-success" ]
        }
    }
    if [eventlog_id] == 4634 {
        mutate {
            add_tag => [ "ad-logoff-success" ]
        }
    }
    if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
        mutate {
            add_tag => [ "ad-logon-failure" ]
        }
    }
    if [eventlog_id] == 4723 {
        mutate {
            add_tag => [ "ad-password-change" ]
        }
    }
    if [eventlog_id] == 4724 {
        mutate {
            add_tag => [ "ad-password-reset" ]
        }
    }
    if "ad-logon-success" in [tags] {
        metrics {
            add_tag => [ "drop", "metric", "ad-logon-success" ]
            meter => "ad-logon-success-metric"
        }
    }
    if "ad-logon-failure" in [tags] {
        metrics {
            add_tag => [ "drop", "metric", "ad-logon-failure" ]
            meter => "ad-logon-failure-metric"
        }
    }
}

IIS (Access) Logs

Here's what one of my IIS access logs looks like, it's in W3C format with all of the fields enabled.

2014-05-31 00:05:01 W3SVC6 SERVER-NAME a.a.a.a GET / - 443 - b.b.b.b HTTP/1.1 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/34.0.1847.131+Safari/537.36 - - www.company.com 200 0 0 11525 333 140  

Luckily for me, the nxlog config that I use to ship the IIS logs already turns all of the parts of the log into fields. So I don't have to do that parsing in Logstash.

First things first, some more personal preferences:

mutate {  
    replace => [ "@message", "%{hostname} %{verb} %{fqdn}%{request}%{querystring} %{httpversion} %{status} %{useragent}" ]
    replace => [ "@source_host", "server-name.ad.company.com" ]
    add_field => { "requesturl" => "%{fqdn}%{request}%{querystring}" }
}

Here I replace @message which starts out as the entire log (like the example above) with just the information that I want at a glance. The second replace sets @source_host to the server's FQDN like the Event Logs from the same server. (Quick Note: there's probably a better way to set this, i.e. not hard-coded, but we only have the one web server so this works well enough) Finally, I add the requesturl field which is the fqdn, request, and querystring fields combined. (Another Note: if querystring was empty in the actual request (like in the example), the field here will be set to "-". It doesn't bother me that much, so I haven't figured out how to fix it.)

Next up:

useragent {  
    source => "useragent"
}

As you might have guessed, the useragent filter decodes the User Agent string from the request (which is stored in the useragent field). It adds os, os_name, browser name, browser minor and major version fields. (Quick Note: it doesn't currently seem to report OS X properly)

And, finally:

geoip {  
    source => "clientip"
}

This magic geoip filter looks up GeoIP data for the IP in the (in my case) clientip field using the Maxmind GeoLite database which ships with Logstash. It adds a bunch of data about the IP's location data, so I'm going to refer you to the filter's docs for that.

Now the "iislog" section is complete! Here's what it looks like:

if [type] == "iislog"  
{
    mutate {
        replace => [ "@message", "%{hostname} %{verb} %{fqdn}%{request}%{querystring} %{httpversion} %{status} %{useragent}" ]
        replace => [ "@source_host", "server-name.ad.company.com" ]
        add_field => { "requesturl" => "%{fqdn}%{request}%{querystring}" }
    }
    useragent {
        source => "useragent"
    }
    geoip {
        source => "clientip"
    }
}

...There's one more thing

The last part of the filter section is another metrics filter, this time for keeping track of the total number of events processed by Logstash. (Note the lack of a conditional, so this metrics is triggered by all events.)

metrics {  
    meter => "events"
    add_tag => [ "drop", "metric", "events-metric" ]
}

And that's it! Here's the whole filter section:

filter {  
    if [type] == "syslog" {
        grok {
            match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{DATA:syslog_program}\[%{NUMBER:syslog_pid}\]\: %{GREEDYDATA:syslog_message}" }
            add_field => [ "received_at", "%{@timestamp}" ]
        }
        syslog_pri { }
        date {
            match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
        }
        mutate {
            replace => [ "@source_host", "firewall-name.ad.company.com" ]
            replace => [ "@message", "%{syslog_message}" ]
            remove  => [ "syslog_message", "syslog_timestamp" ]
        }
        kv {
            source => "@message"
        }
    }

    if [type] == "eventlog" {
        # Incoming Windows Event logs from nxlog
        mutate {
            # Lowercase some values that are always in uppercase
            lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]
        }
        mutate {
            # Set source to what the message says
            rename => [ "Hostname", "@source_host" ]
        }
        date {
            # Convert timestamp from integer in UTC
            match => [ "EventReceivedTime", "UNIX" ]
        }
        mutate {
            # Rename some fields into something more useful
            rename => [ "Message", "@message" ]
            rename => [ "Severity", "eventlog_severity" ]
            rename => [ "SeverityValue", "eventlog_severity_code" ]
            rename => [ "Channel", "eventlog_channel" ]
            rename => [ "SourceName", "eventlog_program" ]
            rename => [ "SourceModuleName", "nxlog_input" ]
            rename => [ "Category", "eventlog_category" ]
            rename => [ "EventID", "eventlog_id" ]
            rename => [ "RecordNumber", "eventlog_record_number" ]
            rename => [ "ProcessID", "eventlog_pid" ]
        }
        mutate {
            # Remove redundant fields
            remove => [ "SourceModuleType", "EventTimeWritten", "EventTime", "EventReceivedTime", "EventType" ]
        }
        if [eventlog_id] == 4624 {
            mutate {
                add_tag => [ "ad-logon-success" ]
            }
        }
        if [eventlog_id] == 4634 {
            mutate {
                add_tag => [ "ad-logoff-success" ]
            }
        }
        if [eventlog_id] == 4771 or [eventlog_id] == 4625 or [eventlog_id] == 4769 {
            mutate {
                add_tag => [ "ad-logon-failure" ]
            }
        }
        if [eventlog_id] == 4723 {
            mutate {
                add_tag => [ "ad-password-change" ]
            }
        }
        if [eventlog_id] == 4724 {
            mutate {
                add_tag => [ "ad-password-reset" ]
            }
        }
        if "ad-logon-success" in [tags] {
            metrics {
                add_tag => [ "drop", "metric", "ad-logon-success" ]
                meter => "ad-logon-success-metric"
            }
        }
        if "ad-logon-failure" in [tags] {
            metrics {
                add_tag => [ "drop", "metric", "ad-logon-failure" ]
                meter => "ad-logon-failure-metric"
            }
        }
    }

    if [type] == "iislog"
    {
        mutate {
            replace => [ "@message", "%{hostname} %{verb} %{fqdn}%{request}%{querystring} %{httpversion} %{status} %{useragent}" ]
            replace => [ "@source_host", "server-name.ad.company.com" ]
            add_field => { "requesturl" => "%{fqdn}%{request}%{querystring}" }
        }
        useragent {
            source => "useragent"
        }
        geoip {
            source => "clientip"
        }
    }

    metrics {
        meter => "events"
        add_tag => [ "drop", "metric", "events-metric" ]
    }
}

Outputs

Now we take all of that lovely log data and send it off to be stored permenantly. It's pretty easy, so here's the whole thing up front:

output {  
    if "drop" not in [tags] {
        elasticsearch { }
    }
}

And that's actually it! Ok, not really. But for my pretty basic setup, that really is it.

Now, keen readers might remember that all of the metrics have a "drop" tag which means that they aren't being stored in any way, and that's correct. Typically, you'd send the metrics from Logstash to something like Graphite either directly or through something like Statsd. In my testing VM, Graphite and Statsd were always a bit of a pain-point, either not working entirely, not working intermittently, or causing Logstash to crash randomly. So I haven't set these up on my production box. Also, it's non-trivial to do alerting off of data sent to Graphite. So I'm currently experimenting with using Zabbix to keep track of metrics and do alerting (along with Zabbix's usual functions).

All of that aside, and because I'm a nice guy, here's an example of sending the "events" metric to Graphite using Statsd and the statsd output:

statsd {  
    sender => "events"
    count => [ "rate.1m", "%{events.rate_1m}" ]
    count => [ "rate.5m", "%{events.rate_5m}" ]
    count => [ "rate.15m", "%{events.rate_15m}" ]
}

This will create three series in Graphite, events.rate.1m, events.rate.5m, and events.rate.15m.

And that really is it for this post. I'll try to keep this updated as my configuration evolves, and I'll probably eventually write a post about my Zabbix experiments.

I'm writing this guide as I set ELK up to capture Event Logs from some Windows Server 2008 boxes, and a Sophos UTM (Astaro) firewall. To capture the Event Logs, I'm using nxlog-ce to serialize the Event Logs to JSON and send them to Logstash.

I'm setting ELK up on an older Dell Precision T3400 workstation running a fresh install of Ubuntu Server 14.04 LTS.

Personal note: For whatever reason, Ubuntu still doesn't ship with htop. That's always the first thing that I install: sudo aptitiude install htop

Let's get started!

First, I'm going to add the repositories for Elasticsearch and Logstash. From the Elasticsearch site:
Download and install the Public Signing Key:

wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -  

NOTE: You probably won't see the password prompt for sudo, so just type your password and hit enter when wget looks done.

Edit /etc/apt/sources.list and add the Elasticsearch and Logstash repositories to the end of the file:

deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main  
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main  

Don't forget to run a quick sudo aptitude update after adding these new repositories!

Now to actually install some things!

To get started, I'm going to install Logstash, who's current version at the time I'm writing this is 1.4.1.

sudo aptitude install logstash  

Logstash (and Elasticsearch) requires Java, so this will take some time depending on your internet connection

Logstash requires specific versions of Elasticsearch (for the native Elasticsearch output), and for v1.4.1 that's ES 1.1.1. So I'll install that next:

sudo aptitude install elasticsearch=1.1.1  

As you might have noticed in the Aptitude output, Elasticsearch won't be started at boot, by default. To fix this, run their suggested command:

sudo update-rc.d elasticsearch defaults 95 10  

Next, I'm going to tell APT to not upgrade Logstash or Elasticsearch because there could be breaking changes between versions.

sudo aptitude hold elasticsearch logstash  

Now, a quick test to make sure that Elasticsearch was installed correctly and works. To do that, start Elasticsearch with sudo service elasticsearch start and run curl http://localhost:9200. Curl should output something like:

{
  "status" : 200,
  "name" : "Karthon the Quester",
  "version" : {
    "number" : "1.1.1",
    "build_hash" : "f1585f096d3f3985e73456debdc1a0745f512bbc",
    "build_timestamp" : "2014-04-16T14:27:12Z",
    "build_snapshot" : false,
    "lucene_version" : "4.7"
  },
  "tagline" : "You Know, for Search"
}

Last, but definitely not least, we need to install a web server to host Kibana. I'm going to be using nginx, so:

sudo aptitude install nginx  

Quick overview of important locations for files and what-not:

Elasticsearch:

• Binaries and stuff: /usr/share/elasticsearch
• Plugin manager: /usr/share/elasticsearch/bin/plugin
• Configuration: /etc/elasticsearch/elasticsearch.yml
• Data: /var/lib/elasticsearch/<cluster-name>

Logstash

• Binaries and stuff: /opt/logstash
• Configuration: /etc/logstash/conf.d
• Logs: /var/log/logstash

Kibana

Kibana is 'just' some html and javascript, so download the tarball and extract it to where you want to put it. I'm going to put it in /srv/www/kibana, so I had to make those directories and change their owner to www-data:

sudo mkdir -p /srv/www/kibana  
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz  
sudo tar xf kibana-3.1.0.tar.gz -C /srv/www/  
sudo chown -R www-data:www-data /srv/www/  

So, kibana will actually be extraced to /srv/www/kibana-3.1.0 and that's fine with me.

Configurations

Elasticsearch

ES has sane defaults that work well for Logstash, so I'm pretty much going to leave it alone. But, there are some plugins that I like to have installed:

Bigdesk:

In simple words bigdesk makes it very easy to see how your Elasticsearch cluster is doing. Just install it as an Elasticsearch plugin, download locally or run online from the web, then point it to the Elasticsearch node REST endpoint and have fun.

To install, run:

sudo /usr/share/elasticsearch/bin/plugin -install lukas-vlcek/bigdesk/2.4.0  

We want v2.4.0 for ES 1.1.1

Bigdesk will be accessible at http://your-es-host:9200/_plugin/bigdesk/

elasticsearch-head:

elasticsearch-head is a web front end for browsing and interacting with an Elastic Search cluster.

To install, run:

sudo /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head  

elasticsearch-head will be accessible at http://your-es-host:9200/_plugin/head/

Kibana

Kibana's defaults work great as well, so I won't be changing its configuration either. But I do have to tell nginx where to find and host Kibana.

/etc/nginx/sites-available/default

server {  
        listen 80 default_server;

        root /srv/www;
        index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name localhost;

        location / {
                try_files $uri $uri/ =404;
        }
        location /kibana {
                alias /srv/www/kibana-3.1.0/;
                try_files $uri $uri/ =404;
        }
}

Don't forget to make a copy of the actual default config, you might want it again later!
Then, a quick sudo service nginx reload will (should) have it working!

Logstash

This is the biggie! When logstash is started using its initscript, it'll simply check /etc/logstash/conf.d for configuration files and load them in. I highly recommend reading through the documentation on Logstash's website, especially the Configuration Overview page. It's pretty good and gives a good overview of how Logstash works, and how to configure it. However, I'll cover my actual Logstash configuration in another post.

Once you've written your configuration, you can test it by running /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/. Then, when the config-test succeeds, just run sudo service logstash start to get going, and tail -f /var/log/logstash/logstash.log to make sure that everything is hunky-dory (there might not be any output if everything is OK). As you might have noticed when running the config-test, Logstash takes some time to start up, so be patient!

All that's left is to head on over to Kibana, choose the included "Logstash Dashboard" and look at all your pretty logs!

Node.js: https://launchpad.net/~chris-lea/+archive/node.js/
Redis: https://launchpad.net/~chris-lea/+archive/redis-server

Adding them to the computer for use just involves running
sudo apt-add-repository ppa:chris-lea/nodejs for Node.js, and
sudo apt-add-repository ppa:chris-lea/redis-server for Redis.