Trainings

Tuesday, February 6, 2018

sending Log file / CSV data to Elasticsearch without header row - PART I

Lets take an example , i am having below sample csv data

stock,open,close,date
icici,240,350,05-02-2014
sbi,140,250,05-02-2014
infy,950,1150,05-02-2014
tcs,2400,3500,05-02-2014

and when using below logstash config file "header row" is going to Standard output( we can replace this with elasticsearch . for demonstration purpose using stdout plugin)

logstash.conf

input {
file {
        path => "C:/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
output {
  stdout {
codec => rubydebug
   }
}

when running command "logstash -f logstash.conf" getting below output.

Now our requirement that we don't want to send "header row" i.e. "stock,open,close,date"
So lets modify our logstash.conf to below
input {
file {
        path => "C:/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
filter{
if [message] =~ /^stock/
{
drop {}
}
}
output {
  stdout {
codec => rubydebug
   }
}

so here we are using filter plugin where we are checking that if message contains "stock" string then drop that row. and boila see below output

So crux of this post is use filter plugin and check any of the column value and drop that row.
filter{
if [message] =~ /^stock/
{
drop {}
}
}

In my next post , i will walk through if i read this file as a csv i.e. comma delimited list of column in that case how we will ignore first row.