Trainings

Monday, February 5, 2018

Use multiple logstash config file at a time on single command line or as a service

Problem Statement- having multiple logstash config file(As there is differet data configured in each file) for posting data from different machines in cluster which requires to open as many command line instances as number of config files. So is it possible to run all the config files from single instance or anything similar.

Answer: Either put all files in a directory and run Logstash with -f path/to/directory
or use multiple -f options that each point to one of the files i.e.
Command- logstash -f Sample1.conf -f Sample2.conf

Note: Keep in mind that Logstash has a single event pipeline (i.e. internally all configuration files concatenated and treated as a single big file.) and apply to all filters and outputs events unless you use conditionals to select how they apply..

need to use conditionals to select which filters and outputs to apply to which events.

filter {

if [type] == "event" {

# do stuff

}

else if [type] == "data" {

# do other stuff

}

}

OR

input {
file {
path => "BatchData\Batch_Raw_Data.csv"
tags => [ "batchdata" ]
start_position => "beginning"
}
}
output {
if "batchdata" in [tags]{
elasticsearch {
action => "index"
index => "IndexName"
}
}}

If you really need multiple event pipelines that can be stopped and restarted individually you will indeed have to run multiple instances of Logstash.