[ElasticStack-1] Logstash configure file form (based on 5.x ~ 6.x versions)

Logstash는 raw data를 Elasticsearch로 업로드하는 기능을 수행한다.

기본 구조는 다음과 같다.

(input, filter, output)

 

# Context is written in configure file (ex: test.conf)

input {
	file {
    	path => "FILE_PATH"									# can use asterisk (*) beside file name; e.g. test_1234.csv, test_23456.csv -> test_*.csv 
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
	csv {
    	separator => ","									# value type is string. default is ","
        columns => [ "COL1", "COL2", ... ,"COLN"]
        skip_empty_columns => true 							# default is false
        remove_field => [ "COL1", "COL2", ... ]  			# default value is []
        # default value of 'convert'is {}
        convert => {
        	"COL1" => "string"
            "COL2" => "integer"
            "COL3" => "float"
            "Lat_Info" => "float"
            "Lon_Info" => "float"
        }
	}

    date {
        match => [ "recordTime", "MMM dd yyyy HH:mm:ss" ]   # default is []
        target => [ "recordTime" ]                          # default value is "@timestamp"
    }

    mutate {
        rename => {
            "Lat_Info" => "[location][lat]"
            "Lon_Info" => "[location][lon]"
        }
    }

}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "elasticsearch_index"                      # Index naming rule is lowercase only
        template => "./sample.json"                         # template call by file path
    }

    stdout {
        codec => rubydebug
    }

    file {
        codec => rubydebug
        path => "./output.txt"                              # Create text file that lists Logstash log
    }

}

 

 

 

 

reference)

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-convert

https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

https://www.elastic.co/guide/en/logstash/current/configuration.html

https://www.elastic.co/guide/en/logstash/current/config-examples.html