Log Parser

The LOG parser plugin is used to take in a string and extract the fields when they originally resemble a system log line.

Parameters

pattern: The pattern of the log line. The Apache access log, for example* - * [*] "* *" * *.

Syntax

! and * are wildcards that represent a field to be extracted. The * is for wanted fields that map to a field in the RioDB stream, whereas ! is a wildcard for fields to be tossed (not used).

For example, consider this Apache access.log

10.10.10.10 - Scott [23/Aug/2019:21:17:34 +0000] "GET /news/excitingstuff.html HTTP/1.1" 200 559654

To extract the fields: ip_address, user, timestamp, method, message, response_code, response_size, we would use:
* - * [*] "* *" * *

But if we only want the ip_address and the method, the pattern would use the ‘!’ wildcard for the unwanted fields:
* - ! [!] "* !" ! !

PARSER log (
    pattern '<your pattern string>'
)

Example

To parse apache access log such as
10.10.10.10 - Scott [23/Aug/2019:21:17:34 +0000] "GET /news/excitingstuff.html HTTP/1.1" 200 559654

CREATE STREAM apache_access_log (
    ip STRING,
    user STRING,
    date TIMESTAMP 'dd/MMM/yyyy:HH:mm:ss Z',
    method STRING,
    message STRING,
    response_code NUMBER,
    response_size NUMBER
)
INPUT UDP(
    port 2334
)
PARSER LOG (
    pattern '* - * [*] "* *" * *'
);

Use the ‘!’ wildcard to ommit fields:

CREATE STREAM apache_access_log (
    ip STRING,
    date TIMESTAMP 'dd/MMM/yyyy:HH:mm:ss Z',
    method STRING,
    response_code NUMBER
)
INPUT UDP(
    port 2334
)
PARSER LOG (
    pattern '* - ! [*] "* !" * !'
);