Create Stream

The CREATE STREAM command is used for defining a stream in RioDB. This will basically serve as a data source that RioDB will use to capture stream traffic. It relies on an INPUT PLUGIN. The input plugin TYPE (TCP, UDP, HTTP, KAFKA, etc.) will start listening on ports or will do whatever is necessary to acquire the stream data for RioDB to process it.

Access Rule

Requires STREAM or ADMIN role.

Syntax

CREATE STREAM <stream name> (  # 1
   <field_name>  <data_type>,
   <field_name>  <data_type>,
   ...
   <field_name>  <data_type>
)  
INPUT <input plugin name> (    # 2
   <parameter>  <value>
   <parameter>  <value>
   ...
   <parameter>  <value>
)  
PARSER <parser plugin name> (  # 3
   <parameter>  <value>
   <parameter>  <value>
   ...
   <parameter>  <value>
);  

Assigning a field as the message TIMESTAMP:

CREATE STREAM <stream name> (  # 1
   <field_name>  <data_type> TIMESTAMP <optional FORMAT> ,
   <field_name>  <data_type>,
   ...
   <field_name>  <data_type>
)  
INPUT <input plugin name> (    # 2
   <parameter>  <value>
   <parameter>  <value>
   ...
   <parameter>  <value>
)  
PARSER <parser plugin name> (  # 3
   <parameter>  <value>
   <parameter>  <value>
   ...
   <parameter>  <value>
);  

Numbered items explained:

1. The stream must be given a name so that it can be referenced by queries. The name is alphanumeric, no spaces or special characters, and cannot be a reserved word. In parenthesis, you must specify the fields and datatypes.
   One of the fields can be specified as the TIMESTAMP field for time-sensitive windows. If none is specified, then messages will be tagged with the timestamp from the system clock, jeopardising time accuracy.
2. The stream must specify an INPUT plugin. This will configure RioDB to process incoming data. INPUT plugins can be HTTP, TCP, UDP etc.
3. The stream should optionally specify a PARSER plugin. By default, RioDB uses tab-delimited parser. If you need a different parser, then specify what parser plugin to use.

Note, an instance of RioDB can process multiple streams.

Examples

Create Stream using auto-assigned timestamp (system clock):
When a timestamp field is not specified, RioDB will assign a timestamp from the sytem clock, rounded to the second.
The reason it’s not rounded to nanosecond or millisecond is because network traffic delays and plugin lag make millisecond precision less relevant. If you require greater precision, consider having the stream producer (whatever is sending data to RioDB) include a true timestamp field in their outgoing data.

CREATE STREAM my_stock_stream (
  symbol   STRING,
  bid      NUMBER,
)
INPUT udp (
    port 5000
)
PARSER delimited (
    delimiter ','
)

Create Stream using a timestamp field provided by the incoming data:
If the incoming messages contain a timestamp field, then RioDB can use the timestamp from the message instead of the system clock.
The timestamp field in the incoming message can be a long number (epoch), or human-readable date format. When creating the stream, users can specify the date/time format. If a date/time format mask is not specified, then RioDB will assume that the data is in numeric format, in milliseconds since the epoch.

Example specifying the timestamp field, and it’s date format:

CREATE STREAM my_udp_stream (
  time             TIMESTAMP 'yyyy-mm-dd hh24:mm:ss',
  symbol           STRING,
  bid              NUMBER
)
INPUT udp(
    port 5000
)
PARSER delimited (
    delimiter ','
);

Example specifying the timestamp field, which is in LONG milliseconds since opoch:

CREATE STREAM stock_bids (
  time_epoch  TIMESTAMP,
  symbol      STRING,
  bid         NUMBER
)
INPUT udp(port 5000);

Note, again, if a PARSER plugin is not provided, RioDB deploys tab-delimited parsing by default.