Basic Stream Processing
- RioDB receives a stream of data from one or many data producers. RioDB can either open a listening port or actively pull the data from a remote system (like Kafka subscription).
- RioDB continuously executes queries upon receiving each individual record. These queries may include computing custom formulas or aggregations such as AVERAGE, COUNT, SLOPE, MAX, MIN, MEDIAN, STD_DEV, etc.
- When a pattern of interest if found, RioDB automatically triggers a notification to another system, which may include a dashboard, or a cloud function, or database, or a notification system.
Sidecar Stream Processor
- A two-way traffic exists between clients and an application server (such as HTTP request & response).
- The application server fulfills requests, but reports a stream to RioDB (such as Apache/nginx access logs via UDP).
- RioDB queries the data continuously. When a pattern of interest is found, it automatically triggers a notification to another system, which may include a dashboard, or a cloud function, or database, or a notification system.
- It may result in a change to the application server context, such as adding a client IP to a blacklist, updating a list of most-popular products in an online store, or killing the session of a cheater in an online game.
Parallel Stream Processing
- Traffic is split (or mirrored) by a network collector or pub/sub system such as Kafka. A copy is consumed by your main application, and a fork of the traffic is sent to RioDB for additional real-time analysis. Your main application continues to do its thing regardless of what RioDB is doing with the data.
- RioDB queries the data continuously. When a pattern of interest is found, it automatically triggers a notification to another system, which may include a dashboard, or a cloud function, or database, or a notification system.
- Results may include changes to the context of your main application.
This is different from the Sidecar model since the traffic between clients and your main application is one-way, indicating a workflow. RioDB could obtain a fork of the traffic either before the workflow begins, or in-between workflow steps. RioDB will have no impact in the execution of one particular request through workflow, but through step #3, it can help set parameters and thresholds that govern the workflow system.
Traffic Filtering or Enrichment
- All one-way traffic is routed through RioDB as a proxy.
- RioDB runs queries upon receiving each record. These queries may include computing custom formulas or aggregations such as AVERAGE, COUNT, SLOPE, MAX, MIN, MEDIAN, STD_DEV and others.
- Records are then forwarded to their final destination. However, records are now augmented with statistical aggregations or formula results, which may be useful to the destination system. Records may also be filtered and forwarded only when a certain criteria is met.
Edge Collector
- You have data producers throughout the world. Instead of having all clients blasting stream data to a global application, you deploy local RioDB instances in each region. All local data producers send their data to the local RioDB instance first.
- Each RioDB instance examines the stream received from all local producers in real-time, and only sends relevant information to your global application. This can be more than just filtering individual records. It could compute aggregations and notify the main application with summarized events when certain threshholds are exceeded, or periodically.
Anomaly Detection with Machine Learning
- A stream of data is sent to a RioDB instance.
- RioDB (optionally enriching the data with aggregated values and formula results) forwards data to a Machine Learning system of your choice (Google BigQuery ML, ElasticSearch, Databricks… you pick).
- The ML system periodically examines the data, with the objective of dynamically identifying anomalies never before seen.
- The ML system issues a SQL statement into RioDB, which will instruct RioDB to scan the stream (#1) for a new type of anomaly going forward. Creating this SQL statement may involve human intervention, or be fully automated if your application is customized to generate SQL statements.
- RioDB now runs additional queries to scan the original data stream received from step #1, and when the “new” anomaly is detected, it automatically triggers a notification to another system, which may include a dashboard, or a cloud function, or database, or a notification system.
