Tantor PipelineDB

Main article: Database Management System (DMS)

2024: Porting to PostgreSQL (13, 15 and 16 versions)

On October 1, 2024, Tantor Labs announced a large-scale revision of the PipelineDB extension, a time series aggregation tool compatible with PostgreSQL and Tantor DBMS. The open source solution for continuous execution of SQL queries on data streams with incremental storage of results in tables has received support for new versions of the DBMS and a number of current functions.

PipelineDB is a PostgreSQL extension that allows you to quickly and flexibly, in real time, process streaming data, as well as combine it with historical data for comparison and analysis. With it, using only SQL queries, you can transfer data directly to databases and continuously transform them. PipelineDB stores only the output of continuous requests, which are gradually updated as data is received. In this case, the size of the database does not depend on the amount of data processed over time.

Historically, the solution was developed as an OpenSource project PipelineDB Core - a fork of PostgreSQL. In 2018, the project was transformed into an extension of PostgreSQL, which greatly simplified its use. However, it was closed in 2019. The last version of Postgres that PipelineDB worked with was the 11th.

Specialists "Tantor Labs" made great efforts to revive the project. They ported PipelineDB to PostgreSQL versions (13, 15, and 16), added a feature extension, and fixed many bugs.

As of October 2024, the basic functionality of Tantor PipelineDB includes continuous aggregation and filtering of stream data, queries with specified time intervals, merging streams with tables and support for probabilistic data structures for approximate calculations on streams (Bloom filters, count-min sketch, Filtered-Space-Saving top-k, HyperLogLog, and t-digest for approximate calculations on large streams).