Tick architecture: simplicity and speed, the kdb+ way | KX

Skip to content

Free trial

Book a demo

Search

Search for:

Search

Tick architecture: simplicity and speed, the kdb+ way

Financial services

Developer

kdb+

Author

Laura Kerr

Community Management Lead

Published

25 June, 2025

Reading Time

Developer

Key Takeaways

Start your own kdb+ real-time data capture architecture in 5 minutes, no prior experience needed!

Get a clear overview of each core process, what it does, how it works, and how they all connect.

Build a fast, scalable, and resilient tick system for real-time and historical analytics using a single stack.

If you’ve ever worked on a production data platform, you know the drill. To accomplish anything meaningful, you could be expected to know half a dozen systems, one for ingestion, another for streaming, and sometimes several for storage, and that’s just the basic stack. At times, it can feel more like navigating an obstacle course than handling data.

But what if you could do more by doing less? What if you could learn how to set up a full tick architecture that handles data front to back, without stitching together multiple components or context-switching between tools?

In this blog, I’m going to show you how quickly you can set up a simple but powerful tick architecture using q, kdb’s vector functional language.  I will demonstrate how each q process in the kdb+ tick architecture has a distinct role, and how they are designed to work together. Don’t worry if you’re new to kdb+/q; we’ll be following the kdb+ architecture course outline, which has been created for complete beginners.

If you would like to follow along, you can either launch a Sandbox from the kdb+ architecture course or on your own environment via git clone https://github.com/KxSystems/kdb-architecture-course/tree/master. This, however, will require kdb+ to be installed with a valid license and the q alias set to invoke kdb+.

Within the repository, you will see a list of scripts, each designed to set up a specific process. You will also need to open a new terminal window per process you wish to start.

Tickerplant

The tickerplant has three integral roles in a kdb+ system:

Publish the data it receives to all subscribers.

Log every tick of data into a TP log in case of outage or system failure to ensure no data loss.

Kick off the end-of-day procedure for the whole system.

The tickerplant is the time-keeper of the roll over (end of day process) of the system, and the guarantor of data integrity and resilience. To create a tickerplant, launch a q process, loading in the tick.q script:

q tick.q sym . -p 5010

The tickerplant process loads the sym.q file, which contains the schema definitions for the entire system. This way, you define the schemas you need once and use them across the architecture, leaving no room for mismatch errors during data ingestion.

The sym.q file looks like this:

trade:([]time:`timespan$();sym:`g#`symbol$();price:`float$();side:`symbol$());

Setting the port number (using -p 5010 in this example) when starting each process allows the components within the system to communicate via Interprocess communication (IPC). The tickerplant has an inbuilt function .u.w , which will show how many processes have subscribed to it, so at any given time you can check your system is still connected and running as it should.

Real-time database (RDB)

The RDB exists entirely in memory; and serves a transitional role in the pipeline:

It receives and stores the intra-day data in memory until the end-of-day procedure has run.

It supports fast ad-hoc queries on in-memory data.

At end-of-day, it writes intra-day data to disk and signals the HDB to reload once complete.

The RDB also allows graceful recovery using the TP logs after failure, avoiding data loss.

To create a real-time database, open a new terminal window and launch a q process reading in the rdb.q script:

q tick/rdb.q -p 5011

When the RDB starts up (using -p 5011 to set the port), it connects to the tickerplant to retrieve the schema of the tables it will receive and the location of the TP log file. Running tables[] on the RDB will show what tables the RDB has subscribed to:

tables[]<br>,'trade

The tables remain empty until the tickerplant starts to publish, which can be seen by querying the table:

Historical database (HDB)

The Historical Database is where all historical data, spanning days, weeks, or even years, is saved. It is designed to serve large-scale queries at speed, without requiring the entire dataset to be pulled into memory. Its columnar format and memory mapping allow it to scan disk-based data with astonishing efficiency. Attributes and enumerations (e.g. partitioned by data, sorted by time, grouped by symbol) optimize both storage and retrieval, returning queries in milliseconds, not minutes, even over years of data.

With thoughtful design, the HDB ensures performance doesn’t degrade as data accumulates.

To create...