Managing storage

There is nothing required to manage xDP’s storage.

xDP persists data via a “durable queue”. Durable queues write events to files on disk using a “commit log”. This means that events are appended to a file. When a program subscribes to a queue aka “topic”, it reads events continuously from an offset within the file. Publishing is referred to as “heading” and consuming is referred to as “tailing”.

If a program had to tail from the beginning of a queue each time it started then it might take some time to consume to the head, particularly after a year or two or events. Programs therefore persist an offset to a queue so that they need not tail from the start. This offset is read from the head of a queue instead of the tail and therefore allows a program to resume where it left off very quickly.


There is no log compaction within xDP at present i.e. data is retained forever. This may at first seem daunting, but when sizing assumes 1,000 sensors of 3 types, each one sending observations 30 times per day, it would take over 300 years to fill up a 1TB USB storage device.

Another upside of not needing to purge data is that, from a security perspective, xDP retains a complete audit trail of every event that occurs within the system.