Im developing something vaguely in the same space (tamper proof audit log) - (or several things since I also have a custom temporal object mapper for postgres); what is the advantage of this project? It doesnt seem sql-compliant (afaik bitemporal operations are part of the SQL-2011 standard), and what this could add -tamper detection - seems absent. Not criticizing, just trying to understand the use case.
It's first and foremost a document store / so for JSON or XML currently, that's why it's not SQL compliant. I think other query languages as JSONiq are much more tailored to this, albeit a niche of course (based on XQuery).
Regarding tamper proof audit logs not much is missing. Cryptographic hashes instead of XXH3, a commit hash chain and signed commits. Actually, I think that's a great addition with minimal changes needed.
What you can audit currently is "who changed what" for instance.
SQL was designed for tables and does this job indisputably well. However, even adding dots, lateral joins, and variants, it was not made for nested, heterogeneous data and reaches its limits with many levels of nestedness or high sparsity/extra fields (aka, denormalized data, aka semi-structured data).
The underlying constructs behind JSONiq and XQuery (which are 99% the same, differing only on the "JSON finish"), in particular the FLWOR expressions (which support pipe-syntax-like clauses natively), were designed in a W3C standardization working group by some of the same experts who also contributed to or edited SQL.
Besides, documents are split into fine granular nodes, so we have "no" upper limit besides running out of 48bit nodeKeys/node identifiers at some point maybe.
Furthermore, a (versioned, as always) path summary keeps track of all distinct paths, which is a key ingredient for the optional secondary indexes, which can index paths or paths and content among other stuff as simply indexing fields. Optionally you can also add indexes to speed up aggregate queries (which are basically column projections).
Furthermore, the whole storage can self-validate through checksums stored in parent pages up to the root as in ZFS for instance.
DeweyIDs can be optionally stored for each node, which can speed up the comparison of subtrees between revisions (in order to detect simply which changesets belong to a certain subtree without having to traverse ancestor chains). They lend themselves well for compression.
We even have importers which can identify based on heuristics minimal edit operations to import existing revisions of XML or JSON documents and to commit these with hopefully a minimal or close to minimal set of update operations, but it depends heavily on the data.
Im developing something vaguely in the same space (tamper proof audit log) - (or several things since I also have a custom temporal object mapper for postgres); what is the advantage of this project? It doesnt seem sql-compliant (afaik bitemporal operations are part of the SQL-2011 standard), and what this could add -tamper detection - seems absent. Not criticizing, just trying to understand the use case.
It's first and foremost a document store / so for JSON or XML currently, that's why it's not SQL compliant. I think other query languages as JSONiq are much more tailored to this, albeit a niche of course (based on XQuery).
Regarding tamper proof audit logs not much is missing. Cryptographic hashes instead of XXH3, a commit hash chain and signed commits. Actually, I think that's a great addition with minimal changes needed.
What you can audit currently is "who changed what" for instance.
+1 on Johannes
SQL was designed for tables and does this job indisputably well. However, even adding dots, lateral joins, and variants, it was not made for nested, heterogeneous data and reaches its limits with many levels of nestedness or high sparsity/extra fields (aka, denormalized data, aka semi-structured data).
The underlying constructs behind JSONiq and XQuery (which are 99% the same, differing only on the "JSON finish"), in particular the FLWOR expressions (which support pipe-syntax-like clauses natively), were designed in a W3C standardization working group by some of the same experts who also contributed to or edited SQL.
I explain the reasons why SQL is not appropriate for denormalized data in my MSc course at ETH: https://www.youtube.com/watch?v=WBe6MlCM9EY&list=PLs5KPrcFtb...
You will also find our VLDB conference paper on data independence for messy data here: https://www.vldb.org/pvldb/vol14/p498-muller.pdf
I hope it helps!
Oh, and Sirix optionally already stores a rolling merkle hash over the data.
Besides, documents are split into fine granular nodes, so we have "no" upper limit besides running out of 48bit nodeKeys/node identifiers at some point maybe.
Furthermore, a (versioned, as always) path summary keeps track of all distinct paths, which is a key ingredient for the optional secondary indexes, which can index paths or paths and content among other stuff as simply indexing fields. Optionally you can also add indexes to speed up aggregate queries (which are basically column projections).
Furthermore, the whole storage can self-validate through checksums stored in parent pages up to the root as in ZFS for instance.
DeweyIDs can be optionally stored for each node, which can speed up the comparison of subtrees between revisions (in order to detect simply which changesets belong to a certain subtree without having to traverse ancestor chains). They lend themselves well for compression.
We even have importers which can identify based on heuristics minimal edit operations to import existing revisions of XML or JSON documents and to commit these with hopefully a minimal or close to minimal set of update operations, but it depends heavily on the data.
Congratulations, Johannes!