This is interesting! I understand the reference implementation of the client being in a specific language, since it kind of has to be to be runnable. But I’m a little off put by the selection of a Python-only serialization format as well. So to implement this in another language, both the Jinbase model concepts as well as the serializer/deserializer have to be implemented. Wouldn’t it have made more sense to use MessagePack or Protobuf or one of the other formats that already has ubiquitous language support? I understand you are the author of the Paradict format but it seems like a non-starter for wide adoption of Jinbase across languages.
Thank you for your comment ! Indeed, for wide adoption across languages, we will need to port at least Paradict as that is the format in which BLOBs are serialized.
Protobuf relies heavily on predefined schemas and this rigidity goes against the flexibility of Jinbase's schema-less philosophy.
MessagePack (or CBOR) seems more convincing but Paradict has some subtleties that I don't find there. For example, Paradict preserves UTC offsets [1], handles integer bases, allowing for the representation of integers in decimal, binary, octal, and hexadecimal formats, has an extension mechanism that I find more interesting, etc.
Soon I will be adding a command line interface to Jinbase. From the CLI one will be able to read and write any type of data and this will only be possible because Paradict has a twin text format. MessagePack, from what I know, started with 1:1 compatibility with JSON and then over time it added things that are not present in JSON, thus breaking the 1:1 compatibility with JSON.
If I understand Peter Naur's take on programming [2] correctly, Jinbase is a software idea that I'm trying to implement (bring to life) one iteration at a time, and for that I need to have some level of control over the components (like the serialization format) so that I can adjust things accordingly. For example, the Paradict binary format is originally intended to serialize and deserialize only dictionaries (P...dict), but I changed that detail so that Jinbase users can freely store other things than dictionaries.
Once the core idea is fully implemented, we will see how to reproduce it elsewhere, one contribution/compilation after another...
How does Jinbase handle concurrency? Since it’s built on SQLite, does it inherit SQLite’s locking mechanisms, or have you implemented additional features to manage concurrent access across the different data models?
Jinbase is designed to be thread-safe, ensuring it can be reliably used in a multithreaded context.
To interact with SQLite, Jinbase uses LiteDBC [1] an SQL interface compliant with the DB-API 2.0 specification described by PEP 249 [2], itself wrapping Python's sqlite3 [3] module for a more intuitive interface and multithreading support by default. I wrote for LiteDBC a stress test [4] involving concurrency with Asyncpal [5].
This is interesting! I understand the reference implementation of the client being in a specific language, since it kind of has to be to be runnable. But I’m a little off put by the selection of a Python-only serialization format as well. So to implement this in another language, both the Jinbase model concepts as well as the serializer/deserializer have to be implemented. Wouldn’t it have made more sense to use MessagePack or Protobuf or one of the other formats that already has ubiquitous language support? I understand you are the author of the Paradict format but it seems like a non-starter for wide adoption of Jinbase across languages.
Thank you for your comment ! Indeed, for wide adoption across languages, we will need to port at least Paradict as that is the format in which BLOBs are serialized.
Protobuf relies heavily on predefined schemas and this rigidity goes against the flexibility of Jinbase's schema-less philosophy.
MessagePack (or CBOR) seems more convincing but Paradict has some subtleties that I don't find there. For example, Paradict preserves UTC offsets [1], handles integer bases, allowing for the representation of integers in decimal, binary, octal, and hexadecimal formats, has an extension mechanism that I find more interesting, etc.
Soon I will be adding a command line interface to Jinbase. From the CLI one will be able to read and write any type of data and this will only be possible because Paradict has a twin text format. MessagePack, from what I know, started with 1:1 compatibility with JSON and then over time it added things that are not present in JSON, thus breaking the 1:1 compatibility with JSON.
If I understand Peter Naur's take on programming [2] correctly, Jinbase is a software idea that I'm trying to implement (bring to life) one iteration at a time, and for that I need to have some level of control over the components (like the serialization format) so that I can adjust things accordingly. For example, the Paradict binary format is originally intended to serialize and deserialize only dictionaries (P...dict), but I changed that detail so that Jinbase users can freely store other things than dictionaries.
Once the core idea is fully implemented, we will see how to reproduce it elsewhere, one contribution/compilation after another...
[1] https://codeblog.jonskeet.uk/2019/03/27/storing-utc-is-not-a...
[2] https://news.ycombinator.com/item?id=26027448
How does Jinbase handle concurrency? Since it’s built on SQLite, does it inherit SQLite’s locking mechanisms, or have you implemented additional features to manage concurrent access across the different data models?
Jinbase is designed to be thread-safe, ensuring it can be reliably used in a multithreaded context.
To interact with SQLite, Jinbase uses LiteDBC [1] an SQL interface compliant with the DB-API 2.0 specification described by PEP 249 [2], itself wrapping Python's sqlite3 [3] module for a more intuitive interface and multithreading support by default. I wrote for LiteDBC a stress test [4] involving concurrency with Asyncpal [5].
[1] https://github.com/pyrustic/litedbc
[2] https://peps.python.org/pep-0249/
[3] https://docs.python.org/3/library/sqlite3.html
[4] https://github.com/pyrustic/litedbc/blob/master/tests/test_s...
[5] https://news.ycombinator.com/item?id=41404020