Thanks, that is definitely a downside to the shift operator overloading approach. I'll take that onboard and investigate whether a single operator to handle both would mesh with the current design.
You can just use the boost.pfr technique to iterate fields though.
Or if you want, starting from C++26 and e.g. clang-21: https://gcc.godbolt.org/z/G1TqP3a8P
I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant.
EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I guess I'm always thinking about having it run on client machines.
Thanks. The documentation could definitely be fleshed out with some more examples.
You'd likely want to always use that API (or layer something on top of it) unless you're in control of both ends and know they were built with the same toolchain & settings. One area where I've skipped over it is by writing a basic code gen tool (albeit unfinished as most personal projects) that generates the serialisation functions at compile-time from a very basic DSL that describes the network structures (of a game protocol I don't control). If it detects that the current toolchain is going to generate a binary-compatible struct layout and there aren't any variable length fields in there (no strings, basically), it'll generate a memcpy (via using get/put on the stream) rather than per-field (de)serialisation. If it can guarantee alignment of the buffer, which is a tougher requirement to meet, it'll give you a view directly into the network buffer so you effectively have zero-overhead deserialisation. Very much a work in progress but there's scope for making things quite efficient with just a few basic building blocks.
I know it's a convention since the inception of the language, but the operator overload abuse of the bitshift operator still makes me sad every time I see it :(
You are not alone. many on the standard committee are trying to get rid of it. std::print is the new way to do io instead of cout in part so you don't have to abuse shift for io. This is new in c++23 though so few people know about it.
Bjarne appears to prefer cout though, so it isn't universal.
With more complex structures, you need to specify how it should behave. The definition for 'more complex' here is basically no virtual functions, virtual base classes, is trivially copyable and constructible and a few others.
Basically, if it seems like memcpying the structure might be a reasonable thing to do, it'll work. This is why types like std::array will work but std::vector and std::string won't. It can handle those types when inserted individually but not in aggregate since there's no reflection.
The compiler barf does tell the user why it was rejected but... average C++ errors, even with concepts. Not the greatest.
main.cpp:136:52: note: the expression ‘is_trivial_v [with T = UserPacket]’ evaluated to ‘false’
136 | concept pod = std::is_standard_layout_v<T> && std::is_trivial_v<T>;
I'll likely add additional functionality for specifying both operations with a single function since it's been mentioned a few times. Thanks for the repos.
By the way I looked through the code, and had to read about metaprogramming in C++. I wonder why is it so complicated? For example, why constraints like std::is_integral are represented by structs. Doesn't make much sense to me. A function wouldn't be better here?
While this is true, you can do so much these days with functions with 'auto' return types (function templates), constexpr functions/lambdas and "if constexpr"
Sure, auto constexpr stuff can express some things. Not most things though, at least in my experience. Perhaps a skill issue on my part. Or things might have changed again. I'm "still" using C++20 after all.
> What does that mean
Have you ever noticed that the (compile time) "rules" for interacting with templated functions are somewhat different from those of non-templated functions? I don't know if "functions as types" is entirely fair but there is definitely some weirdness.
Practically, it's all through this `type_traits` header that (often) end up in unreadable messes. It's all possible because of the catchy acronym SFINAE. It doesn't make much sense to me either, so I avoid it :)
These days, whenever i read "headet only" i immediately get scared about compile times. Does using this library make compilation expensive in the way that eg protobuf or nlohmann_json do?
I don't use the amalgamated version, though (that only exists for this standalone version) and the library overall is significantly smaller than either of those and doesn't drag in nearly as many standard library headers.
This looks very cool. Based on the examples, you might like XDR.
It’s far better than the other binary serialization protocols I’ve looked at / implemented. NFSv3 uses it, and it is compatible with a lot of the tricks you play, like in-place endian translation, branch avoidance, zero allocation use cases, etc:
Agreed but just to clarify, Hexi sits at a level below something like MessagePack because it doesn't impose any particular encoding on you since the use-case was handling arbitrary binary protocols that you might not have any control over. You could build support for MessagePack on top of Hexi but not the other way around. They're all very different use-cases.
Correct but it doesn't claim to be zero-copy overall. Apologies if it was misleading. In the README, zero-copy is mentioned in context of using view() and span() to obtain views into the buffer, which does allow for it. Very much a "there be dragons but if you're sure..." feature but it's there as an option.
I've also built some tooling on top that makes use of those functions to do zero-copy deserialisation where viable, so it is possible in the right scenarios with a bit of work but it definitely isn't going to always fit.
incidentally, the block allocator implementation fails to properly account for alignment requirements:
since the underlying storage is std::array<char, ...>, it's alignment may be less that the required alignment of the requested type and that of the pointers being stored in the free list.
Your lib requires manually creating both a serializing and deserializing function. If the functions are out of sync, bad things happen.
Consider copying Cereal, which solves this problem by requiring you to create a single templated function ( https://uscilab.github.io/cereal/ )
Thanks, that is definitely a downside to the shift operator overloading approach. I'll take that onboard and investigate whether a single operator to handle both would mesh with the current design.
You can just use the boost.pfr technique to iterate fields though. Or if you want, starting from C++26 and e.g. clang-21: https://gcc.godbolt.org/z/G1TqP3a8P
or with boost.pfr as a polyfill until then, which allows to do this back to C++14like boost serialize, which overloads the & operator
Thanks again for this comment. Consider Cereal copied, now only a single function is required.
Wow that api looks fantastic! Bravo!
I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant.
EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I guess I'm always thinking about having it run on client machines.
Thanks. The documentation could definitely be fleshed out with some more examples.
You'd likely want to always use that API (or layer something on top of it) unless you're in control of both ends and know they were built with the same toolchain & settings. One area where I've skipped over it is by writing a basic code gen tool (albeit unfinished as most personal projects) that generates the serialisation functions at compile-time from a very basic DSL that describes the network structures (of a game protocol I don't control). If it detects that the current toolchain is going to generate a binary-compatible struct layout and there aren't any variable length fields in there (no strings, basically), it'll generate a memcpy (via using get/put on the stream) rather than per-field (de)serialisation. If it can guarantee alignment of the buffer, which is a tougher requirement to meet, it'll give you a view directly into the network buffer so you effectively have zero-overhead deserialisation. Very much a work in progress but there's scope for making things quite efficient with just a few basic building blocks.
That code-gen would be fantastic. I have commercial applications for this, so I'll keep an eye on your space.
I know it's a convention since the inception of the language, but the operator overload abuse of the bitshift operator still makes me sad every time I see it :(
You are not alone. many on the standard committee are trying to get rid of it. std::print is the new way to do io instead of cout in part so you don't have to abuse shift for io. This is new in c++23 though so few people know about it.
Bjarne appears to prefer cout though, so it isn't universal.
Danish Bjarne may have his < right next to lshift. He needs to use shift+. to enter colon.
On US layout colon is a single keypress but < is shift+.
This may explain the discrepancy.
—- from someone who read Bjarne at 16yo. All hail the Bjarne
On the plus side, it's optional. The same thing can be achieved with put()/get() equivalents.
>operator overload abuse
Array programming languages smugly enter the chat
What are the exact constraints on the struct contents, i.e. what is it that your library can't serialize?
I tried adding std::string to the UserPacket (from the README)
and the compilation fails - https://onlinegdb.com/B_RJd5UwsWith more complex structures, you need to specify how it should behave. The definition for 'more complex' here is basically no virtual functions, virtual base classes, is trivially copyable and constructible and a few others.
Basically, if it seems like memcpying the structure might be a reasonable thing to do, it'll work. This is why types like std::array will work but std::vector and std::string won't. It can handle those types when inserted individually but not in aggregate since there's no reflection.
The compiler barf does tell the user why it was rejected but... average C++ errors, even with concepts. Not the greatest.
main.cpp:136:52: note: the expression ‘is_trivial_v [with T = UserPacket]’ evaluated to ‘false’ 136 | concept pod = std::is_standard_layout_v<T> && std::is_trivial_v<T>;
In the same vein, but without needing to create separate de- and serialize functions:
https://github.com/eliasdaler/MetaStuff
Another take on the same idea with even simpler interface:
https://github.com/apankrat/cpp-serializer
I'll likely add additional functionality for specifying both operations with a single function since it's been mentioned a few times. Thanks for the repos.
By the way I looked through the code, and had to read about metaprogramming in C++. I wonder why is it so complicated? For example, why constraints like std::is_integral are represented by structs. Doesn't make much sense to me. A function wouldn't be better here?
Because the only way to do metaprogramming in C++ is via the type system. Thismakes it so you need to implement 'functions' as types.
While this is true, you can do so much these days with functions with 'auto' return types (function templates), constexpr functions/lambdas and "if constexpr"
What does that mean, and is it even true, given template value parameters or constexpr for example?
Sure, auto constexpr stuff can express some things. Not most things though, at least in my experience. Perhaps a skill issue on my part. Or things might have changed again. I'm "still" using C++20 after all.
> What does that mean
Have you ever noticed that the (compile time) "rules" for interacting with templated functions are somewhat different from those of non-templated functions? I don't know if "functions as types" is entirely fair but there is definitely some weirdness.
Practically, it's all through this `type_traits` header that (often) end up in unreadable messes. It's all possible because of the catchy acronym SFINAE. It doesn't make much sense to me either, so I avoid it :)
https://en.cppreference.com/w/cpp/language/sfinae
You don't really need to use sfinae anymore, concepts are cleaner and easier to follow, also this library appears to use concepts
Fun! It reminds me of my own attempt at this: https://github.com/louisabraham/ubuf
It can generate efficient JS and C++ from a simple YAML file.
These days, whenever i read "headet only" i immediately get scared about compile times. Does using this library make compilation expensive in the way that eg protobuf or nlohmann_json do?
I'm biased but in my experience, no, not at all.
I don't use the amalgamated version, though (that only exists for this standalone version) and the library overall is significantly smaller than either of those and doesn't drag in nearly as many standard library headers.
This looks very cool. Based on the examples, you might like XDR.
It’s far better than the other binary serialization protocols I’ve looked at / implemented. NFSv3 uses it, and it is compatible with a lot of the tricks you play, like in-place endian translation, branch avoidance, zero allocation use cases, etc:
https://www.rfc-editor.org/rfc/rfc1014
Thanks for the interesting link. :)
It's been a while since I saw a new library with such a clean interface. Congrats!
Thanks very much. :)
If you need schema-less serialiation there’s MessagePack.
But soon you’ll be bitten by the fact you don’t have a schema and so you’ll move to something like Protobuf or the more efficient FlatBuffers
Agreed but just to clarify, Hexi sits at a level below something like MessagePack because it doesn't impose any particular encoding on you since the use-case was handling arbitrary binary protocols that you might not have any control over. You could build support for MessagePack on top of Hexi but not the other way around. They're all very different use-cases.
Why .h for a CPP library and not .hpp? Threw me off as I usually expect .h to be associated with C files, opening it I find modern C++.
.h is the more common convention even in the C++ world.
Cool!! Did not know :)
It doesn't look like zero-copy though in this example:
That is at least one copy.Correct but it doesn't claim to be zero-copy overall. Apologies if it was misleading. In the README, zero-copy is mentioned in context of using view() and span() to obtain views into the buffer, which does allow for it. Very much a "there be dragons but if you're sure..." feature but it's there as an option.
I've also built some tooling on top that makes use of those functions to do zero-copy deserialisation where viable, so it is possible in the right scenarios with a bit of work but it definitely isn't going to always fit.
incidentally, the block allocator implementation fails to properly account for alignment requirements:
since the underlying storage is std::array<char, ...>, it's alignment may be less that the required alignment of the requested type and that of the pointers being stored in the free list.
Damn, the frog reminded me to unload my dishwasher which I really have to do
Lovely API, great work on that.
Semi off-topic, but I just love the header image and the advice frog in the readme. Makes reading the documentation more fun and enjoyable.
Thanks, that was the hope! :)
I would have liked a different froggy reaction for each section but the project budget was zero. :^)
The problem is that without image captions or subtitles, you lose accessibility for anyone with a screen reader.
I made sure to set the alt text for the images for that reason.