This seems to suffer a bit from the same problem that affects a lot of "new" robotics frameworks: the beginnings are simple, and it's easy at that stage to think that everything else is over-complicated.
The comparison with ROS 2 is a bit questionable. Comparing a single-process-only (Copper) approach using shared memory with a multi-process system (ROS 2) using default DDS settings really isn't comparing the same thing. There are ways to make the ROS 2 system much faster if you're willing to be limited to similar constraints (single process with components, or local-system shared-memory transport) but most people don't because those constraints are very limiting in practical applications.
Note that Copper can be distributed too, we do support iceoryx2 for example if you need to split it in 2 or more processes.
We have ~100x less latency and ~12x faster logging also because we have adopted a data oriented architecture: the tasks outputs are written back to back in memory, all the IOs are linear (we could log to a block device straight, we don't even need a filesystem).
I am not sure it is possible to touch this with ROS just because of its "everything is asynchronous" design pattern.
So the question is more about those limits in practical applications: do you have use cases where you absolutely need every single component deployed as a kind of micro service in a robot?
I will say the default network-transparency of ROS is probably more of a long-term hindrance than a help to applications built on top of it. A lot of projects would get a lot further if they were designed from the outset with clear boundaries where network communication was required (and what can be done to deal with what that implies), and it would be nice if components intended to be used in those systems were designed to be composed with that in mind, instead of requiring a network-based pub-sub system with not particularly great latency or reliability guarantees, even if it has a mode where those are tightened up a lot.
I disagree that the ROS2 comparison is questionable. If ROS want's to be seen as performant, they need to use a performant middleware by default. You shouldn't have to fiddle endlessly with it to get reasonable latencies.
Having worked on both large ROS systems and large monolithic robotics systems, the monolith wins hands down. The ROS paradigm that every node is it's own process is frankly insane.
.. was SOA about 10 years ago.
At this point any comparison to ROS1.0 is just strawman.
Nowadays nodelets and Ros2 eschew that approach to a larger degree. But you're right. Plenty of shops still use ROS1.0.
ROS was ... first, so they set the standard, and were AFAICT the introduction to distributed and message-passing systems for most roboticists coming out of school/Academia. They all graduated and had to relearn things over time (myself included). A plain-jane shared memory paradigm is just simpler and easier once you get a basic framework together for most things. There certainly are situations where you want process partitions, e.g., plug-and-play payloads. But even there, ROS plug-and-play is atrocious, since this is akin to a network drop, which it just doesn't handle. So, everyone just writes bridge nodes anyway.
Then don't get me started on multi-agent systems. At one point it was honestly believed that all agents would use a common Ros Master. Laughable!
While it may not be required anymore, one node per process still looks to me like the default paradigm in ROS2. If people are actually eschewing that approach, one would expect the ROS to update their docs to reflect these new best practices. Yet, their beginner tutorial and robot example still use that as the model as do many of the open source projects I have seen. I can't comment on how industry practices have evolved around ROS2, since my job still uses ROS1.
Tangent this got me thinking about: Why are the robotics and embedded communities generally separate, with little overlap in people and tech? For example, this project seems unrelated to the rust embedded tooling. And the components categories are also unfamiliar, e.g. the IMUs and ADCs in the readme, using a SBC vice a MCU etc.
Another angle: `ROS` and `RTOS` share letters, but not much else!
I gather that robotics is a fusion of embedded and mechanical engineering; I refer to the former.
I suppose this lends itself to a hybrid scenario. I suspect most mature robotics projects would have both types of computation. Likely bare metal or RTOS as the default, with delegation to GPOS computer for high bandwidth, and high computation tasks such as CV.
I also imagine distributed computing (for example: over CAN or another bus) would be a useful pattern.
Typically with robots you need more power than what embedded provides. We build on top of commodity x86 systems a lot of the time. Other times we use SOCs like the Nvidia Jetson line, which also doesn't require a lot of embedded skills; you get a full operating system and a userspace so it's more like programming on a typical unix-based system. You find more embedded in the automation side of things. Wearables, IOT, always connected devices -- things that are too small and low powered to afford an OS.
There's a lot of embedded skills in pretty much any hardware platform. Maybe this is selection bias from being an embedded-focused person in robotics, but there's plenty of jobs out there that need a toolchain wrangler who can deal with cost optimization.
I'm not saying skills you pick up doing embedded work aren't useful in robotics contexts. I'm saying roboticists often have the luxury of an OS, and they take advantage of it, so the worries you have doing embedded work without an OS are mitigated.
I would say tho there are different "kinds" of roboticists. Just as you have backend and frontend devs in the web world, you might consider people who work on hardware or software to be the robotics equivalent. Hardware people are going to be low level and working with embedded devices and may even program them. But higher up in the robotics stack most of that embedded know-how (toolchain wrangling as you put it) is much less important. The full stack robotics engineer has hardware and software knowledge/experience, but you don't usually see roles that ask for both, so people tend to specialize into one or the other.
OTOH the OS can very much get in the way. If you're using ROS for instance, it's easy to fall into apt package dependency hell - upgrading CUDA means upgrading the OS, which means upgrading ROS, which means upgrading all deps including OpenCV, etc, etc, etc.
that's the part I hate about ros and a whole lot of other libraries that shouldn't have been so tightly coupled - you should be able to run most of ros in one env and then for the performance critical or hardware facing stuff, run them each in their own env and have lightweight communication between them.
I think if something annoys me enough I'll end up using nanomsg or mqtt as a bridge, but it's a pain.
Xenomai or rt extensions to linux, basically the rtlinux legacy, where the os is a low priority thread while your latency critical code runs bare metal or scheduled out of band
My experience is that there's a major difference between "R&D project" mentality to mass production and big install bases. The embedded stuff really starts to leaking into everything during the latter once you start evaluating how much it'd cost to rewrite the high level stuff to fit cheaper hardware, or resolve the intermittent issues that big customer is complaining about.
It’s great to see more innovation in the robotics framework space, it’s sorely needed.
Deterministic log replay is a killer feature to have baked in from the start - many autonomous vehicle & robotics companies have invested years of effort to achieve that nirvana, while the most popular robotics framework today (ROS) makes it borderline impossible.
If this pics up support for like a open-library of assorted components and people shared them it has a lot of potential especially if the sim env and replays continues to be improved. I love the idea and looks gerat!
Nice release - deterministic replay isn't easy - I still need to go back and pull our implementation forward to match the new features in the core framework. I'm not sure if you've seen Basis, but we're somewhat "competing" in the same space. You're all in on Rust, we're C++ with bindings to other languages (eventually). I don't see this as a bad thing - more tools in the space is a win, eventually the industry will settle on something better than ROS.
Having maintained something similar myself, how has your experience been with exact determinism in C++? C++ compilers tend to be a bit of a free-for-all when it comes to bit determinism, whereas the rustc folks are very good about spending the design time to preemptively avoid many of the issues you'd need workarounds for in other languages.
My goal is determinism specifically around message timings/execution in replay. We don't currently advertise runtime determinism, we also don't promise fixes for determinism in your code. For message timing, the algorithm should be stable - it generates a timeline for when messages should be executed by each message handler, and will pause further executions to wait on handlers that "should have" finished, but haven't due to running too slow. Ties are broken by the handler name or data source name. I wrote a bit about it here https://basisrobotics.tech/2024/09/02/determinism/ but didn't go deep into the implementation. If your own code isn't deterministic you will have differences, but this eliminates the biggest classes of nondeterminism when running tests (threading timings, message handling races).
I don't think Copper claims to offer more than this either, but I can't speak for them.
Do I understand that at each tick, every task outputs a single message, and can depend on the outputs of other tasks? I.e., nodes and topics are merged into tasks, executed in topological order?
This reminds me, I've been messing around with Bevy recently and have been wondering if ECS systems might be suited to hardware. On the surface it seems like a good ergonomic fit. Not sure if ECS is a performance/determinism killer but since they're used in gaming where performance is often critical I imagine it's surmountable?
I believe that at least the initial version of ISAAC from Nvidia was based on an ECS. I don't think it is a determinism killer, this is often a well ordered pipeline within a game loop.
In terms of architecture though, it looks like it makes a ton of sense inside something like a tracker (ie the entity is basically a track) but for other parts like a vision pipeline, sensor fusion etc... I don't see how it helps.
Tell me if I am missing something.
Talking about Bevy, fitting Copper within Bevy to build this little simulation example happened super naturally: Copper is a System querying the Entities within the virtual world after each simulation tick.
As far as how it helps, I was mostly thinking from a dev ergonomics perspective. For things like ROS/mavlink it can sometimes be hard (for me) to think through how all the systems are interacting but for whatever reason ECSs feel like a natural way to think about systems with simultaneousish inputs and outputs.
I'm not an authority on the topic, but a somewhat similar concept to ECS is state charts, which I find far better suited to hardware.
Some might verbally assault me for calling these two things similar, but I mean that loosely. Someone who is new to both might notice that they compose systems which can communicate and respond to inputs, which is conceptually similar. State charts offer far more guarantees, consistency, reliability, and predictability though.
I suppose one is about propagating data (not state, specifically), the other is about state and state control. Both are hierarchical in a sense, but ECS doesn't place as much importance on hierarchy.
This seems to suffer a bit from the same problem that affects a lot of "new" robotics frameworks: the beginnings are simple, and it's easy at that stage to think that everything else is over-complicated.
The comparison with ROS 2 is a bit questionable. Comparing a single-process-only (Copper) approach using shared memory with a multi-process system (ROS 2) using default DDS settings really isn't comparing the same thing. There are ways to make the ROS 2 system much faster if you're willing to be limited to similar constraints (single process with components, or local-system shared-memory transport) but most people don't because those constraints are very limiting in practical applications.
Note that Copper can be distributed too, we do support iceoryx2 for example if you need to split it in 2 or more processes.
We have ~100x less latency and ~12x faster logging also because we have adopted a data oriented architecture: the tasks outputs are written back to back in memory, all the IOs are linear (we could log to a block device straight, we don't even need a filesystem). I am not sure it is possible to touch this with ROS just because of its "everything is asynchronous" design pattern.
So the question is more about those limits in practical applications: do you have use cases where you absolutely need every single component deployed as a kind of micro service in a robot?
I will say the default network-transparency of ROS is probably more of a long-term hindrance than a help to applications built on top of it. A lot of projects would get a lot further if they were designed from the outset with clear boundaries where network communication was required (and what can be done to deal with what that implies), and it would be nice if components intended to be used in those systems were designed to be composed with that in mind, instead of requiring a network-based pub-sub system with not particularly great latency or reliability guarantees, even if it has a mode where those are tightened up a lot.
I disagree that the ROS2 comparison is questionable. If ROS want's to be seen as performant, they need to use a performant middleware by default. You shouldn't have to fiddle endlessly with it to get reasonable latencies.
Having worked on both large ROS systems and large monolithic robotics systems, the monolith wins hands down. The ROS paradigm that every node is it's own process is frankly insane.
TBF
> every node is it's own process
.. was SOA about 10 years ago. At this point any comparison to ROS1.0 is just strawman.
Nowadays nodelets and Ros2 eschew that approach to a larger degree. But you're right. Plenty of shops still use ROS1.0.
ROS was ... first, so they set the standard, and were AFAICT the introduction to distributed and message-passing systems for most roboticists coming out of school/Academia. They all graduated and had to relearn things over time (myself included). A plain-jane shared memory paradigm is just simpler and easier once you get a basic framework together for most things. There certainly are situations where you want process partitions, e.g., plug-and-play payloads. But even there, ROS plug-and-play is atrocious, since this is akin to a network drop, which it just doesn't handle. So, everyone just writes bridge nodes anyway.
Then don't get me started on multi-agent systems. At one point it was honestly believed that all agents would use a common Ros Master. Laughable!
While it may not be required anymore, one node per process still looks to me like the default paradigm in ROS2. If people are actually eschewing that approach, one would expect the ROS to update their docs to reflect these new best practices. Yet, their beginner tutorial and robot example still use that as the model as do many of the open source projects I have seen. I can't comment on how industry practices have evolved around ROS2, since my job still uses ROS1.
Tangent this got me thinking about: Why are the robotics and embedded communities generally separate, with little overlap in people and tech? For example, this project seems unrelated to the rust embedded tooling. And the components categories are also unfamiliar, e.g. the IMUs and ADCs in the readme, using a SBC vice a MCU etc.
Another angle: `ROS` and `RTOS` share letters, but not much else!
I gather that robotics is a fusion of embedded and mechanical engineering; I refer to the former.
It almost feels like we would need an "in between" category.
If I paint a broad brush:
- embedded is all about latency, low bandwidth
- computers are all about bandwidth with often terrible latencies.
A modern robot needs both low latency and high bandwidth.
I suppose this lends itself to a hybrid scenario. I suspect most mature robotics projects would have both types of computation. Likely bare metal or RTOS as the default, with delegation to GPOS computer for high bandwidth, and high computation tasks such as CV.
I also imagine distributed computing (for example: over CAN or another bus) would be a useful pattern.
Typically with robots you need more power than what embedded provides. We build on top of commodity x86 systems a lot of the time. Other times we use SOCs like the Nvidia Jetson line, which also doesn't require a lot of embedded skills; you get a full operating system and a userspace so it's more like programming on a typical unix-based system. You find more embedded in the automation side of things. Wearables, IOT, always connected devices -- things that are too small and low powered to afford an OS.
There's a lot of embedded skills in pretty much any hardware platform. Maybe this is selection bias from being an embedded-focused person in robotics, but there's plenty of jobs out there that need a toolchain wrangler who can deal with cost optimization.
I'm not saying skills you pick up doing embedded work aren't useful in robotics contexts. I'm saying roboticists often have the luxury of an OS, and they take advantage of it, so the worries you have doing embedded work without an OS are mitigated.
I would say tho there are different "kinds" of roboticists. Just as you have backend and frontend devs in the web world, you might consider people who work on hardware or software to be the robotics equivalent. Hardware people are going to be low level and working with embedded devices and may even program them. But higher up in the robotics stack most of that embedded know-how (toolchain wrangling as you put it) is much less important. The full stack robotics engineer has hardware and software knowledge/experience, but you don't usually see roles that ask for both, so people tend to specialize into one or the other.
OTOH the OS can very much get in the way. If you're using ROS for instance, it's easy to fall into apt package dependency hell - upgrading CUDA means upgrading the OS, which means upgrading ROS, which means upgrading all deps including OpenCV, etc, etc, etc.
that's the part I hate about ros and a whole lot of other libraries that shouldn't have been so tightly coupled - you should be able to run most of ros in one env and then for the performance critical or hardware facing stuff, run them each in their own env and have lightweight communication between them.
I think if something annoys me enough I'll end up using nanomsg or mqtt as a bridge, but it's a pain.
Xenomai or rt extensions to linux, basically the rtlinux legacy, where the os is a low priority thread while your latency critical code runs bare metal or scheduled out of band
My experience is that there's a major difference between "R&D project" mentality to mass production and big install bases. The embedded stuff really starts to leaking into everything during the latter once you start evaluating how much it'd cost to rewrite the high level stuff to fit cheaper hardware, or resolve the intermittent issues that big customer is complaining about.
This is currently part of my $dayjob.
It’s great to see more innovation in the robotics framework space, it’s sorely needed.
Deterministic log replay is a killer feature to have baked in from the start - many autonomous vehicle & robotics companies have invested years of effort to achieve that nirvana, while the most popular robotics framework today (ROS) makes it borderline impossible.
True. All of it. https://edms.etas.com for example.
> Unlike a game engine we use a data oriented approach to minimize latency and maximize throughput.
Don't you mean "like"? I thought game engines were all about data oriented approach.
Indeed thanks. "Not unlike" :)
I know nothing about robotics.
Is deterministic log replay really a differentiating factor? My naive assumption would be that this is table stakes for pretty much any software.
ROS is probably the biggest robotics framework and doesn't have deterministic playback.
It's important for safety critical to be sure, but you can get surprisingly far without it.
If this pics up support for like a open-library of assorted components and people shared them it has a lot of potential especially if the sim env and replays continues to be improved. I love the idea and looks gerat!
Nice release - deterministic replay isn't easy - I still need to go back and pull our implementation forward to match the new features in the core framework. I'm not sure if you've seen Basis, but we're somewhat "competing" in the same space. You're all in on Rust, we're C++ with bindings to other languages (eventually). I don't see this as a bad thing - more tools in the space is a win, eventually the industry will settle on something better than ROS.
Having maintained something similar myself, how has your experience been with exact determinism in C++? C++ compilers tend to be a bit of a free-for-all when it comes to bit determinism, whereas the rustc folks are very good about spending the design time to preemptively avoid many of the issues you'd need workarounds for in other languages.
My goal is determinism specifically around message timings/execution in replay. We don't currently advertise runtime determinism, we also don't promise fixes for determinism in your code. For message timing, the algorithm should be stable - it generates a timeline for when messages should be executed by each message handler, and will pause further executions to wait on handlers that "should have" finished, but haven't due to running too slow. Ties are broken by the handler name or data source name. I wrote a bit about it here https://basisrobotics.tech/2024/09/02/determinism/ but didn't go deep into the implementation. If your own code isn't deterministic you will have differences, but this eliminates the biggest classes of nondeterminism when running tests (threading timings, message handling races).
I don't think Copper claims to offer more than this either, but I can't speak for them.
Do I understand that at each tick, every task outputs a single message, and can depend on the outputs of other tasks? I.e., nodes and topics are merged into tasks, executed in topological order?
Exactly.
Technically Copper is a compiler taking your graph with those constraints in mind and building a game loop out of it.
This reminds me, I've been messing around with Bevy recently and have been wondering if ECS systems might be suited to hardware. On the surface it seems like a good ergonomic fit. Not sure if ECS is a performance/determinism killer but since they're used in gaming where performance is often critical I imagine it's surmountable?
I believe that at least the initial version of ISAAC from Nvidia was based on an ECS. I don't think it is a determinism killer, this is often a well ordered pipeline within a game loop.
In terms of architecture though, it looks like it makes a ton of sense inside something like a tracker (ie the entity is basically a track) but for other parts like a vision pipeline, sensor fusion etc... I don't see how it helps.
Tell me if I am missing something.
Talking about Bevy, fitting Copper within Bevy to build this little simulation example happened super naturally: Copper is a System querying the Entities within the virtual world after each simulation tick.
Very cool about ISAAC. I didn't know.
As far as how it helps, I was mostly thinking from a dev ergonomics perspective. For things like ROS/mavlink it can sometimes be hard (for me) to think through how all the systems are interacting but for whatever reason ECSs feel like a natural way to think about systems with simultaneousish inputs and outputs.
I'm not an authority on the topic, but a somewhat similar concept to ECS is state charts, which I find far better suited to hardware.
Some might verbally assault me for calling these two things similar, but I mean that loosely. Someone who is new to both might notice that they compose systems which can communicate and respond to inputs, which is conceptually similar. State charts offer far more guarantees, consistency, reliability, and predictability though.
I suppose one is about propagating data (not state, specifically), the other is about state and state control. Both are hierarchical in a sense, but ECS doesn't place as much importance on hierarchy.
Apologies if I'm dead wrong.
Such a great two sentence intro to the project:
“ Copper is a user-friendly runtime engine for creating fast and reliable robots. Copper is to robots what a game engine is to games.”
Other projects should take notice.
Oh wow thanks, I have to tell you something... It is really hard to explain what a piece of middleware is and why it is important :)