I'm curious how your engine decides to trigger rollbacks without explicit knowledge of which parts of the game state each input can actually affect. Naively rolling back on every missed input can and will be abused - in early versions of Street Fighter V players found out that while being comboed you could churn out inputs as fast as possible to cause rollbacks for your opponent, even though none of those inputs could actually do anything (this was made worse by other issues with that game's inability to keep opponent game clocks synced and constant one-sided rollbacks).
Yes, it's true Easel does rollback on every input. I may be able to improve this. One great thing about controlling the programming language is there are a lot more places I can hook into for things like this. For example, Easel does detect at compile time which types inputs are used somewhere in the codebase and only sends those across the network.
Easel constantly synchronises the clocks (there's an interesting algorithm for this which I will write up at some point). It also adaptively assigns two different kinds of delay to every client - command delay and display delay. Command delay is related to how much lag you are introducing into the game. Basically people take on their own lag. It can be different amounts for different people in the game. The display delay is where the rollback netcode kicks in. It keeps track of how much rollback your computer can handle imperceptibly. If your computer can't handle it, then you won't get as much rollback (and will just experience more input latency). But in either case, whatever number it picks, it should be smooth.
Theres some missing information here. A "rollback that doesn't do anything" shouldn't be noticeable to the user. It would only be noticeable if the game simulation can't keep up with the # of frames it's being asked to simulate. And in street fighter, the simulation is ridiculously simple. There should be no reason why street fighter wouldn't be able to "rollback and resimulate" dozens if not hundreds of times per frame. There's no way that game is CPU bound... Am I missing something?
I think you are underestimating the per frame computation cost or massively overestimating the base model PS4. Remember that all fighting games have to lock 60fps, 16ms is not a ton of time for the handoff from input interpreter (SF has to parse backwards over the past several frames to see if a forward input is completing a quarter circle, double quarter circle, dash, etc) > game state update tick > graphical and sfx update pipeline. The issue with clock syncing meant that the worse your machine was, the more your opponent would roll back and the bigger the rollback windows would be because every single frame of your inputs would come late.
Eventually PC players learned that they could "fix" this by alt-tabbing out of the game, taking away dedicated GPU processing so they could be the peer that was falling behind.
I can't really speak for Street Fighter V, but I can say that Easel would assign command delay to the person who is introducing it. So if a person is somehow forcing their inputs to arrive later than they should, they would be assigned more input latency so that the other players would be unaffected. Easel also has a lag spike protection system where the server will just forcibly reschedule inputs if they arrive really late for some reason.
Yeah, most of these are relatively solved problems, but they are nontrivial enough that one of the industry giants couldn't get it right on its first few tries.
I don't think I am underestimating anything. Think about how simple SFV's simulation is compared to Battlefield 4. Way more players, dynamic objects, huge map, players leaving and joining, shooting projectiles, etc. SFV's simulation is so basic, is fundamentally the same game as SF2 that ran on the super Nintendo.
As far as overestimating the base model PS4, I've shipped rollback multiplayer games that support 32 players that run on Android Web Browsers. I would love to see a frame profiler of SFV. I have years of experience optimizing multiplayer games to run on far worse platforms than the PS4. 16ms is a lot of time.
I bet I could make SFV's simulation run in less than a millisecond on an Android phone from Walmart. It couldn't render on that device, but it could run the raw simulation.
This was exactly the premise behind the multiplayer SDK I made[1] but I moved to a middle ground (after feedback from devs) so to support existing languages and game engines single player game devs are familiar with but still easier than many other options out there.
We have now scaled it to many millions of players and it has proved worthy!
Wow, playroom looks awesome! It is great that it lets people integrate with all the existing libraries like ThreeJS for example. Glad to hear it has found its audience!
Unfortunately, this is not compatible with the rollback netcode model. However, all the world state is stored in WASM linear memory and so it is not the most straightforward to decode.
The rollback netcode model is necessary to make the multiplayer invisible to the developer. The other client-server/state-synchronization approach which is used in other multiplayer games, while great in many ways, requires you to assign authorities to every entity and to send a remote procedure call when attempting to affect entities you do not control. Rollback netcode was the only way for me to achieve the primary mission.
I may look into supporting the client-server/state-synchronization multiplayer model in the future though, which would enable "need to know" style revelations. Given we already have a deterministic programming language it is not at all infeasible as a future project.
From my experience running a multiplayer game, I only had 1 in every 50000 players attempt to perform some kind of hack and it was faster and more reliable for me to shadow IP ban the players. That feature is built into Easel. There is also a replays system built-in which makes it easy for people to submit evidence of people hacking. This is the current pragmatic solution that I suggest.
> requires you to assign authorities to every entity and to send a remote procedure call when attempting to affect entities you do not control.
You can consider every object to be server authoritative and then only send inputs from the client to the server. Other clients don't need to know about other client inputs. They only need to see state changes from the server. Clients functionally only have authority over their own input. This is a simple model of state synchronization that doesn't have the extra complexity of having authority over random objects. In this model, you only do a rollback if the server state differs from your predicted representation of the state. Besides, your game tick should be fast enough that you can rollback and resimulate every frame multiple times anyways.
WASM-4 has a simiar multiplayer model. There's some things you still have to be careful with, like delaying screen transitions so you don't have a "you win" screen flash for a few frames when the other player didn't lose but their input hasn't gotten t o you yet.
Yes. I've made a dozen games using rollback and it really is magical. I built a strategy game in three weeks and didn't test it in multiplayer until the weekend of release. It Just Worked.
But it's not a total silver bullet from a UX perspective when rollbacks happen.
Showing a player dying and then come back to life and actually you're dead will absolutely happen. It's very weird if they were ragdolling.
If players don't have high inertia, like a platformer, they'll teleport around the place as you get the information that actually they aren't falling they jumped 100ms ago.
This is all fixable, but requires first class confirmation (i.e
has the player you shot been dead longer than the max rollback window), and hand tuned interpolation on critical entities.
Luckily, I'm sure it's possible to add them to this engine.
I'm curious as to why a custom programming language was designed if the system uses WASM anyway - which you can make deterministic.
I wrote my system in C# and it worked great if you Followed The Rules (eg you must use immutable data structures). But WASM would have been a big step up.
Great to meet another rollback fan! It sounds like you have had a lot of experience with it. I'm sure I can add those things which you have mentioned - that's why we're in the beta testing phase to figure out things like this!
Why did I make a custom programming language? Well, making multiplayer automatic was only half of my mission when creating Easel.
It's a bit of a long story but the modding tools for my previous game were surprisingly successful. It became the first experience of any form of coding for a lot of people. The tool used JSON, which might sound primitive, but actually if you look past the JSON what it was really doing was defining a hierarchical declarative language for behaviour. There's something magic about that shape which allowed first-time coders to tinker without much help or documentation. (I have many theories as to why, one of them is that the hierarchy eliminates a lot of indirection that you might see in normal game programming, which means you can just kind of look at it and figure it out without jumping around. Everything is direct and in-place.)
The thing that irked me for years was, the limitations of that modding language limited not just what could be made, but what people could learn. I kept wondering what would happen if people were presented with a programming language in the same shape, but with unlimited power. Could it lay down a path for non-coders to follow all the way until they became expert coders, almost accidentally? Easel is my attempt to marry that magical hierarchical-declarative style with imperative programming in order to make a powerful language that still is extremely accessible.
I hope that, the accessibility and power of the programming language, combined with its ability to make multiplayer games automatically, will make it a super engaging choice for a first programming language for many people. I would love to see it used in schools to teach programming.
Experienced developers are probably better off targeting WASM with Rust, but an easy on ramp to programming is definitely something that can justify a new language.
WASM-4 is great, I remember hearing about it when it was released. It is a similar model where it makes the multiplayer basically invisible to the programmer, which is cool :)
A few quick q's, since I'm working on a game with a hand-rolled rollback impl (we have state copying, so we can do some nice tricks):
- Is there anywhere we can follow you about the clock-sync trick? I'd definitely love to be notified
- On the adaptive delay, are there gameplay or rollback engine implications to variable delays? Seems somewhat "unfair" for a player to be penalised for poor network conditions, but maybe it's better than them teleporting around everywhere.
Good luck with the project! I'll hopefully have a fiddle around with it soon :)
I think maybe the Twitter might be the best place to follow for blog updates: https://x.com/MadeWithEasel - I will definitely be posting about every blog there once I get there.
On the adaptive delays: I have a multiplayer game which gets about 100 players a day and it has been interesting seeing how they have all reacted to various iterations of the netcode. The overarching thing I've learned is that latency is quite psychological, it's the difference between expectation and reality that matters. In other words, high latency doesn't necessary mean an unhappy player, if they are expecting the high latency.
First though, the amount of rollback is limited to what the player's device is able to handle (there's yet another algorithm I've made for collecting statistics and estimating this!) Some devices cannot handle any rollback at all and so unfortunately sometimes rollback netcode isn't solving anything for those players. I think these are the cases where the adaptive latency is more important.
We sometimes would have games where we have 3 people from the US playing happily together, and then 15 minutes later 1 person from South Korea would join, and the latency would jump up dramatically for everyone. The US players would feel the difference and become unhappy. The simplest way to explain what Easel does now is it places the server at the weighted-average midpoint between all the players. So in this case, you can imagine that the server started in the US, and the moved 1/4 of the way towards South Korea (since 1 out of 4 players are in South Korea). I have found this to be the most fair and the key thing is the players find it to be fair. It matches how they think the latency should be apportioned and so they are okay with it.
Recently though I added a feature which splits the world up into regions (it's more complicated than it sounds because the regions flex around the players a bit, see https://easel.games/docs/learn/multiplayer/regions). By default, you only play with players near your region, but you can switch into Roaming Mode and play with people all over the world. The trick here is, when the player chooses Roaming Mode, they are explicitly choosing high latency, which changes their expectations. When they get high latency, they expect it, and so they are happy. The funny thing is, the algorithm used to automatically assign them the same high latency in these situations but players didn't like it because they didn't have the choice.
Speaking of async coroutines, my belief is that they don't get used enough in other game engines because their lifetimes are not tied to anything - you have this danger where they can outlive their entities and crash your game.
Async coroutines in the way you are describing have terrible/unpredictable cache/memory access behaviour which leads to bad performance. Every time you switch coroutines you need load memory from (most likely) an unrelated region causing slowdowns.
Yes, I do get this. I made choices to prioritise an abstraction that, in my opinion, makes you more productive. It's not going to work for certain kinds of people or games.
One of my original motivations for creating Easel came from my experience playing (and making) webgames, which in general are coded in JavaScript (or TypeScript). I love webgames as a method of delivering multiplayer because the biggest problem is getting players, and I think the low-friction zero-download really helps with that. So this is the world I am trying to target. When I remade my old webgame in Easel, I found it to be many times more performant and am now able to target much lower spec devices. Not to mention, determinism is a non-issue now.
I get that some people are going to love Easel and some are going to hate it, and that's okay.
yep, Im not saying its a bad design or a dealbreaker it is probably find for the vast majority of games and can provide a nice api for people coming from web games.
How do you abstract away the multi tenancy for your developer/user? As in different rooms or instances. Or similar.
I'm asking because I'm building a visual development tool for web apps, and all such abstractions, as in computer powers to non developers, are interesting for me to see how they are done.
Yes, all the Easel games are sharing infrastructure, which is much more cost-effective for all of us. In general it's not really something that really affects players or developers.
When a game developer chooses to publish their game, they choose either a subdomain (e.g. https://pewpew.easel.games) or a subpath (https://easel.games/@alzarath/snake). It just takes one click and then they have a URL to share with other people.
When players request to join a multiplayer game, they are matched to other players who are playing the same game, who are close to them geographically, and who are wanting to play the same game mode with the same parameters. A single game may have many lobbies running at once.
To the server, all inputs have the same format, regardless of game, so it's really efficient and just churning through relaying the inputs from one player to another. Because determinism is guaranteed for Easel, the server does not actually do any simulation at all, it just relays inputs, so I don't currently have to manage issues with one "noisy" tenant overwhelming the server and degrading the experience for others.
Not really sure if any of this answers your question!
It still is. There's been attempts at porting the game to modern engines (Space Station 14 having the most progress), but SS13 is still alive and well.
/tg/station's github is very interesting, too. About a dozen pull requests every week adding random features and bug fixes from random contributors. The game is constantly in flux, but they manage to keep everything feeling the same.
As a formally very active BYOND community member (joined 2003), BYOND's heyday was probably 15 years ago. Space Station 13 is its last vestige of relevance unfortunately. I've always wished for a new/updated equivalent version of it to crop up but nothing has really filled the same niche. It used to be a pretty vibrant community of (very) amateur game developers.
Basically it is a fully contained multi-user game development environment created by two friends, Dan and Tom around 2000. It has its own language, DM (DreamMaker), roughly based on C and Python. It has a map editor and server software. You would write a game and then host it and then anyone else who had the BYOND client software could connect to the server and run it. The server acted as the source of truth and the clients were effectively dumb clients just rendering. It had limitations (frankly a lot of limitations), but for the time it ran pretty damn well for reasonably sized 2d top down games. While you could publish your game to the BYOND site, you could also run it will no connection to the centralized BYOND hub.
The secret sauce was the combined game dev environment and how simple it made networking. You basically didn't have to think about any of it because the total game state was run by a server the you or someone else would host. The client just had to download the interface.
The development environment was bare bones but the language was so simple that it didn't really matter. Over simplifying but basically movable objects were of type "mob" and background tiles were of type "turf". Both inherited from "atom". You could also write like 5 lines of code defining 1 mob and 1 turf and walk around in your new world with others online. You could then add "verbs" basically functions that the mob could take and "procs" which are just all other functions you'd want to write.
- Forum entry from 2000 by one of the creators https://web.archive.org/web/20250325092124/http://www.byond.com/forum/post/194515
- Getting started writing games post (formatting is all broken on this archived version but the text is there): https://web.archive.org/web/20190402034657/http://www.byond.com/forum/post/46230
- Interesting crash course on everything from 2023: https://www.youtube.com/watch?v=TzAzMtWa0u0
- Probably the biggest game that made it out of just cult following was NEStalgia: http://www.silkgames.com/nestalgia/
Yes, I think it is super important that Easel games last forever and that has always been the plan. My long term plan is to create a standalone version of Easel that you can run on your server forever, regardless of what happens to Easel itself.
I'm not yet sure of the details of how an escrow service might work, but honestly, I would be willing to look into it, that could be a good answer. I really do plan on Easel as a platform lasting forever. This is my life's work as much as it is yours.
It seems that Github's successor feature only allows your successor to re-assign ownership for your public repositories. I'm currently using a private repository so this sadly does not work for me. Otherwise I would grab this feature with both hands!
Online multiplayer games? AFAIK they’re all dead unless the server has been reverse engineered like WoW.
Offline games continue being playable only because of software/hardware emulation — it’s not a product of the game design/engine/language or anything else really.
This is cool, but its hard to make a game engine full featured enough to be worth developing for. Have you thought about making this an addon for another engine like Godot or Bevy? Godot in particular is a nice engine to develop on, but the multiplayer support could be improved.
You are right. I am insane. I've had many existential crises about this exact issue over the past 3 years.
I just had a very very particular idea of what I wanted to do and nothing else would do. Making automatic multiplayer was only half my mission. The other half was creating the perfect first programming language. (I talked about this in another comment: https://news.ycombinator.com/item?id=44001047 )
I would love for Easel to inspire other game engines about how multiplayer could or should be done. But I just can't personally do this any other way. It's like, sometimes I don't know if I chose this project or it chose me.
I'm curious how your engine decides to trigger rollbacks without explicit knowledge of which parts of the game state each input can actually affect. Naively rolling back on every missed input can and will be abused - in early versions of Street Fighter V players found out that while being comboed you could churn out inputs as fast as possible to cause rollbacks for your opponent, even though none of those inputs could actually do anything (this was made worse by other issues with that game's inability to keep opponent game clocks synced and constant one-sided rollbacks).
Yes, it's true Easel does rollback on every input. I may be able to improve this. One great thing about controlling the programming language is there are a lot more places I can hook into for things like this. For example, Easel does detect at compile time which types inputs are used somewhere in the codebase and only sends those across the network.
Easel constantly synchronises the clocks (there's an interesting algorithm for this which I will write up at some point). It also adaptively assigns two different kinds of delay to every client - command delay and display delay. Command delay is related to how much lag you are introducing into the game. Basically people take on their own lag. It can be different amounts for different people in the game. The display delay is where the rollback netcode kicks in. It keeps track of how much rollback your computer can handle imperceptibly. If your computer can't handle it, then you won't get as much rollback (and will just experience more input latency). But in either case, whatever number it picks, it should be smooth.
Theres some missing information here. A "rollback that doesn't do anything" shouldn't be noticeable to the user. It would only be noticeable if the game simulation can't keep up with the # of frames it's being asked to simulate. And in street fighter, the simulation is ridiculously simple. There should be no reason why street fighter wouldn't be able to "rollback and resimulate" dozens if not hundreds of times per frame. There's no way that game is CPU bound... Am I missing something?
I think you are underestimating the per frame computation cost or massively overestimating the base model PS4. Remember that all fighting games have to lock 60fps, 16ms is not a ton of time for the handoff from input interpreter (SF has to parse backwards over the past several frames to see if a forward input is completing a quarter circle, double quarter circle, dash, etc) > game state update tick > graphical and sfx update pipeline. The issue with clock syncing meant that the worse your machine was, the more your opponent would roll back and the bigger the rollback windows would be because every single frame of your inputs would come late.
Here's an example of the constant one sided rollback: https://www.youtube.com/watch?v=aSB_JlJK_Ks
and an example of how players were aware that mashing caused frequent rollbacks: https://youtu.be/_jpg-ZiE70c?t=105
Eventually PC players learned that they could "fix" this by alt-tabbing out of the game, taking away dedicated GPU processing so they could be the peer that was falling behind.
I can't really speak for Street Fighter V, but I can say that Easel would assign command delay to the person who is introducing it. So if a person is somehow forcing their inputs to arrive later than they should, they would be assigned more input latency so that the other players would be unaffected. Easel also has a lag spike protection system where the server will just forcibly reschedule inputs if they arrive really late for some reason.
Yeah, most of these are relatively solved problems, but they are nontrivial enough that one of the industry giants couldn't get it right on its first few tries.
I don't think I am underestimating anything. Think about how simple SFV's simulation is compared to Battlefield 4. Way more players, dynamic objects, huge map, players leaving and joining, shooting projectiles, etc. SFV's simulation is so basic, is fundamentally the same game as SF2 that ran on the super Nintendo.
As far as overestimating the base model PS4, I've shipped rollback multiplayer games that support 32 players that run on Android Web Browsers. I would love to see a frame profiler of SFV. I have years of experience optimizing multiplayer games to run on far worse platforms than the PS4. 16ms is a lot of time.
I bet I could make SFV's simulation run in less than a millisecond on an Android phone from Walmart. It couldn't render on that device, but it could run the raw simulation.
I wonder if SAT solvers could prove the maximum "reach" of inputs into the state...
Pretty cool work, kudos!
This was exactly the premise behind the multiplayer SDK I made[1] but I moved to a middle ground (after feedback from devs) so to support existing languages and game engines single player game devs are familiar with but still easier than many other options out there.
We have now scaled it to many millions of players and it has proved worthy!
1. https://joinplayroom.com
Wow, playroom looks awesome! It is great that it lets people integrate with all the existing libraries like ThreeJS for example. Glad to hear it has found its audience!
How do you handle hidden information, or "need to know"/"potentially visible set" style revelations to player clients?
Unfortunately, this is not compatible with the rollback netcode model. However, all the world state is stored in WASM linear memory and so it is not the most straightforward to decode.
The rollback netcode model is necessary to make the multiplayer invisible to the developer. The other client-server/state-synchronization approach which is used in other multiplayer games, while great in many ways, requires you to assign authorities to every entity and to send a remote procedure call when attempting to affect entities you do not control. Rollback netcode was the only way for me to achieve the primary mission.
I may look into supporting the client-server/state-synchronization multiplayer model in the future though, which would enable "need to know" style revelations. Given we already have a deterministic programming language it is not at all infeasible as a future project.
From my experience running a multiplayer game, I only had 1 in every 50000 players attempt to perform some kind of hack and it was faster and more reliable for me to shadow IP ban the players. That feature is built into Easel. There is also a replays system built-in which makes it easy for people to submit evidence of people hacking. This is the current pragmatic solution that I suggest.
> requires you to assign authorities to every entity and to send a remote procedure call when attempting to affect entities you do not control.
You can consider every object to be server authoritative and then only send inputs from the client to the server. Other clients don't need to know about other client inputs. They only need to see state changes from the server. Clients functionally only have authority over their own input. This is a simple model of state synchronization that doesn't have the extra complexity of having authority over random objects. In this model, you only do a rollback if the server state differs from your predicted representation of the state. Besides, your game tick should be fast enough that you can rollback and resimulate every frame multiple times anyways.
This is all very impressive and combines ideas I tinkered with in C on and off for years, into a cohesive product. Very impressive.
Oh thank you!
WASM-4 has a simiar multiplayer model. There's some things you still have to be careful with, like delaying screen transitions so you don't have a "you win" screen flash for a few frames when the other player didn't lose but their input hasn't gotten t o you yet.
Yes. I've made a dozen games using rollback and it really is magical. I built a strategy game in three weeks and didn't test it in multiplayer until the weekend of release. It Just Worked.
But it's not a total silver bullet from a UX perspective when rollbacks happen.
Showing a player dying and then come back to life and actually you're dead will absolutely happen. It's very weird if they were ragdolling.
If players don't have high inertia, like a platformer, they'll teleport around the place as you get the information that actually they aren't falling they jumped 100ms ago.
This is all fixable, but requires first class confirmation (i.e has the player you shot been dead longer than the max rollback window), and hand tuned interpolation on critical entities.
Luckily, I'm sure it's possible to add them to this engine.
I'm curious as to why a custom programming language was designed if the system uses WASM anyway - which you can make deterministic.
I wrote my system in C# and it worked great if you Followed The Rules (eg you must use immutable data structures). But WASM would have been a big step up.
Great to meet another rollback fan! It sounds like you have had a lot of experience with it. I'm sure I can add those things which you have mentioned - that's why we're in the beta testing phase to figure out things like this!
Why did I make a custom programming language? Well, making multiplayer automatic was only half of my mission when creating Easel.
It's a bit of a long story but the modding tools for my previous game were surprisingly successful. It became the first experience of any form of coding for a lot of people. The tool used JSON, which might sound primitive, but actually if you look past the JSON what it was really doing was defining a hierarchical declarative language for behaviour. There's something magic about that shape which allowed first-time coders to tinker without much help or documentation. (I have many theories as to why, one of them is that the hierarchy eliminates a lot of indirection that you might see in normal game programming, which means you can just kind of look at it and figure it out without jumping around. Everything is direct and in-place.)
The thing that irked me for years was, the limitations of that modding language limited not just what could be made, but what people could learn. I kept wondering what would happen if people were presented with a programming language in the same shape, but with unlimited power. Could it lay down a path for non-coders to follow all the way until they became expert coders, almost accidentally? Easel is my attempt to marry that magical hierarchical-declarative style with imperative programming in order to make a powerful language that still is extremely accessible.
I hope that, the accessibility and power of the programming language, combined with its ability to make multiplayer games automatically, will make it a super engaging choice for a first programming language for many people. I would love to see it used in schools to teach programming.
That makes sense to me!
Experienced developers are probably better off targeting WASM with Rust, but an easy on ramp to programming is definitely something that can justify a new language.
WASM-4 is great, I remember hearing about it when it was released. It is a similar model where it makes the multiplayer basically invisible to the programmer, which is cool :)
A few quick q's, since I'm working on a game with a hand-rolled rollback impl (we have state copying, so we can do some nice tricks):
- Is there anywhere we can follow you about the clock-sync trick? I'd definitely love to be notified - On the adaptive delay, are there gameplay or rollback engine implications to variable delays? Seems somewhat "unfair" for a player to be penalised for poor network conditions, but maybe it's better than them teleporting around everywhere.
Good luck with the project! I'll hopefully have a fiddle around with it soon :)
Sorry for the slow reply!
I think maybe the Twitter might be the best place to follow for blog updates: https://x.com/MadeWithEasel - I will definitely be posting about every blog there once I get there.
On the adaptive delays: I have a multiplayer game which gets about 100 players a day and it has been interesting seeing how they have all reacted to various iterations of the netcode. The overarching thing I've learned is that latency is quite psychological, it's the difference between expectation and reality that matters. In other words, high latency doesn't necessary mean an unhappy player, if they are expecting the high latency.
First though, the amount of rollback is limited to what the player's device is able to handle (there's yet another algorithm I've made for collecting statistics and estimating this!) Some devices cannot handle any rollback at all and so unfortunately sometimes rollback netcode isn't solving anything for those players. I think these are the cases where the adaptive latency is more important.
We sometimes would have games where we have 3 people from the US playing happily together, and then 15 minutes later 1 person from South Korea would join, and the latency would jump up dramatically for everyone. The US players would feel the difference and become unhappy. The simplest way to explain what Easel does now is it places the server at the weighted-average midpoint between all the players. So in this case, you can imagine that the server started in the US, and the moved 1/4 of the way towards South Korea (since 1 out of 4 players are in South Korea). I have found this to be the most fair and the key thing is the players find it to be fair. It matches how they think the latency should be apportioned and so they are okay with it.
Recently though I added a feature which splits the world up into regions (it's more complicated than it sounds because the regions flex around the players a bit, see https://easel.games/docs/learn/multiplayer/regions). By default, you only play with players near your region, but you can switch into Roaming Mode and play with people all over the world. The trick here is, when the player chooses Roaming Mode, they are explicitly choosing high latency, which changes their expectations. When they get high latency, they expect it, and so they are happy. The funny thing is, the algorithm used to automatically assign them the same high latency in these situations but players didn't like it because they didn't have the choice.
Speaking of async coroutines, my belief is that they don't get used enough in other game engines because their lifetimes are not tied to anything - you have this danger where they can outlive their entities and crash your game.
Async coroutines in the way you are describing have terrible/unpredictable cache/memory access behaviour which leads to bad performance. Every time you switch coroutines you need load memory from (most likely) an unrelated region causing slowdowns.
Yes, I do get this. I made choices to prioritise an abstraction that, in my opinion, makes you more productive. It's not going to work for certain kinds of people or games.
One of my original motivations for creating Easel came from my experience playing (and making) webgames, which in general are coded in JavaScript (or TypeScript). I love webgames as a method of delivering multiplayer because the biggest problem is getting players, and I think the low-friction zero-download really helps with that. So this is the world I am trying to target. When I remade my old webgame in Easel, I found it to be many times more performant and am now able to target much lower spec devices. Not to mention, determinism is a non-issue now.
I get that some people are going to love Easel and some are going to hate it, and that's okay.
yep, Im not saying its a bad design or a dealbreaker it is probably find for the vast majority of games and can provide a nice api for people coming from web games.
I exclusively make games that include online multiplayer, so this really caught my eye. Really looking forward to digging in.
Woohoo! I'm excited for you to dig in!
How do you abstract away the multi tenancy for your developer/user? As in different rooms or instances. Or similar. I'm asking because I'm building a visual development tool for web apps, and all such abstractions, as in computer powers to non developers, are interesting for me to see how they are done.
Yes, all the Easel games are sharing infrastructure, which is much more cost-effective for all of us. In general it's not really something that really affects players or developers.
When a game developer chooses to publish their game, they choose either a subdomain (e.g. https://pewpew.easel.games) or a subpath (https://easel.games/@alzarath/snake). It just takes one click and then they have a URL to share with other people.
When players request to join a multiplayer game, they are matched to other players who are playing the same game, who are close to them geographically, and who are wanting to play the same game mode with the same parameters. A single game may have many lobbies running at once.
To the server, all inputs have the same format, regardless of game, so it's really efficient and just churning through relaying the inputs from one player to another. Because determinism is guaranteed for Easel, the server does not actually do any simulation at all, it just relays inputs, so I don't currently have to manage issues with one "noisy" tenant overwhelming the server and degrading the experience for others.
Not really sure if any of this answers your question!
How does this compare to something like BYOND? I realize it’s dated now but conceptually there are some similarities.
I hadn't heard of BYOND until just then. Sadly its website seems to be down, I would like to learn more about it.
Space station 13 was the big game for it.
It still is. There's been attempts at porting the game to modern engines (Space Station 14 having the most progress), but SS13 is still alive and well.
/tg/station's github is very interesting, too. About a dozen pull requests every week adding random features and bug fixes from random contributors. The game is constantly in flux, but they manage to keep everything feeling the same.
https://github.com/tgstation/tgstation
As a formally very active BYOND community member (joined 2003), BYOND's heyday was probably 15 years ago. Space Station 13 is its last vestige of relevance unfortunately. I've always wished for a new/updated equivalent version of it to crop up but nothing has really filled the same niche. It used to be a pretty vibrant community of (very) amateur game developers.
It seems it is going through a DDOS attack right now. https://www.reddit.com/r/BYOND/comments/1kl4bjs/byond_hub_do...
Some history of BYOND here: https://tig.fandom.com/wiki/BYOND
Basically it is a fully contained multi-user game development environment created by two friends, Dan and Tom around 2000. It has its own language, DM (DreamMaker), roughly based on C and Python. It has a map editor and server software. You would write a game and then host it and then anyone else who had the BYOND client software could connect to the server and run it. The server acted as the source of truth and the clients were effectively dumb clients just rendering. It had limitations (frankly a lot of limitations), but for the time it ran pretty damn well for reasonably sized 2d top down games. While you could publish your game to the BYOND site, you could also run it will no connection to the centralized BYOND hub.
The secret sauce was the combined game dev environment and how simple it made networking. You basically didn't have to think about any of it because the total game state was run by a server the you or someone else would host. The client just had to download the interface.
The development environment was bare bones but the language was so simple that it didn't really matter. Over simplifying but basically movable objects were of type "mob" and background tiles were of type "turf". Both inherited from "atom". You could also write like 5 lines of code defining 1 mob and 1 turf and walk around in your new world with others online. You could then add "verbs" basically functions that the mob could take and "procs" which are just all other functions you'd want to write.
You can read the language reference here on internet archive while the main site is down: https://web.archive.org/web/20200229044355/http://www.byond....
Some other links:
If I created a game in this, what gurantee do I have that someone will be able to play this game in 100 years?
I think a chess board is more what you are looking for. :)
But seriously, it wouldn't hurt to have some kind of escrow service for products like this.
Yes, I think it is super important that Easel games last forever and that has always been the plan. My long term plan is to create a standalone version of Easel that you can run on your server forever, regardless of what happens to Easel itself.
I'm not yet sure of the details of how an escrow service might work, but honestly, I would be willing to look into it, that could be a good answer. I really do plan on Easel as a platform lasting forever. This is my life's work as much as it is yours.
I believe Github has tools built for leaving a successor in the event of the developers death thats relatively straight forwars.
It seems that Github's successor feature only allows your successor to re-assign ownership for your public repositories. I'm currently using a private repository so this sadly does not work for me. Otherwise I would grab this feature with both hands!
Consider just releasing it under a BSL license forbidding commercial hosting? If that works with your business model.
No, like 90% of video games could in theory be played 100 years from now.
If you can't answer that question, why should I trust your for 3 years? Or 10?
Here's a list of lost video games. Hard to prove it's complete of course. https://lostmediawiki.com/Category:Lost_video_games
FYI, the person you're replying to is not OP. Their answer is here: https://news.ycombinator.com/item?id=44000198
Online multiplayer games? AFAIK they’re all dead unless the server has been reverse engineered like WoW.
Offline games continue being playable only because of software/hardware emulation — it’s not a product of the game design/engine/language or anything else really.
For a long time most PC games came bundled with a server. Consoles being the exception, but many games were released on PC as well.
This is cool, but its hard to make a game engine full featured enough to be worth developing for. Have you thought about making this an addon for another engine like Godot or Bevy? Godot in particular is a nice engine to develop on, but the multiplayer support could be improved.
You are right. I am insane. I've had many existential crises about this exact issue over the past 3 years.
I just had a very very particular idea of what I wanted to do and nothing else would do. Making automatic multiplayer was only half my mission. The other half was creating the perfect first programming language. (I talked about this in another comment: https://news.ycombinator.com/item?id=44001047 )
I would love for Easel to inspire other game engines about how multiplayer could or should be done. But I just can't personally do this any other way. It's like, sometimes I don't know if I chose this project or it chose me.
[dead]