I'm not sure I follow with the LGPL requirement.. how do you envision downstream users complying with the relinking requirement, particularly in cases where Sidequest is orchestrating jobs across tightly-coupled backend infrastructure?
AFAIK, the relinking requirement only applies if you distribute the software to someone.
If you’re running Sidequest entirely on your own infrastructure to orchestrate jobs across your backend, you’re not distributing the software at all, you’re providing a service. The tight coupling does not itself trigger extra obligations. What matters legally is distribution, not architecture.
Edgecase is if you give your software to a customer to run on their own servers (self‑hosted deployment/docker image shipped to customer). In those cases, you would need to allow them to replace Sidequest.js (ie, not obfuscating it away).
Someone more knowledgeable can correct me, if I'm wrong
Layman here also, but I think you are correct about GPL and LGPL (this case), but not for AGPL which adds a requirement that: "... If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source"
My guess is the 2-level separation. You WILL need to make some part in your system LGPL by using "Sidequest.js" tightly, but then expose very simple APIs (like /start /status etc..) that will stop the LGPL from "infecting" other parts, as they can replace the "linking" with anything that will support those simple APIs.
Which is a good way to get improvements other are doing that relate to your source code through LGPL's source exposing, while not forcing it everywhere (GPL case). Especially, for backend libs.
And since AGPL will essentially make it non viable for SaaS (as network separation won't do), LGPL is what left.
This is great! I sometimes have projects that could use SQLite, but they're just big enough to need a persistent job queue. Great to have an option for that.
I also like that there's a dashboard. That's really important when a project gets serious, and a surprising number of job queue libraries don't have admin tools.
Looks really interesting, especially with the dashboards.
Just to validate an idea here:
I’m using k8s (20+ services) and trying to stick to the 12factor design pattern by having all the baking/companion services, also like cron jobs deployed from within the service directory. Right now I’m using k8s cron services for cron jobs and log the steps and observe with DataDog. Using k8s cron services feels right somehow but the observability with DataDog not. So, would it make sense for me, to deploy a baking k8s web service running sidequest instead?
Well, Sidequest comes with a comprehensive built-in dashboard that provides real-time monitoring, job management, and performance analytics, so there's that.
You can probably deploy Sidequest a job processor for all your 20+ services. Each service can enqueue the jobs and you can run as many machines as you want with Sidequest just to run the jobs. You maintain the 12-factor principle - your services remain stateless and delegate scheduled work to Sidequest.
The only requirement here is that the scripts to be executed must be in the same path in all machines. As long as you follow that, you can deploy it and run your jobs :)
Really like your approach of using existing Postgres/MySQL instead of dragging in Redis. It feels genuinely drop-in, but still Sidekiq-class. I know it's a bit early to ask about production patterns, but I was curious: if the worker thread flood hits the same Postgres that serves the web API, how do the job-fetch queries avoid contending with OLTP traffic? Does Sidequest follow Oban's advisory-lock approach or use a different throttling strategy?
Sidequest uses transaction-level row locks (`SELECT ... FOR UPDATE SKIP LOCKED`) when fetching jobs. This means each worker thread only locks the specific job rows it’s about to process, minimizing lock contention and avoiding blocking other queries. This pattern is inspired by Oban’s advisory-lock approach, but instead of using explicit advisory locks, Sidequest relies on the database’s built-in row locking mechanism.
The only drawback is that Sidequest will require one or two connections to your DB. If you enqueue jobs from within other jobs, then each job that requests an enqueue will also connect to your DB (lazily connected upon request - if your job does not enqueue, no connection is created). However, you can configure the number of concurrent workers per queue and globally, so you control how much load Sidequest puts on your database.
Well, you don't need to worry. Sidequest only locks the rows it's gonna use, i.e., the job rows it manages. Any other data in your DB is not touched by Sidequest, so you should be safe :)
Nice to know SQLite will be useful.
Just be mindful that SQLite does not work well in highly concurrent systems. So, if you have a single job runner process, it will work wonderfully. However, if you have many workers, then you might see a few issues.
If you have heard of pg-boss
[1], how does sidequest compare to it? I’m about to embark on some « jobification » on some flows and I’d love to have your opinion (possibly biased, but still)!
Thanks for the question! I just checked out pg-boss. Solid library if you're fully on Postgres. Sidequest.js takes a broader and more flexible approach. Key differences:
Database agnostic: Sidequest isn't tied to Postgres. It also works with MySQL, MongoDB, and SQLite, which helps if your stack isn’t Postgres-based.
Job isolation: Jobs run in worker threads, so heavy tasks won’t block your main Node.js process. Express APIs stay responsive.
Distributed coordination: Designed to run across multiple instances. One node picks up the job, and if it fails, another can retry. This is built-in.
Built-in dashboard: Includes a web UI to inspect, retry, or cancel jobs.
More than queues: Supports cron jobs, uniqueness constraints, per-queue concurrency, and configuration. Some of this overlaps with pg-boss, but the intent behind Sidequest is to provide a complete solution for job processing.
If you just need simple queues in a Postgres-only setup, pg-boss is great. If you want more flexibility, tooling, and backend options, Sidequest may be a better fit.
I'm a big fan of ActiveJob in Rails. I was considering building a version inspired by it in Node but now it looks like I don't have to. Thank you for building this.
Excellent! I’m a .NET developer who dabbles in Node and have been looking for a Hangfire alternative for a while. This looks like just what I would want.
There are a few key differences. I plan to write a proper blog post about it soon.
The main conceptual one is that BullMQ enqueues plain objects, and your handler just receives that data. In Sidequest, you enqueue full class instances, so when the job runs you can call this.someMethod() directly inside run().
Sidequest also runs jobs in isolated worker threads out of the box, has a built-in dashboard in the OSS version, and works with Postgres, MySQL, SQLite, or MongoDB.
That said, I still want to explore BullMQ more deeply to write a complete comparison.
I'm not sure I follow with the LGPL requirement.. how do you envision downstream users complying with the relinking requirement, particularly in cases where Sidequest is orchestrating jobs across tightly-coupled backend infrastructure?
AFAIK, the relinking requirement only applies if you distribute the software to someone.
If you’re running Sidequest entirely on your own infrastructure to orchestrate jobs across your backend, you’re not distributing the software at all, you’re providing a service. The tight coupling does not itself trigger extra obligations. What matters legally is distribution, not architecture.
Edgecase is if you give your software to a customer to run on their own servers (self‑hosted deployment/docker image shipped to customer). In those cases, you would need to allow them to replace Sidequest.js (ie, not obfuscating it away).
Someone more knowledgeable can correct me, if I'm wrong
Layman here also, but I think you are correct about GPL and LGPL (this case), but not for AGPL which adds a requirement that: "... If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source"
https://www.gnu.org/licenses/gpl-faq.html#AGPLv3ServerAsUser
The AGPL license makes me so confused every time I read it. This "network effect" really is so deeply confusing.
My guess is the 2-level separation. You WILL need to make some part in your system LGPL by using "Sidequest.js" tightly, but then expose very simple APIs (like /start /status etc..) that will stop the LGPL from "infecting" other parts, as they can replace the "linking" with anything that will support those simple APIs.
Which is a good way to get improvements other are doing that relate to your source code through LGPL's source exposing, while not forcing it everywhere (GPL case). Especially, for backend libs.
And since AGPL will essentially make it non viable for SaaS (as network separation won't do), LGPL is what left.
This is great! I sometimes have projects that could use SQLite, but they're just big enough to need a persistent job queue. Great to have an option for that.
I also like that there's a dashboard. That's really important when a project gets serious, and a surprising number of job queue libraries don't have admin tools.
Looks really interesting, especially with the dashboards.
Just to validate an idea here: I’m using k8s (20+ services) and trying to stick to the 12factor design pattern by having all the baking/companion services, also like cron jobs deployed from within the service directory. Right now I’m using k8s cron services for cron jobs and log the steps and observe with DataDog. Using k8s cron services feels right somehow but the observability with DataDog not. So, would it make sense for me, to deploy a baking k8s web service running sidequest instead?
Well, Sidequest comes with a comprehensive built-in dashboard that provides real-time monitoring, job management, and performance analytics, so there's that.
You can probably deploy Sidequest a job processor for all your 20+ services. Each service can enqueue the jobs and you can run as many machines as you want with Sidequest just to run the jobs. You maintain the 12-factor principle - your services remain stateless and delegate scheduled work to Sidequest.
The only requirement here is that the scripts to be executed must be in the same path in all machines. As long as you follow that, you can deploy it and run your jobs :)
How does Sidequest compare to Graphile Worker https://worker.graphile.org/ ?
Interesting lib!
Graphile Worker: PostgreSQL only
Sidequest: Multiple backends (PostgreSQL, MySQL, SQLite, MongoDB)
Graphile Worker: No built-in dashboard - you need external monitoring
Sidequest: Comprehensive built-in web dashboard for job monitoring
Graphile Worker: Single queue with job prioritization
Sidequest: Multiple queues with individual: i) Concurrency limits; ii) Priority levels; iii) State management (active/paused); iv) Isolated workloads.
Graphile Worker: Direct PostgreSQL integration, very lightweight
Sidequest: Worker threads for non-blocking processing, more comprehensive job lifecycle management
Graphile Worker: Optimized for PostgreSQL performance (3ms latency)
Sidequest: Balanced performance with rich feature set
I hope that helps answering your question.
Really like your approach of using existing Postgres/MySQL instead of dragging in Redis. It feels genuinely drop-in, but still Sidekiq-class. I know it's a bit early to ask about production patterns, but I was curious: if the worker thread flood hits the same Postgres that serves the web API, how do the job-fetch queries avoid contending with OLTP traffic? Does Sidequest follow Oban's advisory-lock approach or use a different throttling strategy?
Hello! One of the creators of Sidequest here.
Great question!
Sidequest uses transaction-level row locks (`SELECT ... FOR UPDATE SKIP LOCKED`) when fetching jobs. This means each worker thread only locks the specific job rows it’s about to process, minimizing lock contention and avoiding blocking other queries. This pattern is inspired by Oban’s advisory-lock approach, but instead of using explicit advisory locks, Sidequest relies on the database’s built-in row locking mechanism.
The only drawback is that Sidequest will require one or two connections to your DB. If you enqueue jobs from within other jobs, then each job that requests an enqueue will also connect to your DB (lazily connected upon request - if your job does not enqueue, no connection is created). However, you can configure the number of concurrent workers per queue and globally, so you control how much load Sidequest puts on your database.
I hope that answers your question :)
Oban doesn't use advisory locks for fetching jobs (unless there is uniqueness involved)—it uses `FOR UPDATE SKIP LOCKED` as well to pull jobs.
Thanks for the clarification. That's a clean approach. I just stared your repo. Looking forward to seeing where sidequest.js goes :)
Ah!Nice, was looking for that as all are either redis or postgres (and we don't use postgres and prefer not to use redis). So thanks for that.
This looks like exactly what I need, but our DB is Postgres on Supabase and I don’t think their version supports transaction locking…
Well, you don't need to worry. Sidequest only locks the rows it's gonna use, i.e., the job rows it manages. Any other data in your DB is not touched by Sidequest, so you should be safe :)
https://github.com/timgit/pg-boss https://github.com/hatchet-dev/hatchet are both on top of Postgres, but I like the SQLite here for a project I have in mind.
Nice to know SQLite will be useful. Just be mindful that SQLite does not work well in highly concurrent systems. So, if you have a single job runner process, it will work wonderfully. However, if you have many workers, then you might see a few issues.
Looks really neat! Starred on GitHub!
If you have heard of pg-boss [1], how does sidequest compare to it? I’m about to embark on some « jobification » on some flows and I’d love to have your opinion (possibly biased, but still)!
[1] https://github.com/timgit/pg-boss
Thanks for the question! I just checked out pg-boss. Solid library if you're fully on Postgres. Sidequest.js takes a broader and more flexible approach. Key differences:
Database agnostic: Sidequest isn't tied to Postgres. It also works with MySQL, MongoDB, and SQLite, which helps if your stack isn’t Postgres-based.
Job isolation: Jobs run in worker threads, so heavy tasks won’t block your main Node.js process. Express APIs stay responsive.
Distributed coordination: Designed to run across multiple instances. One node picks up the job, and if it fails, another can retry. This is built-in.
Built-in dashboard: Includes a web UI to inspect, retry, or cancel jobs.
More than queues: Supports cron jobs, uniqueness constraints, per-queue concurrency, and configuration. Some of this overlaps with pg-boss, but the intent behind Sidequest is to provide a complete solution for job processing.
If you just need simple queues in a Postgres-only setup, pg-boss is great. If you want more flexibility, tooling, and backend options, Sidequest may be a better fit.
Thanks for the thorough reply!
I'm all in with Postgres, but the job isolation + built-in dashboard seem really appealing. I'll definitely give it a try!
Keep up the great work, love to see such high quality codebase / documentation / tooling!
does this work with cloudflare workers? I'm assuming if its using worker threads it might be some node specific API?
or at least the putting jobs into a queue side running within cloudflare?
I'm a big fan of ActiveJob in Rails. I was considering building a version inspired by it in Node but now it looks like I don't have to. Thank you for building this.
Excellent! I’m a .NET developer who dabbles in Node and have been looking for a Hangfire alternative for a while. This looks like just what I would want.
Hi, how would this compare to something like BullMQ (apart from it using your exising DB instead of i.e. redis).
There are a few key differences. I plan to write a proper blog post about it soon.
The main conceptual one is that BullMQ enqueues plain objects, and your handler just receives that data. In Sidequest, you enqueue full class instances, so when the job runs you can call this.someMethod() directly inside run().
Sidequest also runs jobs in isolated worker threads out of the box, has a built-in dashboard in the OSS version, and works with Postgres, MySQL, SQLite, or MongoDB.
That said, I still want to explore BullMQ more deeply to write a complete comparison.
This looks really cool.
This is awesome. Nice and simple. Thanks.