These last days I tried to use Sidekiq on different servers than the Rails app one.
First, What is Sidekiq?
Sidekiq is a
Simple, efficient background processing for Ruby.
With more words, it’s a library who allows you to create job classes. A job contains the code pieces you want to perform later. It’s a kind of code decentralization.
And in this test the goal is to try running the job in different server than the web server.
Why is it better to separate worker of web server?
Imagine you need to do processes, so you use memory and CPU. In this case that means you take web server resources. So your web app could be slower.
By using different servers, it uses separate server resources. If process needs time, it takes the time it needs. And you could eventually decide to use a bigger server later.
So it’s just easier to scale web and worker servers separately with you VPS or cloud hosting provider.
Test in pictures
To do this test I used Vagrant with 3 VMs. One for the Rails
web server and two others for Sidekiq named
The example is a simple snippets application.
1. Set data into form
This is the “homepage” and also the form to send the code of our snippet.
2. After data posted
After posted our snippet we could see the code. Currently the code is still raw. We have to wait the job is processed.
3. After worker processed
Now the worker processed the job. The code is full of colors. And we could see that the IP is a different one and the hostname as well.
4. Retry a new once
In my example I instantiate two worker servers. So I do others tests.
5. After worker processed on a different worker
In this new test the IP is not the same than the other time. It’s not the same worker server who processed the job.
We have the list of the Sidekiq processes who are working. The two (or more) workers are listed here :)
Let see with scheme how it looks for the servers configuration.
My current test with Vagrant
For the test I did with Vagrant, it looked like this.
So we have the
web server who also contains the Redis server and the Rails app. Because this is a test, I run the application in development mode with Rack.
The application is exactly the same used for the 3 servers. So I used an Sqlite as a file for simplicity.
workers it uses the same Redis server and database. And I run the Sidekiq root process as a daemon.
So, what about a production configuration? It could be something like this.
And have separate
web servers only contain the application with the http server. If you want multiple
web servers, you have to install a proxy server as HAProxy.
If you want to test it, all the code is available on GitHub.
After clone or download you only have to install Vagrant (if you haven’t yet) and just
Have fun !