Learn Managing Laravel Work Queues With Beanstalk and Supervisor on Ubuntu 16.04
Table of Contents
Beanstalk is a fast and simple work queue. It allows you to run time-consuming tasks asynchronously, such as sending emails, connecting to external APIs or processing images. By doing so, you will reduce your web app latency. Laravel provides out-of-the-box support for beanstalkd.
In this tutorial we will install beanstalkd, setup a demo Laravel application and manage the queue workers through Supervisor. The demo application will get a list of available OSes from the IT Web Services API and
randomly choose one.
https://www.itweb.services/tutorials/linux-guides/installing-and-configuring-supervisor-on-ubuntu-16-04″>Installing and Configuring Supervisor on Ubuntu 16.04
It is also assumed that you have SSH access to your IT Web Services instance.
Installing Beanstalk
The first step to take is to install beanstalkd.
sudo apt-get update
sudo apt-get install beanstalkd
Start the service.
sudo systemctl start beanstalkd
You may also enable the service to start upon system initialization.
sudo systemctl enable beanstalkd
Check the service status by issuing the following command.
sudo systemctl status beanstalkd
By default, beanstalkd listens on the port 11300. Beanstalk uses a simple text-based protocol described in it’s github repository. You can test it’s protocol by running telnet.
telnet localhost 11300
Write the following and hit ENTER.
list-tubes
You should see a list of tubes available on the server:
OK 14
---
- default
To close the connection simply type quit and then press ENTER.
Tubes in Beanstalk represent work queues. Beanstalk is composed basically by producers, consumers, jobs and tubes. Producers put jobs into a tube to be consumed (processed) by any number of consumers. Note that both producers and consumers are simply clients of the Beanstalk server and are totally independent of each other. In practical terms this means that by using Beanstalk you may produce your jobs in your PHP application and have it processed in a NodeJS app for example. Luckily, Laravel abstracts all of this and provides us with a very simple API to dispatch and handle jobs, as we will see next.
Sample project
To build our sample project, we first need to install its dependencies. We are going to install PHP and Composer.
sudo apt-get install php php-mbstring php-xml php-common php-zip composer
Now, create the project based on Laravel 5.5.
composer create-project --prefer-dist laravel/laravel itweb.services "5.5.*"
Next cd into the recently created folder. We will be working in this directory from now on. This folder may be referenced later as PROJECT_ROOT.
cd itweb.services/
To use Beanstalk in Laravel we need to install one more dependency, the PHP Beanstalk client.
composer require pda/pheanstalk ~3.0
We also need to create a Job representation. In Laravel, this is a class inside the app/Jobs folder. Laravel has a console command to help us create a Job class. Let’s create our sample Job.
php artisan make:job FindFavoriteOS
Update the app/Jobs/FindFavoriteOS.php file to the following.
<?php
namespace AppJobs;
use IlluminateBusQueueable;
use IlluminateQueueSerializesModels;
use IlluminateQueueInteractsWithQueue;
use IlluminateContractsQueueShouldQueue;
use IlluminateFoundationBusDispatchable;
use IlluminateSupportFacadesLog;
class FindFavoriteOS implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct()
{
//
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
$rawData = file_get_contents('https://api.itweb.services.com/v1/os/list');
$list = json_decode($rawData, true);
shuffle($list);
$key = array_rand($list);
$favorite = $list[$key];
Log::info('My Favorite OS is: ' . $favorite['name']);
}
}
The handle method is what will be effectively executed when the job is consumed. Here, we fetch data from the IT Web Services API, randomly pick an OS and write the chosen OS to the log file. The log file is located at storage/logs/laravel.log.
We have already installed Beanstalk and defined the Job to be executed by it. Now it is time to tell Laravel to use Beanstalk as the default work queue. Copy the default .env file provided by the Laravel installation.
cp .env.example .env
Now open the .env file and update the line where the queue driver is specified.
QUEUE_DRIVER=beanstalkd
Finally, generate an application key.
php artisan key:generate
We are now ready to dispatch jobs to the Beanstalk work queue.
Dispatching and running jobs
Dispatching a job in Laravel is quite simple. Update the routes/web.php file.
<?php
Route::get('/', function () {
for ($i = 0; $i < 50; $i++) {
AppJobsFindFavoriteOS::dispatch();
}
return '50 Jobs dispatched!';
});
Despite not being recommended to run the built-in provided server in production, we will be using it here for the sake of brevity.
php artisan serve --host 0.0.0.0 --port 8000
Now in a web browser, navigate to http://[itweb.services-instance-ip]:8000. You will see the following message.
50 Jobs dispatched!
Open a new SSH connection to your server and inside of our project root, execute the following.
php artisan queue:work --once
This is the expected output:
[2018-02-14 00:03:52] Processing: AppJobsFindFavoriteOS
[2018-02-14 00:03:53] Processed: AppJobsFindFavoriteOS
Confirm that the log was generated.
cat storage/logs/laravel.log
Configuring Supervisor
To avoid the need to manually process the queue, we will use supervisord. Create the following program configuration in /etc/supervisor/conf.d/itweb.services.conf.
[program:itweb.services]
process_name=%(program_name)s_%(process_num)02d
command=php [PROJECT_ROOT]/artisan queue:work
autostart=true
autorestart=true
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/worker.log
Notice that the right path to put the Supervisor configuration file will depend on your setup. Also, remember to replace [PROJECT_ROOT] with the full path to the PROJECT_ROOT on your system.
Here we are configuring Supervisor to automatically start processing the queue and, in case of the script breaking, to restart it. Also note that we are not instantiating a single worker, but 8 processes. You are free to instantiate however many processes you find necessary depending on your application.
To allow Supervisor to manage our worker, force it to reread its configuration.
sudo supervisorctl reread
sudo supervisorctl update
If the Supervisor service has not been started, you many need to start it.
sudo systemctl start supervisord
Now let’s see if the jobs are being processed.
tail -f storage/logs/laravel.log
On a web browser, navigate to http://[itweb.services-instance-ip]:8000. You will see the logs being generated on your console.
Conclusion
We have successfully configured a Laravel application to use Beanstalk as a work queue. Also, we demonstrated how to use Supervisor to manage workers.
Need help?
Do you need help setting up this on your own service?
Please contact us and we’ll provide you the best possible quote!