Introduction
In this article we look at how to automatically to automatically start a set of programs upon startup using ‘Supervisor’ process control tool for embedded
Typically in any embedded application we require a set of services to be running upon startup or boot into operating system.
sudo apt-get install supervisor
Supervisor is a client/server system that allows its users to control of processes on UNIX-like operating systems. It can be user to start/stop/restart/monitor a single or group of processes
Supervisor consists of following important components
- supervisord - Server
- supervisorctl - commandline client
- Web Server
Installing
If the Python interpreter you’re using has Setuptools installed, and the system has internet access, you can download and install supervisor in one step using easy_install.
easy_install supervisor
Depending on the permissions of your system’s Python, you might need to be the root user to install Supervisor successfully using easy_install.
Or you can download the supervisor package from http://pypi.python.org/pypi/supervisor and install it. After unpacking the downloaded software archive, run
python setup.py install
It will download and install all distributions depended upon by Supervisor and finally install Supervisor itself.
Starting supervisor upon startup
In Unix-based computer operating systems, init (short for initialization) is the first process started during booting of the computer system. Init is a daemon process that continues running until the system is shut down
Init scripts are the scripts located in /etc/init.d. These scripts are part of the bootup sequence of Ubuntu.
During boot, they are not called directly, but through a structure of symbolic links which manage the services which are to be started in a particular runlevel. The scripts which are symlinked from /etc/rcS.d are executed first. Then the scripts in /etc/rcN.d/ are executed.
The naming convention of symlinks are /etc/rc[L].d/[S/K][NN]name
-
L is the chosen runlevel (default 2)
runlevel S is one of the runlevels supported by init, namely, ` 0123456789S`
Runlevel 0 is used to halt the system and 6 to reboot the system and 2 is default runlevel ,1 is runlevel to boot into single user more
-
NN is the two-digit sequence number that determines where in the sequence init will run the scripts.
-
The K links are responsible for killing services and the S link for starting services upon entering the runlevel.
After installation we can find the file supervisor in the /etc/init.d/ directory indicating that it can be accessed as a service
root@prasad-TA790GX-A3:/media/sda2# ls -alrt /etc/rc2.d/S20supervisor lrwxrwxrwx 1 root root 20 Dec 5 13:01 /etc/rc2.d/S20supervisor -> ../init.d/supervisor root@prasad-TA790GX-A3:/media/sda2#
we can also see that symlink for supervisor is present in rc2.d directory.Runlevel 2 is default runlevel in Ubuntu OS.Indicating that the service will start upon boot.And through supervisor we can control various processes being managed by supervisor.This include starting a set of processes at boot and stopping a set of processes at shutdown/reboot.
####Creating a Configuration File Once the Supervisor installation has completed, run echo_supervisord_conf. This will print a “sample” Supervisor configuration file to your terminal’s stdout.
Once you see the file echoed to your terminal
echo_supervisord_conf > supervisord.conf.
Once you have a configuration file on your filesystem, you can begin modifying it to your liking.
####Adding a Program
To add a program, you’ll need to edit the supervisord.conf file. The section relevant to present article is ` [program:x] Section ` of configuration file .
The program section will define a program that is run and managed when you invoke the supervisord command.
The configuration file must contain one or more program sections in order for supervisord to know which programs it should start and control
The header value is composite value. It is the word “program”, followed directly by a colon, then the program name
[program:foo] command="/bin/echo %(program_name)s_%(process_num)02d " numprocs=1 process_name=%(program_name)s redirect_stderr: true autorestart: true
In the above example a header value of [program:foo] describes a program with the logical name of “foo”
Command
The command section value will contain the command that will be run when this program is started.
command line arguments can be passed to program .In the above configuration the command line arguments are predefined string expressions supported by supervisor
Now starting the supervisor service will create a file in the tmp directory
foo-stdout---supervisor-XXX.log and foo-stderr---supervisor-a35d4Z.log
The contents of /tmp/foo-stdout---supervisor-XXX.log will contain the output of echo command which is pushed onto the stdout
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
specified in the supervisord section of the configuration file defines where the log files will be created
foo_00 foo_00 foo_00 foo_00
Thus we can see that logical program name is foo and value assigned to process_num is a 2 digit decimal value 00 due to %(process_num)02d string literal expression.
The section also include autorestart parameter which If set to true will cause the process to be unconditionally restarted when it exits, without regard to its exit code.We can see that it will restart the above program a fixed number of times .However if the program has some fatal error due to which it restarts too many times,it will stop the restart process .
Since echo command terminates immediately,it will stop after some attempts.In present case after 4 attempts.We can observe the following lines in /tmp/supervisord.log file which indicate why program was terminated
2014-12-05 14:39:43,153 CRIT Supervisor running as root (no user in config file) 2014-12-05 14:39:43,155 INFO daemonizing the supervisord process 2014-12-05 14:39:43,157 INFO supervisord started with pid 5344 2014-12-05 14:39:44,161 INFO spawned: 'foo' with pid 5345 2014-12-05 14:39:44,172 INFO exited: foo (exit status 0; not expected) 2014-12-05 14:39:45,176 INFO spawned: 'foo' with pid 5346 2014-12-05 14:39:45,188 INFO exited: foo (exit status 0; not expected) 2014-12-05 14:39:47,193 INFO spawned: 'foo' with pid 5347 2014-12-05 14:39:47,206 INFO exited: foo (exit status 0; not expected) 2014-12-05 14:39:50,211 INFO spawned: 'foo' with pid 5349 2014-12-05 14:39:50,221 INFO exited: foo (exit status 0; not expected) 2014-12-05 14:39:51,222 INFO gave up: foo entered FATAL state, too many start retries too quickly
Multiprocess applications
A [program:x] section can also be used to represents a “homogeneous process group” to supervisor (as of 3.0). The members of the group are defined by the combination of the numprocs and process_name parameters in the configuration
[program:foo1] command=/bin/echo %(program_name)s_%(process_num)02d numprocs=3 process_name=%(program_name)s_%(process_num)02d redirect_stderr: true autorestart: false startretries=0
numprocs Supervisor will start as many instances of this program as named by numprocs.This can be used in case of multiprocess applications.
process_name
A Python string expression that is used to compose the supervisor process name for this process. This is mandatory if numproc section value >1.
The default value is :%(program_name)s
It can contain string expression values : group_name, host_node_name, process_num, program_name
Example
But for instance, if you have a [program:foo1] section with a numprocs of 3 and a process_name expression of %(program_name)s_%(process_num)02d, the “foo1” group will contain three processes, named foo1_00, foo1_01, and foo1_02 as seen below
-rw------- 1 root root 16 Dec 5 15:08 foo1_02-stdout---supervisor-AkgbT1.log -rw------- 1 root root 16 Dec 5 15:08 foo1_01-stdout---supervisor-LYqA26.log -rw------- 1 root root 16 Dec 5 15:08 foo1_00-stdout---supervisor-xRoT5o.log
The startretries=0 and autorestart=false cause the program to not be restarted upon exit.
This makes it possible to start a multiprocess application from a single program configuration
####Running supervisord
To start supervisord, run supervisord. The resulting process will daemonize itself and detach from the terminal. It keeps an operations log at supervisor.log file in the directory specified by configuration files
sudo supervisord -c "$HOME/supervisord.conf"
While starting the supervisord service using init.d scripts the command line arguments cannot be passed.In which case it searches the configuration file in default location /etc/supervisord.conf
Web Server
The various processes managed by the supervisor can be accessed by webservice.
[inet_http_server] ; inet (TCP) server disabled by default port=*:9001 ; (ip_address:port specifier, *:port for all iface) username=user ; (default is no username (open server)) password=123 ; (default is no password (open server))
The following section must be present in the supervisord.conf file so that webserver is started when supervisord service is started
The webservice can be accessed from the browser by accessing the url http://${hostname}:9001/
for example in the present case http://192.168.1.3:9001
The sample supervisord.conf files and sample script used in article can be found at tutorials/supervisor tree path of github pyVision repository