Master Slave Communication

This part of Python Testing Infrastructure talks about how master sends job requests to the slave and how slave picks up job request

Queue

The communication between master and slave is done using AMQP protocol implemented by RabbitMQ server. The Queue is asynchronous and persistent(i.e job requests will still remain in the queue even if Master shutdowns)

Master

Master processes the PyPI request into meaningful job requests so that slave can be able to read it and execute it easily. So master looks up at setup.cfg file of the PyPI project and writes a config which consists of the project name, project version, platform to be run on, platform variants (which means what does the VM require for the project to run , for eg some projects may need MySQL server) and recipe identifier(which defines what tests have to be conducted on the project)

Master then converts this config into JSON format and puts it in the Queue::

>>> from pyti.dispatcher.dispatcher import Queue, Serializer
>>> s = Serializer('demo_project', '1.0', 'debian', None, 'recipe')
>>> data = s.serialize_data()
>>> q = Queue('demo')
>>> q.send(data)

Slave

The slave side is more sophisticated than the master. A slave can have multiple VMs of multiple platforms. So each time it looks up in the Queue for wanting a job, it has to take care of VMs of different platforms running in the slave. So when slave requires jobs, it looks up in the Queue if there are available jobs and once it gets the job request , it has to check if VMs for the platform to be tested on are busy or not. If they are busy, slave releases the job request back to the Queue:

>>> from pyti.slave.manager import Queue
>>> q = Queue('demo')
>>> q.listen()