Masterprocess

This module contains the default architecture for master process.

The main task for masterprocesses is to create and run the Frontier. Starting a master involves the following steps:

1. Bind to the configured ZeroMQ sockets
2. Start the management interface
3. Create the frontier
4. Start the master

Once the master is up and you have configured a settings.MASTER_CALLBACK, this method will be called before the master is really started, i.e. before the IOLoop.start() is called. This will allow you to insert Seed URLs, e.g.

spyder.masterprocess.create_frontier(settings, log_handler)

Create the frontier to use.

spyder.masterprocess.create_master_management(settings, zmq_context, io_loop)

Create the management interface for master processes.

spyder.masterprocess.main(settings)

Main method for master processes.

ZeroMQ Master

A ZeroMQ master, i.e. the producer of URIs.

class spyder.core.master.ZmqMaster(settings, identity, insocket, outsocket, mgmt, frontier, log_handler, log_level, io_loop)

This is the ZMQ Master implementation.

The master will send DataMessage object to the workers and receive the processed messages. Unknown links will then be added to the frontier.

close()

Close all open sockets.

finished()

Return true if all uris have been processed and the master is ready to be shut down.

shutdown()

Shutdown the master and notify the workers.

start()

Start the master.

stop()

Stop the master gracefully, i.e. stop sending more URIs that should get processed.