Content Fetcher

Module for downloading content from the web.

TODO: document pycurls features, i.e. what it can download.

class spyder.processor.fetcher.FetchProcessor(settings, io_loop=None)[source]

A processing class for downloading all kinds of stuff from the web.

spyder.processor.fetcher.extract_info_from_response(response, msg)[source]

Extract the interesting information from a HTTPResponse.

spyder.processor.fetcher.handle_response(msg, out_stream)[source]

Decorator for the actual callback function that will extract interesting info and forward the response.

spyder.processor.fetcher.prepare_headers(msg)[source]

Construct the HTTPHeaders with all the necessary information for the request.

Previous topic

Workerprocess

Next topic

Link Extractors

This Page