Logo Search packages:      
Sourcecode: harvestman version File versions  Download package

HarvestMan::crawler::HarvestManBaseUrlCrawler Class Reference

Inheritance diagram for HarvestMan::crawler::HarvestManBaseUrlCrawler:

HarvestMan::crawler::HarvestManUrlCrawler HarvestMan::crawler::HarvestManUrlFetcher HarvestMan::crawler::HarvestManUrlDownloader HarvestMan::crawler::HarvestManUrlDownloader

List of all members.


Detailed Description

Base class to do the crawling and fetching of internet/intranet urls.
This is the base class with no actual code apart from the threading or
termination functions. 

Definition at line 41 of file crawler.py.


Public Member Functions

def __init__
def __str__
def action
def append_to_buffer
def crawl_url
def get_index
def get_role
def get_status
def get_status_string
def get_url
def get_url_object
def has_work
def is_locked
def process_url
def push_buffer
def run
def set_download_flag
def set_index
def set_role
def set_url
def set_url_object
def stop
def terminate

Public Attributes

 buffer

Private Member Functions

def _initialize

Private Attributes

 _configobj
 _crawlerqueue
 _download
 _endflag
 _index
 _isThread
 _loops
 _role
 _status
 _url
 _urlobject

The documentation for this class was generated from the following file:

Generated by  Doxygen 1.6.0   Back to index