JobsΒΆ

Warning

Mouldwarp is currently under development. This section describes what Mouldwarp will look like rather than what it currently is.

Mouldwarp is the generic crawling component of Molescrape. To create a crawl job you have to define a start URL and the maximum depth to follow. The crawler will then start at your defined start URL and follow each link until the maximum depth has been reached.

To create a new crawl job run:

mouldwarp create-job my-job-name http://example.com/

This will create a crawl job that is only executed once. To create a recurrent job use:

mouldwarp create-job --recurrency 60 my-job-name http://example.com/

This job will be repeated each hour.