Mouldwarp is currently under development. This section describes what Mouldwarp will look like rather than what it currently is.

Mouldwarp is the generic crawling component of Molescrape. To create a crawl job you have to define a start URL and the maximum depth to follow. The crawler will then start at your defined start URL and follow each link until the maximum depth has been reached.

To create a new crawl job run:

mouldwarp create-job my-job-name

This will create a crawl job that is only executed once. To create a recurrent job use:

mouldwarp create-job --recurrency 60 my-job-name

This job will be repeated each hour.