https://github.com/andrefs/derzis
Tip revision: b735226ae5281132295cffd958274039b8111d9d authored by André Santos on 30 May 2021, 17:36:12 UTC
Extract jobs from Manager to CurrentJobs class
Extract jobs from Manager to CurrentJobs class
Tip revision: b735226
README.md
# derzis-crawler
A path-aware distributed linked data crawler.
## Run
The easiest way of deploying Derzis is using Docker, but it can also
be executed without using containers.
### Docker
#### Dependencies
* docker
* docker-compose
#### Running
* Edit `data/seeds.txt` and add the IRIs for the seed resources.
* Edit `common/config.js`, `worker/config.js` or `manager/config.js`
to modify the crawler parameters.
* Run `docker-compose up`.
### Locally
#### Dependencies
* MongoDB
* Redis
* Node.js v16
* Run `yarn install` in each of the folders `common`, `worker` and
`manager`.
#### Running
* Open a terminal and run `node manager/bin/manager.js`.
* Open another terminal and run `./worker/bin/worker-pool`.