using the datalad crawler for LORIS

[thomasbeaudry]

When trying to use the datalad crawler i am running into this error:

$ datalad crawl
[INFO ] Loading pipeline specification from ./.datalad/crawl/crawl.cfg
[ERROR ] could not find pipeline for loris [pipeline.py:load_pipeline_from_template:472] (PipelineNotSpecifiedError)

My crawl.cfg looks like:
[crawl:pipeline]
template = loris
_url = https://XXXXXXXX/api/v0.0.3-dev/projects/loris/images
_apibase = https://XXXXXXX/api/v0.0.3-dev/

Thanks!

[kyleam]

The error occurs because pipeline:_find_pipeline doesn't locate a pipeline with the name loris.py in any of the locations it expects. It doesn't look like gh-13 was merged yet, so I believe you'd need to check out that PR locally to crawl with the LORIS API.

[thomasbeaudry]

@kyleam Hi, yeah i had already done a git checkout 1c07bfc to get that pipeline. The error is with that branch so it has the loris.py

[kyleam]

Hmm, I'm not able to trigger that error locally. With 1c07bfc checked out and the config you posted saved to the dataset's .datalad/crawl/crawl.cfg, _find_pipeline is able to locate REPO/datalad_crawler/pipelines/loris.py. I'd suggest stepping through _find_pipeline and seeing if you can spot any issue with the candidates, particularly opj(dirname(__file__), 'pipelines', name).