Pyscenic ctx taking a long time

[willrosenow]

I have been running pyscenic ctx for over 5 days. My unfiltered loom file contains ~7k cells. How long does this step typically take? I am running it from a slurm script on my HPC using the following script:

pyscenic ctx files/adj.csv
cisTarget_databases/mm10__refseq-r80__500bp_up_and_100bp_down_tss.mc9nr.feather
--annotations_fname cisTarget_databases/motifs-v9-nr.mgi-m0.001-o0.0.tbl.txt
--expression_mtx_fname files/ob_unfiltered.loom
--output files/ob_pyscenic.csv
--mask_dropouts
--num_workers 39

In the output, it gets to the Calculating regulons step, but stops there.

2021-02-08 08:30:12,939 - pyscenic.cli.pyscenic - INFO - Calculating regulons.

Do you have any recommendations on making this script run faster?

Thanks,
Will

[cflerin]

There's not really enough information here to help. At minimum you should provide the following from the bug report template:

• pySCENIC version: [e.g. 0.10.0, use pyscenic -h at the command line]
• Installation method: [e.g. Conda, Pip, Docker, Singularity]
• Run environment: [e.g. Jupyter notebook, CLI, Nextflow, HPC script]
• OS: [e.g. Ubuntu]
• Package versions: [obtain using pip freeze, conda list, or skip this if using Docker/Singularity]:

...

[willrosenow]

Here are the details:

pySCENIC version: 0.10.4

Installation method: Pip

Run environment: HPC slurm script

OS: CentOS

Package versions:
adjustText==0.7.3
alabaster==0.7.12
anaconda-client==1.7.2
anaconda-navigator==1.9.7
anaconda-project==0.8.3
anndata==0.7.5
arboreto==0.1.5
asn1crypto==1.0.1
astroid==2.3.1
astropy==3.2.2
atomicwrites==1.3.0
attrs==19.2.0
Babel==2.7.0
backcall==0.1.0
backports.os==0.1.1
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.8.0
biothings-client==0.2.3
bitarray==1.0.1
bkcharts==0.2
blaze==0.11.3
bleach==3.1.0
bokeh==1.3.4
boltons==20.1.0
boto==2.49.0
Bottleneck==1.2.1
cellbrowser==0.7.15
cellex==1.2.1
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
Click==7.0
cloudpickle==1.2.2
clyent==1.2.2
colorama==0.4.1
conda==4.8.0
conda-build==3.18.11
conda-package-handling==1.6.0
conda-verify==3.1.1
contextlib2==0.6.0
cryptography==2.7
cycler==0.10.0
Cython==0.29.13
cytoolz==0.10.0
dask==1.0.0
datashape==0.5.4
decorator==4.4.0
defusedxml==0.6.0
descartes==1.1.0
dill==0.3.1.1
distributed==1.28.1
docutils==0.15.2
entrypoints==0.3
et-xmlfile==1.0.1
fa2==0.3.5
fastcache==1.1.0
fenics-dijitso==2019.1.0
fenics-ffc==2019.1.0.post0
fenics-fiat==2019.1.0
fenics-ufl==2019.1.0
filelock==3.0.12
Flask==1.1.1
Flask-Cors==3.0.8
frozendict==1.2
fsspec==0.5.2
future==0.18.2
get-version==2.1
gevent==1.4.0
glob2==0.7
gmpy2==2.0.8
greenlet==0.4.15
h5py==2.10.0
HeapDict==1.0.1
html5lib==1.0.1
idna==2.8
imageio==2.6.0
imagesize==1.1.0
importlib-metadata==0.23
interlap==0.2.6
ipykernel==5.1.2
ipyparallel==6.2.4
ipython==7.8.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
isort==4.3.21
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.15.1
jeepney==0.4.1
Jinja2==2.10.3
joblib==0.13.2
json5==0.8.5
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.3
jupyter-console==6.0.0
jupyter-core==4.5.0
jupyterlab==1.1.4
jupyterlab-launcher==0.13.1
jupyterlab-server==1.0.6
jupytext==1.3.2
keyring==18.0.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.2
legacy-api-wrap==1.2
leidenalg==0.8.3
libarchive-c==2.8
lief==0.9.0
llvmlite==0.29.0
locket==0.2.0
loompy==3.0.6
lxml==4.4.1
MarkupSafe==1.1.1
matplotlib==3.2.1
mccabe==0.6.1
metakernel==0.24.3
mistune==0.8.4
mizani==0.7.2
mkl-fft==1.0.14
mkl-random==1.1.0
mkl-service==2.3.0
mock==3.0.5
more-itertools==7.2.0
mpmath==1.1.0
msgpack==0.6.1
MulticoreTSNE==0.1
multipledispatch==0.6.0
multiprocessing-on-dill==3.5.0a4
mygene==3.1.0
natsort==7.0.1
navigator-updater==0.2.1
nb-conda-kernels==2.2.2
nbconvert==5.6.0
nbformat==4.4.0
networkx==2.3
nltk==3.4.5
nose==1.3.7
notebook==6.0.1
numba==0.45.1
numexpr==2.7.0
numpy==1.17.2
numpy-groupies==0+unknown
numpydoc==0.9.1
odo==0.5.1
olefile==0.46
opencv-python==4.1.2.30
openpyxl==3.0.0
packaging==19.2
palettable==3.3.0
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.1
partd==1.0.0
path.py==12.0.1
pathlib2==2.3.5
patsy==0.5.1
pbr==5.4.2
pep8==1.7.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.2.0
pkgconfig==1.5.1
pkginfo==1.5.0.1
plotnine==0.7.1
pluggy==0.13.0
ply==3.11
portalocker==1.5.2
prometheus-client==0.7.1
prompt-toolkit==2.0.10
psutil==5.6.3
ptyprocess==0.6.0
py==1.8.0
pyarrow==0.16.0
pycodestyle==2.5.0
pycosat==0.6.3
pycparser==2.19
pycrypto==2.6.1
pycurl==7.43.0.3
pyflakes==2.1.1
Pygments==2.4.2
pylint==2.4.2
pyodbc==4.0.27
pyOpenSSL==19.0.0
pyparsing==2.4.2
pyrsistent==0.15.4
pysam==0.16.0.1
pyscenic==0.10.4
PySocks==1.7.1
pytest==5.2.1
pytest-arraydiff==0.3
pytest-astropy==0.5.0
pytest-doctestplus==0.4.0
pytest-openfiles==0.4.0
pytest-remotedata==0.3.2
python-dateutil==2.8.0
python-igraph==0.8.3
pytz==2019.3
PyWavelets==1.0.3
PyYAML==5.1.2
pyzmq==18.1.0
QtAwesome==0.6.0
qtconsole==4.5.5
QtPy==1.9.0
requests==2.22.0
rope==0.14.0
ruamel-yaml==0.15.46
scanpy==1.7.0rc1
scikit-image==0.15.0
scikit-learn==0.21.3
scipy==1.6.0
scvelo==0.2.2
seaborn==0.9.0
SecretStorage==3.1.1
Send2Trash==1.5.0
setuptools-scm==3.5.0
simplegeneric==0.8.1
sinfo==0.3.1
singledispatch==3.4.0.3
six==1.12.0
snowballstemmer==2.0.0
sortedcollections==1.1.2
sortedcontainers==2.1.0
soupsieve==1.9.3
Sphinx==2.2.0
sphinxcontrib-applehelp==1.0.1
sphinxcontrib-devhelp==1.0.1
sphinxcontrib-htmlhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.2
sphinxcontrib-serializinghtml==1.1.3
sphinxcontrib-websupport==1.1.2
spyder==3.3.6
spyder-kernels==0.5.2
SQLAlchemy==1.3.9
statsmodels==0.12.1
stdlib-list==0.8.0
sympy==1.4
tables==3.6.1
tbb==2020.0.133
tblib==1.4.0
terminado==0.8.2
testfixtures==6.10.0
testpath==0.4.2
texttable==1.6.3
toolz==0.10.0
tornado==6.0.3
tqdm==4.36.1
traitlets==4.3.3
typing==3.6.4
umap-learn==0.4.2
unicodecsv==0.14.1
urllib3==1.24.2
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.16.0
widgetsnbextension==3.5.1
wrapt==1.11.2
wurlitzer==1.0.3
xarray==0.14.1
xlrd==1.2.0
XlsxWriter==1.2.1
xlwt==1.3.0
zict==1.0.0
zipp==0.6.0

[cflerin]

Thanks for the info. I would recommend:

• Update to pySCENIC 0.11.0 (there are package updates that could resolve this issue)
• If the new version still doesn't help, you could try the multiprocessing version instead of using dask. You can do this by using the --mode "custom_multiprocessing" option in the CLI command.

(similar to #142)

[willrosenow]

Thank you @cflerin !

I'll try both of those suggestions.

[mingshuang1]

Hi! Are your problems resolved? I am facing the same issue now but neither methods work for me