[Errno 2] No such file or directory: '/data/training/ctc-data/dataset.py'

[baronfairy]

(modelenv) lsl@asus:~$ bonito train --epochs 1 --lr 5e-4 --pretrained [email protected] --directory /data/training/ctc-data /data/training/fine-tuned-model
[loading model]
[using pretrained model [email protected]]
[loading data]
Traceback (most recent call last):
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/cli/train.py", line 57, in main
train_loader_kwargs, valid_loader_kwargs = load_numpy(
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/data.py", line 40, in load_numpy
train_data = load_numpy_datasets(limit=limit, directory=directory)
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/data.py", line 66, in load_numpy_datasets
chunks = np.load(os.path.join(directory, "chunks.npy"), mmap_mode='r')
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/numpy/lib/npyio.py", line 427, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/data/training/ctc-data/chunks.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/mnt/raid/lsl/miniconda3/envs/modelenv/bin/bonito", line 8, in
sys.exit(main())
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/init.py", line 32, in main
args.func(args)
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/cli/train.py", line 61, in main
train_loader_kwargs, valid_loader_kwargs = load_script(
File "/mnt/raid/lsl/miniconda3/envs/modelenv/lib/python3.10/site-packages/bonito/data.py", line 31, in load_script
spec.loader.exec_module(module)
File "", line 879, in exec_module
File "", line 1016, in get_code
File "", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/data/training/ctc-data/dataset.py'

[iiSeymour]

@baronfairy can you post the output of ls /data/training/ctc-data please?

[LiPYlpy]

Hello, I encountered the same situation when I ran
bonito train /data/training/model --directory /data/training/dna_10
The error message is:
[loading model]
[loading data]
Traceback (most recent call last):
File "/home/bonito/bonito/cli/train.py", line 57, in main
train_loader_kwargs, valid_loader_kwargs = load_numpy(
File "/home/bonito/bonito/data.py", line 40, in load_numpy
train_data = load_numpy_datasets(limit=limit, directory=directory)
File "/home/bonito/bonito/data.py", line 66, in load_numpy_datasets
chunks = np.load(os.path.join(directory, "chunks.npy"), mmap_mode='r')
File "/opt/conda/envs/basecaller/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/data/training/dna_10/chunks.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/envs/basecaller/bin/bonito", line 33, in
sys.exit(load_entry_point('ont-bonito', 'console_scripts', 'bonito')())
File "/home/bonito/bonito/init.py", line 32, in main
args.func(args)
File "/home/bonito/bonito/cli/train.py", line 61, in main
train_loader_kwargs, valid_loader_kwargs = load_script(
File "/home/bonito/bonito/data.py", line 31, in load_script
spec.loader.exec_module(module)
File "", line 839, in exec_module
File "", line 975, in get_code
File "", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/data/training/dna_10/dataset.py'
The output of ls /data/training/dna_10 is
chunks.npy reference_lengths.npy references.npy validation
that is the one of datasets download by default.