Master's thesis of Theodor Amariucai, supervised by Alexander Warstadt and Prof. Ryan Cotterell.
- Text-vision pre-training works directly through HuggingFace and WaB.
- Log in with HuggingFace:
huggingface-cli login - Log in with Weights and Biases:
wandb login
poetry build
poetry install
poetry shell
cd alkmi/callbacks/evaluation-pipeline
pip install -e ".[dev]"
unzip filter-data.zip
# Test run:
bash run_locally.sh configs/flava/debug.yaml
- Evaluation involves the evaluation-pipeline project
for BLiMP. Note: this requires >=
python3.9.
- For sweeps with WandB on the WiT configuration, run e.g.:
cd alkmi
wandb sweep configs/flava/wit_final_sweep_config.yaml
bash run_sweep_on_cluster.sh NR_AGENTS SWEEP_ID- To connect to the Euler compute node, first ssh into the login node then run e.g.:
# An already-running job
srun --interactive --jobid JOBID --pty bash
# Interactive node with 2 GPUs (20GBs VRAM each) and a lot of RAM for the (initial) WiT collapsing
srun --gpus=2 \
--gres=gpumem:40g \
--ntasks-per-node=2 \
--job-name "interactive" --cpus-per-task=4 --mem-per-cpu=15000 --nodes=1 --time=4:00:00 --pty --preserve-env $SHELL
# Interactive node with 1 GPU and a lot of RAM for the (initial) WiT collapsing
srun --gpus=1 \
--gres=gpumem:40g \
--ntasks-per-node=1 \
--job-name "interactive" --cpus-per-task=4 --mem-per-cpu=15000 --nodes=1 --time=4:00:00 --pty --preserve-env $SHELL
# Processing node
srun --time=24:00:00 --ntasks-per-node=1 --cpus-per-task=16 --mem-per-cpu=16000 --nodes=1 --pty --preserve-env $SHELL
- Check space:
du -h -d 2 ~ | sort -h # home directory
du -h -d 2 /cluster/scratch/tamariucai/ | sort -h # scratch space
du -h -d 2 /cluster/work/cotterell/tamariucai/ | sort -h # work directory- For faster access, set up a
~/init.sh:
env2lmod
module purge
module load eth_proxy gcc/8.2.0 python_gpu/3.10.4 cuda/11.8.0 cudnn/8.8.1.3 # should move to python_gpu/3.11.2!
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256
export PATH="$HOME/.local/bin:$PATH"
export HF_DATASETS_CACHE="/cluster/scratch/tamariucai/HuggingfaceDatasets"
export HF_HOME="/cluster/work/cotterell/tamariucai/HuggingfaceHome"
export WANDB_CACHE_DIR="/cluster/scratch/tamariucai/WandbCache"
export WANDB_DIR="/cluster/work/cotterell/tamariucai/WandbDir"
export PYTHONPATH=/cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge/:/cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge/alkmi/callbacks/evaluation-pipeline
cd /cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge
poetry shell
cd alkmi
To work with submodules:
- Pull library updates with
git submodule update --remote --rebase. This might also helpgit submodule update --init --force --remote. - Push changes (including to the library) with
git push --recurse-submodules=check