Code for EMNLP 2023 paper Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning (Arxiv).
We propose to retrieve similar examples from the target unlabeled corpus to serve as the context of a source query and perform adaptive in-context learning by concatenating the source query and target contexts as the input prompt. And we propose a domain-adaptive in-context learning (DAICL) framework for different LM architectures, including encoder-only and decoder-only models.
NER:
- WNUT2016 Data
- CoNLL2003 Data Other
- Financial NER Data
- WNUT2017 Data
- BioNER Data
- You can also download our processed NER data Here
SA:
- Amazon Benchmark 2-classes Data
- Amazon Review 3-classes Data
- You can also download our processed SA data Here
python NER_Datasets/retrieval_cross_domain.pyTrain LLaMA:
CUDA_VISIBLE_DEVICES=0 python finetune_new.py \
--base_model 'yahma/llama-7b-hf' \
--data_path 'NER_Datasets/llama_train_data/gold_demo/wnut16_gold_demo.json' \
--output_dir 'model/ner_conll03-wnut16_gold_demo_lr3e-4_r16_alpha16_toi_aet_0' \
--batch_size 256 \
--micro_batch_size 4 \
--num_epochs 5 \
--learning_rate 3e-4 \
--cutoff_len 512 \
--val_set_size 1000 \
--warmup_steps 20 \
--logging_steps 4\
--eval_steps 50 \
--save_steps 50 \
--lora_r 16 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[q_proj,k_proj,v_proj,o_proj]' \
--train_on_inputs \
--add_eos_tokenLLaMA Inference:
CUDA_VISIBLE_DEVICES=1 python eval_generate.py \
--load_8bit \
--base_model "yahma/llama-7b-hf" \
--lora_weights 'model/ner_conll03-wnut16_gold_demo_lr3e-4_r16_alpha16_toi_aet_0' \
--eval_path "NER_Datasets/llama_inf_data/gold_demo//wnut16_gold_demo.json" \
--eval_result_path "NER_Datasets/llama_inf_data/eval_result/ner_conll03-wnut16_gold_demo_lr3e-4_r16_alpha16_toi_aet_0/wnut16_gold_demo.txt" \
--eval_batch_size 3Run Roberta NER
CUDA_VISIBLE_DEVICES=0 python roberta_ner/train.py \
--config config/conll03-wnut16_cl_kl.yamlPlease cite the following paper if you found the resources in this repository useful.
@inproceedings{long-etal-2023-adapt,
title = "Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning",
author = "Long, Quanyu and
Wang, Wenya and
Pan, Sinno",
editor = "Bouamor, Houda and
Pino, Juan and
Bali, Kalika",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.emnlp-main.402",
pages = "6525--6542",
}
This project is implemented based on alpaca-lora and CLNER source code