Skip to content

Commit

Permalink
default tokenizer_type should be string
Browse files Browse the repository at this point in the history
For BPE tokenization, `tokenizer_type` argument should be a string instead of the list.
  • Loading branch information
mayankjobanputra committed Jul 8, 2023
1 parent 700219b commit 90df174
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion megatron/tokenizer/train_tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def parse_args():
type=str,
help="type of tokenizer to train, currently only BPE is supported",
choices=["BPE"],
default=["BPE"],
default="BPE",
)
parser.add_argument(
"-v",
Expand Down

0 comments on commit 90df174

Please sign in to comment.