Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Remove the NeoX implementation of GPT2Tokenizer (#1042)
* Try out just using the HF implementation Signed-off-by: Dashiell Stander <[email protected]> * Rely solely on HF tokenizer. Signed-off-by: Dashiell Stander <[email protected]> * Update NeoXArgs docs automatically --------- Signed-off-by: Dashiell Stander <[email protected]> Co-authored-by: github-actions <[email protected]>
- Loading branch information