Turn your local or Github repository into a single text file including a tree, and the contents of all the files. Ignores certain formats, and you can edit these in the blacklist. Your Github token (if used) will be stored in a local JSON.
Program also outputs an estimated token length to judge suitability for prompting with language models such as Claude 3.5, GPT4o, or Gemini Ultra.
Concatenate files from a local directory or GitHub repository and estimate token count. Claude 3.5 Sonnet has a 120k limit, but on free mode you can enter messages up to 15-20k tokens. Strip down your repo in order to get a viable length for prompting and receiving answers, and set the limit to be added to the file.
Token refers to your Github auth token: navigate to "Settings > Developer settings > Personal access tokens > Tokens (classic)" to generate one and pass it as a command line argument -t.
General usage:
usage: text_repo.py [-h] [-t TOKEN] [-l LIMIT] path
Concatenate files from a local directory or GitHub repository.
positional arguments:
path Local directory path or GitHub repository full name (e.g., 'owner/repo')
options:
-h, --help show this help message and exit
-t TOKEN, --token TOKEN
GitHub Personal Access Token (required for GitHub repositories)
-l LIMIT, --limit LIMIT
Token limit for partial concatenation (default: 15000)
Local example:
git clone https://github.com/haelyons/text_repo.git
cd ./text_repo
python text_repo.py ~/your_repo_path/
Github example:
git clone https://github.com/haelyons/text_repo.git
cd ./text_repo
# Example using a sample, randomly generated token - replace the repo with one of your own (or a public repo)
python text_repo.py -t ghp_Rx9fKmLpYjHqNvS3Zw7eBcAd5UoT1iW8nDgE haelyons/challenger_rp2040_wearable
# Future usage won't require your token again
python text_rep.py haelyons/challenger_rp2040_wearable
Run the following, depends on your Python installation.
pip install python-docx
pip install requests github pygithub
- Exclude .git from tree
- Exclude local program outputs from scrape (add destination param)