This is the code that backs the blog post at: https://rushabhdoshi.com/posts/2023-09-18-llm-leverage/
This is based on the GPTs are GPTs paper. I was interested in building an intuition around the broad impacts of LLMs and used that as a starting point.
This repo is essentially three parts:
- Building up a sqlite db from SQL files from O*Net. The actual files aren't included because they're large. You can download them here: https://www.onetcenter.org/database.html#all-files
- Classifying the DWAs from the O*Net using OpenAI. This is done in
notebooks/classification.ipynb
. The results are stored indata/readonly
- Analyzing the results. This is done in
notebooks/analysis.ipynb
. The results are stored indata/output
To get up and running, after downloading the repo, do:
poetry install
I prefer to run jupyter notebooks in VSCode but YMMV.
Contributions are welcome, especially if you see bugs and issues. Please open a PR.
- Replicate with GPT-4 (costs quite a bit more)
- Look at
gpt-3.5-turbo-instruct
with log-probs, to see if it improves classification - Classify tasks instead of DWAs
- Classify occupations instead of DWAs