GPT implementation in pure Python.

You've seen nanoGPT.

You've seen picoGPT.

Now, imagine you're alone on a desert island 🏝️ with a computer, Python installed and no Internet (you can't do pip install torch 🥲).

A crazy idea occurs to you: what if I created a GPT model... from scratch?

Here is the result of this work: femtoGPT, a GPT implementation in pure Python, without any dependencies... and with it's own autograd engine (inspired by micrograd).

• No tensor ❌
• Not fast ❌
• No batch (one sample at a time) ❌
• No GPU support ❌

A pure learning experience.

Enjoy!

Install the package:

git clone https://github.com/cthiriet/femtoGPT
pip install .

Train a femto GPT model on a dataset, that is actually...just a string:

# Train on the "hello world" sentence
python femtogpt/train.py --dataset "hello world"

pytest