LocalMacLLM
Make the model small enough to understand.
An independent learning project that follows a tiny GPT-style language model from raw text through tokenization, training, evaluation, and generation on Apple Silicon.
Follow the work behind the answer.
Using a language model can hide almost everything that makes it work. I wanted to understand that machinery by assembling a small version whose data, architecture, and training process I could inspect.
LocalMacLLM follows the complete path from a text dataset to generated tokens. The small scale keeps the moving parts within reach: the tokenizer, the attention mechanism, the training loop, checkpoints, and the sampling choices.
Sean Goedecke’s guide to training a model on a MacBook Pro was the starting point. I developed this educational implementation with AI assistance, using Python and Apple’s MLX framework.
Understanding a model means understanding the path that produces it.
One pipeline. Six things to understand.
The project uses TinyStories as its source material. A SentencePiece byte-pair tokenizer builds the vocabulary, and a packing stage turns the tokenized text into fixed-length sequences for the model.
Training, evaluation, and generation are separate steps. Keeping those boundaries visible made it easier to trace where a result came from and which part of the process to investigate next.
- 01
Prepare
TinyStories
Download the text and separate training, validation, and test data.
- 02
Tokenize
SentencePiece BPE
Train a vocabulary that maps pieces of text to token IDs.
- 03
Pack
256-token sequences
Arrange the token stream into examples the model can process.
- 04
Train
Python + MLX
Predict the next token and update the model from its errors.
- 05
Evaluate
Held-out text
Inspect validation loss and perplexity; retain checkpoints.
- 06
Generate
Sample the next token
Turn a prompt into a sequence using temperature and sampling controls.
A compact transformer, opened up.
The default configuration uses seven transformer blocks, four attention heads, a model width of 128, a vocabulary of 1,024 tokens, and a 256-token context. Together, the configured layers make a model of roughly 1.5 million parameters.
Token and position embeddings feed the transformer blocks. Causal attention limits each position to earlier tokens; a feed-forward network and residual connections carry the representation forward. The output projection shares its weights with the token embeddings.
Those choices are visible in the implementation. The scale makes it possible to move between an architectural idea and the code that expresses it.
From the project source
Small, by design.
The configuration is a concrete place to connect the dimensions of the model to its implementation.
Read the model implementationsrc/model/gpt_mlx.py
@dataclass
class ModelConfig:
vocab_size: int = 1024
seq_len: int = 256
d_model: int = 128
n_layers: int = 7
n_heads: int = 4
dropout: float = 0.0Keep the experiment on the machine in front of you.
Apple Silicon and MLX provided the setting for a local experiment. The repository includes dataset preparation, memory-mapped sequence loading, an AdamW training loop, learning-rate scheduling, checkpointing, evaluation, and a command-line generation path.
Working through that pipeline made the supporting decisions visible. Vocabulary size changes the representation; context length changes what the model can see; sampling changes how a prediction becomes a piece of text.
The project is an educational implementation. Its value here is the inspectable path through the system and the understanding gained by building it.
The useful part was the whole path.
The most useful lesson was how much of the work sits around the model itself. Data preparation, packing, evaluation, and preserving a usable checkpoint all influence whether the experiment can be understood and repeated.
It also gave me a more concrete relationship with the AI tools I use in other projects: a way to ask what the system is doing, where its limits come from, and what evidence would support the next change.