This is Alan's token embedding table (4096 tokens x 256 dims) in 3D space.
Every point in the graph below is one of the Alan's tokens (a GPT2 like model). Hover a point to view the token's text.
Left click to rotate, right click to pan, scroll to zoom.
Type something and press trace to run the text through the model and see the residual stream move through
this space, layer by layer, plus which MLP neurons fire hardest at each layer.
booting...
More information
This is the base pretraining checkpoint after being int8-quantized.
Trajectories and activations approximate the float32 training checkpoint.
The trajectory shown in the graph is the last token's residual stream (embedding and each of 6 blocks).
The heatmap is that same token's post GELU MLP activation per layer (6 x 1024 neurons).
The 5 orange diamonds are the model's actual top-5 next-token predictions.
The attention view is a chosen block (slider) at each head's causal attention matrix over the traced tokens