This is Alan's token embedding table (4096 tokens x 256 dims) in 3D space.
Every point in the graph below is one of the Alan's tokens. Hover a point to view the token's text.
Left click to rotate, right click to pan, scroll to zoom.
Type something and press trace to run the text through the model and see the residual stream move through
this space, layer by layer, plus which MLP neurons fire hardest at each layer.
booting...
More information
This is the base pretraining checkpoint after being int8-quantized.
Trajectories and activations approximate the float32 training checkpoint.
The trajectory shown in the graph is the last token's residual stream (embedding and each of 12 blocks).
The heatmap is that same token's post GELU MLP activation per layer (12 x 512 neurons).
The 5 orange diamonds are the model's actual top-5 next-token predictions.
The attention view is a chosen block (slider) at each head's causal attention matrix over the traced tokens
row = the token attending,
column = the token attended to
brighter = more weight
Model trained on data from, but not endorsed by, Wikipedia