Real-time Explainability:
Watching a CNN Learn MNIST in 3D
When we talk about "training a neural network," we usually mean a set of numbers drifting around in a high-dimensional space until accuracy looks good. That's powerful—but it's also abstract. We wanted something more direct: a way to watch a model's internal representation reorganize itself as it learns.
So we built a live visualization that shows a convolutional neural network (CNN) learning to classify handwritten digits (MNIST) in real time—by projecting its evolving latent space into an interactive 3D t-SNE plot. As training progresses, points that initially overlap and mix begin to separate into clean class-specific clusters.
In parallel, we stream standard training metrics (loss and accuracy) so you can connect "what the model feels like inside" with "what the scoreboard says."
What You're Seeing
Our interface pairs two complementary views that together provide a complete picture of the learning process. The most critical view is the latent space projection itself:
The Latent Space in 3D
Each dot is a fixed MNIST sample (e.g., 1,000 samples), colored by its ground-truth digit class (0–9). At each update, we:
- 1Run those samples through the model
- 2Extract the activation vector at the selected layer
- 3Reduce the high-dimensional vectors to 3D using t-SNE
- 4Render the result as an interactive scatter plot
The Evolution Over Training
Over time, as the CNN learns, you typically see a clear progression in how the data organizes itself:
Colors are Mixed
Classes overlap heavily; the model hasn't yet learned to distinguish between different digits. Points representing 4s and 9s, or 3s and 5s, are intermingled.
Islands Begin Forming
Some digits separate faster than others—often those with distinctive visual patterns like "1" or "7". You start seeing coherent clusters emerge from the noise.
Clusters Tighten
Most classes form distinct, compact groups. Ambiguous digits may remain closer together (e.g., 4 vs 9, 3 vs 5)—revealing where the model's uncertainty persists.
Why Clustering Happens
Classification models don't just "memorize labels." They learn intermediate representations that make the final decision easier. A helpful mental model:
- The CNN transforms an input image
xinto a feature vectorh(x) - The last linear layer separates classes by learning decision boundaries in that feature space
If training succeeds, the latent vectors become increasingly linearly separable by class—meaning samples from the same class compress together, and different classes spread apart. The 3D t-SNE view gives you an intuitive window into that process.
Real-time Metrics, Side-by-Side
A key part of this tool is that it doesn't replace standard metrics—it complements them. Alongside the 3D latent plot, we stream and overlay training metrics:
- Training Loss
- Validation Loss
- Training Accuracy
- Validation Accuracy
This makes it easy to spot patterns like:
- Loss decreases but clusters remain messy: Model may be learning shortcuts or not building robust features yet
- Training accuracy rises while validation stalls: Clusters may look tight on training samples but drift/overlap on validation → potential overfitting
- Sudden metric spikes: Often correspond to visible latent "shifts" (e.g., learning rate changes, batch norm effects, optimizer transitions)
"The geometry of the representation allows us to literally see the model 'thinking' – separating distinct concepts into orthogonal directions."
Keeping the Visualization Meaningful
t-SNE is great for visual intuition, but it comes with a subtle caveat: it's not a stable coordinate system by default. If you re-run t-SNE from scratch every time, the plot can "spin" or rearrange in ways that look like learning—but are really just the optimizer finding a different projection.
To make the animation interpretable, we:
- Keep a fixed set of samples across time (same 1,000 points)
- Update embeddings on a schedule (every N iterations, not every mini-batch)
- Reuse the previous projection as an initialization so motion reflects changes in representations rather than random re-layout
The goal isn't to claim t-SNE is a perfect microscope. It's to provide a consistent, intuitive signal for how separability emerges during training.
What You Can Learn from Watching the Latent Space
Even on a "toy" dataset like MNIST, the real-time view is surprisingly informative:
Which Classes Are Learned First
Some digits separate early because their visual patterns are distinctive (e.g., "1"). Others take longer because they share strokes and shapes.
Where the Model Is Confused
If two colors repeatedly overlap or form adjacent clusters, the model may be conflating those classes. That's a strong hint to inspect samples, check augmentation, or adjust architecture.
Feature Learning by Layer
Switching the "magnifying glass" layer reveals a progression: earlier layers show more mixed shape/edge features, later layers show cleaner class separation.
Why We Built This (Beyond MNIST)
MNIST is the perfect sandbox: fast training, clear classes, and immediate feedback. But the real point is broader:
Modern model development needs better real-time interpretability tools. We should be able to see representation quality while training—not only after the fact.
This visualization is a starting point for:
Bonus: Build the Probe
Want to run this experiment yourself? Construct the exact architecture used above. The code generator below provides the PyTorch implementation instantly.
Live Graph2Code
Layers
Common
Activation
Operations
Live Graph2Code
Drag and drop the layers to build your neural network — your changes are instantly reflected in the generated code!
Generated Code