AI RESEARCH FUNDAMENTALS
Master fundamentals through bite-sized lessons, toy models, and hands-on experiments—all built on our Orchestra platform.
No posts found for this filter.
An intuitive exploration of gradient clipping through toy experiments. We discover that clipping acts as a velocity limiter in weight space, protecting your model from rare but dangerous gradient spikes.
A controlled experiment on the vanishing gradient problem. We demonstrate why a simple "+x" skip connection enables training of arbitrarily deep networks.
(Part 1) A comprehensive empirical study of activation functions through controlled experiments. We explore gradient flow, dead neurons, training stability, and representational capacity across 8 activation functions.
A controlled experiment on width-dependent training dynamics. We demonstrate why standard parameterization breaks when scaling models, and how μP (Maximal Update Parameterization) fixes it.
