Convolutional Networks and Their Layers

The CNN in Action

This network was trained to recognise a circle, a square or a triangle in a 16×16 image (see the pipeline demo). It has only 1,619 parameters: 8 filters of 3×3, a max-pooling layer and a dense layer. Draw in the grid, shift your drawing, add noise, and watch every step of the computation update.

Convolution

A 3×3 filter (or kernel) slides over the image. At each position, the 9 covered pixels are multiplied by the filter's 9 weights and summed: the result is one pixel of the feature map. The same filter is used everywhere (weight sharing), so there are very few parameters: 9 weights + 1 bias per filter and input channel. Padding adds zeros around the image, stride skips positions, and dilation spreads the filter's taps apart to see wider with no extra weights.

Pick a classic kernel (the Sobel edge detectors, for instance), one of the CNN's 8 learnt filters, or type your own values.

Activation Functions

Without a non-linear function between them, several convolution layers would collapse into a single convolution. ReLU (max(0, x)) is the default: simple, fast, and it zeroes negative responses. Leaky ReLU keeps a small slope to avoid "dead" neurons. Sigmoid and tanh squash values into a bounded range. GELU, a smoothed ReLU, is used in Transformers.

Pooling

Pooling summarises each window with a single value: the maximum ("is the pattern present anywhere here?") or the average. It shrinks the maps, has no parameters, and makes the network less sensitive to small shifts: compare how much the values change when the image moves by one pixel, before and after pooling (max versus average). Global pooling reduces each map to a single number, often just before the classification layer.

Stacking Layers: the Receptive Field

A neuron in a deep layer depends on a larger and larger area of the input image: its receptive field. With kernel size k, each layer grows it by (k − 1) × j, where j is the cumulative jump, multiplied by each layer's stride. Two 3×3 convolutions see as much as one 5×5, with fewer weights (18 versus 25) and one more non-linearity. Pooling doubles the jump, so the following layers see much further.

Dropout

During training, dropout randomly switches off a fraction p of the neurons in every batch, and multiplies the others by 1 / (1 − p) to keep the same mean. The network can no longer rely on any single neuron: it is a regularisation against overfitting. At inference, it is turned off. Our CNN was not trained with dropout: see how much a single mask disturbs its prediction, and how averaging many masks stabilises it.

Batch Normalisation

Batch normalisation centres and scales a filter's outputs over the current batch: x̂ = (x − μ) / σ, then applies a learnt scale γ and shift β. The following layers thus receive values with a stable distribution, which allows higher learning rates. At inference, μ and σ are replaced by averages measured during training.

Softmax and Temperature

The last layer produces scores (logits) that the softmax turns into probabilities: pi = ezi/T / Σj ezj/T. The temperature T does not change the predicted class, only the confidence: T < 1 sharpens the gap, T > 1 flattens it. It is the same knob that makes a language model more or less "creative".

Summary: the Demo CNN

Notice that 95% of the parameters are in the dense layer. Replacing Flatten with global average pooling would leave the dense layer with only 8 × 3 + 3 = 27 parameters: the choice made by modern architectures (ResNet, MobileNet…).

Other Layers to Know