Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Neural Networks and Backpropagation

Mahmood Amintoosi, Fall 2026

Computer Science Dept, Ferdowsi University of Mashhad

I should mention that the original material was from Tomas Beuzen’s course.

English · فارسی

Lecture Learning Objectives


  • Describe the difference between numpy and torch arrays (np.array vs. torch.Tensor)

  • Explain fundamental concepts of neural networks such as layers, nodes, activation functions, etc.

  • Create a simple neural network in PyTorch for regression or classification

  • Derive the gradients of a small network by hand and check them with backward()

Imports


Notebook Cell
Notebook Cell

1. Introduction


  • PyTorch is a Python-based tool for scientific computing that provides several main features:

    • torch.Tensor, an n-dimensional array similar to that of numpy, but which can run on GPUs

    • Computational graphs for building neural networks

    • Automatic differentiation for training neural networks (more on this next lecture)

  • You can install PyTorch from: https://pytorch.org/

2. PyTorch’s Tensor


  • In PyTorch a tensor is just like NumPy’s ndarray that we have become so familiar with.

  • A key difference between PyTorch’s torch.Tensor and numpy’s np.array is that torch.Tensor was constructed to integrate with GPUs and PyTorch’s computational graphs (more on that next lecture though)

2.1. ndarray vs tensor

  • Creating and working with tensors is much the same as with numpy ndarrays

  • You can create a tensor with torch.tensor():

tensor([1, 2, 3]), dtype: torch.int64
tensor([1., 2., 3.]), dtype: torch.float32
tensor([1, 2, 3], dtype=torch.int32), dtype: torch.int32
  • PyTorch also comes with most of the numpy functions we’re already familiar with:

tensor([[0., 0.], [0., 0.]])
tensor([[1., 1.], [1., 1.]])
tensor([[ 0.3388, 0.0180], [-0.1241, 0.9065], [-1.6244, 0.1037]])
tensor([[[0.8185, 0.1055], [0.7195, 0.2424], [0.8114, 0.7107]], [[0.8324, 0.9311], [0.1115, 0.8091], [0.0066, 0.2076]]])
  • Just like in NumPy we can look at the shape of a tensor with the .shape attribute:

torch.Size([2, 3, 2, 2])
4

2.2. Tensors and Data Types

  • Different dtypes have different memory and computational implications (see NumPy’s data types documentation for more)

  • In Pytorch we’ll be building networks that require thousands or even millions of floating point calculations!

  • In such cases, using a smaller dtype like float32 can significantly speed up computations and reduce memory requirements

  • The default float dtype in pytorch float32, as opposed to numpy’s float64

  • In fact some operations in Pytorch will even throw an error if you pass a high-memory dtype!

float64
torch.float32
  • But just like in numpy, you can always specify the particular dtype you want using the dtype argument:

torch.float64

2.3. Operations on Tensors

  • Tensors operate just like ndarrays and have a variety of familiar methods that can be called off them:

tensor([[0.6376, 0.8497, 0.5364], [0.1417, 0.3538, 0.0404], [0.8795, 1.0915, 0.7782]])
tensor([[5.5718e-02, 1.6878e-01, 1.7360e-03], [3.8856e-03, 1.1771e-02, 1.2106e-04], [8.0994e-02, 2.4535e-01, 2.5235e-03]])
tensor(0.1415)
tensor(0.4244)

2.4. Indexing

  • Once again, same as NumPy!

tensor([[0.5512, 0.5119],
        [0.8011, 0.4224],
        [0.1395, 0.0438],
        [0.5987, 0.7250],
        [0.2124, 0.4778]])
tensor([0.5512, 0.5119])
tensor([0.5512, 0.5119])
tensor([0.5512, 0.8011, 0.1395, 0.5987, 0.2124])

2.5. Numpy Bridge

  • Sometimes we might want to convert a tensor back to a NumPy array

  • We can do that using the .numpy() method

<class 'torch.Tensor'>
<class 'numpy.ndarray'>

2.6. GPU and CUDA Tensors

  • GPU is a graphical processing unit (as opposed to a CPU: central processing unit)

  • GPUs were originally developed for gaming, they are very fast at performing operations on large amounts of data by performing them in parallel (think about updating the value of all pixels on a screen very quickly as a player moves around in a game)

  • More recently, GPUs have been adapted for more general purpose programming

  • Neural networks can typically be broken into smaller computations that can be performed in parallel on a GPU

  • PyTorch is tightly integrated with CUDA - a software layer that facilitates interactions with a GPU (if you have one)

  • You can check if you have GPU capability using:

False
  • When training on a machine that has a GPU, you need to tell PyTorch you want to use it

  • You’ll see the following at the top of most PyTorch code:

cpu
  • You can then use the device argument when creating tensors to specify whether you wish to use a CPU or GPU

  • Or if you want to move a tensor between the CPU and GPU, you can use the .to() method:

cpu
  • We’ll revisit GPUs later in the course when we are working with bigger datasets and more complex networks

  • For now, we can work on the CPU just fine

3. Neural Network Basics


  • You’ve already learned about several machine learning algorithms (kNN, Random Forest, SVM, etc.)

  • Neural networks are simply another algorithm and actually one of the simplest in my opinion!

  • As we’ll see, a neural network is just a sequence of linear and non-linear transformations

  • Often you see something like this when learning about/using neural networks:

  • So what on Earth does that all mean? Well we are going to build up some intuition one step at a time

3.1. Simple Linear Regression with a Neural Network

  • Let’s create a simple regression dataset with 500 observations:

Image produced in Jupyter
  • We know how to fit a simple linear regression to this data using sklearn:

Image produced in Jupyter
  • Here are the parameters of that fitted line:

w_0: -0.77 (bias/intercept)
w_1: 45.50
  • As an equation, that looks like this:

y^=−0.77+45.50X\hat{y}=-0.77 + 45.50X
  • Or in matrix form:

[y1^y2^⋮yn^]=[1x11x2⋮⋮1xn][−0.7745.50]\begin{bmatrix} \hat{y_1} \\ \hat{y_2} \\ \vdots \\ \hat{y_n} \end{bmatrix}=\begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} -0.77 \\ 45.50 \end{bmatrix}
  • Or in graph form I’ll represent it like this:

3.2. Linear Regression with a Neural Network in PyTorch

  • So let’s implement the above in PyTorch to start gaining an intuition about neural networks!

  • Every neural network model you build in PyTorch will inherit from torch.nn.Module

  • Inheritance allows us to inherit commonly needed functionality without having to write code ourselves! W3 Schools has a good mini tutorial for class inhertance in Python.

We’ll explore torch.nn.Module more in the lab but if it helps, think about sklearn models: they all inherit common methods like .fit(), .predict(), .score(), etc. When creating a neural network, we define our own architecture but still want common functionality which we inherit from torch.nn.Module.

  • Let’s create a model called linearRegression and then I’ll talk you through the syntax:

Let’s step through the above:

class linearRegression(nn.Module):
    def __init__(self, input_size, output_size):
        super().__init__()

Here we’re creating a class called linearRegression and inheriting the methods and attributes of nn.Module (hint: try typing help(linearRegression) to see all the things we inheritied from nn.Module).

        self.linear = nn.Linear(input_size, output_size)

Here we’re defining a “Linear” layer, which just means wX + b, i.e., the weights of the network, multiplied by the input features plus the bias.

    def forward(self, x):
        out = self.linear(x)
        return out

PyTorch networks created with nn.Module must have a forward() method. It accepts the input data x and passes it through the defined operations. In this case, we are passing x into our linear layer and getting an output out.

  • After defining the model class, we can create an instance of that class:

  • We can check out our model using print():

linearRegression(
  (linear): Linear(in_features=1, out_features=1, bias=True)
)
  • Or the more useful summary() (which we imported at the top of this notebook with from torchsummary import summary):

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Linear-1                    [-1, 1]               2
================================================================
Total params: 2
Trainable params: 2
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.00
Estimated Total Size (MB): 0.00
----------------------------------------------------------------
  • Notice how we have two parameters? We have one for the weight (w1) and one for the bias (w0)

  • These were initialized randomly by PyTorch when we created our model. They can be accessed with model.state_dict():

OrderedDict([('linear.weight', tensor([[0.3072]])), ('linear.bias', tensor([-0.9891]))])
  • Okay, before we move on, our x and y data are currently numpy arrays but they need to be PyTorch tensors

  • Let’s convert them:

  • We have a working model right now and could tell it to give us some output with this syntax:

Predicted: -0.80
   Actual: 31.08
  • Our prediction is pretty bad because our model is not trained/fitted yet!

  • As we learned in the past few lectures, to fit our model we need:

    1. a loss function (called “criterion” in PyTorch) to tell us how good/bad our predictions are - we’ll use mean squared error, torch.nn.MSELoss()

    2. an optimization algorithm to help optimise model parameters - we’ll use GD, torch.optim.SGD()

  • Before we train I’m going to create a “data loader” to help batch my data

  • Data loaders are just generators that yield data to us on request.

  • We’ll use a BATCH_SIZE = 50 (which should give us 10 batches because we have 500 data points)

  • We should have 10 batches:

Total number of batches: 10
  • We can look at a batch using this syntax:

 Shape of feature data (X) in batch: torch.Size([50, 1])
Shape of response data (y) in batch: torch.Size([50])
  • With those, let’s train our simple network for 5 epochs of SGD!

    We open this block line by line in Section 5; for now just note that it computes a loss, asks for gradients and takes a step.

epoch: 1, loss: 648.7896
epoch: 2, loss: 99.0878
epoch: 3, loss: 94.1367
epoch: 4, loss: 93.6165
epoch: 5, loss: 93.4385
  • Now our model has been trained, our parameters should be different than before:

OrderedDict([('linear.weight', tensor([[45.7395]])), ('linear.bias', tensor([-0.7178]))])
  • Comparing to our sklearn model, we get the same answer:

Loading...
  • We got pretty close! We could do better by changing the number of epochs or the learning rate

  • So here is our simple network once again:

  • By the way, check out what happens if we run trainer() again:

epoch: 1, loss: 93.5241
epoch: 2, loss: 94.1431
epoch: 3, loss: 93.9253
epoch: 4, loss: 93.4623
epoch: 5, loss: 93.8662
  • Our model continues where we left off!

  • This may or may not be what you want. We can start from scratch by re-making our model and optimizer.

3.3. Multiple Linear Regression with a Neural Network

  • Okay, let’s do a multiple linear regression now with 3 features

  • So our network will look like this:

  • Let’s go ahead and create some data:

  • And let’s create the above model:

  • We should now have 4 parameters (3 weights and 1 bias)

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Linear-1                    [-1, 1]               4
================================================================
Total params: 4
Trainable params: 4
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.00
Estimated Total Size (MB): 0.00
----------------------------------------------------------------
  • Looks good to me! Let’s train the model and then compare it to sklearn’s LinearRegression():

epoch: 1, loss: 1007.8107
epoch: 2, loss: 110.3681
epoch: 3, loss: 102.1059
epoch: 4, loss: 101.9216
epoch: 5, loss: 102.1261
Loading...

3.4. Non-linear Regression with a Neural Network

  • Okay so we can make a simple network to imitate simple and multiple linear regression

  • You’re probably thinking, so what? But we’re getting to the good stuff I promise!

  • For example, what happens when we have more complicated datasets like this?

Image produced in Jupyter
  • This is obviously non-linear, and we need to introduce some non-linearities into our network

  • These non-linearities are what make neural networks so powerful and they are called “activation functions”

  • We are going to create a new model class that includes a non-linearity - a sigmoid function:

S(X)=11+e−xS(X)=\frac{1}{1+e^{-x}}
Image produced in Jupyter
  • We’ll talk more about activation functions later, but note how the sigmoid function non-linearly maps x to a value between 0 and 1

  • Okay, so let’s create the following network:

  • All this means is that the value of each node in the hidden layer will be transformed by the “activation function”, thus introducing non-linear elements to our model!

  • There’s two main ways of creating the above model in PyTorch, I’ll show you both:

  • Note how our forward() method now passes x through the nn.Sigmoid() function after the hidden layer

  • The above method is very clear and flexible, but I prefer using nn.Sequential() to combine my layers together in the constructor:

  • Let’s make an instance of our new class and confirm it has 10 parameters (6 weights + 4 biases):

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Linear-1                    [-1, 3]               6
           Sigmoid-2                    [-1, 3]               0
            Linear-3                    [-1, 1]               4
================================================================
Total params: 10
Trainable params: 10
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.00
Estimated Total Size (MB): 0.00
----------------------------------------------------------------
  • Okay, let’s train:

epoch: 1, loss: 32.5818
epoch: 2, loss: 14.8140
epoch: 3, loss: 7.6448
epoch: 4, loss: 5.2685
epoch: 5, loss: 4.8211
epoch: 6, loss: 4.7319
epoch: 7, loss: 4.4413
epoch: 8, loss: 4.2567
epoch: 9, loss: 4.3213
epoch: 10, loss: 4.2639
Image produced in Jupyter
  • Take a look at those non-linear predictions! Cool!

  • Our model is not great and we could make it better soon by adjusting the learning rate, the number of nodes, and the number of epochs

  • But I really want you to see how each of our hidden nodes is “engineering a non-linear feature” to be used for the predictions, check it out:

Image produced in Jupyter

3.5. Multi Layer Perceptron

  • You’ve probably heard the magic term “MLP” and you’re about to find out what it means!

  • Really, it’s just a neural network with more than 1 hidden layer! Easy!

  • Let’s create an “MLP” network of 2 layers:

Image produced in Jupyter
  • But just because the network has more layer, doesn’t always mean it’s better!

4. Activation Functions


  • As we learned above, activation functions are what allow us to model complex, non-linear functions

  • There are many different activations functions:

Image produced in Jupyter
  • Activation functions should be non-linear and tend to be monotonic and continuously differentiable (smooth)

  • But as you can see with the ReLU function above, that’s not always the case!

  • I wanted to point this out because it highlights how much of an art deep learning really is.

  • Here’s a great quote from Yoshua Bengio (famous for his work in AI and deep learning) on his group experimenting with ReLU:

“...one of the biggest mistakes I made was to think, like everyone else in the 90s, that you needed smooth non-linearities in order for backpropagation to work. because I thought that if we had something like rectifying nonlinearities, where you have a flat part, that it would be really hard to train, because the derivative would be zero in so many places. And when we started experimenting with ReLU, with deep nets around 2010, I was obsessed with the idea that, we should be careful about whether neurons won’t saturate too much on the zero part. But in the end, it turned out that, actually, the ReLU was working a lot better than the sigmoids and tanh, and that was a big surprise...it turned out to work better, whereas I thought it would be harder to train!”

  • Anyway, TL;DR ReLU is probably the most popular these days, but you can treat activation functions as hyperparams (to be optimized)

5. Backpropagation


  • Every network we have fitted so far was trained by the same short block of code, hidden inside trainer():

optimizer.zero_grad()        # 1. forget the gradients left by the previous batch
y_hat = model(X).flatten()   # 2. forward pass: compute the prediction
loss  = criterion(y_hat, y)  # 3. how wrong are we?
loss.backward()              # 4. backward pass: compute the gradients
optimizer.step()             # 5. move the parameters against the gradient
  • The loss.backward() line is the one we have been treating as a black box. It is backpropagation: the chain rule, applied backwards through the network, one layer at a time.

  • The trick is that the derivative of the loss with respect to the output is computed once and then reused by every parameter that contributed to that output. That is why one backward pass costs about as much as one forward pass, instead of one forward pass per parameter.

  • To see this concretely, take the smallest network that is still non-linear: one input, two sigmoid hidden units and one output.

5.1. The forward pass, by hand

  • Fix the weights and biases as in the figure below, and feed the network a single observation x=1x=1 with target y=3y=3.

  • A forward pass is multiply, add, squash — twice:

z1=w1x+b1=2,a1=σ(z1)=0.88z_1=w_1x+b_1=2,\qquad a_1=\sigma(z_1)=0.88
z2=w2x+b2=1,a2=σ(z2)=0.73z_2=w_2x+b_2=1,\qquad a_2=\sigma(z_2)=0.73
y^=w3a1+w4a2+b3=1.34,L=(y^−y)2=2.75\hat{y}=w_3a_1+w_4a_2+b_3=1.34,\qquad \mathcal{L}=(\hat{y}-y)^2=2.75
  • Nothing here is new: a linear layer, a sigmoid, another linear layer and a squared error, chained together.

5.2. The backward pass, by hand

  • Start at the loss and ask how L\mathcal{L} changes when y^\hat{y} changes. For the squared error, ∂L/∂y^=2(y^−y)=−3.31\partial\mathcal{L}/\partial\hat{y}=2(\hat{y}-y)=-3.31.

  • That single number is the upstream gradient. Every parameter below it gets its own gradient by multiplying the upstream gradient by its local derivative:

∂L∂w3=∂L∂y^ a1=−2.91,∂L∂w4=∂L∂y^ a2=−2.42,∂L∂b3=∂L∂y^=−3.31\frac{\partial\mathcal{L}}{\partial w_3}=\frac{\partial\mathcal{L}}{\partial\hat{y}}\,a_1=-2.91,\qquad \frac{\partial\mathcal{L}}{\partial w_4}=\frac{\partial\mathcal{L}}{\partial\hat{y}}\,a_2=-2.42,\qquad \frac{\partial\mathcal{L}}{\partial b_3}=\frac{\partial\mathcal{L}}{\partial\hat{y}}=-3.31
  • For the hidden layer we keep walking backwards: the output layer hands back the gradient with respect to each activation, and the sigmoid contributes its own derivative σ′(z)=σ(z)(1−σ(z))\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big). With σ′(z1)=0.105\sigma'(z_1)=0.105 and σ′(z2)=0.197\sigma'(z_2)=0.197:

∂L∂a1=∂L∂y^ w3,∂L∂z1=∂L∂a1 σ′(z1),∂L∂w1=∂L∂z1 x,∂L∂b1=∂L∂z1\frac{\partial\mathcal{L}}{\partial a_1}=\frac{\partial\mathcal{L}}{\partial\hat{y}}\,w_3,\qquad \frac{\partial\mathcal{L}}{\partial z_1}=\frac{\partial\mathcal{L}}{\partial a_1}\,\sigma'(z_1),\qquad \frac{\partial\mathcal{L}}{\partial w_1}=\frac{\partial\mathcal{L}}{\partial z_1}\,x,\qquad \frac{\partial\mathcal{L}}{\partial b_1}=\frac{\partial\mathcal{L}}{\partial z_1}
  • The same recipe on the second hidden unit gives ∂L/∂w2=∂L/∂b2=−1.30\partial\mathcal{L}/\partial w_2=\partial\mathcal{L}/\partial b_2=-1.30, and the first one gives ∂L/∂w1=∂L/∂b1=−0.35\partial\mathcal{L}/\partial w_1=\partial\mathcal{L}/\partial b_1=-0.35.

5.3. The same gradients from PyTorch

  • We can set the network to exactly the weights of the figure and let PyTorch do the bookkeeping. The numbers below should match the hand calculation to two decimals.

5.4. What this means for trainer()

  • Why the graph exists. Parameters created by a nn.Module come with requires_grad=True, so PyTorch records every operation applied to them. Nothing else needs to be tracked: we want derivatives with respect to the parameters, not the data.

  • What backward() fills. Before the call, p.grad is None; afterwards it holds ∂L/∂p\partial\mathcal{L}/\partial p for the current batch. That is the whole output of backpropagation.

  • Why zero_grad() is there. Gradients accumulate by default. If you forget it, the next batch adds its gradient to the previous one and the effective learning rate drifts.

  • What step() does. For plain SGD, p←p−η p.gradp \leftarrow p-\eta\, p.\mathrm{grad}, which is exactly Step 4 of the gradient descent procedure from the regression lecture.

  • Backpropagation is not the optimizer. It only computes gradients; the same numbers can be handed to SGD, Adam or anything else.

6. Classification with Neural Networks


6.1. Binary Classification

  • This will actually be the easiest part of the lecture

  • Up until now, we’ve been looking at developing networks for regression tasks, but what if we want to do binary classification?

  • In logistic regression we passed the output of a linear model through the sigmoid to get a probability between 0 and 1. A network does the same thing, with a hidden layer in front of the sigmoid.

  • Let’s create a toy dataset:

Image produced in Jupyter
  • Let’s create this network to model that dataset:

  • I’m going to start using ReLU as our activation function(s) and Adam as our optimizer because these are what are currently, commonly used in practice.

  • We are doing classification now so we’ll need to use log loss (binary cross entropy) as our loss function:

f(w)=∑x,y∈D−ylog(y^)−(1−y)log(1−y^)f(w) = \sum_{x,y \in D} -y log(\hat{y}) - (1-y)log(1-\hat{y})
  • In PyTorch, binary cross entropy loss criterion is torch.nn.BCELoss

  • The formula expects a “probability” which is why we add a Sigmoid function to the end of out network.

  • BUT WAIT!

  • While we can do the above and then train with a torch.nn.BCELoss loss function, there’s a better way!

  • We can omit the Sigmoid function and just use torch.nn.BCEWithLogitsLoss (which combines a Sigmoid layer and the BCELoss)

  • Why would we do this? It’s numerically stable! (Did you do the log-sum-exp question in Lab 1? We use it here for stability!)

  • From the docs:

“This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.”

  • So actually, here’s our model (no Sigmoid layer at the end because it’s included in the loss function we’ll use):

  • Let’s train the model:

epoch: 1, loss: 0.6842
epoch: 2, loss: 0.6534
epoch: 3, loss: 0.6097
epoch: 4, loss: 0.5676
epoch: 5, loss: 0.5343
epoch: 6, loss: 0.4916
epoch: 7, loss: 0.4410
epoch: 8, loss: 0.3871
epoch: 9, loss: 0.3299
epoch: 10, loss: 0.2833
epoch: 11, loss: 0.2467
epoch: 12, loss: 0.2135
epoch: 13, loss: 0.1992
epoch: 14, loss: 0.1889
epoch: 15, loss: 0.1800
epoch: 16, loss: 0.1695
epoch: 17, loss: 0.1698
epoch: 18, loss: 0.1687
epoch: 19, loss: 0.1541
epoch: 20, loss: 0.1510
Image produced in Jupyter
  • To be clear, our model is just outputting some number between -∞ and +∞ (we aren’t applying Sigmoid), so:

    • To get the probabilities we would need to pass them through a Sigmoid;

    • To get classes, we can apply some threshold (usually 0.5)

  • For example, we would expect the point (0,0) to have a high probability and the point (-1,-1) to have a low probability:

tensor([[  4.9344],
        [-11.3356]])
tensor([[9.9286e-01],
        [1.1939e-05]])
[[1]
 [0]]

6.2. Multiclass Classification (Optional)

  • For multiclass classification, remember softmax?

σ(z⃗)i=ezi∑j=1Kezj\sigma(\vec{z})_i=\frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}}
  • It basically makes sure all the outputs are probabilities between 0 and 1, and that they all sum to 1.

  • torch.nn.CrossEntropyLoss is a loss that combines a softmax with cross entropy loss.

  • Let’s try a 4-class classification problem using the following network:

Image produced in Jupyter
  • Let’s train this model:

epoch: 1, loss: 1.1576
epoch: 2, loss: 0.7209
epoch: 3, loss: 0.4848
epoch: 4, loss: 0.3222
epoch: 5, loss: 0.1632
epoch: 6, loss: 0.0652
epoch: 7, loss: 0.0157
epoch: 8, loss: 0.0085
epoch: 9, loss: 0.0040
epoch: 10, loss: 0.0016
Image produced in Jupyter
  • To be clear once again, our model is just outputting some number between -∞ and +∞, so:

    • To get the probabilities we would need to pass them to a Softmax;

    • To get classes, we need to select the largest probability.

  • For example, we would expect the point (-1,-1) to have a high probability of belonging to class 1, and the point (0,0) to have the highest probability of belonging to class 2.

tensor([[-27.5895,  19.3565,   1.1430, -33.1879],
        [-12.4731, -22.3387,   2.0026,  20.8498]])
  • Note how we get 4 predictions per data point (a prediction for each of the 4 classes)

tensor([[4.0889e-21, 1.0000e+00, 1.2302e-08, 1.5144e-23],
        [3.3731e-15, 1.7517e-19, 6.5275e-09, 1.0000e+00]])
  • The predictions should now sum to 1:

tensor([1., 1.])
  • We can get the class with maximum probability using argmax():

tensor([1, 3])

7. Lecture Exercise: True/False Questions


Answer True/False for the following:

  1. Neural networks can be used for both regression and classification. (True)

  2. For fully connected neural networks, the number of parameters ≥\geq the number of features. (True)

  3. Neural networks are parametric models. (True)

  4. Any neural network with 3 hidden layers will have more parameters than any neural network with 2 hidden layers. (False)

  5. The architecture of a neural network (number of hidden layers and hidden nodes) is a hyperparameter. (True)

  6. Like linear regression or logistic regression, with neural networks we can interpret each feature’s weight value as a measure of the feature’s importance. (False)

8. The Lecture in Three Conjectures


  1. PyTorch is a neural network software based on “tensors” (like NumPy arrays on steroids).

  2. Neural Networks are simply:

    • Composed of an input layer, 1 or more hidden layers, and an output layer, each with 1 or more nodes.

    • The number of nodes in the Input/Output layers is defined by the problem/data. Hidden layers can have an arbitrary number of nodes.

    • Activation functions in the hidden layers help us model non-linear data.

    • Feed-forward neural networks are just a combination of simple linear and non-linear operations.

  3. Activation functions allow the network to learn non-linear function

Exercises

  1. Derive the gradients of a one-hidden-layer network y^=w2⊤σ(W1x+b1)+b2\hat y = w_2^\top\sigma(W_1x+b_1)+b_2 with respect to W1,b1,w2,b2W_1,b_1,w_2,b_2, then check each of the four against autograd on one fixed input.

  2. Gradient check: compare the analytic gradient of one weight with a central finite difference (f(w+h)−f(w−h))/2h(f(w+h)-f(w-h))/2h, plot the relative error against hh from 10-12 to 10-1, and explain the U-shape of the curve.

  3. Replace the hidden activation by the identity, retrain the non-linear regression example of §3.4, and report the training and test error. Which property of the network does the collapse show?

  4. A stack of linear layers is a single matrix: show that W3W2W1x=WxW_3W_2W_1x = Wx and count the parameters of the deepest versus the shallowest version when the input and output both have dimension 10. Which one has more freedom?