Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Linear Regression, Part 1

English · فارسی
English · فارسی

In this chapter we will start with a quick walkthrough of the mathematics behind this well-known problem, before moving on to see how linear models can be generalized to account for more complicated patterns in data.

We begin with the standard imports:

Source

Simple Linear Regression

We will start with the most familiar linear regression, a straight-line fit to data. A straight-line fit is a model of the form:

y=ax+by = ax + b

where aa is commonly known as the slope, and bb is commonly known as the intercept.

Consider the following data, which is scattered about a line with a slope of 2 and an intercept of –5 (see the following figure):

<Figure size 1200x800 with 1 Axes>

We can use Scikit-Learn’s LinearRegression estimator to fit this data and construct the best-fit line, as shown in the following figure:

Source
<Figure size 1200x800 with 1 Axes>

The slope and intercept of the data are contained in the model’s fit parameters, which in Scikit-Learn are always marked by a trailing underscore. Here the relevant parameters are coef_ and intercept_:

Source
Model slope:     2.0272088103606944
Model intercept: -4.9985770855532

We see that the results are very close to the values used to generate the data, as we might hope.

The LinearRegression estimator is much more capable than this, however—in addition to simple straight-line fits, it can also handle multidimensional linear models of the form:

y=a0+a1x1+a2x2+⋯y = a_0 + a_1 x_1 + a_2 x_2 + \cdots

where there are multiple xx values. Geometrically, this is akin to fitting a plane to points in three dimensions, or fitting a hyperplane to points in higher dimensions.

The multidimensional nature of such regressions makes them more difficult to visualize, but we can see one of these fits in action by building some example data, using NumPy’s matrix multiplication operator:

0.5000000000000033
[ 1.5 -2.   1. ]

Here the yy data is constructed from a linear combination of three random xx values, and the linear regression recovers the coefficients used to construct the data.

In this way, we can use the single LinearRegression estimator to fit lines, planes, or hyperplanes to our data. It still appears that this approach would be limited to strictly linear relationships between variables, but it turns out we can relax this as well.

Polynomial Regression

One trick you can use to adapt linear regression to nonlinear relationships between variables is to transform the data according to basis functions. We have seen one version of this before, in the PolynomialRegression pipeline used in Hyperparameters and Model Validation and Feature Engineering. The idea is to take our multidimensional linear model:

y=a0+a1x1+a2x2+a3x3+⋯y = a_0 + a_1 x_1 + a_2 x_2 + a_3 x_3 + \cdots

and build the x1,x2,x3,x_1, x_2, x_3, and so on from our single-dimensional input xx. That is, we let xn=fn(x)x_n = f_n(x), where fn()f_n() is some function that transforms our data.

For example, if fn(x)=xnf_n(x) = x^n, our model becomes a polynomial regression:

y=a0+a1x+a2x2+a3x3+⋯y = a_0 + a_1 x + a_2 x^2 + a_3 x^3 + \cdots

Notice that this is still a linear model—the linearity refers to the fact that the coefficients ana_n never multiply or divide each other. What we have effectively done is taken our one-dimensional xx values and projected them into a higher dimension, so that a linear fit can fit more complicated relationships between xx and yy.

Polynomial Basis Functions

This polynomial projection is useful enough that it is built into Scikit-Learn, using the PolynomialFeatures transformer:

array([[ 2., 4., 8.], [ 3., 9., 27.], [ 4., 16., 64.]])

We see here that the transformer has converted our one-dimensional array into a three-dimensional array, where each column contains the exponentiated value. This new, higher-dimensional data representation can then be plugged into a linear regression.

As we saw in Feature Engineering, the cleanest way to accomplish this is to use a pipeline. Let’s make a 7th-degree polynomial model in this way:

With this transform in place, we can use the linear model to fit much more complicated relationships between xx and yy. For example, here is a sine wave with noise (see the following figure):

<Figure size 1200x800 with 1 Axes>

Our linear model, through the use of seventh-order polynomial basis functions, can provide an excellent fit to this nonlinear data!

Regularization

The introduction of basis functions into our linear regression makes the model much more flexible, but it also can very quickly lead to overfitting (see Hyperparameters and Model Validation for a discussion of this).

Highly flexible models can fit the training data by combining many components with large, opposing coefficients, producing unstable fits that may generalize poorly. This is a common manifestation of overfitting. A natural remedy is to constrain the model explicitly by penalizing large parameter values, so that the fitted function is discouraged from relying on such unstable cancellations. Such a penalty is known as regularization, and it comes in several forms.

Ridge Regression (L2L_2 Regularization)

Perhaps the most common form of regularization is known as ridge regression or L2L_2 regularization (sometimes also called Tikhonov regularization). This proceeds by penalizing the sum of squares (2-norms) of the model coefficients θn\theta_n. In this case, the penalty on the model fit would be:

P=α∑n=1Nθn2P = \alpha\sum_{n=1}^N \theta_n^2

where α\alpha is a free parameter that controls the strength of the penalty. This type of penalized model is built into Scikit-Learn with the Ridge estimator (see the following figure):

The α\alpha parameter is essentially a knob controlling the complexity of the resulting model. In the limit α→0\alpha \to 0, we recover the standard linear regression result; in the limit α→∞\alpha \to \infty, all model responses will be suppressed. One advantage of ridge regression in particular is that it can be computed very efficiently—at hardly more computational cost than the original linear regression model.

Generate Synthetic Data

Let’s create some synthetic data that follows a non-linear pattern with some noise:

Source
<Figure size 1000x600 with 1 Axes>

Understanding Overfitting

Let’s implement functions to fit both standard linear regression and Ridge regression with polynomial features:

Source

Comparing Linear Regression with Different Polynomial Degrees

Let’s visualize how standard linear regression behaves with increasing polynomial degrees:

Source
Number of columns in DataFrame: 17
<Figure size 1200x500 with 3 Axes>

Analyzing Model Complexity and Overfitting

In this example, we’ll explore how increasing the complexity of a polynomial regression model affects its performance on both training and test data. We’ll:

  1. Split our synthetic data into training (50%) and test (50%) sets

  2. Fit polynomial regression models with degrees from 1 to 15

  3. Visualize the model predictions for degrees 1, 3, 6, 12, and 15

  4. Plot the training and test errors (RSS) against model complexity

Key observations to look for:

  • How well does each model fit the training data?

  • Does increasing polynomial degree always lead to better predictions?

  • Can you identify where overfitting begins to occur?

  • Notice how the training error consistently decreases while the test error might start increasing

Source
<Figure size 1500x1000 with 6 Axes>

Ridge regression adds a penalty term to the cost function, which is the sum of the squared coefficients multiplied by a regularization parameter alpha. This helps prevent overfitting by keeping the coefficients small.

The cost function for Ridge regression is:

J(θ)=MSE+α∑i=1nθi2J(\theta) = MSE + \alpha \sum_{i=1}^{n} \theta_i^2

Where:

  • MSE is the mean squared error

  • α (alpha) is the regularization parameter

  • θᵢ are the model coefficients (excluding the intercept)

Let’s compare Ridge regression with different alpha values for a high-degree polynomial:

Source
c:\programs\anaconda3\envs\pth\lib\site-packages\sklearn\linear_model\_ridge.py:216: LinAlgWarning: Ill-conditioned matrix (rcond=1.187e-18): result may not be accurate.
  return linalg.solve(A, Xy, assume_a="pos", overwrite_a=True).T
<Figure size 1500x1000 with 3 Axes>

Visualizing the Effect of Alpha on Coefficients

Let’s see how the coefficients change with different alpha values:

Source
<Figure size 1200x600 with 1 Axes>

Lasso Regression (L1L_1 Regularization)

Another common type of regularization is known as lasso regression or L1L_1 regularization involves penalizing the sum of absolute values (1-norms) of regression coefficients:

P=α∑n=1N∣θn∣P = \alpha\sum_{n=1}^N |\theta_n|

Though this is conceptually very similar to ridge regression, the results can differ surprisingly. For example, due to its construction, lasso regression tends to favor sparse models where possible: that is, it preferentially sets many model coefficients to exactly zero.

With the lasso regression penalty, the majority of the coefficients are exactly zero, with the functional behavior being modeled by a small subset of the available basis functions. As with ridge regularization, the α\alpha parameter tunes the strength of the penalty and should be determined via, for example, cross-validation (see Hyperparameters and Model Validation for a discussion of this).

For further information about related materials see Mathematics for Machine Learning

Example: Predicting Bicycle Traffic

As an example, let’s take a look at whether we can predict the number of bicycle trips across Seattle’s Fremont Bridge based on weather, season, and other factors. We already saw this data in Working With Time Series, but here we will join the bike data with another dataset and try to determine the extent to which weather and seasonal factors—temperature, precipitation, and daylight hours—affect the volume of bicycle traffic through this corridor. Fortunately, the National Oceanic and Atmospheric Administration (NOAA) makes its daily weather station data available—I used station ID USW00024233—and we can easily use Pandas to join the two data sources. We will perform a simple linear regression to relate weather and other information to bicycle counts, in order to estimate how a change in any one of these parameters affects the number of riders on a given day.

In particular, this is an example of how the tools of Scikit-Learn can be used in a statistical modeling framework, in which the parameters of the model are assumed to have interpretable meaning. As discussed previously, this is not a standard approach within machine learning, but such interpretation is possible for some models.

Let’s start by loading the two datasets, indexing by date:

For simplicity, let’s look at data prior to 2020 in order to avoid the effects of the COVID-19 pandemic, which significantly affected commuting patterns in Seattle:

Next we will compute the total daily bicycle traffic, and put this in its own DataFrame:

We saw previously that the patterns of use generally vary from day to day. Let’s account for this in our data by adding binary columns that indicate the day of the week:

Similarly, we might expect riders to behave differently on holidays; let’s add an indicator of this as well:

We also might suspect that the hours of daylight would affect how many people ride. Let’s use the standard astronomical calculation to add this information (see the following figure):

(8.0, 17.0)
<Figure size 1200x800 with 1 Axes>

We can also add the average temperature and total precipitation to the data. In addition to the inches of precipitation, let’s add a flag that indicates whether a day is dry (has zero precipitation):

Finally, let’s add a counter that increases from day 1, and measures how many years have passed. This will let us measure any observed annual increase or decrease in daily crossings:

Now our data is in order, and we can take a look at it:

Loading...

With this in place, we can choose the columns to use, and fit a linear regression model to our data. We will set fit_intercept=False, because the daily flags essentially operate as their own day-specific intercepts:

Finally, we can compare the total and predicted bicycle traffic visually (see the following figure):

<Figure size 1200x800 with 1 Axes>

From the fact that the data and model predictions don’t line up exactly, it is evident that we have missed some key features. Either our features are not complete (i.e., people decide whether to ride to work based on more than just these features), or there are some nonlinear relationships that we have failed to take into account (e.g., perhaps people ride less at both high and low temperatures). Nevertheless, our rough approximation is enough to give us some insights, and we can take a look at the coefficients of the linear model to estimate how much each feature contributes to the daily bicycle count:

Mon 504.882756 Tue 610.233936 Wed 592.673642 Thu 482.358115 Fri 177.980345 Sat -1103.301710 Sun -1133.567246 holiday -1187.401381 daylight_hrs 128.851511 PRCP -65.436504 dry day 547.698592 Temp (C) 65.162791 annual 26.942713 dtype: float64

These numbers are difficult to interpret without some measure of their uncertainty. We can compute these uncertainties quickly using bootstrap resamplings of the data:

With these errors estimated, let’s again look at the results:

              effect  uncertainty
Mon            505.0         86.0
Tue            610.0         83.0
Wed            593.0         83.0
Thu            482.0         85.0
Fri            178.0         81.0
Sat          -1103.0         80.0
Sun          -1134.0         83.0
holiday      -1187.0        163.0
daylight_hrs   129.0          9.0
PRCP           -65.0          6.0
dry day        548.0         33.0
Temp (C)        65.0          4.0
annual          27.0         18.0

We first see that there is a relatively stable trend in the weekly baseline: there are many more riders on weekdays than on weekends and holidays. We see that for each additional hour of daylight, 129 ± 9 more people choose to ride; a temperature increase of one degree Celsius encourages 65 ± 4 people to grab their bicycle; a dry day means an average of 548 ± 33 more riders, and each inch of precipitation means 665 ± 62 more people leave their bike at home. Once all these effects are accounted for, we see a modest increase of 27 ± 18 new daily riders each year.

Our model is almost certainly missing some relevant information. For example, nonlinear effects (such as effects of precipitation and cold temperature) and nonlinear trends within each variable (such as disinclination to ride at very cold and very hot temperatures) cannot be accounted for in this model. Additionally, we have thrown away some of the finer-grained information (such as the difference between a rainy morning and a rainy afternoon), and we have ignored correlations between days (such as the possible effect of a rainy Tuesday on Wednesday’s numbers, or the effect of an unexpected sunny day after a streak of rainy days). These are all potentially interesting effects, and you now have the tools to begin exploring them if you wish!

Perceptron

  • Represents a single neuron (node) with inputs xix_i, a bias w0w_0, and output yy

  • Each connection has a (synaptic) weight wiw_i. The node outputs y^=∑inxiwi+w0\hat{y} = \sum_{i}^n x_{i}w_i + w_0

  • The activation function predicts 1 if xw+w0>0\mathbf{xw} + w_0 > 0, -1 otherwise

  • Weights can be learned with (stochastic) gradient descent and Hinge(0) loss

    • Updated only on misclassification, corrects output by ±1\pm1

    LPerceptron=max(0,−yi(wxi+w0))\mathcal{L}_{Perceptron} = max(0,-y_i (\mathbf{w}\mathbf{x_i} + w_0))
    ∂LPerceptron∂wi={−yixiyi(wxi+w0)<00otherwise\frac{\partial \mathcal{L_{Perceptron}}}{\partial w_i} = \begin{cases}-y_i x_i & y_i (\mathbf{w}\mathbf{x_i} + w_0) < 0\\ 0 & \text{otherwise} \\ \end{cases}

Logistic regression

  • Aims to predict the probability that a point belongs to the positive class

  • Converts target values {negative (blue), positive (red)} to {0,1}

  • Fits a logistic (or sigmoid or S curve) function through these points

    • Maps (-Inf,Inf) to a probability [0,1]

    y^=logistic(fθ(x))=11+e−fθ(x)\hat{y} = \textrm{logistic}(f_{\theta}(\mathbf{x})) = \frac{1}{1+e^{-f_{\theta}(\mathbf{x})}}
  • E.g. in 1D: logistic(x1w1+w0)=11+e−x1w1−w0 \textrm{logistic}(x_1w_1+w_0) = \frac{1}{1+e^{-x_1w_1-w_0}}