Linear Regression with PyTorch

      Linear regression is a fundamental supervised learning technique used for predicting a continuous target variable based on one or more input features. In this tutorial, we'll learn how to implement linear regression using PyTorch deep learning framework. We'll cover the following topics:

1. Introduction to linear regression 
2. Preparing data
   
3. Building the linear regression model
4. Training the model
5. Evaluating the model
6. Conclusion 
7. Source code listing
   

     Let's get started.

Introduction to linear regression

    Linear regression is a simple yet powerful statistical method used for modeling the relationship between two or more variables. It assumes a linear relationship between the input features and the target variable. The model aims to find the best-fitting line that minimizes the difference between the observed and predicted values.

 

Preparing data

    We'll begin by loading the necessary libraries for this tutorial.

 

import torch

import torch.nn as nn

import numpy as np

from sklearn import datasets

from sklearn.model_selection import train_test_split

from sklearn.metrics import mean_squared_error  

import matplotlib.pyplot as plt 

 

    We'll use a synthetic regression dataset generated using the make_regression function from scikit-learn. The dataset contains 400 samples with 2 input features and a noise level of 20.

 

# Generate synthetic regression data

x_data, y_data = datasets.make_regression(n_samples=400, n_features=2, noise=20, random_state=4)

     Then we split data into train and test parts. Here, we use 20 percent of data as test data.

# Split data into training and testing sets

xtrain, xtest, ytrain, ytest = train_test_split(x_data, y_data, test_size=0.2)

     The input data needs to be converted to PyTorch tensors. Below, we convert the training and test input data into tensors.

 

# Convert data to PyTorch tensors

X_train_ts = torch.from_numpy(xtrain.astype(np.float32))

y_train_ts = torch.from_numpy(ytrain.astype(np.float32)).view(-1, 1)

x_test_ts = torch.from_numpy(xtest.astype(np.float32))

 

Building the linear regression model

     We'll define a simple linear regression class using PyTorch's nn.Linear module. The model class consists of a single linear layer without any activation function. Inside the constructor, a linear transformation layer (nn.Linear) is created and assigned to the self.linear attribute of the object. The forward method defines the forward pass of the model. It takes an input tensor x as input and passes it through the linear layer (self.linear) to compute the output.

 

# Define the linear regression model

class LinearRegressionModel(nn.Module):

def __init__(self, input_size, output_size):

super(LinearRegressionModel, self).__init__()

self.linear = nn.Linear(input_size, output_size)

def forward(self, x):

return self.linear(x)

     Next, we define model parameters and hyperparameters.

    Model parameters:

• input_size: This represents the number of features in the input data.
• output_size: This specifies the dimensionality of the output, which is usually 1 for regression tasks since we are predicting a single target variable.

    Hyperparameters:

• learning_rate: The learning rate determines the step size at which the model parameters are updated during training.
• num_epochs: The number of epochs refers to the number of times the entire dataset is passed forward and backward through the model during training.

# Model parameters and hyperparameters

input_size = X_train_ts.shape[1]

output_size = 1

learning_rate = 0.01

num_epochs = 200

 

 

Training the model

    Next, we'll train the linear regression model on the training data. We'll define the loss function (mean squared error) and the optimizer (Stochastic Gradient Descent).

# Instantiate the model, loss function, and optimizer

model = LinearRegressionModel(input_size, output_size)

criterion = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

    To train the model, the dataset is iterated for a fixed number of epochs. An epoch is one complete pass through the entire training dataset. 

    Inside the loop, we perform a forward pass through the model using the training data (X_train_ts). This generates predicted values (y_predicted) for the target variable. We then calculate the loss between the predicted values (y_predicted) and the actual target values (y_train_ts). The loss represents how far off the model's predictions are from the true values. 

    After computing the loss, we perform a backward pass through the model to compute gradients with respect to each parameter. This is done using the backward() method on the loss tensor. Then, we use the optimizer to update the model parameters based on these gradients, using the step() method. 

    We print the current value of the loss every 10 epochs to monitor the training progress and ensure that the loss is decreasing over time.   

 

# Training loop

for epoch in range(num_epochs):

# Forward pass

y_predicted = model(X_train_ts)

# Compute loss

loss = criterion(y_predicted, y_train_ts)

# Backward pass and optimization

optimizer.zero_grad()

loss.backward()

optimizer.step()

# Print loss every 10 epochs

if (epoch + 1) % 10 == 0:

print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {loss.item():.4f}')

 

Evaluating the model

    After training, we'll evaluate the model's performance on the test data. The torch.no_grad() is a context manager that ensures PyTorch operations do not track gradients inside this context manager. This is useful during model evaluation, where we do not need to compute gradients during inference. By disabling gradient tracking, we save memory and computational resources. 

    Then, we calculate the mean squared error and visualize the actual and predicted data on a graph.

 

# Evaluation

with torch.no_grad():

test_pred = model(x_test_ts).detach().numpy()

# Calculate Mean Squared Error

mse = mean_squared_error(ytest, test_pred)

print("MSE: %.4f" % mse)

# Plot actual vs. predicted

x = range(len(ytest))

plt.scatter(x, ytest, color='red', marker='o', label='Actual')

plt.plot(x, test_pred, color='blue', linewidth=2, label='Predicted')

plt.xlabel('X')

plt.ylabel('y')

plt.title('Linear Regression')

plt.legend()

plt.grid()

plt.show()

 

The result is as follows.

 

Epoch [10/200], Loss: 9233.3242
Epoch [20/200], Loss: 6646.3257
Epoch [30/200], Loss: 4822.3564
Epoch [40/200], Loss: 3533.9141
Epoch [50/200], Loss: 2622.0962
Epoch [60/200], Loss: 1975.6768
Epoch [70/200], Loss: 1516.6350
Epoch [80/200], Loss: 1190.1326
Epoch [90/200], Loss: 957.5449
Epoch [100/200], Loss: 791.6177
Epoch [110/200], Loss: 673.0816
Epoch [120/200], Loss: 588.2905
Epoch [130/200], Loss: 527.5629
Epoch [140/200], Loss: 484.0189
Epoch [150/200], Loss: 452.7618
Epoch [160/200], Loss: 430.3016
Epoch [170/200], Loss: 414.1467
Epoch [180/200], Loss: 402.5165
Epoch [190/200], Loss: 394.1366
Epoch [200/200], Loss: 388.0937
MSE: 401.0717 

 

Conclusion

 

    In this tutorial, we learned how to implement linear regression using PyTorch. We prepared a synthetic regression dataset, built a linear regression model, trained it on the training data, and evaluated its performance on the test data. The full source code is listed below. 

 

Source code listing

 

import torch

import torch.nn as nn

import numpy as np

from sklearn import datasets

from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt

from sklearn.metrics import mean_squared_error

# Generate synthetic regression data

x_data, y_data = datasets.make_regression(n_samples=400, n_features=2, noise=20, random_state=4)

# Split data into training and testing sets

xtrain, xtest, ytrain, ytest = train_test_split(x_data, y_data, test_size=0.2)

# Convert data to PyTorch tensors

X_train_ts = torch.from_numpy(xtrain.astype(np.float32))

y_train_ts = torch.from_numpy(ytrain.astype(np.float32)).view(-1, 1)

x_test_ts = torch.from_numpy(xtest.astype(np.float32))

# Define the linear regression model

class LinearRegressionModel(nn.Module):

def __init__(self, input_size, output_size):

super(LinearRegressionModel, self).__init__()

self.linear = nn.Linear(input_size, output_size)

def forward(self, x):

return self.linear(x)

# Model parameters and hyperparameters

input_size = X_train_ts.shape[1]

output_size = 1

learning_rate = 0.01

num_epochs = 200

# Instantiate the model, loss function, and optimizer

model = LinearRegressionModel(input_size, output_size)

criterion = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# Training loop

for epoch in range(num_epochs):

# Forward pass

y_predicted = model(X_train_ts)

# Compute loss

loss = criterion(y_predicted, y_train_ts)

# Backward pass and optimization

optimizer.zero_grad()

loss.backward()

optimizer.step()

# Print loss every 10 epochs

if (epoch + 1) % 10 == 0:

print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {loss.item():.4f}')

# Evaluation

with torch.no_grad():

test_pred = model(x_test_ts).detach().numpy()

# Calculate Mean Squared Error

mse = mean_squared_error(ytest, test_pred)

print("MSE: %.4f" % mse)

# Plot actual vs. predicted

x = range(len(ytest))

plt.scatter(x, ytest, color='red', marker='o', label='Actual')

plt.plot(x, test_pred, color='blue', linewidth=2, label='Predicted')

plt.xlabel('X')

plt.ylabel('y')

plt.title('Linear Regression')

plt.legend()

plt.grid()

plt.show()

 

References: