Gradient Descent with Linear Regression model in R

   The concept of gradient descent is to improve the model accuracy by minimizing the cost function through the iterative loop. The cost function represents the difference between the actual and the predicted value. The lower the difference provides more accurate the model performance.

   In this tutorial, you'll briefly learn the gradient descent algorithm through the linear regression model in R. 

    We'll start preparing x and y data for this tutorial. The linear function gives y outputs for each given x value.

y = c(1, 2, 4, 3, 4, 5, 3, 5, 7, 6)
x = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) 

Next, we'll define the linear model and fit it on x, y data.

model_lm = lm(y~x)

print(model_lm)

 
Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)            x  
     1.1333       0.5212  
 

A linear regression model is based on y = a + b * x  formula. Here, a is an intercept and b is a slope. We'll extract the intercept and the slope values from the coefficients of the fitted model.

a = model_lm$coefficients[1]  
b = model_lm$coefficients[2]

We can calculate the yhat prediction by using the above formula and get the yhat predicted results.

yhat = a + b * x
print(yhat)   

[1] 1.654545 2.175758 2.696970 3.218182 3.739394 4.260606 4.781818 5.303030
 [9] 5.824242 6.345455 

Or predict() function can be used. As you have seen below both results are the same.

pred_y = predict(model_lm, newdata = data.frame(x))
print(pred_y)

        1        2        3        4        5        6        7        8 
1.654545 2.175758 2.696970 3.218182 3.739394 4.260606 4.781818 5.303030 
       9       10 
5.824242 6.345455

We check the difference by visualizing the y and yhat in a graph.

plot(x, y, col="blue", type="l")
lines(x, yhat, col="red", lwd=2)

Next, we calculate a and b coefficients by using the gradient descent method which calculates the error for each learning rate in an iteration.

n = length(x)
a = 0
b = 0
lr = 0.005     # learning rate
for(j in 1:100)    # epoch or repitition set
{
  for(i in 1:n)
  {
    yp = a + b * x[i]   # formula 
    err = yp - y[i]     # calculating error 
    a = a - lr * err    
    b = b - lr * err * x[i]
  }
  cat("epoch:",j," a:",a,"  b:",b,"\n")
}

This time we use calculated a and b values and find y output values.

yyhat  =  a + b*x

Next, we'll check the RMSE of both the yhat and yyhat. The result shows that error rates are almost the same.

yhat_rmse = sqrt(sum((yhat - y)^2)/n)
yyhat_rmse = sqrt(sum((yyhat - y)^2)/n)
 
cat("yhat_rmse RMSE:", yhat_rmse, "\n")

yhat RMSE: 0.8710843
 
cat("yyhat_rmse RMSE:", yyhat_rmse, "\n")
 

yyhat_rmse RMSE: 0.8739416  

Finally, we'll visualize the results in a graph.

plot(x, y, col="blue")
lines(x, yhat, col="red")  
lines(x, yyhat, col="blue")
  

 

  In this tutorial, we've briefly learned how to calculate the gradient descent by using a linear model in R. The full source code is listed below.


Source code listing

y = c(1, 2, 4, 3, 4, 5, 3, 5, 7, 6)
x = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
 
model_lm = lm(y~x)
print(model_lm) 
 
a = model_lm$coefficients[1]  
b = model_lm$coefficients[2]
 
yhat = a + b * x
print(yhat) 
 
pred_y = predict(model_lm, newdata = data.frame(x))
print(pred_y)

 
plot(x, y, col="blue", type="l")
lines(x, yhat, col="red", lwd=2) 
 
n = length(x)
a = 0
b = 0
lr = 0.005     # learning rate
for(j in 1:100)    # epoch or repitition set
{
  for(i in 1:n)
  {
    yp = a + b * x[i]   # formula 
    err = yp - y[i]     # calculating error 
    a = a - lr * err    
    b = b - lr * err * x[i]
  }
  cat("epoch:",j," a:",a,"  b:",b,"\n")
} 
 
yyhat  =  a + b*x
 
yhat_rmse = sqrt(sum((yhat - y)^2)/n)
yyhat_rmse = sqrt(sum((yyhat - y)^2)/n)
 
cat("yhat_rmse RMSE:", yhat_rmse, "\n")
cat("yyhat_rmse RMSE:", yyhat_rmse, "\n") 
 
plot(x, y, col="blue")
lines(x, yhat, col="red")  
lines(x, yyhat, col="blue")