Multi-output Multi-step Regression Example with Keras SimpleRNN in Python

   In previous posts, we saw the multi-output regression data analysis with CNN and LSTM methods. In this tutorial, we'll learn how to implement multi-output and multi-step regression data with Keras SimpleRNN class in Python. This method can be applied to time-series data too. Multi-output data contains more than one output value for a given dataset. To predict data we'll use multiple steps to train the output data. The tutorial covers:

1. Preparing the data
2. Defining and fitting the model
3. Predicting and visualizing the results
4. Source code listing

We'll start by loading the required libraries of Python and Keras API for this tutorial.

from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from numpy import array, sqrt, array
from numpy.random import uniform
from numpy import hstack
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error

Preparing the data

We'll create a multi-output dataset for this tutorial. It is randomly generated data with a simple rule. You can check the logic of data generation in a function below. There are three inputs and two outputs in this dataset.

def create_data(n):
 x1 = array([i/100-uniform()*10 for i in range(n)]).reshape(n,1)
 x2 = array([i/100+uniform()*5 for i in range(n)]).reshape(n,1)
 x3 = array([i/200+uniform()-5 for i in range(n)]).reshape(n,1)

 y1= [x1[i]/2+x2[i]+x3[i]+uniform() for i in range(n)]
 y2= [x1[i]-x2[i]*2-x3[i]/2-5-uniform() for i in range(n)]
 X = hstack((x1, x2, x3))
 Y = hstack((y1, y2))
 return X, Y

 
Now, we'll set N total number of samples and generate the dataset. You can draw X or Y data in a plot to check it visually.

N = 420
x, y = create_data(N)

plt.plot(y)
plt.show()

Now, we'll split the dataset into the train and test parts. Here, we'll define n number of train samples.

n = 400

xtrain, xtest = x[0:n,:], x[n:N,:]
ytrain, ytest = y[0:n,:], y[n:N,:]

Next, we'll define the function to prepare training data. It creates x and y data for a given step window.

def convertData(datax, datay, step):
 X, Y = [], []
 for i in range(len(datax)-step):
  d = i+step  
  X.append(datax[i:d,])
  Y.append(datay[d])
 return array(X), array(Y)

We'll define step value and create train and test data. You can check the dimensions of the data.

step = 2
testx,testy = convertData(xtest,ytest, step)
trainx,trainy = convertData(xtrain,ytrain, step)

print("test-x:", testx.shape, "test-y:", testy.shape)

print("train-x:", trainx.shape, "trian-y:", trainy.shape)

test-x: (18, 2, 3) test-y: (18, 2)

train-x: (398, 2, 3) trian-y: (398, 2)

Defining and fitting the model

Input and output shapes can be extracted from the input and output training data.

in_dim = trainx.shape[1:3]
out_dim = trainy.shape[1]

print(in_dim)

(2, 3)
print(out_dim)

2 

We'll define a sequential model and add the SimpleRNN layer by defining the input shapes. We'll add Dense layers with ReLU activations, set the output layer dimension, and compile the model with Adam optimizer.

model = Sequential()
model.add(SimpleRNN(units=100, input_shape=in_dim, activation="relu")) 
model.add(Dense(16, activation="relu")) 
model.add(Dense(out_dim))
model.compile(loss='mse', optimizer='adam') 

model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
simple_rnn_1 (SimpleRNN)     (None, 100)               10400     
_________________________________________________________________
dense_2 (Dense)              (None, 16)                1616      
_________________________________________________________________
dense_3 (Dense)              (None, 2)                 34        
=================================================================
Total params: 12,050
Trainable params: 12,050
Non-trainable params: 0
_________________________________________________________________

We'll fit the model with train data and check the training score.

model.fit(trainx,trainy, epochs=500, verbose=2)
trainScore = model.evaluate(trainx, trainy, verbose=0)
print(trainScore)

4.617212070292564 

Predicting and visualizing the results

Finally, we'll predict test data and check to RMSE rate.

predtest= model.predict(testx)

rmse_y1 = sqrt(mean_squared_error(testy[:,0], predtest[:,0]))
rmse_y2 = sqrt(mean_squared_error(testy[:,1], predtest[:,1]))
 

print("RMSE y1: %.2f y2: %.2f" % (rmse_y1, rmse_y2))

RMSE y1: 3.12 y2: 6.54 

We can visualize the y1 and y2 predicted and original values in a plot.

x_ax = range(len(testx))
plt.plot(x_ax, testy[:,0],  label="y1-test",color="c")
plt.plot(x_ax, predtest[:,0], label="y1-pred")
plt.plot(x_ax, testy[:,1],  label="y2-test",color="m")
plt.plot(x_ax, predtest[:,1], label="y2-pred")
plt.legend()
plt.show() 

   In this tutorial, we've briefly learned how to fit and predict multi-output and multi-step regression data with Keras SimpleRNN model. The full source code is listed below.


Source code listing

from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from numpy import array, sqrt, array
from numpy.random import uniform
from numpy import hstack
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error


def create_data(n):
 x1 = array([i/100-uniform()*10 for i in range(n)]).reshape(n,1)
 x2 = array([i/100+uniform()*5 for i in range(n)]).reshape(n,1)
 x3 = array([i/200+uniform()-5 for i in range(n)]).reshape(n,1)

 y1= [x1[i]/2+x2[i]+x3[i]+uniform() for i in range(n)]
 y2= [x1[i]-x2[i]*2-x3[i]/2-5-uniform() for i in range(n)]
 X = hstack((x1, x2, x3))
 Y = hstack((y1, y2))
 return X, Y

N=420
n=400
x, y = create_data(N)
plt.plot(y)
plt.show()

xtrain, xtest = x[0:n,:], x[n:N,:]
ytrain, ytest = y[0:n,:], y[n:N,:]

print("xtrain:", xtrain.shape, "ytrian:", ytrain.shape)

def convertData(datax,datay,step):
 X, Y = [], []
 for i in range(len(datax)-step):
  d = i+step  
  X.append(datax[i:d,])
  Y.append(datay[d])

 return array(X), array(Y)

step=2
testx,testy = convertData(xtest,ytest, step)
trainx,trainy = convertData(xtrain,ytrain, step)
print("test-x:", testx.shape, "test-y:", testy.shape)
print("train-x:", trainx.shape, "trian-y:", trainy.shape)

trainx[1:2,]
test[1:3,1:2]
in_dim = trainx.shape[1:3]
out_dim = trainy.shape[1]

model = Sequential()
model.add(SimpleRNN(units=100, input_shape=in_dim, activation="relu")) 
model.add(Dense(16, activation="relu")) 
model.add(Dense(out_dim))
model.compile(loss='mse', optimizer='adam')
model.summary()

model.fit(trainx,trainy, epochs=500, verbose=2)
trainScore = model.evaluate(trainx, trainy, verbose=0)
print(trainScore)

predtest= model.predict(testx)

rmse_y1 = sqrt(mean_squared_error(testy[:,0], predtest[:,0]))
rmse_y2 = sqrt(mean_squared_error(testy[:,1], predtest[:,1]))
print("RMSE y1: %.2f y2: %.2f" % (rmse_y1, rmse_y2))

x_ax = range(len(testx))
plt.plot(x_ax, testy[:,0],  label="y1-test",color="c")
plt.plot(x_ax, predtest[:,0], label="y1-pred")
plt.plot(x_ax, testy[:,1],  label="y2-test",color="m")
plt.plot(x_ax, predtest[:,1], label="y2-pred")
plt.legend()
plt.show()