Machine Learning | Savvas Panagi

Deploying Diverse Regression Machine Learning Algorithms: An Implementation Overview

Sun, 19 Nov 2023 00:00:00 +0000

In this current post, we go into the exploration and analysis of various regression machine learning methods. Our objective is to utilize a range of machine learning techniques to forecast the load demand of the Cyprus power system.

In the initial phase, the ‘data’ dataframe include all essential features and measurements spanning a 16-month period. Throughout the analysis, our focus is directed towards two important features: temperature and hour. While incorporating additional features could potentially enhance result accuracy, for code explanation and methodology simplification, we have opted to retain only these two key features.

import matplotlib.pyplot as plt
import numpy as np

# Create a scatter plot with demand as color
plt.scatter(data['hour'], data['T2M'], c=data['Demand'], cmap='plasma', s=50)

# Add colorbar
cbar = plt.colorbar()
cbar.set_label('Demand (MW)')

# Set labels for axes
plt.xlabel('Hour of the Day')
plt.ylabel('Temperature (°C)')

plt.title('Temperature vs Hour (Color-coded by Demand)')

plt.show()

The first analysis illustrates the fluctuation of demand in megawatts (MW) in response to variations in temperature and hour. In instances of low demand, the figure showcases a darker color, transitioning to brighter hues as demand increases.

Data’s preparation

Prior to normalizing the data, it’s crucial to recognize that the ‘hour’ feature is cyclical. A detailed analysis on handling cyclical features has been discussed in a previous post, accessible . Furthermore, the guidance on scaling data developed in other post , Normalisation and Standardisation method implemented. Moreover, the data’s was divided into test and training data.

from sklearn.preprocessing import StandardScaler
sc=StandardScaler()
data.loc[:,['hour_sin','hour_cos','T2M_norm']]=sc.fit_transform(data.loc[:,['hour_sin','hour_cos','T2M']])

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

Assessing Machine Learning Regression Models: RMSE and R-squared Evaluation

When it comes to checking how well a machine learning regression model is doing, we use two important measures: RMSE and R-squared. RMSE looks at the average difference between the predicted values and the actual values. If the RMSE is low, it means the model is doing a good job. R-squared, on the other hand, tells us how much of the variability in the data our model can explain. A higher R-squared is better, showing that the model fits the data well. So, by looking at both RMSE and R-squared, we can figure out if our regression model is accurate and how well it explains the data.

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, X_test[:,5]))

# Calculate R-squared
r_squared = r2_score(y_test, X_test[:,5])

print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

Current TSO’s demand forecast accuracy: RMSE= 30.194821557544707 , R-squared= 0.9700502334373836

1. Multiple Linear Regression

Multiple Linear Regression is a statistical technique that helps us make predictions by analyzing the relationships between multiple variables. Imagine you have a dataset with one outcome you want to predict (let’s call it ‘Y’) and several factors that might influence it (let’s call them ‘X1,’ ‘X2,’ etc.).

Multiple Linear Regression:

$$ Y = b_0 + b_1X_1 + b_2X_2 + \ldots + b_kX_k $$

The goal is to find the best values for b0,b1,…,bk that minimize the difference between our predicted Y and the actual Y in our dataset. Once we find these values, we can use them to make predictions. For example, if we want to predict someone’s weight (Y), we’d use their height (X1), age (X2), and other relevant factors. The coefficients (b0,b1,…,bk) tell us how much Y is expected to change for a one-unit change in each X variable, holding other variables constant. A positive coefficient means an increase in Y with an increase in the corresponding X, and a negative coefficient means the opposite.

Implementation of Multiple Linear Regression

# Training the Simple Linear Regression model on the Training set
from sklearn.linear_model import LinearRegression
MLR = LinearRegression()
MLR.fit(X_train[:,:3], y_train)

# Predicting the Test set results
y_predMLR = MLR.predict(X_test[:,:3])

from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, y_predMLR))

# Calculate R-squared
r_squared = r2_score(y_test, y_predMLR)

print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

2. Polyonimial Regression

Polynomial regression focuses on capturing nonlinear patterns by introducing higher-degree polynomial terms of a single independent variable. Polynomial regression is particularly useful when the relationship between the variables is nonlinear, providing a more accurate fit to the data compared to the linear assumptions of multiple linear regression.

Polynomial Regression:

$$ Y = b_0 + b_1X + b_2X^2 + \ldots + b_nX^n $$

Implementation of Polyonimial Regression

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Assuming X_train is your feature matrix with 3 columns, and y_train is your target variable

# Create a polynomial regression model with degree 2 (you can change the degree as needed)
degree = 2
PR = make_pipeline(PolynomialFeatures(degree), LinearRegression())

# Train the model
PR.fit(X_train[:,:3], y_train)

# Make predictions
y_predPR = PR.predict(X_test[:,:3])

from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, y_predPR))

# Calculate R-squared
r_squared = r2_score(y_test, y_predPR)

print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

Implementation Support Vector Regression

# Training the SVR model on the whole dataset
from sklearn.svm import SVR
SVR = SVR(kernel = 'rbf')
SVR.fit(X_train[:,:3], y_train)

# Predicting a new result
y_predSVR=SVR.predict(X_test[:,:3])

print(np.concatenate((y_predSVR.reshape(len(y_predSVR),1), y_test.reshape(len(y_test),1)),1))
from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, y_predSVR))
# Calculate R-squared
r_squared = r2_score(y_test, y_predSVR)

print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

Implementation Disicion Tree Regression

# Training the Decision Tree Regression model on the whole dataset
from sklearn.tree import DecisionTreeRegressor
DTR = DecisionTreeRegressor(random_state = 0)
DTR.fit(X_train[:,:3], y_train)
# Predicting a new result
y_predDTR=DTR.predict(X_test[:,:3])
from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, y_predDTR))
# Calculate R-squared
r_squared = r2_score(y_test, y_predDTR)
print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

Implementation Random Forest Regression

# Training the Random Forest Regression model on the whole dataset
from sklearn.ensemble import RandomForestRegressor
RFR = RandomForestRegressor(n_estimators = 10, random_state = 0)
RFR.fit(X_train[:,:3], y_train)
# Predicting a new result
y_predRFR=RFR.predict(X_test[:,:3])
from sklearn.metrics import mean_squared_error
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test, y_predRFR))
# Calculate R-squared
r_squared = r2_score(y_test, y_predRFR)
print(f'RMSE: {rmse}')
print("R-squared:", r_squared)

Results

	TSO’s Forecast	Multiple Linear Regression	Polyonimial Regression	Support Vector Regression	Disicion Tree Regression	Random Forest Regression
RMSE	30.2	109.9	63.1	59.7	27.3	29.4
R-Squared	0.970	0.603	0.869	0.883	0.976	0.972

Your thoughts and questions are important for me. Feel free to share your insights or inquire about anything in the comments section below. Let’s keep the conversation going!

Feature Scaling (Normalisation and Standardisation) / Python implementation

Tue, 29 Aug 2023 00:00:00 +0000

following the previous article about , feature scaling is also important when we are going to use a machine learning algorithms. Is a method to scale the features in a comparative form. But, how can we scaling it ? what is the difference between normalization and standardization ? and when we can use it ? All this questions is gone solve at the current article/blog post.

Equations

Normalization (Min-Max Transform):

$$ X' = \frac{X - X_{\text{min}}}{X_{\text{max}} - X_{\text{min}}} $$

Standardization (Z-Score Transform):

$$ X' = \frac{X - \mu}{\sigma} $$

Base on the very helpful Udemy courses that i have take (Machine learning A-Z) i will try to provide what’s the problem if we are not scaling the features and how scaling can help be giving a simple example. Let’s imagine that we have 2 columns of income and age, also we have 3 different persons as the following table:

Person	Income	Age (Years)
1	70.000€	45
2	60.000€	44
3	52.000€	40

So which 2 of this 3 persons is more similar together? We are clearly sure that the person 1-3 is the worst similar couple that we can take so ignore this pattern. If we try to calculate the difference between the remain combinations, person 1 from 2 has salary difference of 10.000€ and age different of 1 year, but person 2 from 3 has salary different of 8.000€ and age different of 4 years. So numerically the best pair is the person 2-3 which has less totally “distance” (8004<10001). Is this true? No, because is not acceptable to compare income with age. If we want to compare different units of measurement like income with age we have to change the units to a common comparable form.

What is the different of each transform (Normalisation and Standardisation) ?

Normalization, transform the data’s is a scale of [0-1], in contrast the standardisation is not predetermined scale from before and as larger is the standard deviation (σ) as closest to the zero was the standardisation data’s.

When we use each transform ?

Normalisation is recommended when the features follows normal distribution, in contrast the standardisation can be used in the most of situations without any problem. But this universal usage of standardisation does not mean that is always the best choice to scaling the features.

Going back to the problem we have described and try to normalisation the values ,we will have the following table:

Person	Income	Age
1	1	1
2	0.444	0.8
3	0	0

What about the most common pair now?

Person [1-2] = (1-0.444)+(1-0.8) = 0.756

Person [2-3] = (0.444-0)+(0.8-0) = 1.244

So is obvious that the most similar couple is the person 1 with person 2 and not the above results he had calculated before normalization.

To implement this 2 methodologies with python, i will propose to use sklearn.preprocessing library.

Standardisation:

import numpy as np
from sklearn.preprocessing import StandardScaler

# Create a sample dataset with two features (columns)
data = np.array([[10, 2],
 [20, 4],
 [30, 6],
 [40, 8]])

# Create a StandardScaler object
scaler = StandardScaler()

# Fit the scaler to the data and transform the data
normalized_data = scaler.fit_transform(data)

print("Original Data:\n", data)
print("\nNormalized Data:\n", normalized_data)

Normalisation:

import numpy as np
from sklearn.preprocessing import MinMaxScaler

# Create a sample dataset with two features (columns)
data = np.array([[10, 2],
 [20, 4],
 [30, 6],
 [40, 8]])

# Create a MinMaxScaler object
scaler = MinMaxScaler()

# Fit the scaler to the data and transform the data
normalized_data = scaler.fit_transform(data)

print("Original Data:\n", data)
print("\nNormalized Data:\n", normalized_data)

Your thoughts and questions are important for me. Feel free to share your insights or inquire about anything in the comments section below. Let’s keep the conversation going!

How to handle cyclical features, for distance methodologies like k-means machine learning algorithms

Sat, 26 Aug 2023 00:00:00 +0000

Often in situations of use machine learnings methods, we have to consider how to handle the cyclic features. For example in K-Means algorithm it use Euclidean distance in order to sort the available data’s in clusters. In this situations the distance between the hour 0 (00:00) from 23 (23:00) is bigger than what really is. Base on the literature in order to overcome this problem they are use the sin and cosine method to represent each hour in a different cyclic form. With the help of and his corresponding article [1] about i will try to analytical explain the method and with a simple example i will represent how this help in distance measures.

Equations

$$ \begin{align*} x &= \sin\left(\frac{a \cdot 2\pi}{\max(a) + 1}\right) \\ y &= \cos\left(\frac{a \cdot 2\pi}{\max(a) + 1}\right) \end{align*} $$

Base on the above equations we transform real values to a new cyclical form. So, lets think a simple cyclical problem of seasons values and try to solve it for the proof of operation. If we declare the seasons Winter, Spring , Summer and Autumn to a number value it will be 0,1,2 and 3 corresponding. As we know after Autumn the next season is the winter but a distance of 3 (3-0) is the maximum that can occur in our problem if we use the declared number values. After the implementation of transform we have the following form of datas:

Now let’s try to represent the above data’s in x and y axes of scatter plot (x_axes=Sin , y_axes=Cos) :

Finally, how does this help in K-Means algorithm and other distance methods that i will use ? Euclidian distance calculated based on the below formulas.

In our simple problem there are 2D points and after applying the formulas the distance is the following:

So is this Euclidean distance true? if we think the reality this represent exactly how far is each season for each other.

Your thoughts and questions are important for me. Feel free to share your insights or inquire about anything in the comments section below. Let’s keep the conversation going!

References:

[1] :