Introduction to Machine Learning
Last month, I participated in a Machine learning approach Hackathon hosted on Analytics Vidhya’s Datahack platform. Over a weekend, more than 600 participants competed to build and improve their solutions and climb the leaderboard. In this article, I will be sharing my hackathon experience – what worked, what didn’t work, and what I learned from it.
Objective of Machine Learning Approach
Forecasting the demand for car rentals on an hourly basis based on past data.
Loading and Exploring Data
Importing necessary libraries
%matplotlib inline import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import datetime as dt from sklearn.metrics import mean_squared_error
Importing Train and Test Data
train_df = pd.read_csv('train_E1GspfA.csv') test_df = pd.read_csv('test_6QvDdzb.csv')
So basically, we are provided with the train data comprising of hourly rental car demand spanning from mid-Aug 2018 to Feb 2021 (approx 31 months). Our job is to make a prediction on the next 13 months of the test data (ie from Mar 2021 to Mar 2022).
Data Cleaning and Pre-processing
Check for the datatype of the data in the columns
This also tells us that there are no null values in the data.
Check for duplicate entries
As seen, no duplicate data is present.
From visual analysis, I believe that the train dataframe is already sorted, but let’s just do it.
Sorting dataframe by ‘date’ and ‘hour’ —
train_df = train_df.sort_values(by = ['date', 'hour'])
Now that we have organised the data, let’s proceed to the next stage.
As seen we don’t really have many features. ‘hour’ data we already have, let’s see what information we can extract from the ‘date’ column
train_df['date'] = pd.to_datetime(train_df['date']) train_df['Month'] = train_df.date.dt.month train_df['Week'] = train_df.date.dt.week train_df['Day'] = train_df.date.dt.day train_df['Quarter'] = train_df.date.dt.quarter train_df['Weekday'] = train_df.date.dt.dayofweek train_df['Dayofyear'] = train_df.date.dt.dayofyear
This is how our dataframe looks after adding newly engineered features
Repeat the same feature engineering steps on test data
test_df['date'] = pd.to_datetime(test_df['date']) test_df['Month'] = test_df.date.dt.month test_df['Week'] = test_df.date.dt.week test_df['Day'] = test_df.date.dt.day test_df['Quarter'] = test_df.date.dt.quarter test_df['Weekday'] = test_df.date.dt.dayofweek test_df['Dayofyear'] = test_df.date.dt.dayofyear
Note that here ‘Day’ means what day of the month it is, while ‘Dayofyear’, as the name suggests, represents what day of the year it is.
Now that we have extracted additional features from ‘date’, can we do something about the ‘hour’ feature? Can we somehow aggregate it to form a new feature?
Generating new feature ‘time2’ based on what part of the day the ‘hour’ falls in
def time_day if t in [12, 13, 14]: return 'Afteroon' elif t in [15, 16, 17]: return 'Evening' elif t in [18, 19, 20]: return 'Late_evening' elif t in [21, 22, 23]: return 'Night' elif t in [0, 1, 2]: return 'Late_night' elif t in [3, 4, 5]: return 'Dawn' elif t in [6, 7, 8]: return 'Early_morning' elif t in [9, 10, 11]: return 'Morning'
train_df['time2'] = train_df['hour'].apply(lambda x:time_day(x)) test_df['time2'] = test_df['hour'].apply(lambda x:time_day(x))
So far we have extracted the following features from ‘date’ and ‘hour’ data – ‘Month’, ‘Week’, ‘Day’, ‘Quarter’, ‘Weekday’, ‘Dayofyear’, ‘time2’.
Since the beginning, we are assuming that it is possible to forecast the future demand based on the past data. But, can the past data really tell us about the future? Let’s test our hypothesis using Exploratory analysis.
Hypothesis testing – Exploratory Data Analysis
Let’s first divide our train data
1. From March’19 to February’20
2. From March’20 to February’21
# Mar-19 to Feb-20 train_df_19 = train_df[(train_df['date'] >= '01-03-2019') & (train_df['date'] <= '29-02-2020')] # Mar-20 to Feb-21 train_df_20 = train_df[(train_df['date'] >= '01-03-2020') & (train_df['date'] <= '28-02-2021')]
Plotting demand by ‘Hour’, ‘Month’, ‘Week’, ‘Day’, ‘Quarter’, ‘Weekday’, ‘Dayofyear’, ‘time2’ —
If you observe these lineplots, there is a lot of similarity in the demand trend in the years 2019 and 2020 (March to Feb). Especially if we see ‘Hour’, ‘Quarter’, ‘Weekday’, ‘Dayofyear’ and ‘time2’ plots, the peaks and troughs tend to concur.
Now that we are convinced that the newly engineered features along with the existing features can prove useful in predicting the demand for the subsequent year, let’s proceed to the modeling stage.
Hypothesis testing – Exploratory Data Analysis
Let’s observe our train and test data —
Drop redundant columns from train and test data —
X_train = train_df.drop(['date'], axis = 1) X_test = test_df.drop(['date'], axis = 1)
y_train = train_df['demand']
Split training data further into training and validation data
from sklearn.model_selection import train_test_split X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = 0.30, shuffle = False) print(X_train.shape, y_train.shape) print(X_val.shape, y_val.shape)
Before we proceed, recall we had handcrafted a new variable ‘time2’? It is a categorical variable and machine learning models, in general, cannot consume categorical variables directly. So it needs to be encoded. We can do label encoding here, but usually, I do not prefer label encoding unless the variable has some inherent ordering. So what other option do we have? Let’s do response encoding!
Mean-encoding ‘time2’ variable
agg_df = pd.DataFrame(X_train.groupby(['time2']).agg('demand':'mean')).reset_index() agg_df['demand'] = round(agg_df['demand'], 2) agg_dict = dict(agg_df.values) print(agg_dict) X_train['time2'] = X_train['time2'].apply(lambda x:agg_dict[x]) X_val['time2'] = X_val['time2'].apply(lambda x:agg_dict[x])
The above dictionary represents the numerical values that these categorical variables take upon encoding.
Let’s train some tree-based ensembles and perform testing on the validation data in order to select the best model. The metric for scoring is Root mean-squared error (RMSE) —
from xgboost import XGBRegressor from lightgbm import LGBMRegressor from catboost import CatBoostRegressor from sklearn.metrics import mean_squared_error
X_train = X_train.drop(['demand'], axis = 1) X_val = X_val.drop(['demand'], axis = 1)
models = [XGBRegressor(), LGBMRegressor(), CatBoostRegressor()] for model in models: model.fit(X_train, y_train) y_train_pred = model.predict(X_train) y_val_pred = model.predict(X_val) train_error = mean_squared_error(y_train, y_train_pred) validation_error = mean_squared_error(y_val, y_val_pred) print("Model: ", model) print("Train RMSE:", round(np.sqrt(train_error), 4)) print("Validation RMSE:", round(np.sqrt(validation_error), 4))
|Train RMSE||Validation RMSE|
(Here I am displaying the result of the three models in the tabular format for the sake of brevity).
The comparative analysis of the performance of the models on the validation set reveals that the Light-GBM regressor model is the best among the three. So we select LGBMRegressor.
Now repeat all the above steps on the complete training dataset and make the prediction on the test dataset —
X_train = train_df.copy() X_train = X_train.drop(['date'], axis = 1)
# encoding time2 variable agg_df = pd.DataFrame(X_train.groupby(['time2']).agg('demand':'mean')).reset_index() agg_df['demand'] = round(agg_df['demand'], 2) agg_dict = dict(agg_df.values) X_train['time2'] = X_train['time2'].apply(lambda x:agg_dict[x]) X_test['time2'] = X_test['time2'].apply(lambda x:agg_dict[x])
# lgbm regressor y_train = X_train['demand'] X_train = X_train.drop(['demand'], axis = 1) lgbm = LGBMRegressor() lgbm.fit(X_train, y_train) y_train_pred = lgbm.predict(X_train) y_test_pred = lgbm.predict(X_test) train_error = mean_squared_error(y_train, y_train_pred) print("Train RMSE:", round(np.sqrt(train_error), 4))
Finally, make a submission with the predicted values —
submit_df = pd.read_csv('sample_4E0BhPN.csv') submit_df['demand'] = y_test_pred submit_df.to_csv('lgbm_baseline.csv', index = False)
# display the submission file submit_df
Let’s see what all features contributed towards predicting the rental cabs demand and their relative importance.
feat_df = pd.DataFrame('Features': X_train.columns, 'Relative importance': lgbm.feature_importances_) imp_feat_df = feat_df.sort_values('Relative importance', ascending = False) plt.figure(figsize = (8, 6)) sns.barplot(x = 'Relative importance', y = 'Features', data = imp_feat_df) plt.grid() plt.show()
Turns out that ‘Dayofyear’ is the most important feature followed by ‘Day’ and ‘hour’. Also, the feature ‘Quarter’ is negligibly significant for our prediction task, so it can be discarded.
Conclusion on Machine learning
In the beginning, were provided with the hourly data of the Car rentals from mid-Aug 2018 to Feb 2021. We did some high-level analysis, followed by feature engineering. Using newly extracted features we tested our hypothesis by performing exploratory analysis. Then finally we trained a regression model to predict the hourly demand of the rental Cars from Mar 2021 to Mar 2022.
Key Takeaways on machine learning
- Feature engineering is good and can drastically improve the model performance, but overdoing it can lead to overfitting resulting in bad performance on the test data.
- Make sure you keep train and test (or validation) data separate while encoding categorical variables to avoid data leakage issues.
- The rank on the public leaderboard is often deceptive. Here your prediction is partially evaluated so you may end up with a sub-optimal model. Always make sure you test your models on validation data first before deciding which model to finalize.
- Here we used all baseline models. So once the best model is selected, we can try hyperparameter tuning to see if the model performance improves further.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.