Ml Model Not Predicting Properly

I am trying to create an ML model (regression) using various techniques like SMR, Logistic Regression, and others. With all the techniques, I'm not able to get efficiency more than

Solution 1:

As a rule of thumb, we usually follow this convention:

1. For little number of features, go with Logistic Regression.
2. For a lot of features but not a lot of data, go with SVM.
3. For a lot of features and a lot of data, go with Neural Network.

Because your dataset is a 10K cases, it'd be better to use Logistic Regression because SVM will take forever to finish!.

---

Nevertheless, because your dataset contains a lot of classes, there is a chance of classes imbalance in your implementation. Thus I tried to workaround this problem via using the StratifiedKFold instead of train_test_split which doesn't guarantee balanced classes in the splits.

Moreover, I used GridSearchCV with StratifiedKFold to perform Cross-Validation in order to tune the parameters and try all different optimizers!

So the full implementation is as follows:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, StratifiedShuffleSplit
import numpy as np


defgetDataset(path, x_attr, y_attr):
    """
    Extract dataset from CSV file
    :param path: location of csv file
    :param x_attr: list of Features Names
    :param y_attr: Y header name in CSV file
    :return: tuple, (X, Y)
    """
    df = pd.read_csv(path)
    X = X = np.array(df[x_attr]).reshape(len(df), len(x_attr))
    Y = np.array(df[y_attr])
    return X, Y

defstratifiedSplit(X, Y):
    sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
    train_index, test_index = next(sss.split(X, Y))
    X_train, X_test = X[train_index], X[test_index]
    Y_train, Y_test = Y[train_index], Y[test_index]
    return X_train, X_test, Y_train, Y_test


defrun(X_data, Y_data):
    X_train, X_test, Y_train, Y_test = stratifiedSplit(X_data, Y_data)
    param_grid = {'C': [0.01, 0.1, 1, 10, 100, 1000], 'penalty': ['l1', 'l2'],
                  'solver':['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga']}
    model = LogisticRegression(random_state=0)
    clf = GridSearchCV(model, param_grid, cv=StratifiedKFold(n_splits=10))
    clf.fit(X_train, Y_train)
    print(accuracy_score(Y_train, clf.best_estimator_.predict(X_train)))
    print(accuracy_score(Y_test, clf.best_estimator_.predict(X_test)))


X_data, Y_data = getDataset("data - Sheet1.csv", ['distance'], 'orders')

run(X_data, Y_data)

---

Despite all the attempts with all different algorithms, the accuracydidn't exceed 36%!!.

---

Why is that?

If you want to make a person recognize/classify another person by their T-shirt color, you cannot say: hey if it's red that means he's John and if it's red it's Peter but if it's red it's Aisling!! He would say "really, what the hack is the difference"?!!.

And that's exactly what is in your dataset!

Simply, run print(len(np.unique(X_data))) and print(len(np.unique(Y_data))) and you'll find that the numbers are so weird, in a nutshell you have:

NumberofCases: 10000 !!
NumberofClasses: 118 !!
NumberofUniqueInputs (i.e. Features): 66 !!

All classes are sharing hell a lot of information which make it impressive to have even up to 36% accuracy!

In other words, you have no informative features which lead to a lack in the uniqueness of each class model!

---

What to do? I believe you are not allowed to remove some classes, so the only two solutions you have are:

1. Either live with this very valid result.

2. Or add more informative feature(s).

---

Update

Having you provided same dataset but with more features (i.e. complete set of features), the situation now is different.

I recommend you do the following:

1. Pre-process your dataset (i.e. prepare it by imputing missing values or deleting rows containing missing values, and converting dates to some unique values (example) ...etc).

2. Check what features are most important to the Orders Classes, you can achieve that by using of Forests of Trees to evaluate the importance of features. Here is a complete and simple example of how to do that in Scikit-Learn.

3. Create a new version of the dataset but this time hold Orders as the Y response, and the above-found features as the X variables.

4. Follow the same GrdiSearchCV and StratifiedKFold procedure that I showed you in the implementation above.

---

Hint

As per mentioned by Vivek Kumar in the comment below, stratify parameter has been added in Scikit-learn update to the train_test_split function.

It works by passing the array-like ground truth, so you don't need my workaround in the function stratifiedSplit(X, Y) above.