Unable To Evaluate Score Using Decision_function() In Logistic Regression

I'm doing this Univ. Of Washington assignment where i have to predict the score of sample_test_matrix (last few lines) using decision_function() in LogisticRegression . But the err

Solution 1:

This line is causing errors in the subsequent lines:

test_matrix = vectorizer.fit_transform(test_data['review_clean'])

Change the above to this:

test_matrix = vectorizer.transform(test_data['review_clean'])

Explanation: Using fit_transform() will refit the CountVectorizer on the test data. So all the information about the training data will be lost and vocabulary will be calculated only from test data.

Then you are using that vectorizer object to transform the sample_data['review_clean']. So the features in that will be only those which are learnt from test_data.

But the sentiment_model is trained on vocabulary from train_data. Hence the features are different.

Always use transform() on test data, never fit_transform().