The file "airlinetweets.csv" contains sentiments (positive, neutral and negative) of the tweets for different airline companies. You have to learn a model using neural network classifier that is able to classify the tweets into one of the three classes.

Importing libraries

from sklearn.neural_network import MLPClassifier
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score, recall_score, precision_score
import re

Function to clean the tweets

def cleaner(impure_data):
    temp_list = []
    for item in impure_data:
        #finding words which start with @
        item = re.sub('@\S+', '', item)
        
        #finding words which start with http
        item = re.sub('http\S+\s*', '', item)
        
        #finding special characters, but not "emoji"
        item = re.sub('[%s]' % re.escape("""!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""), '', item)
        temp_list.append(item)
    return temp_list

Function to load tweets and vectorize them

def load_tweets():
    #reading the tweets from csv files
    df = pd.read_csv("airlinetweets.csv")
    tweets = df["text"]
    polarity = df["airline_sentiment"].tolist()
    
    #cleaning tweets i.e. removing @mentions, http(s) links and special characters such as punctuations
    clean_tweets = cleaner(tweets)
    
    return(clean_tweets, polarity)

def vectorize_tweets(clean_tweets, polarity):
    #splitting the data into train and test dataset in 70 : 30 ratio at random
    X_train, X_test, Y_train, Y_test = train_test_split(clean_tweets, polarity, test_size = 0.3)
    
    #initializing tf-idf vectorizer
    tf_idfvectorizer = TfidfVectorizer(sublinear_tf=True, use_idf=True)
    
    #vectorizing the training data
    #fit_transform() does two jobs, fit() and transform()
    #fit calculates the statistics of the data
    #transform takes care of any missing values or unexpected values by utilizing statistics calculated by fit
    X_train_vectorized = tf_idfvectorizer.fit_transform(X_train) 
        
    #vectorizing the testing data
    #transform takes care of any missing values or unexpected values based on fit for training data
    X_test_vectorized = tf_idfvectorizer.transform(X_test)
    
    return(X_train_vectorized, X_test_vectorized, X_train, X_test, Y_train, Y_test, tf_idfvectorizer)

Loading tweets and vectorizing them

clean_tweets, polarity = load_tweets()
X_train_vectorized, X_test_vectorized, X_train, X_test, Y_train, Y_test, tf_idfvectorizer = vectorize_tweets(clean_tweets, polarity)

Learning the sentiment model using neural network

#using MLP classifier package to initialize a classifier with two hidden layers
clf = MLPClassifier(hidden_layer_sizes=(5,5))

#fitting the sparse matrix in the classifier with their respective sentiments
clf.fit(X_train_vectorized, Y_train)

#predicting the sentiments for the test dataset
Y_pred = clf.predict(X_test_vectorized)

#this print accuracy score for the test dataset
print("Accuracy",accuracy_score(Y_test,Y_pred))

Accuracy 0.7349726775956285

Function to predict tweet sentiment and store it in a data file

The predicted tweets are stored in "predicted_airlinetweets.csv" file. Classification of the tweets has been done using neural network. Note that it is a multi-class classification task.

#saving the data into a csv file in the current folder
temp_df = pd.DataFrame()
temp_df["Tweet"] = X_test
temp_df["Sentiment"] = Y_test
temp_df["Predicted Sentiment"] = Y_pred
temp_df.to_csv("predicted_airlinetweets.csv")

Predict sentiments

vector = tf_idfvectorizer.transform(["My journey was good. Thanks to your customer service."])
sentiment = clf.predict(vector)
print(sentiment)

['positive']

vector = tf_idfvectorizer.transform(["My journey was horrible because of your customer service."])
sentiment = clf.predict(vector)
print(sentiment)

['negative']

vector = tf_idfvectorizer.transform(["My journey was not good because of your crew."])
sentiment = clf.predict(vector)
print(sentiment)

['positive']

vector = tf_idfvectorizer.transform(["Travel was not okay."])
sentiment = clf.predict(vector)
print(sentiment)

['negative']

vector = tf_idfvectorizer.transform(["Travel was okay."])
sentiment = clf.predict(vector)
print(sentiment)

['neutral']