Machine Learning and Data Science Blog

Natural Language Processing can be implemented in python using Natural Language Toolkit (NLTK) which is a suite of python libraries for text analysis and human language data processing.

Before Starting with NLTK lets understand what is Natural Language Processing (NLP). NLP is extracting information from unstructured data like finding the name of the places, names and named entities from given text and analyzing the linguist structure in the text like semantics analysis.

NLTK is suites of open source libraries in python and using these libraries we do Natural Language Processing in python on human language data in text form. NLTK has over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

How to install NLTK

Data cleaning is the first process you must be doing to get the better insight and better performance of your machine learning models. It's most important in the process of data Data preparation. One of the main challenges in the data cleaning is handling the missing values in the dataset. The missing values is a piece of information that was not available during the data creation or the value is unknown. In simple words it is NULL.

Here I will be explaining the different ways of dealing with missing values in datasets

Ignore observations:

In this approach, we delete the observation rows which has the null values in it and would not be used in model training and testing. This is done before splitting data into training/testing set. This method of dealing with missing doesn't impact the model training much when we have huge and good distributed data sets and number of record which has missing data is very less about less than 5% of the datasets. However, when the data set size not huge, it is not optimal approach because the significant information could be lost when incomplete rows of data are discarded and impact our Learning models.

If we are working with multivariable analysis and there are large no. missing values for that column, it's better to drop that column from the dataset.

Deleting the row with the missing value in python :

       
import pandas as pd
df = pd.read_csv('Sample.csv',encoding='utf-8')
df = df.dropna(how='any',axis=0)
       
 

axis : {0 or ‘index’, 1 or ‘columns’}, default 0
Determine if rows or columns which contain missing values are removed.
0, or ‘index’ : Drop rows which contain missing values.
1, or ‘columns’ : Drop columns which contain missing value.
Deprecated since version 0.23.0:: Pass tuple or list to drop on multiple axes.

how : {‘any’, ‘all’}, default ‘any’
Determine if row or column is removed from DataFrame, when we have at least one NA or all            NA.

Deleting the row with the missing value in R:

       
dataset[complete.cases(dataset), ]
 

note: This will delete the rows having any of column value as NULL. complete.cases also allow deleting only the rows with the certain column with the null value.

       
dataset[complete.cases(dataset[ , 2:3]),]
 

Statistical Approach:

There is another way to handle the missing data without losing the information. The basic idea of this approach is instead of deleting the rows we can substitute the missing value with sensible data.  we Often use mean of the columns to replace null in the dataset but median and mode can also be used.

In python, the scikit-learn library provides the Imputer() pre-processing class that can be used to replace the null with Mean, Median, and Mode

The strategy parameter is set to mean which mean the missing value will be replaced by that column mean. similarly for Median and Mode.

Python 

from sklearn.preprocessing import Imputer
imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)
imputer = imputer.fit(X[:, 1:5])
X[:, 1:5] = imputer.transform(X[:, 1:5])

Note: we can use the different strategy like mean, median and mode.

R

In R, this is different there is no imputer class present

       
emp$age = ifelse(is.na(emp$age), ave(emp$age, FUN = function(x) mean(x, na.rm = TRUE)),emp$age)

emp$age represents the column  name age in the dataset emp

Hope you have liked this post please share your feedback and questions in the comments.