Machine Learning and Data Science Blog

Create a user in Oracle database

We are creating a user in oracle, using this user will connect to DB from python. 

 I have used the name "PYTHON" for the username, you can choose a different name just replace your username with "PYTHON" in the below SQL commands.

open command prompt and type "sqlplus / as sysdba" , this will open sqlplus utility and now execute the below command one by one in exact order.

 Connect to the user using command  conn PYTHON, provide the password PYTHON as shown below

we will create a sample table with data, execute below SQL commands in prompt to create a table and insert data.

now we are done with the Oracle part next, we will install the required python library.

we are going to use cx_Oracle library to connect python and oracle.
like all other libraries, we will install this using PIP command 

open command prompt execute the below pip command 

to see package information type, pip show cx_Oracle, it will list all the details about the package.

Import the required libraries

import cx_Oracle
import pandas as pd

create the connection by providing username, password and server information

conn = cx_Oracle.connect('PYTHON/PYTHON@localhost')

Create a cursor object from the connection

cursor = conn.cursor()

for now, we are using a simple query

query = 'SELECT * FROM WORLD'

execute the query and print the result

cursor.execute(query)

for row in cursor:
    print (row)

Result: -

('AFGHANISTAN', 'ASIA', 652230, 25500100, 20343000000)
('ALBANIA', 'EUROPE', 28748, 2831741, 12960000000)
('ALGERIA', 'AFRICA', 2381741, 37100000, 188681000000)
('ANDORRA', 'EUROPE', 468, 78115, 3712000000)
('ANGOLA', 'AFRICA', 1246700, 20609294, 100990000000)

Close the cursor and connection

cursor.close()
conn.close()

How to load pandas Dataframe from oracle:

What is Dimensionality Reduction

Dimensionality reduction is a technique to reduce the dimensions of the dataset and Dimension of the dataset is the number of features(columns) it has, in other words, Dimensionality reduction means reducing the number of features (columns) of the dataset, still preserving the most of information it has before.

Mathematically, reducing a matrix of MxN to MxP where P<<N.

The need for Dimensionality Reduction.

• Visualization: One of the most important of data analysis is Data Visualisation. In data visualisation, we plot one feature against other features in 2D or 3D space to study the relationship between them how they are related? what is the correlation between them?  what is the pattern? also, it helps in identifying the groups or cluster in data at the very beginning stage of data analysis. we can't apply the visualization to the high dimensional data as it one feature represent one dimension and 1000 features means 1000 dimension to study our data. which doesn't make sense as we are limited up to 3-dimensional only. The best way is to reduce the dimension of the data from 1000 features to 3 features and represent the data on a 3D plot and it will still make sense we have reduced the dimension but we haven't lost much of the information. 

In this post, I am writing about the difference between python 2.X and 3.X and I will try to keep it short and informative.

String Handling

In Python 3, text strings are Unicode by default. In Python 2, strings are stored as ASCII by default–you have to add a “u” if you want to store strings as Unicode in Python 2.x.


Python 3  integer division

Python 2.x rounds your calculation down to the nearest whole number
Python 3, the division expression will return the expected result of without having to worry about adding those extra zeroes.

Print statement 

In Python 3, the print statement in Python 2 is replaced by print() function
Python 2 --> print “hello world!”
Python 3 --> print (“hello world!”)


Python 2 range() vs xrange() and Python 3 range()


In Python 2, range() returns list and xrange() memory efficient iterable and in python 3 xrange() functionality implemented in range(). There is no xrange in python 3

I have started working on Image dataset for one of my projects and one who is working or worked on image dataset can tell how much important is image processing. Like any other dataset type, the image data also need to be converted into computer understandable format, it also has noise which needs to be removed before feature extraction for any application. I use OpenCV for all image-related processing and feature extractions.

Neural Network is a mathematical model that mimics the functionality of an actual biological neuron.  It has the ability to learn just like a biological neuron. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems that it has learned from the dataset provided to the network. Each interconnected processing element (neurons) are capable of processing input information and connection between this neurons help in information signal flow.

To understand this suppose If I give you a typical age problem and in order to solve the problem you may come up with an equation that relates the input and output, this is called logical modeling where we use logic to find the relationship between input and output but in case of Neural Network just feed or train the model with a huge dataset with this relation the Neural Network will just find the logic or will learn the relation between input and output and it will find the pattern. The age problem is very simple if we scale to the more complex problem where we have many inputs and learning the pattern is no longer in the scope traditional logical modeling method. The Neural Network has upper hand when it comes to this cases.  The Neural Network has the capability to learn the complex patterns.

Fig 1

Why we need the activation functions in Neural Networks?

Why we need an activation function in Neural Networks? How our Neural Network is going to behave if we don't use activation function at all, we are going to answer these question in this post. I will try to come with as much clarity of the concepts as much I can and will try to explain by using mathematical equation.

 So we know Neural Network are Universal Function Approximators that means a Neural Network can learn or it can behave like any complex function if we feed the network with enough data. In short, it has the capability of learning complex function from the data itself.  So there has to be a non-linear complex mapping between the input and output of the Neural Network which is achieved by using Non-Linear activation between the layers of the Networks. otherwise, the network reduces to a simple linear transformation system since the sequence of linear operations is a linear operation itself. Let us understand with the below equation.


Consider the Neural Network in fig 1 which as one hidden layer




X = input to the network
W = weight in the layers
B = bias

In equation (a),(b) we have used activation function which is denoted by g. From equation (1) to (4) we have derived some equation that would study how the output is affected when no activation function is applied in the network. In equation (1), the activation is input times(X) weight W of layer 1 and bias B of layer 1, then this signal is used as input to layer 2 activations(from fig 1 we say that the activation in layer 2 is the output of the network). Now the output of layer 1 will be input to layer 2. From equation (3) and (4) we can see that output is just linear dependent on the input X. The network linear transforming the input. The output will be a linear transformation of input, there will be a linear mapping between outputs and inputs and our Neural Network will learn only linear function.


So we can come to the conclusion that we use activation function to introduce non-linearity in the network so it can learn complex non-linear function.

A typical Text classification problem can be solved by the using various machine learning algorithms but prior to this we need to prepare our data, we can't directly feed our data into the machine learning algorithms. The text data needs to be converted into numbers in such form that this numerical data usable for machine learning algorithms, this process of transforming the text data in numerical features is called as feature extraction and also known as vectorization.


The feature extraction or vectorization from text involves tokenizing the string and assigning a unique id to each token, counting the frequencies of each token and normalizing. This technique is also called as "Bag-Of-Word". The two commonly used vectorizers are CountVectorizer and TF-IDF.

TF-IDF is term frequency-inverse document frequency. TF-IDF is used in text analytics to find the high-quality information from text. In this article, I will be explaining TF-IDF and how it is different from CountVectorizer.

To understand TF-IDF first we need to understand CountVectorizer.

CountVectorizer in simple words count the word frequencies and In the sklearn implementation, it Converts a collection of text documents to a matrix of token counts. On feeding the human language text data or corpus in CountVectorizer feature extraction model, it will return a matrix, the columns will be the unique word found in the corpus and one row corresponding to each document in the corpus, the row values will be the count of occurrence of the word(columns) in that document. Thus the weightage is determined by the word count in the document.

however, it may be possible the most frequent word is not very useful in fetching the high-quality information, the words like 'the','a', 'that' occurs very frequently across all documents but doesn't possess any information about the document.

Consider a document about the sports cars and in the document the word "the" has the highest frequency. The word 'the' doesn't give extra information about the document, However, the word "Lamborghini" set context to the document because from the word "Lamborghini" we can say that the document is related to sports cars. You may be thinking why we don't give more weightage to the less occurring word in the document and just reversing the case of the most frequent count and adjust the weightage to the less frequently occurring word but there is a catch in this solution, for example, the word "Jota" occurring one time in the same document and it means a folk dance in northern Spain. now do you think the word "jota" tells something about context?

Nothing!

If we feed this vector to our model, it will give higher weightage to meaningless features and it will miss the important features. So the features in CountVectorizer needs to be normalized.

This problem is solved by using TF-IDF

TF-IDF is the product of term-frequency(TF) and Inverse document frequency (IDF).

Now first let us understand what is term-frequency(TF), TF of a word represents how many times that word appears in a single document.

IDF of a word is the logarithm of the ratio of the total number document in corpus and no. of the document in which that word occurs.

IDF = log(total no. of document/no. of the document in which the word is found)

consider the words "the ", "Lamborghini"  and "jota" and their respective term frequency (tf) is :

tf("the")=100

tf("Lamborghini")=10

tf("jota")=1

and total no. of documents in the corpus is 100, so Inverse document frequency of the words will be

idf("the") = log(100/100) = log(1) = 0

idf("Lamborghini") = log(100/10) = log(10) = 1

idf("jota") = log(100/1) = log(100) = 2 

as we know TF-IDF is the product of TF and IDF so let's calculate the TF-IDF now

tf-idf("the") = 100*0 = 0

tf-idf("Lamborghini") = 10*1 = 10

tf-idf("jota") = 2*1 = 2


it is evident from the above mathematical explanation that using tf-id how we have adjusted the weightage of words. Now the word "the" as 0 zero weightage and the weightage of the word "Lamborghini" is more than that of the word "jota".

However, there is some disadvantage also, the feature extraction is based on the word of bags model, which has nothing to do with the position of the word in the text and semantics and it doesn't explain the co-occurrence with other words how their combination change the context of a sentence.

Hope you liked this post and it was helpful for you, please let me know your feedback and suggestion in the comment section.

Natural Language Processing can be implemented in python using Natural Language Toolkit (NLTK) which is a suite of python libraries for text analysis and human language data processing.

Before Starting with NLTK lets understand what is Natural Language Processing (NLP). NLP is extracting information from unstructured data like finding the name of the places, names and named entities from given text and analyzing the linguist structure in the text like semantics analysis.

NLTK is suites of open source libraries in python and using these libraries we do Natural Language Processing in python on human language data in text form. NLTK has over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

How to install NLTK