Logistic Regression From Scratch

When attempting to identify one class from another, the algorithm for machine learning of logistic regression is applied. A binary segmentation is the most basic situation. This is like a query to which we can only respond “yes” or “no.” There are just two broad classifications: positive and negative. Typically, a classifier denotes the existence of a thing, whereas a critical class denotes its absence.

Given below is the equation of Logistic Regression:

Here,
x = input value
y = predicted output
b0 = bias or intercept term
b1 = coefficient for input (x)

Let’s implement logistic regression from scratch in python without sklearn: Step 1. Import dependent libraries, and generate a simple dataset

To explain firstly, Pandas is to handle Dataframes which is nothing but row and column format of storing data which is one of the characteristic features of Python , Numpy is for matrix calculations, and Matplotlib is for data visualisation. Lastly, Random is to import diverse functions featuring randomness in it.
The generated dataset is very simple, only having two columns; age and whether the person bought a house or not. If the person had one, then 1, if not, then 0.
Step 2. Split dataset into two parts which is Training, and Testing:

For simplification, we only made 27 units in our dataset. Out of the 27, we will leave 7 for testing, to see how our Numpy backed Logistic Regression performs on unfamiliar data. And the remaining 20 will be used to learn the parameters in Logistic Regression model.
Since we will check the performance level of our model after training it, the target value we are aiming is [1 1 0 0 0 1 1] which means first two and the last two of the testing dataset have bought a house.

Step 3: Place an Activation function:

Activation functions are nothing but function gives non-linearity to the linearly computed value. This is one of the super important features of machine learning which you’ll encounter during the neural networks part!
The sigmoid used here is a mathematical function having a characteristic that can take any real value and map it to between 0 to 1 shaped like the letter “S”. The sigmoid function also called a logistic function and known as an activation function which in simple words it decides which value to pass as output and what not to pass.

Step 4: Set up an loss function:

There are literally thousands of loss functions. You can even make your own one if you want, but here we will deploy the most basic and one of the widely used one. Square loss function, which calculates the euclidean distance between the target value and the predicted one.

Step 5: Split X (feature) part and y (target) part:

As you can see to select a column, which could be regarded as a series in python, there are two ways: using a dot to indicate certain column or using square brackets and assigning column name in it as a string value.

Step 6 : Set up the Model and Running it:

‘lr’ in the code signifies ‘learning rate’, and this would be multiplied to the gradient value so that this will determine how much you would like to move from the current spot for better performance of the model. And the gradient, which I mentioned just before shows how much you missed out from the target value. So iteratively updating gradients and fixing it with the tuned learning rate, we could hopefully and finally reach the desired performance of the model.

Step 7 : Test the performance of the model:

After step 6, we would have W and b, which has been updated 10,000 times iteratively. So, what we have is a linear discriminating function whose slope is W and intercept is b.

Output:

What we have concluded is for the 7 people in the test dataset. And the end result is [1, 1, 1, 0, 0, 1, 1], so we have missed the 3rd person’s decision, compared to the target [1 1 0 0 0 1 1].
There are tons of ways to upgrade this simple model. You can change the learning rate or a number of iterations. Also, you can deploy other types of loss functions such as cross-entropy or log-likelihood and so forth.

Applications of Logistic Regression:
1. Credit scoring:
One such stage in the data preparation for credit score modelling is the reduction of correlated variables. If your model has more than 15 variables, it becomes challenging. It is simple to determine which factors have a greater and lesser impact on the predictions’ outcome when using logistic regression. Additionally, using techniques like recursive feature elimination, it is feasible to determine the ideal amount of features and get rid of extraneous variables.

2. Text Editing:
Given that we have been discussing texts, it is important to note that many natural language processing tasks favour logistic regression. Prior to using logistic regression to draw conclusions about a text fragment, text preprocessing is first carried out, followed by feature extraction. Examples of applications where logistic regression performs well include email sorting, subject categorization for inquiries to support, and toxic speech detection. Support vector machines and random forests are two more widely used methods in these applications.

3. Hotel Bookings:
Machine learning techniques are practically used everywhere on Booking.com. They aim to identify things and anticipate user intentions. Where are you going, where do you like to stop, and what are you doing? Even if the user hasn’t yet typed anything in the search line, certain predictions are created. How did they begin to do this, though? No one is capable of creating a large-scale, intricate system using several machine learning algorithms from start. As a first step, they gathered basic facts and made some straightforward models.