This notebook was created by Jean de Dieu Nyandwi for the love of machine learning community. For any feedback, errors or suggestion, he can be reached on email (johnjw7084 at gmail dot com), Twitter, or LinkedIn.

Linear Models for Regression¶

Early in the previous parts, we saw that regression is a supervised learning type in which we are predicting a continous value. Take an example, given the information about a house such as its region, size, number of bedrooms, can you predict the price of the house?

That is in fact what you will see in this lab and to achieve that, we will use Regression models available in Scikit-Learn.

Let's get started. How exciting is that!!

As much as we can, we will try to structure all machine learning projects in accordance to the standard ML worklow. Here are the typical steps that you will see in most ML projects:

• 1. Problem formulation
• 2. Collecting the data
• 3. Exploraroty Data Analysis
• 4. Data Processing
• 5. Choosing and Training a model
• 6. Evaluating a model
• 7. Improving a model

1. Problem Formulation¶

Here is the problem. There is a real state agent who knows that you're a Machine Learning Engineer and would like you to help out with building a machine learning model that can predict the price of the house given the information about that particular house.

The idea of creating a model clicked and replied, oh, yeah, that sounds cool! Let's do it!!

You have understood the problem and probably you already have an idea of type of models you will use. You know there are so many models such as random forests, decision trees, and neural networks but you have learned that it's always okay to start simple and so, now you know you will use Linear Regression as this is not a complex problem.

Understanding the problem well goes beyond determining the right models for the problem, but also to doing effecting data processing and error analysis as you will be finding out.

It's time to collect the data now.

2. Collecting the data¶

Ideally, the real estate agent would hand you his/her housing data but unfortunately he/she told you, that the model will be used in California, so it's okay to use the California housing dataset available on public and free to use.

So, we will collect the data from the internet. Fortunately, Scikit-Learn provides the exact same data. Let's load it but first, let's import all relavant libraries that we will need.

In [2]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline from sklearn.datasets import fetch_california_housing from sklearn.linear_model import LinearRegression

In [3]:

# Loading the data

X, y = fetch_california_housing(return_X_y=True)

# Loading the data X, y = fetch_california_housing(return_X_y=True)

In [4]:

type(X)

type(X)

Out[4]:

numpy.ndarray

In [5]:

type(y)

type(y)

Out[5]:

numpy.ndarray

Now that we have the dataset, we can try to see what it looks like. X is training data and y is training labels.

But wait, the data from Sklearn is a NumPy array and it seems it prepared to be fed to the model directly. That could easier, but quite often, real world data they are not like that. We often have to do our work in order to process it to be fed to the ML model.

Let's get the real data. You can learn more about the data at Kaggle.

In [6]:

import urllib.request 

data_path = 'https://raw.githubusercontent.com/nyandwi/public_datasets/master/housing.csv'

def download_read_data(path):
    
    """
     Function to retrieve data from the data paths
     And to read the data as a pandas dataframe
  
    To return the dataframe
    """ 
    
      ## Only retrieve the directory of the data

    data_path =  urllib.request.urlretrieve(path)[0]
    data = pd.read_csv(path)
    
    return data

import urllib.request data_path = 'https://raw.githubusercontent.com/nyandwi/public_datasets/master/housing.csv' def download_read_data(path): """ Function to retrieve data from the data paths And to read the data as a pandas dataframe To return the dataframe """ ## Only retrieve the directory of the data data_path = urllib.request.urlretrieve(path)[0] data = pd.read_csv(path) return data

Now that we have real world data, let's see how it looks like.

In [7]:

cal_data = download_read_data(data_path)

cal_data = download_read_data(data_path)

In [8]:

cal_data.head()

cal_data.head()

Out[8]:

	longitude	latitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	median_house_value	ocean_proximity
0	-122.23	37.88	41.0	880.0	129.0	322.0	126.0	8.3252	452600.0	NEAR BAY
1	-122.22	37.86	21.0	7099.0	1106.0	2401.0	1138.0	8.3014	358500.0	NEAR BAY
2	-122.24	37.85	52.0	1467.0	190.0	496.0	177.0	7.2574	352100.0	NEAR BAY
3	-122.25	37.85	52.0	1274.0	235.0	558.0	219.0	5.6431	341300.0	NEAR BAY
4	-122.25	37.85	52.0	1627.0	280.0	565.0	259.0	3.8462	342200.0	NEAR BAY

In [9]:

cal_data.tail()

cal_data.tail()

Out[9]:

	longitude	latitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	median_house_value	ocean_proximity
20635	-121.09	39.48	25.0	1665.0	374.0	845.0	330.0	1.5603	78100.0	INLAND
20636	-121.21	39.49	18.0	697.0	150.0	356.0	114.0	2.5568	77100.0	INLAND
20637	-121.22	39.43	17.0	2254.0	485.0	1007.0	433.0	1.7000	92300.0	INLAND
20638	-121.32	39.43	18.0	1860.0	409.0	741.0	349.0	1.8672	84700.0	INLAND
20639	-121.24	39.37	16.0	2785.0	616.0	1387.0	530.0	2.3886	89400.0	INLAND

Information about the features 

1. longitude: A measure of how far west a house is; a higher value is farther west
2. latitude: A measure of how far north a house is; a higher value is farther north
3. housing_median_age: Median age of a house within a block; a lower number is a newer building
4. total_rooms: Total number of rooms within a block
5. total_bedrooms: Total number of bedrooms within a block
6. population: Total number of people residing within a block
7. households: Total number of households, a group of people residing within a home unit, for a block
8. median_income: Median income for households within a block of houses (measured in tens of thousands of US Dollars)
9. median_house_value: Median house value for households within a block (measured in US Dollars)
10. oceanProximity: Location of the house w.r.t ocean/sea

Source: Kaggle

In [10]:

cal_data.info()

cal_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   longitude           20640 non-null  float64
 1   latitude            20640 non-null  float64
 2   housing_median_age  20640 non-null  float64
 3   total_rooms         20640 non-null  float64
 4   total_bedrooms      20433 non-null  float64
 5   population          20640 non-null  float64
 6   households          20640 non-null  float64
 7   median_income       20640 non-null  float64
 8   median_house_value  20640 non-null  float64
 9   ocean_proximity     20640 non-null  object 
dtypes: float64(9), object(1)
memory usage: 1.6+ MB

In [11]:

len(cal_data)

len(cal_data)

Out[11]:

20640

In [12]:

len(cal_data.columns)

len(cal_data.columns)

Out[12]:

10

So, we have 20640 data points and 10 features. In those 10 features, 9 features are input features and the feature median_house_value is the target variable/label.

3. Exploratory Data Analysis¶

This is where we are going to understand more about the data. But before we get there, let's split the data into training and testing sets. This is because on the course of data analysis and processing, there can be data leakage. Put it in other words, we don't want the model to see the data that it will be tested on. If it does, it will show that it can make good predictions (or generalize well) on test data because it saw it, but it will fail on giving accurate predictions on the future data.

As a side notes, training set is used during the model training, and testing set is used during the model evaluation. As we go we will try to explain some terminologies! ML is huge :)

So to split the data, sklearn provides a function for that.

In [13]:

from sklearn.model_selection import train_test_split

train_data, test_data = train_test_split(cal_data, test_size=0.1,random_state=20)

from sklearn.model_selection import train_test_split train_data, test_data = train_test_split(cal_data, test_size=0.1,random_state=20)

In [14]:

print('The size of training data is: {} \nThe size of testing data is: {}'.format(len(train_data), len(test_data)))

print('The size of training data is: {} \nThe size of testing data is: {}'.format(len(train_data), len(test_data)))

The size of training data is: 18576 
The size of testing data is: 2064

As you can see, we have allocated 10 percent of the full data to the test set.

In [15]:

# Let's copy the training data to revert it in case we mess things up

cal_train = train_data.copy()

# Let's copy the training data to revert it in case we mess things up cal_train = train_data.copy()

Checking data statistics¶

In [16]:

# By default, describe shows the stats of the numerical features. 
# include paramater gives us option to show all features 

train_data.describe(include='all').transpose()

# By default, describe shows the stats of the numerical features. # include paramater gives us option to show all features train_data.describe(include='all').transpose()

Out[16]:

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
longitude	18576.0	NaN	NaN	NaN	-119.56753	2.000581	-124.35	-121.79	-118.49	-118.01	-114.49
latitude	18576.0	NaN	NaN	NaN	35.630217	2.13326	32.54	33.93	34.26	37.71	41.95
housing_median_age	18576.0	NaN	NaN	NaN	28.661068	12.604039	1.0	18.0	29.0	37.0	52.0
total_rooms	18576.0	NaN	NaN	NaN	2631.567453	2169.46745	2.0	1445.0	2127.0	3149.0	39320.0
total_bedrooms	18390.0	NaN	NaN	NaN	537.344698	417.672864	1.0	295.0	435.0	648.0	6445.0
population	18576.0	NaN	NaN	NaN	1422.408376	1105.486111	3.0	785.75	1166.0	1725.0	28566.0
households	18576.0	NaN	NaN	NaN	499.277078	379.473497	1.0	279.0	410.0	606.0	6082.0
median_income	18576.0	NaN	NaN	NaN	3.870053	1.900225	0.4999	2.5643	3.5341	4.742725	15.0001
median_house_value	18576.0	NaN	NaN	NaN	206881.011305	115237.605962	14999.0	120000.0	179800.0	264700.0	500001.0
ocean_proximity	18576	5	<1H OCEAN	8231	NaN	NaN	NaN	NaN	NaN	NaN	NaN

Checking Missing Values¶

In [17]:

train_data.isnull().sum()

train_data.isnull().sum()

Out[17]:

longitude               0
latitude                0
housing_median_age      0
total_rooms             0
total_bedrooms        186
population              0
households              0
median_income           0
median_house_value      0
ocean_proximity         0
dtype: int64

We only have missing values in the total_bedrooms feature. To g beyond a little bit, here is the percentage of the missing values in such feature.

In [18]:

print('The Percentage of missing values in total_bedrooms is: {}%'.format(train_data.isnull().sum()['total_bedrooms'] / len(train_data) * 100))

print('The Percentage of missing values in total_bedrooms is: {}%'.format(train_data.isnull().sum()['total_bedrooms'] / len(train_data) * 100))

The Percentage of missing values in total_bedrooms is: 1.0012919896640826%

Checking Values in the Categorical Feature(s)¶

In [19]:

train_data['ocean_proximity'].value_counts()

train_data['ocean_proximity'].value_counts()

Out[19]:

<1H OCEAN     8231
INLAND        5896
NEAR OCEAN    2384
NEAR BAY      2061
ISLAND           4
Name: ocean_proximity, dtype: int64

In [20]:

sns.countplot(data=train_data, x='ocean_proximity')

sns.countplot(data=train_data, x='ocean_proximity')

Out[20]:

<AxesSubplot:xlabel='ocean_proximity', ylabel='count'>

Checking Correlation Between Features¶

In [21]:

correlation = train_data.corr()
correlation['median_house_value']

correlation = train_data.corr() correlation['median_house_value']

Out[21]:

longitude            -0.048622
latitude             -0.142543
housing_median_age    0.105237
total_rooms           0.133927
total_bedrooms        0.049672
population           -0.026109
households            0.065508
median_income         0.685433
median_house_value    1.000000
Name: median_house_value, dtype: float64

In [22]:

#### Visualizing correlation

plt.figure(figsize=(12,7))

sns.heatmap(correlation,annot=True,cmap='crest')

#### Visualizing correlation plt.figure(figsize=(12,7)) sns.heatmap(correlation,annot=True,cmap='crest')

Out[22]:

<AxesSubplot:>

Some features like total_bedrooms and households are highly correlated. Same things for total_bedrooms and total_rooms and that makes sense because for many houses, the number of people who stay in that particular house (households) goes with the number of available rooms(total_rooms) and bed_rooms.

The other interesting insights is that the price of the house is closely correlated with the median income, and that makes sense too. For many cases, you will resonably seek house that you will be able to afford based on your income.

Plotting Geographical Features¶

Since we have latitude and longitude, let's plot it. It can help us to know the location of certain houses on the map and hopefully this will resemble California map.

In [23]:

plt.figure(figsize=(12,7))

sns.scatterplot(data = train_data, x='longitude', y='latitude')

plt.figure(figsize=(12,7)) sns.scatterplot(data = train_data, x='longitude', y='latitude')

Out[23]:

<AxesSubplot:xlabel='longitude', ylabel='latitude'>

In [24]:

plt.figure(figsize=(12,7))

sns.scatterplot(data = train_data, x='longitude', y='latitude', hue='median_house_value')

plt.figure(figsize=(12,7)) sns.scatterplot(data = train_data, x='longitude', y='latitude', hue='median_house_value')

Out[24]:

<AxesSubplot:xlabel='longitude', ylabel='latitude'>