Machine learning algorithms

linear regression

Linear regression helps predict continuous outcomes, such as house prices, based on input features like size, income, or number of rooms.

For this linear regression example, we use a real dataset of U.S. housing data available on https://www.kaggle.com/datasets/vedavyasv/usa-housing

The dataset, called USA Housing, contains information about house prices and their related features such as area population, number of rooms, and average income.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the dataset
houses = pd.read_csv("USA_Housing.csv")

# Display the first 20 rows
print(houses.head(20))
# General information
print(houses.info())

Before modeling, it is useful to visualize distributions and correlations to understand relationships between variables. distribution plot

sns.histplot(houses["Price"], bins=30, kde=True)
plt.title("Distribution of House Prices")
plt.show()

correlation heatmap

A heatmap helps identify which features are most correlated with the target variable (`Price`).

corr_matrix = houses.corr()
sns.heatmap(corr_matrix, annot=True, cmap="coolwarm")
plt.title("Correlation Heatmap")
plt.show()

The strongest correlation is typically between Average Area Income and Price, meaning wealthier areas tend to have more expensive houses. preparing the data

preparing the data

Before training the model, we split the dataset into features (X) and target (y).

# Define features (independent variables)
X = houses[["Avg. Area Income", "Avg. Area House Age",
            "Avg. Area Number of Rooms", "Avg. Area Number of Bedrooms",
            "Area Population"]]

# Define target (dependent variable)
y = houses["Price"]

print(X.head())
print(y.head())

splitting data for training and testing

We divide the dataset into training data (used to build the model) and testing data (used to evaluate it).

from sklearn.model_selection import train_test_split

# Split data: 70% for training, 30% for testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

# Training samples: 3500
# Testing samples: 1500

training the model

We now create and train a Linear Regression model using scikit-learn.

from sklearn.linear_model import LinearRegression

# Create model
model = LinearRegression()

# Train the model
model.fit(X_train, y_train)

# Display coefficients and intercept
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

# Create a DataFrame for better readability
coef_df = pd.DataFrame(model.coef_, X.columns, columns=["Coefficient"])
print(coef_df)

                                Coefficient
Avg. Area Income                  21.625799
Avg. Area House Age           165590.392746
Avg. Area Number of Rooms     119827.783390
Avg. Area Number of Bedrooms    2361.095262
Area Population                   15.216581

Each coefficient represents how much the Price changes with one unit of change in the corresponding feature.

making predictions

making predictions

We can now predict house prices using the test data.

# Predict on the test set
predictions = model.predict(X_test)

# Compare actual vs predicted
comparison = pd.DataFrame({"Actual": y_test, "Predicted": predictions})
print(comparison.head())

            Actual     Predicted
1501  1.339096e+06  1.308536e+06
2586  1.251794e+06  1.237123e+06
2653  1.340095e+06  1.243836e+06
1055  1.431508e+06  1.229242e+06
705   1.042374e+06  1.059353e+06

visualizing predictions

We can visualize how close the predictions are to the actual prices. If the points align closely to the diagonal line, the model predictions are accurate.

scatter plot (real vs predicted)

plt.scatter(y_test, predictions)
plt.xlabel("Actual Prices")
plt.ylabel("Predicted Prices")
plt.title("Actual vs Predicted House Prices")
plt.show()

histogram of residuals

Residuals (errors) are the differences between actual and predicted prices. A normal distribution of residuals indicates a good fit.

sns.histplot((y_test - predictions), bins=30, kde=True)
plt.title("Distribution of Residuals")
plt.xlabel("Prediction Error")
plt.show()

evaluating the model

We use three evaluation metrics to measure model performance:

1. MAE (Mean Absolute Error) average of absolute errors.

2. MSE (Mean Squared Error) average of squared errors.

3. RMSE (Root Mean Squared Error) square root of MSE.

from sklearn import metrics

mae = metrics.mean_absolute_error(y_test, predictions)
mse = metrics.mean_squared_error(y_test, predictions)
rmse = np.sqrt(mse)

print("Mean Absolute Error (MAE):", mae)
print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)

The smaller these values are, the better the model performs.

A well-trained linear regression model for this dataset typically achieves a very low error, indicating good predictive accuracy.

logistic regression

Logistic regression is a supervised learning algorithm used for classification problems, where the target variable is categorical, meaning it can take a limited number of discrete values.

It is used to model the probability that an event occurs based on one or more independent variables.

Typical applications include:

- Detecting spam vs. non-spam emails

- Predicting whether a loan applicant will repay or default

- Determining if a patient has a certain disease or not

These are examples of binary classifications, where the result can be either 1 (yes) or 0 (no).

what is a logistic function

The logistic function produces an S-shaped curve that converts any real-valued number into a value between 0 and 1.

If the output is ≥ 0.5, we classify it as 1 (event occurs), otherwise as 0 (event does not occur).

This makes it ideal for probabilistic models in classification tasks.

confusion matrix

A confusion matrix is a table used to evaluate classification performance.

It compares predicted values with actual values and shows:

- True positives (TP): correctly predicted positive cases

- True negatives (TN): correctly predicted negative cases

- False positives (FP): predicted positive but actually negative

- False negatives (FN): predicted negative but actually positive

From the confusion matrix, we can calculate:

- Accuracy = (TP + TN) / Total

- Error rate = (FP + FN) / Total

For example, in a spam detection test with 200 emails:

TP = 30

- TN = 150

- FP = 5

- FN = 15

Then:

Accuracy = (30 + 150) / 200 = 0.9 → 90%

Error rate = (5 + 15) / 200 = 0.1 → 10%

example of logistic regression with the Titanic dataset

We will now create a logistic regression model using the Titanic dataset from Kaggle.

This dataset contains information about Titanic passengers, including age, class, gender, and whether they survived.

importing required libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the training dataset
train = pd.read_csv("train.csv")

# Display the first rows
print(train.head())

exploring the data

We can check for missing values and visualize them.

# Check for missing values
print(train.isnull().sum())

# Visualize missing data
sns.heatmap(train.isnull(), cbar=False, yticklabels=False)
plt.title("Missing values heatmap")
plt.show()

Columns like Age and Cabin often have missing values. Cabin usually has too many nulls and can be removed.

analyzing survival data

We can analyze the number of survivors and non-survivors.

sns.countplot(x="Survived", data=train)
plt.title("Count of survivors vs non-survivors")
plt.show()

# Survival by gender
sns.countplot(x="Survived", hue="Sex", data=train)
plt.title("Survival rate by gender")
plt.show()

# Survival by class
sns.countplot(x="Survived", hue="Pclass", data=train)
plt.title("Survival rate by passenger class")
plt.show()

We observe that:

- Most non-survivors were men

- Most survivors were women

- Passengers in first class had a higher survival rate than those in third class

age distribution

We can visualize the distribution of passengers' ages.

sns.histplot(train["Age"].dropna(), bins=30, kde=False)
plt.title("Distribution of passenger ages")
plt.xlabel("Age")
plt.ylabel("Count")
plt.show()

Most passengers were between 20 and 40 years old.

cleaning the data

Before training the model, we must clean and prepare the dataset.

# Fill missing ages based on passenger class (Pclass)
def fill_age(cols):
    age = cols[0]
    pclass = cols[1]
    if pd.isnull(age):
        if pclass == 1:
            return 38
        elif pclass == 2:
            return 30
        else:
            return 25
    else:
        return age

train["Age"] = train[["Age", "Pclass"]].apply(fill_age, axis=1)

# Drop 'Cabin' due to too many missing values
train.drop("Cabin", axis=1, inplace=True)

# Drop irrelevant columns
train.drop(["Name", "Ticket", "PassengerId"], axis=1, inplace=True)

encoding categorical variables

We need to convert text columns into numeric values. For example, we can convert gender into 1 (male) and 0 (female).

# Convert 'Sex' to numeric
sex = pd.get_dummies(train["Sex"], drop_first=True)
train = pd.concat([train, sex], axis=1)
train.drop("Sex", axis=1, inplace=True)

# Convert 'Embarked' to numeric
embark = pd.get_dummies(train["Embarked"], drop_first=True)
train = pd.concat([train, embark], axis=1)
train.drop("Embarked", axis=1, inplace=True)

print(train.head())

Now the dataset is completely numeric and ready for modeling.

splitting the data

We separate our data into features (X) and target (y), then split it into training and testing sets.

from sklearn.model_selection import train_test_split

X = train.drop("Survived", axis=1)
y = train["Survived"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

training the model

training the model

We create and train a logistic regression model.

from sklearn.linear_model import LogisticRegression

# Initialize and train model
log_model = LogisticRegression(max_iter=1000)
log_model.fit(X_train, y_train)

# Make predictions
predictions = log_model.predict(X_test)
print(predictions[:10])

evaluating the model

We can now evaluate the model using various metrics.

from sklearn.metrics import classification_report, confusion_matrix

# Classification report
print(classification_report(y_test, predictions))

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print(cm)

sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title("Confusion matrix")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

Typical output:

Precision for class 0 (did not survive): 0.88 Precision for class 1 (survived): 0.75

This means the model correctly predicts non-survivors about 88% of the time and survivors about 75% of the time.

K-Nearest Neighbors (KNN)

K-Nearest Neighbors (KNN) is a supervised classification algorithm used to estimate the probability that an element *X* belongs to a class *C*, based on its proximity to other known examples in the dataset. It is simple, intuitive, and works with any number of classes.

The idea is to classify a new element by looking at the K nearest points (neighbors) in the training data and assigning it the class most common among them.

how it works

Imagine we want to determine the class of a green circle based on nearby red triangles and blue squares.

The KNN algorithm:

1. Calculates the distance between the green circle and all other points.

2. Sorts them from closest to farthest.

3. Looks at the K nearest data points.

4. Assigns the green circle to the class that appears most frequently among those neighbors.

For example:

- If K = 3 and two of the closest points are red triangles and one is a blue square, the circle is classified as a red triangle.

- If K = 5 and three closest points are blue squares, then the circle is classified as a blue square.

Choosing the right value for K is crucial, too small can lead to noise sensitivity, too large can oversimplify the classification.

example with vehicle type prediction

We will demonstrate KNN using the Vehicle Silhouettes dataset, which predicts the type of vehicle based on its physical characteristics.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
data = pd.read_csv("vehicles.csv")

# Display first rows
print(data.head())

# Show the unique vehicle classes
print(data["Vehicle Class"].unique())

The dataset contains various vehicle features and a target column called Vehicle Class, which indicates the type of vehicle (e.g., Bus, Opel, Saab, Van).

preparing the data

We define the features (X) and the target variable (y).

# Define features (all columns except 'Vehicle Class')
X = data.drop("Vehicle Class", axis=1)

# Define target
y = data["Vehicle Class"]

print("Features shape:", X.shape)
print("Target shape:", y.shape)

splitting data into training and testing

We divide our dataset into two parts: - Training data to train the model - Testing data to evaluate its performance

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=45
)

print("Training samples:", X_train.shape[0])
print("Testing samples:", X_test.shape[0])

training the model

We train a KNN classifier with an initial K value of 1.

from sklearn.neighbors import KNeighborsClassifier

# Create model with K=1
knn = KNeighborsClassifier(n_neighbors=1)

# Train model
knn.fit(X_train, y_train)

# Make predictions
predictions = knn.predict(X_test)

# Display sample predictions
print(predictions[:10])

evaluating the model

To measure model performance, we use a confusion matrix and a classification report.

from sklearn.metrics import confusion_matrix, classification_report

# Confusion matrix
cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:")
print(cm)

sns.heatmap(cm, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=1")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report:")
print(classification_report(y_test, predictions))

This report shows precision, recall, and F1-score for each vehicle class.

For example, the model might correctly predict:

- Bus: 82% accuracy

- Opel: 48% accuracy

- Saab: 38% accuracy

- Van: 78% accuracy

These results can improve by finding a better value for K.

optimizing the K value

We can test multiple K values (from 1 to 30) to find the one with the lowest error rate.

error_rate = []

# Loop through K values from 1 to 30
for i in range(1, 31):
    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train, y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

# Plot error rate vs K
plt.figure(figsize=(10, 5))
plt.plot(range(1, 31), error_rate, color="green", marker="o",
         markerfacecolor="red", markersize=8)
plt.title("Error rate vs K value")
plt.xlabel("K")
plt.ylabel("Error rate")
plt.show()

The optimal K corresponds to the lowest point on this graph, typically, K = 3 in this dataset.

retraining with optimal K

We now retrain the model with the optimal K = 3 and re-evaluate its performance.

# Train with K=3
knn_optimal = KNeighborsClassifier(n_neighbors=3)
knn_optimal.fit(X_train, y_train)

# Make predictions
predictions_opt = knn_optimal.predict(X_test)

# Evaluate
cm_opt = confusion_matrix(y_test, predictions_opt)
print("Confusion matrix (K=3):")
print(cm_opt)

sns.heatmap(cm_opt, annot=True, cmap="Blues", fmt="d")
plt.title("Confusion matrix for K=3 (optimized model)")
plt.xlabel("Predicted")
plt.ylabel("Actual")
plt.show()

# Classification report
print("Classification report for K=3:")
print(classification_report(y_test, predictions_opt))

After optimization, the accuracy of all vehicle classes improves, and the error rate decreases significantly.

conclusion

K-Nearest Neighbors (KNN) is a simple yet effective algorithm for classification.

It makes predictions based on proximity between data points, using a distance metric such as Euclidean distance.

The key points to remember:

- It is non-parametric, no assumptions about the data distribution.

- It is instance-based, stores all training data for comparison.

The choice of K greatly affects accuracy:

 - Small K → sensitive to noise.  
 - Large K → oversimplifies classification.

By tuning K and testing error rates, we can build a reliable model for classifying objects like vehicles, customers, or any feature-based categories.