PCA-Based Anomaly Detection in Python

     Anomaly detection is a technique used to identify unusual patterns that do not conform to expected behavior. Principal Component Analysis (PCA) is a dimensionality reduction technique that can be used for anomaly detection by projecting data into a lower-dimensional space and identifying anomalies as points that deviate significantly from the projected data. 

    In this tutorial, we will learn how to perform PCA-based anomaly detection using Python. We will generate synthetic 3D data, apply PCA, and detect anomalies based on the reconstruction error. Finally, we will evaluate the performance using a confusion matrix and classification report and visualize the results in a 3D plot.

    The tutorial covers:

1. Introduction to PCA and Anomaly detection
2. Generating test data
3. Applying PCA
   
4. Detecting anomalies
   
5. Conclusion
   
6. Source code listing 
   

 

Introduction to PCA and anomaly detection

   PCA is a statistical technique that transforms the data into a new coordinate system, where the first coordinate (principal component) explains the most variance in the data, the second coordinate explains the second most variance, and so on. By reducing the dimensionality of the data, PCA can help in identifying patterns and anomalies.
 

Anomaly Detection
    Anomaly detection involves identifying data points that deviate significantly from the majority of the data. In the context of PCA, anomalies are points that have a high reconstruction error when projected back to the original space.

 

Generating test data

    Before we start, make sure you have the necessary libraries installed. You can install those libraries using pip command.

 

pip install numpy pandas scikit-learn matplotlib

 

     We import the required libraries for this tutorial.

 

import numpy as np

import pandas as pd

from sklearn.decomposition import PCA

from sklearn.preprocessing import StandardScaler

import matplotlib.pyplot as plt

from sklearn.metrics import confusion_matrix, classification_report

 

    Next, we'll generate synthetic 3D data with a small portion of anomalies.

 

# Generate simple 3D random scattered data

np.random.seed(42)

n_samples = 300

n_features = 3 # Only 3 features for simplicity

# Generate normal data (centered around 0)

normal_data = np.random.normal(loc=0, scale=1, size=(n_samples, n_features))

# Introduce some anomalies (located at the edges)

# Uniform distribution for edge anomalies

anomalies = np.random.uniform(low=-5, high=5, size=(20, n_features)) 

data = np.vstack([normal_data, anomalies])

# Convert to a DataFrame for easier manipulation

df = pd.DataFrame(data, columns=[f'feature_{i}' for i in range(n_features)])

# Add ground truth labels (1 for anomalies, 0 for normal)

df['ground_truth'] = 0 # Initialize all as normal

df.iloc[-20:, df.columns.get_loc('ground_truth')] = 1 # Last 20 rows are anomalies

 

print(df.head())

First 5 rows of  'df' data frame.

 

feature_0 feature_1 feature_2 ground_truth 0 0.496714 -0.138264 0.647689 0 1 1.523030 -0.234153 -0.234137 0 2 1.579213 0.767435 -0.469474 0 3 0.542560 -0.463418 -0.465730 0 4 0.241962 -1.913280 -1.724918 0  

Applying PCA

    PCA is sensitive to the scale of the data, so it's important to standardize the data before applying PCA.

 

scaler = StandardScaler()

data_scaled = scaler.fit_transform(df)

 

    Now we can apply PCA to the scaled data. We will apply PCA to reduce the dimensionality of the data to 3 components for better visualization purpose.

  

# Apply PCA with 3 components

pca = PCA(n_components=3)

data_pca = pca.fit_transform(data_scaled)

# Convert to DataFrame for easier manipulation

df_pca = pd.DataFrame(data_pca, columns=['PC1', 'PC2', 'PC3'])

Detecting anomalies

    Anomalies can be detected by calculating the reconstruction error. The idea is that normal data points will have a low reconstruction error, while anomalies will have a high reconstruction error. 

    We will reconstruct the data from the PCA space and calculate the reconstruction error.

 

# Reconstruct the data from the PCA space

data_reconstructed = pca.inverse_transform(data_pca)

# Calculate the reconstruction error

reconstruction_error = np.sum((data_scaled - data_reconstructed) ** 2, axis=1)

# Add the reconstruction error to the DataFrame

df['reconstruction_error'] = reconstruction_error

 

    We can identify anomalies by setting a threshold on the reconstruction error. We can use the 95th percentile as the threshold.

# Determine anomalies based on reconstruction error

threshold = np.percentile(df['reconstruction_error'], 95) # 95th percentile as threshold

df['anomaly'] = df['reconstruction_error'] > threshold

# Add anomaly labels to the PCA DataFrame

df_pca['anomaly'] = df['anomaly'] 

 

    We will evaluate the performance of the anomaly detection using a confusion matrix and classification report.

 

# Evaluate performance

print("Confusion Matrix:")

print(confusion_matrix(df['ground_truth'], df['anomaly']))

print("\nClassification Report:")

print(classification_report(df['ground_truth'], df['anomaly']))

 

The output looks:

Confusion Matrix: [[297 3] [ 7 13]] Classification Report: precision recall f1-score support 0 0.98 0.99 0.98 300 1 0.81 0.65 0.72 20 accuracy 0.97 320 macro avg 0.89 0.82 0.85 320 weighted avg 0.97 0.97 0.97 320 

 

    Using the extracted anomalous points, we can visualize them in a a 3D plot, highlighting the anomalies

 

# Visualize the data in 3D

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(111, projection='3d')

# Plot normal points

ax.scatter(df_pca[~df['anomaly']]['PC1'],

df_pca[~df['anomaly']]['PC2'],

df_pca[~df['anomaly']]['PC3'],

color='blue', label='Normal', alpha=0.6)

# Plot anomalies

ax.scatter(df_pca[df['anomaly']]['PC1'],

df_pca[df['anomaly']]['PC2'],

df_pca[df['anomaly']]['PC3'],

color='red', label='Anomaly', alpha=0.6)

ax.set_xlabel('Principal Component 1')

ax.set_ylabel('Principal Component 2')

ax.set_zlabel('Principal Component 3')

ax.set_title('3D PCA-Based Anomaly Detection')

ax.legend()

plt.show()

 

Conclusion 

    In this tutorial, we applied PCA-based anomaly detection to synthetic 3D data. We evaluated the performance using a confusion matrix and classification report and visualized the results in a 3D plot. 

    PCA-based anomaly detection is a powerful technique, especially when dealing with high-dimensional data. It's important to note that the effectiveness of this method depends on the choice of the threshold and the nature of the data. Full source code is provided below.

Source code listing 

 

import numpy as np

import pandas as pd

from sklearn.decomposition import PCA

from sklearn.preprocessing import StandardScaler

import matplotlib.pyplot as plt

from sklearn.metrics import confusion_matrix, classification_report

# Create simpler 3D random scattered data

np.random.seed(42)

n_samples = 300

n_features = 3 # Only 3 features for simplicity

# Generate normal data (centered around 0)

normal_data = np.random.normal(loc=0, scale=1, size=(n_samples, n_features))

# Introduce some anomalies (located at the edges)

# Uniform distribution for edge anomalies 

anomalies = np.random.uniform(low=-5, high=5, size=(20, n_features))

data = np.vstack([normal_data, anomalies])

# Convert to a DataFrame for easier manipulation

df = pd.DataFrame(data, columns=[f'feature_{i}' for i in range(n_features)])

# Add ground truth labels (1 for anomalies, 0 for normal)

df['ground_truth'] = 0 # Initialize all as normal

df.iloc[-20:, df.columns.get_loc('ground_truth')] = 1 # Last 20 rows are anomalies

print(df.head())

# Scale the data

scaler = StandardScaler()

data_scaled = scaler.fit_transform(df.drop(columns=['ground_truth']))

# Apply PCA with 3 components

pca = PCA(n_components=3)

data_pca = pca.fit_transform(data_scaled)

# Convert to DataFrame for easier manipulation

df_pca = pd.DataFrame(data_pca, columns=['PC1', 'PC2', 'PC3'])

# Reconstruct the data from the PCA space

data_reconstructed = pca.inverse_transform(data_pca)

# Calculate the reconstruction error

reconstruction_error = np.sum((data_scaled - data_reconstructed) ** 2, axis=1)

# Add the reconstruction error to the DataFrame

df['reconstruction_error'] = reconstruction_error

# Determine anomalies based on reconstruction error

threshold = np.percentile(df['reconstruction_error'], 95) # 95th percentile as threshold

df['anomaly'] = df['reconstruction_error'] > threshold

# Add anomaly labels to the PCA DataFrame

df_pca['anomaly'] = df['anomaly']

# Evaluate performance

print("Confusion Matrix:")

print(confusion_matrix(df['ground_truth'], df['anomaly']))

print("\nClassification Report:")

print(classification_report(df['ground_truth'], df['anomaly']))

# Visualize the data in 3D

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(111, projection='3d')

# Plot normal points

ax.scatter(df_pca[~df['anomaly']]['PC1'],

df_pca[~df['anomaly']]['PC2'],

df_pca[~df['anomaly']]['PC3'],

color='blue', label='Normal', alpha=0.6)

# Plot anomalies

ax.scatter(df_pca[df['anomaly']]['PC1'],

df_pca[df['anomaly']]['PC2'],

df_pca[df['anomaly']]['PC3'],

color='red', label='Anomaly', alpha=0.6)

ax.set_xlabel('Principal Component 1')

ax.set_ylabel('Principal Component 2')

ax.set_zlabel('Principal Component 3')

ax.set_title('3D PCA-Based Anomaly Detection')

ax.legend()

plt.show()