Machine Learning
Data Structures
Algorithms
Problem Solving
Home >
Machine Learning >
Anomaly Detection using Autoencoders
Post author: Ujjwal
Post published: May 1, 2021
Post category: Machine Learning / Projects
Post comments: 0 Comments
An anomaly is a data point or a set of data points in our dataset that is different from the rest of the dataset. It may either be a too large value or a too-small value. Anomalies describe many critical incidents like technical glitches, sudden changes or plausible opportunity in the market. Anomalies are a very small fraction of the entire dataset. In this article, we use autoencoders to detect anomalies.
Autoencoders belong to an unsupervised class of learning algorithms. An autoencoder consists of two parts – an encoder and a decoder . The encoder transforms an input to a low dimensional latent vector and the decoder tries to reconstruct the original input with the help of the latent vector. The difference between the original input and the reconstructed data is called reconstruction loss.
In anomaly detection, we train the autoencoder on normal data points. This way the autoencoder learns to reconstruct the normal data points and has a low reconstruction loss. And, when we feed an anomaly to this trained autoencoder, it fails to reconstruct the anomaly data point and thus has a large reconstruction loss. This way we are able to detect an anomaly in our dataset.
We are going to make an anomaly detector using autoencoders in TensorFlow and Python. The dataset that we will use is the ECG5000 dataset which is available on this link to download. We will consider this as a simple problem and not as a time series problem. We use Jupyter Notebook to run the code.
Here we import the libraries and download the dataset. Then we extract the dataset and concatenate the train and test files that we will use for our purpose. We scale the dataset and split it into the train and test parts. Our dataset contains 141 columns of which 1st column is the label column. There are five different types of labels ranging from 1 to 5. We will consider label 1 as normal data points and group the rest of the data points with labels 2,3,4, and 5 as anomaly data points.
The plot for normal data points is as shown below.
The plot for anomaly data points can be seen as below.
As can be seen, the plots are quite different for the normal data points and anomaly data points. This difference is what we will utilize to detect anomalies.
We have used dense layers to build the autoencoder for this example. One can use LSTMs and Convolution layers too to build an autoencoder.
The encoder part consists of 4 Dense layers of decreasing number of units of which the last Dense layer is the Latent vector.
The decoder part consists of 4 Dense layers of an increasing number of units of which the last Dense layer outputs the reconstructed data.
Each Dense layer has a ‘relu’ activation except for the last Dense layer in Decoder which has a ‘sigmoid’ activation for the reconstructed data point.
We compile the model with ‘adam’ optimizer and ‘mean absolute error’ as loss function. The autoencoder is trained for a total of 50 epochs on the normal data points and validated on all the data points (consists of normal as well as anomaly data points) to have a well-trained model. We have also instantiated an early stopping with the patience of 2 monitoring the validation loss. If the validation loss does not decrease for two epochs straight, the training will stop. We predict the normal test data points and plot one of them and so we do with the anomaly test data point.
The plot for the normal test data point and its prediction (or rather say reconstruction) by the autoencoder is as shown below. The blue line shows the original test data point and the red one the reconstructed one by the autoencoder. We see that they overlap quite well but still, there is little reconstruction loss.
The plot for the anomaly test data point and its prediction (or rather say reconstruction) by the autoencoder is as shown below. The blue line shows the original test data point and the red one the reconstructed one by the autoencoder. We see that they do not overlap quite well and there is a lot of reconstruction loss.
The reconstruction loss for the normal and test data are calculated. We set the threshold value of the loss to classify a data point as normal or anomaly according to our problem and need. We set the threshold here as the mean of loss plus two standard deviations of the loss for normal data points. If the loss for a data point is less than the threshold it will be a normal data point else it will be an anomaly. Lastly we calculate the number of normal test data points and anomaly data points that are classified correctly.
We plot the reconstruction losses of the normal and anomaly test data points with the threshold to see if the threshold we have set is good or it need readjustments. The plot can be seen below.
This way we can utilize an autoencoder to detect anomaly in our dataset.
You can see the GitHub repo containing the notebook that shows the code used here at https://github.com/hellomlorg/Anomaly-Detection-using-Autoencoders .
Hope you enjoyed this article. For more such amazing articles check out hello ML . If you would like to improve this article or report something incorrect, please do let us know in the comments below.
If you have any copyright infringement claims, kindly send a mail to [email protected] . We deal with plagiarism very strictly and if our authors are found to be involved in plagiarism, we will take appropriate action against them after reviewing the claim.
Tweet
Telegram
Register
Lost your password?
Deque
Comb Sort
Long Short-Term Memory Cell (LSTM)
Ways to Write n as a Sum of Two or More Positive Integers
Gnome Sort
Internship Guidelines at hello ML
Privacy Policy