Anomaly Detection and Root cause analysis using Spotfire Analytics and Microsoft Cognitive Services

This article supports the community Exchange item Anomaly Detection and Root cause analysis using Spotfire Analytics and Microsoft Cognitive Services.

Anomaly detection using equipment maintenance dashboard

This anomaly detection solution includes Microsoft Cognitive Services container deployment with anomaly detection, text mining and root cause analysis. Anomaly detection and analysis provides value across nearly every industry including energy, financial fraud and risk, algorithmic insurance, connected vehicles, healthcare and insurance claims, and manufacturing fault detection and yield optimization.  

This article focuses on energy, identifying anomalies for asset management. This specific example uses machine learning techniques to detect anomalies, understand root cause from related text data, and alert case managers when sensor readings are deviating from expected patterns by recommending a suggested maintenance action. This enables operators to implement predictive-based maintenance actions before equipment failure, and to prevent costly manufacturing process shutdown. 

Reference Architecture

 The solution is built out in 3 phases; 

Phase 1 - Anomaly Detection on historical data

Spotfire Data Science platform is used for detecting anomalies across all the sites of a power plant. The input file is power plant data consisting of a timestamp column and several other numeric sensor readings. The response variable in this case is 'Prodperminute' (sensor reading tracking production of power per minute). The workflow consists of multiple steps including data pre-processing, transformation of time-series data, filtering and ultimately calling the MS Azure services from a TIBCO Data Science custom anomaly detection operator designed to invoke the services. The results from the model (sbdf file, inferred data) are made available offline which the maintenance engineer uses to carry along to the remote site.

Phase 2 - Detecting Anomalies at remote site

Once the maintenance engineer is at the remote site that may not be connected to the internet, the first step is to perform anomaly detection analysis using the Spotfire equipment maintenance dashboard. The dashboard can run entirely in an offline mode so the engineer at the site can continue their critical inspection regardless of access to network connection.

The user selects a response variable(in the demo: 'Prodperminute') and a time granularity. Anomalies are then detected using historical data collected at the particular site. This produces two visualizations, the original data readings along with the expected values provided from the containerized service, and the difference between the original and expected values over time. The red markers indicate the anomalies detected by the Anomaly Detection container.

Phase 3 - Performing root cause analysis

The site engineer investigates anomalies from a certain time window to perform a root cause analysis. As a part of root cause analysis, key driving factors indicate what factors are contributing to the anomalies in production per minute for the time window selected by an engineer using the Spotfire's brush linking capabilities.

At this point, the site engineer digs deeper to understand the maintenance action items performed prior to these anomalies occurring by getting insights from log data using key phrase extraction from the Text Analytics container. An example of the key phrases generated from the text log, 'Complaint of Faulty BaroPressure Meter Reading' would be ['Complaint of Faulty BaroPressure Meter Reading', 'Needle bearing', 'diagnostics', 'SNO 476BD78', 'PCB', 'replacement', 'Restart PCU unit', 'psi']

From here, the equipment maintenance dashboard generates a recommendation for the next steps by further pre-processing the key phrases extracted from the logs. In the example shown in the demo, the maintenance logs from the selected time window indicate 'BaroPressure' sensor was restarted 5 times vs. being recommended for replacement just once. As this sensor was one of the top key driving factors, the recommendation to replace the 'BaroPressure' sensor is made.

The solution can be adapted to data from different domains. To learn more and to download this item go to the community Exchange item.

This use case is just an illustrative example and one of the many ways how customers use Spotfire's analytics and data science capabilities to solve some of the most complex business problems such as pricing and promotion optimization, supply chain analytics, fraud detection, yield enhancement, and real-time process control.

To learn more about Spotfire's anomaly detection solution, visit Anomaly Detection at Spotfire.