Thoroughly exploring data with AI before building a predictive model is one way to ensure that all important factors in complex data sets will be tracked.
Predictive modeling typically involves gathering available data and assessing which human behaviors, atmospheric conditions, or consumer trends in that record are likely to recur. This has worked great for things like automated online shopping recommendations, weather forecasts, and optimizing truck routes and schedules.
But the present is not always an echo of the past. In times of disruption, it is much more difficult to predict what lies ahead. This is when predictive models are most likely to go wrong.
Sometimes unintended consequences are harmless, like increase the number of home runs in MLB which are associated with climate change. Other times they are devastating, like the failure to price the tail risks of subprime mortgages, which led to the Great Recession of 2008.
AI is now used to improve predictive modeling by detecting and making sense of all the characteristics of a scenario. before a model is designed. This “pattern precognition” allows data analysts to understand all the features at play in huge data sets. By detecting factors that are not intuitive or have not been significant in the past, analysts uncover insights that teams can use to create predictive models that track what is truly important and quickly iterate so that they remain true.
Here’s how this AI technique, intelligent mining, helps data teams better see the future and build predictive models that prevent performance degradation over time.
When history doesn’t repeat itself, predictive models can short-circuit
As a member of the U.S. Department of Defense community in 2019, I was part of a team trying to help alleviate some of the profound disruption caused by COVID. We were tasked with examining where our predictive supply chain models were failing and how to improve them in the future. It turned out that there were two main gaps.
The first was that supply chain models were often built around the concept of just-in-time inventory management. JIT has been an effective tool for reducing waste and increasing productivity. But a loophole emerged during COVID; our JIT models did not sufficiently take fragility into account. Lean pipelines were great for efficiency, but the resulting limited inventory made supply chains prone to disruption in unpredictable ways.
The supply of toilet paper is a simple illustration of this. It is a large and relatively inexpensive product, so no retailer was accustomed to keeping a large quantity in stock. Moreover, its consumption patterns are generally so stable that there is little need to overstock it. Then COVID caused a huge setback. Demand wasn’t stable at all, shelves were emptied due to panic buying, and stores had to start rationing rolls because production couldn’t keep up.
Assumptions about directionality were another dimension poorly represented by the predictive models we used. N95 masks were one of the key strategic shortages we encountered early in the pandemic. Our models were based on China being the largest supplier of surgical masks, which it has been historically. But suddenly this exporter became a major importer of masks, particularly from the United States and Europe, breaking the predictive models we used to ensure sufficient supply of masks.
See also: Predictive intelligence only works with high-quality data
AI-Driven Data Mining: A Leap Forward for Predictive Models
Modeling can be time-consuming and expensive, which can make it difficult to abandon a predictive model that no longer works as it should. AI-powered data mining can help alleviate this problem by ensuring the right things are modeled in the first place. It can highlight all the characteristics of a supply chain (or maintenance program, manufacturing line, or financial ecosystem) before generating a predictive model.
How? Intelligent Exploration uses AI to examine all important dimensions of an organizational problem or system so users can see and evaluate the knowns, the unknown unknowns, and everything in between. Here’s a simple overview of how this evolves the process of developing predictive models.
How Intelligent Exploration Improves Predictive Modeling
| Before AI-powered data mining | With intelligent exploration | |
| 1. | The analyst is given a huge data set and must determine where to start and how to apply all the data. The analysis is often limited to a handful of factors. | The analyst receives a huge data set. Ready-to-use AI routines quickly explore complex data sources by including all potentially relevant dimensions. |
| 2. | The data team goes through cycles of queries looking for direction. The team formulates a hypothesis or problem to solve to make the work manageable. | The analysis simultaneously considers dozens of dimensions within the data. AI reveals what’s important. Ideas are rooted in data, not assumptions. The analyst gets immediate direction. |
| 3. | The results of a best estimate analysis form the basis for building a predictive model. Biases can be incorporated. Relationships with significant business impact may be overlooked. | The results reveal driving factors, anomalies, relationships and patterns within the data. The analyst receives recommendations that target the most important issues. The data team builds a predictive model taking these factors into account. |
| 4. | The predictive model answers the original question, but high-value factors are not examined. | The predictive model targets the main current issues and all impacting factors, including the most unexpected ones. |
Other Ways AI Data Mining is Advancing Modeling
AI-powered data mining works at scale. Hundreds of columns of data can be analyzed simultaneously. This makes it much more likely that ideas, areas of opportunity, risks and recommendations accurately reflect reality.
Additionally, results are delivered in simple language and intuitive formats such as 3D visualizations that reveal the interactions between all this complex data. This transparency is important because AI is neither magical nor perfect. Analysts, subject matter experts, and data teams can more easily spot anything that seems illogical and review it before building a predictive model.
Intelligent exploration also enables a virtuous cycle of improved predictive modeling. Solutions with the highest earning potential are identified first. As things evolve, new areas of improvement can be discovered and added to the model.
Predictive models that solve problems in a changing world
In a model that considers a large system like a global supply chain, even small miscalculations can have an impact. butterfly Effect— a cascade of consequences that are extremely difficult to predict. Thoroughly exploring data with AI before building a predictive model is one way to ensure that all important factors in complex data sets will be tracked. The models can then point to better decisions, policies and initiatives.
Additionally, AI data mining detects signals you may not be looking for, such as anomalies that indicate a change is occurring. It can warn you in advance that it’s time to update your predictive model. This way, predictions stay on track even when events deviate from the norm.
