AI’s true super power: Counter-intuitive insights that actually make sense

When encountering AI systems for the first time, users frequently encounter results which they regard as counter-intuitive. AI engineers expect this kind of feedback because machine learning does not mirror the thought processes of the human brain. 

This is what makes AI so useful: it helps us think outside the box.

The most frequently encountered example of the counter-intuitive nature of AI is in game playing. Here AI systems like AlphaGo have shown strategies that human players did not consider viable, leading to surprising victories against top players.

In the context of AI, therefore, users should not expect outcomes that will be immediately obvious based on human reasoning or traditional logic. This is not to say that AI will consistently diverge from human expectation, far from it, but some results will jar. Much of this has to do with the way that AI learns and certain biases in the way we humans think. 

AI models are complex

AI learns from data rather than relying solely on programmed rules. Models based on deep learning operate through complex algorithms that process vast amounts of data. The relationships within the data are often non-linear and intricate, making it difficult for humans to predict how inputs, or changes in inputs, will affect output. 

This complexity can lead to counter-intuitive results, such as when even small input changes yield disproportionately large changes in results, or when seemingly good characteristics, such as low debt, can sometimes be flagged as a risk. 

In a nutshell, AI can uncover hidden patterns and correlations that humans might overlook. This ability of AI to derive insights from large datasets often leads to conclusions that seem counter-intuitive at first glance but are statistically valid upon further examination.

AI challenges human biases

Humans have inherent biases and heuristics that shape their understanding of the world. Common examples would include:

• Confirmation bias refers to a tendency to search for, interpret, favor, and recall information in a way that confirms one’s preexisting beliefs or hypotheses. This is arguably the most prevalent form of bias in financial analysis. 

• Anchoring bias occurs when we rely too heavily on the first piece of information encountered (the “anchor”) when making decisions. In finance, the purchase price of an investment determines whether an investment is in-the-money or out-of-the-money frequently affects investment decisions: ignoring market signals, holding a losing asset too long, and ignoring opportunities because we missed buying at a lower price.

• The availability heuristic refers to a mental shortcut that relies on immediate examples that come to mind when evaluating a specific topic or decision. Companies that are market darlings tend to get disproportionate positive coverage in the financial media and investment discussions. This can lead to a misperception that such companies are safe or have good governance when evidence exists to the contrary.

• Self-serving bias is the tendency for individuals to attribute positive outcomes to their own actions while attributing negative outcomes to external factors. Simply put, when our investments go up, we tend to congratulate ourselves but when they go down we blame external factors beyond our control.  

These and other biases influence the way that we view companies. It explains why companies such as Enron, Wirecard and others were beloved by investors just months before they collapsed. 

AI systems challenge these biases by presenting data-driven insights that sometimes contradict popular beliefs or assumptions. For example, our Transparently risk engine, which measures the quality of company accounts, saw significant risk in electric battery makers and some AI stocks when they were at the height of their mania. Certainly, the system saw many problems with some of the most famous and popular companies that collapsed due to accounting scandals. 

In the context of risk assessment or investment, the true value of AI is that it provides a unique perception. Views concordant with human thinking add to conviction. Views that conflict with human thinking ostensibly should encourage us to seek deeper understanding. But human biases are hard to overcome. And in fact, humans can introduce bias into AI models. 

AI is difficult to trust

Working with AI systems can be challenging. While the output may be accurate, the underlying processes can appear convoluted, making it difficult to trust the output at times. The implication is that the user needs to become familiar with how the system “thinks.” 

Over time, output that appears to lack explainability becomes more intuitive. In the Transparently system, for example, the risk engine will sometimes flag risk due to an absence of debt. This can happen when a company fails to generate cash flow consistent with its reported sales and margins, especially in relation to peers. 

Or it can happen when sales rise and margins widen but cash flow declines and sales of peers weaken. In this situation, the system is simply questioning why there is not greater cash flow, especially if a company has significantly less debt than peers. 

The problem of overfitting

Not all AI systems are created equal. Training an AI system requires considerable knowledge and experience. Overfitting can be a problem. This occurs when a machine learning model learns its training data too well, including noise and irrelevant details, instead of capturing the underlying patterns that generalize to new, unseen data. 

An overfit model performs exceptionally well on the training dataset but poorly on validation or test datasets because it fails to recognise beyond the specific examples it has seen. 

Say you’re training a model to recognise cats. It learns the training examples (the first cat breed) perfectly, but it can't recognize new, different examples (other cat breeds) because it hasn't learned the general characteristics of "cat." It's too focused on the specifics of what it was initially trained on.

To mitigate overfitting at Transparently, we conduct extensive out-of-sample testing, which involves reserving a portion of data not used during training to test our model's performance. By testing the model on data it has never seen before, you can get a realistic assessment of how well the model will perform in real-world scenarios when exposed to new inputs.

We have great confidence in our output. In out-of-sample testing, our system was able to predict accounting manipulation with a high degree of proximity to the in-sample results.

Out-of-sample testing is a cornerstone of reliable machine learning model evaluation, and is essential for building trust in AI systems. That’s especially crucial in high-stakes domains like finance, where errors can lead to significant losses or reputational damage.

Conclusion

AI output can be counter-intuitive. Expect unexpected outcomes. These arise due to AI’s reliance on data-driven learning processes, challenges to human biases, and complexity in explaining output. 

Perhaps one day AI will be able to fully explain its output in human terms, but until that day arrives, we humans need to invest effort to understand why the system is generating certain results. The user will find that these tend to recur in certain areas and can be explained as one gains familiarity.