Is expert knowledge still relevant for tackling fraud in the age of advanced machine learning?
A discussion about the pros and cons of expert knowledge and machine learning models when tackling fraud
“You’re not honestly selling rules engines” - business rule models have become an increasingly unappealing concept in an age where Google and IBM seem to be successfully automating everything with self-learning, self-tuning software.
Often customers have implemented rules systems which were based on some potentially biased “expert opinions” - unsupported by any kind of science. Furthermore, after buying expensive top of the range software, customers may find that a few years later the knowledge they implemented is outdated, resulting in them paying for an expensive and time consuming upgrade.
The silver bullet alternative is every customer’s dream; a bottomless pool of shiny new tools such as neural networks and classifying algorithms, which claim to give accurate results and even improve with time! You can almost hear the sales pitch.
Advanced machine learning
Incredibly complex problems are being solved by modern artificial intelligence; reading handwriting, diagnosing illnesses, highlighting cyber security risks, even beating world champions at a multitude of complex logic games. Price non-withstanding, I would trust a human translator over an online translator, but it goes without saying, machine learning is becoming increasingly prominent in our day to day lives – even driving us home after a day’s work.
However, all of these applications of machine learning have a few things in common. They have rich and abundant outcome data (i.e. there are many documented examples of peoples’ symptoms and the eventual medical diagnosis, and there are many examples of the moves which led to victories or defeats in chess matches). The examples I mention also have somewhat logical rules that lead to their outcomes (i.e. every time a patient has a diagnosis/outcome of appendicitis their symptoms are likely to have shown common traits).
These applications of machine learning are far from trivial and each give their own challenges, but they do have vast amounts of accurate outcome data and logical patterns for the models to uncover.
The problem with fraud
I have spent many hours with Data Scientists, Fraud Experts and Business Analysts assisting in the creation of various types of models for use in detecting fraudulent behaviour in large volumes of financial data. To summarise my findings, fraud is the ultimate machine learning nightmare; fuzzy/no outcome data (often 99% of all the data is completely unclassified), complex illogical cases and missing/inaccurate data.
Detecting fraud can often feel like you’re fighting in a continuously evolving battlefield where you can only see less than 1% of the terrain. It is easy for machines to draw inaccurate correlations, or point out uninteresting anomalies in this kind of data landscape.
Of course, some fraud can be found with machine learning models, but the understanding of the cause and effect of a fraudster’s behaviour is where the accuracy and benefits can be felt. For example, you often see an increase in the number of fraudulent sickness claims occurring on Mondays, due to workers alleging that an accident which happened at the weekend actually happened at work, but on a bank holiday this effect is delayed. Machine learning models will likely show the correlation with Mondays, but not be prepared for the bank holidays since the number of reported cases is so low. In my opinion a fusion of human intelligence and machine learning is necessary to create effective models.
Ways around the problems
I have explored a number of ways to improve the results of machine learning models, the two prominent ideas revolve around extrapolating the outcomes (oversampling) and combining multiple algorithms (ensembling) in order to utilise the combined strengths of numerous different approaches. These two techniques can help with the issues faced due to low outcome data and improve accuracy.
The greatest way to truly improve the field of vision in the fraud warzone is to combine as much data from different sources as possible. This is because by combining two or more companies’ data, you gain knowledge about third parties which you previously knew nothing about – giving you a more holistic view and additional outcome data. Data sharing faces legal and political battles limiting its use, but still there are more and more collaborative projects taking place globally.
The silver bullet?
These techniques alone do not overcome the problem of correlation vs causation. By understanding the cause of the correlations highlighted by machine learning models, you can enhance them.
Throughout my experience I have felt the benefits of machine learning as both a tool to detect fraudulent patterns of behaviour, highlight unusual behaviour (potentially leading to the finding and understanding of new fraud patterns) and also to inform decisions when writing complex business rules models.
Simple logistic regression models have also been used to prove certain industry biases have no correlation with fraudulent outcomes. Combining the outcome of pure machine learning models with business rules models has yielded incredible results.
Although combining and sharing data across industries is not always possible, experts can act as a knowledge reservoir, gathering experience and war stories, which can help customers protect against fraud scenarios that they may not be prepared for.
Still room for an expert in this field?
I am a scientist at heart, but I simply cannot move away from the fundamental fact that modelling fraud, even with the big data and artificial intelligence movement we have today, is still in its infancy.
Both approaches have strengths and shortcomings; human opinions are heavily influenced by past experiences (which can be beneficial or detrimental) and prejudice, but machines struggle with the intricacies of modelling complex human behaviours, especially when there is little data for it to learn from. Supervised and unsupervised learning is without a doubt a powerful way to create models solving many of today’s problems, but rules models can also be extremely effective by combining experts broad understanding of fraud with supervised models’ insights.
Knowledge, years of experience, hunches, hints and tips vs modern advanced analytics – I think the answer lies in a fusion of the two – at least for now.
About the author
Alexandrea Ridden is a contractor for Knowit advising on and implementing big data counter-fraud or risk detection solutions.
Alex is a counter-fraud modelling expert specialising in insurance (motor, property, commercial and liability), trading and banking.
If you are interested in working with Alex, please contact her via LinkedIn.