Ten steps to help auditors evole and adapt to AI reserving systems

In this article, Gary Stakem, actuarial director on our insurance team, outlines ten steps for adapting the audit approach to meet emerging industry needs whilst maintaining accuracy, compliance and trust in financial reporting.

1. Upskilling audit specialist teams

Insurance audit teams have traditionally relied upon actuarial specialists to assess complex reserving models. Actuarial teams bring a deep understanding of the commercial and regulatory environments in which insurers operate. However, actuarial audit teams must now upskill and embrace machine learning analysis techniques, ensuring algorithms are explainable, transparent and compliant with regulatory and accounting standards.

2. Enhanced risk assessments

Risk assessments are a critical step in the audit planning process, helping actuaries to identify significant risks within reserves and design appropriate audit procedures.

With the advent of AI, risk assessments must go beyond financial risks. Model bias, overfitting, data quality and model transparency must be considered as significant in-scope risks.

3. Data governance expectations

Traditional actuarial models are inherently interpretable – for example, audit teams can easily visually inspect non-life claim development patterns. In life insurance, plots of mortality or lapse experience often uncover discrepancies in underlying data.

The shift towards less transparent black boxes will limit the ability to deploy these ‘in-model’ controls. This dramatically increases the importance of pre-model data governance and hence audit procedures must respond accordingly. Manual interventions to cleanse or transform data are potential sources of management bias. Feature engineering, a step whereby raw data is transformed into features that enable more effective machine learning, will likely become a significant audit focus.

4. Benchmarking against traditional methods

Traditional actuarial reserving techniques have long been part of the auditor's toolkit and this well-understood approach is unlikely to fully disappear anytime soon – it will provide an established benchmark for assessing black box model results.

But what happens when black box models claim superior results that cannot be validated by traditional methods alone? Actuarial audit teams will need to exercise expert judgment and professional scepticism, knowing when to bring more sophisticated techniques into the audit process.

5. Black box vs black box

The notion of using one AI machine to challenge another may sound like a scenario somewhat reminiscent of a sci-fi film, yet it could become commonplace in the actuarial audit domain.

This independent approach could prove critical in identifying potential user bias in the selection and design of AI models. Such a technique underscores the need for actuarial audit teams to enhance their coding skills to develop credible AI algorithms independently.

6. Model architecture and hyperparameter assessment

AI models have gained popularity based on their ability to self-learn, refining their model parameters to optimise accuracy. Nevertheless, initial human oversight is required to design an appropriate architecture and to set external model parameters that are not machine adjustable – often referred to as ‘hyperparameters’. Designing a suitable architecture is a subjective art and introduces a new source of potential management bias. Partition of actuarial data into subsets, for training and testing of AI models, will also be subject to audit attention.

Benchmarking and sensitivity testing of key assumptions is no stranger to actuarial audits. This practice now looks set to evolve, with assessing variations of hyperparameters becoming a new norm.

7. Explainable AI (XAI) testing

Explainable AI, or ‘XAI’, is fundamental to understanding, trusting and effectively communicating the behaviour of black box reserving models. The modern audit process will include both global and local XAI techniques.

Global techniques examine a model’s overall behaviour providing insights into the relationship between black box inputs and predicted outputs. The use of surrogate models is an example of one such common technique whereby simplified, more interpretable models are fit to broadly replicate an insurer’s more sophisticated black box model.

Local XAI techniques focus on explaining individual policy projections. Although sampling of individual policies is not an unfamiliar concept in insurance audits, new AI interpretability tools such as ‘LIME’ and ‘SHAP’ will better equip actuarial teams seeking to understand individual results.

Deploying a combination of global and local XAI techniques can increase credibility and confidence in black box models.

8. Expert judgments and model overrides

Challenging the actuarial function’s expert judgment is already central to the audit process. Reserving models often need to be adjusted in response to emerging risks not reflected in historic data sets. Covid-19, judicial developments and high inflation rates have been prime examples in recent years. However, with sophisticated AI models, it will no longer be straightforward to ‘go-inside’ models and tweak the parameters.

The use of synthetic or augmented datasets could become one workaround solution. Alternatively, post-processing algorithmic intervention could be utilised. Evidencing the appropriateness of these bespoke adjustments will become more challenging if underlying models are less transparent.

Auditors are accustomed to challenging insurers’ actuaries on expert judgements regarding the commercial and external environment – this assessment must now also extend to machine learning judgements. The scarcity of this specific blend of skills and domain knowledge could see insurers place ever-increasing reliance on a small group of management experts. This will call for audit specialists with equally nuanced skills capable of balancing precision and pragmatism.

9. Model reliability, stability and reproducibility

Consistency and reliability are the bedrock of actuarial reserving models – qualities that are non-negotiable even as business landscapes evolve. Auditors must adapt and employ rigorous tests to ensure AI models can withstand the scrutiny of varied datasets. Models should be capable of producing reasonable results even in obscure scenarios in which they were not trained.

While AI models, such as neural networks, cannot be identically replicated, audit teams should at least be capable of reproducing reasonable imitations. This will require insurers to maintain robust documentation on code management, model logs and deployment procedures.

10. Stakeholder collaboration

Deployment of AI in actuarial reserving is still a field in its infancy, with the development of commonly accepted AI audit standards even less mature. Audit firms must stay engaged with actuarial and accountancy professional bodies, keeping abreast of emerging standards, legislation and regulatory expectations.

Other stakeholders, such as boards and audit committees, will expect regular and transparent communication through periods of change.

The transition towards integrating AI into actuarial practices demands a proactive approach from both insurers and auditors alike, emphasising the need for transparency, collaboration and ongoing education to maintain confidence in financial disclosures. It is sound advice for insurers to initiate a conversation with their auditor from the outset of their journey.

Click the link below to sign up for our insurance regulation newsletter and keep up to date with all the latest news in the sector.