Model risk management – Model development, implementation and use

5 December 2024

In PS6/23 and SS1/23 – ‘Model risk management principles for banks’, the PRA outlines five Principles designed to support effective model risk management. In this article, we outline the key elements of the third principle, model development, implementation and use.

In this principle, the regulator highlights the importance of a robust model development process which revolves around testing and documentation. Banks may rely on third parties to develop certain models, however, there will almost certainly be instances in which internally developed models (e.g., spreadsheets) are being used. In addition, where the development of models is being outsourced to a third party, firms must still ensure:

the model meets the intended objective
that testing is robust and meets their own standards
that any limitations are clear, understood and documented

It’s therefore important to fully understand the guidance set out in this principle and to have proportionate practices in place to manage model risk.

Model development can be thought of as a lifecycle consisting of designing, testing, adapting, and monitoring the model. The figure below describes the key steps of the model development lifecycle:

Model risk management – Model development, implementation and use - graph.jpg

What should firms be doing to effectively implement this principle?

Establishing standards around model design, development, testing and performance.
Ensuring that input data is appropriate for the model’s purpose.
Ensuring documentation and communication are robust, especially with respect to model limitations, adjustments and expert judgement.

1. Establishing standards around model design, development, testing and performance

The first step in the development process is the assignment of responsibility. Model development lies within the “core modelling” process and those involved will typically sit within ‘first line’ functions and should have the necessary degree of technical knowledge and expertise.

Each model must have a clear statement of purpose, which then guides the process from beginning to end; knowing the ‘why’ behind the model is necessary to know the ‘how’. In other words, this means understanding what the problem is that needs addressing and then identifying how to reach the desired business outcome. Model developers should liaise with the model owner and users throughout this stage to ensure that outcomes are clear.

Model testing ensures the best modelling approach is selected, and that the model meets the objectives set out in the first stages. Firms should have a set of minimum requirements for testing as part of their model risk management frameworks, covering:

Clear test procedures, measurement and reporting of the outcomes of testing against pre-defined success criteria.
Forward-looking performance tests should employ ‘sensitivity analysis’ to assess the extent to which inputs can change (largely due to economic and market conditions) whilst ensuring the model still performs within the acceptable threshold.
Backwards-looking performance tests should use observations across a variety of economic and market conditions that are relevant to the model’s intended use.
Testing should be undertaken every time a material change occurs.

2. Ensuring that input data is suitable for the model’s purpose

A crucial element in model development is understanding the data. This involves preparing the data: sourcing, reviewing, cleaning and transforming so that the data used in the model is of the best possible quality. Following this, the data should be analysed to identify correlations between variables, any irregularities and/or patterns that might support the model-build process.

The outcome of these steps will allow model developers to determine if the data available is suitable for model development i.e. captures relevant behaviours and aligns with the business’ objectives. This is a key expectation set out by the PRA in SS1/23, where it highlights the data used to develop the model should be suitable for the intended use and consistent with the chosen methodology, whilst also being representative of underlying portfolios, products, assets or customer base the model is aimed at.

Careful consideration should be given if there are elements of the data that are not representative if there is complexity arising from data sources, and/or if any limitations are identified. These must be fully understood and if necessary, result in an adjustment of the model’s tier classification to reflect the higher risk.

3. Ensuring documentation and communication is robust, especially with respect to model limitations, adjustments and expert judgement

The regulator emphasises the need to communicate the different advantages and limitations of a model under different conditions as well as the sensitivities of model output to changes in inputs so that model users and other stakeholders fully understand the theory and underlying logic. This understanding and communication is key as it will enable the business to choose to implement the model if it meets the criteria required, or trigger model changes/decommissioning where required.

Linked to this, appropriately documenting the outcomes from the model development stage is crucial as it enables the business to meet the PRA’s expectations and will also allow for independent validation to be conducted thoroughly. This should consist of:

The use of data - including description of data sources, data proxies, data quality and relevance tests.
Methodology choice – modelling techniques used, justification of assumptions made and mathematical specification amongst other choices.
Testing outcomes – forward and backward testing as well as detailed criteria used to assess the model.
Model limitations and use of expert judgement.

In practice, this information should be documented within the firm’s model inventory and leveraged for reporting purposes on model risk.