Facilitating ongoing monitoring in Model Risk Management

Verify that your model is performing as intended

A banking institution’s model governance structure is designed to provide comfort that models comply with applicable laws and regulations, model data has quality and relevance, and conceptual soundness has been validated. With these processes for managing model risk in place, it is easy to overlook another crucial element of Model Risk Management (MRM): ongoing monitoring. The ongoing monitoring process assesses whether the model is performing as intended. In addition, a key objective of ongoing monitoring is verifying an organization’s ongoing compliance with any changes to the regulatory requirements for its models.

Weakness in an organization’s ongoing monitoring of its models can lead to significant financial and reputational consequences. One of the largest U.S. banks suffered a $2 billion loss in 2012 due to its implementation of a new Value-at-Risk model that was later deemed inadequate. The new Value-at-Risk model masked losses by artificially depressing the potential risks the bank was exposed to under specific market risk conditions. Proper surveillance of model outputs and auditing of the trading strategy’s effects on the model would have provided more transparency for decision-makers and may have prevented the losses.

Successful MRM requires several processes to operate at optimum capacity. Ongoing monitoring establishes a framework that perpetually surveys the areas of model governance, data, performance and conceptual soundness. This practice allows for timely discovery of issues, necessary adjustments of models, and compliance with both internal policies and external regulation. It is important to establish ongoing monitoring during the model’s inception and throughout its life cycle. According to the Federal Reserve’s Supervisory Guidance on Model Risk Management (SR 11-7), it is essential to evaluate whether changes in products, exposures, activities, clients or market conditions necessitate adjustment, redevelopment or replacement of the model and to verify that any extension of the model usage beyond its original scope is valid.

4 components of ongoing monitoring

Firms are obligated now more than ever to ensure their MRM function is robust and that they have established effective and efficient ongoing monitoring routines. 

To maintain successful governance of model risk, key risk indicators (KRIs) must be embedded into the overall MRM framework. KRIs allow a firm to measure the probability of an event that will adversely impact the organization as well as measure what the consequences of the event will be. For example, the population stability index (PSI) is a KRI that measures how much data has shifted over time. As time goes on and data populations shift, outputs can become inaccurate. Firms can be proactive by using PSI scores to determine shifts in data population and deciding if adjustments are required.

2. Outputs

Ongoing assessment of model key performance indicators (KPIs) allows organizations to obtain insight on underperforming measurements, outdated measurements, and relevance of measurements. KPIs must be adjusted to reflect current and forecasted market conditions, customer behaviors, product offerings and more. KPIs should be frequently compared to the model’s goals and evaluated for alignment. By comparing predicted to actual results, a firm can better discern whether the model is performing as intended. Additionally, ongoing scrutiny of risk measurements may limit model malfunctions and in turn mitigate model risk. 

3. Model risk assessment and regulatory requirements

Model risk assessments are used to identify risks for individual models as well as the aggregate risks posed by all the models housed in the model inventory. Model risk assessments should be in sync with the organization’s model validation procedures, both in cadence and in the risk ratings for the models. The higher the risk rating for a model, the more frequent and detailed the monitoring efforts should be.

With the emergence of the Federal Reserve’s Supervisory Guidance, MRM has become a focal point for regulators and organizations. Adherence to the holistic MRM regulations and guidance is as important as the business-specific regulatory requirements applicable to specific products.

4. Reporting

Every organization should have a framework in place that contains provisions on how to respond to and escalate model errors, risks and other issues. A clear framework of roles and responsibilities in the escalation structure, reporting cadence and report components prevents ambiguity when adverse events surface. Escalation should lead up to the board to make timely, informed decisions. There is no one-size-fits-all approach to an escalation framework; however, senior management and the board should constantly be kept informed of errors, risks and other issues associated with models and the MRM function in totality.