SIRA, data completeness and tuning
According to the participants, the SIRA, data completeness and tuning are the three major challenges faced when validating a transaction monitoring model. This was the result of a short survey held during the ACAMS Netherlands Chapter webinar about the validation of AML transaction monitoring models. The webinar was held on the 10th of February 2021, presented by my colleagues Renske van Hooff and Jori van Schijndel and attended by over 240 participants from a great variety of financial institutions. During the webinar, the various pillars of KPMG's model validation methodology were explained.
Level of detail of the SIRA
In order to allow for effective monitoring, the SIRA (Systematic Integrity Risk Assessment) should be used as a basis for the scenarios in the transaction monitoring model. The lack of quality of the SIRA was presented as one of the challenges in the validation of the conceptual soundness of the transaction monitoring model. Almost half of the respondents recognized this challenge and indicated that the risks in their SIRA are not specific enough to be used as direct input for the model. When the SIRA lacks detail, it is difficult to use it as a basis for an adequate design of a transaction monitoring model. Furthermore, some relevant typologies may not be covered or scenarios may not be properly configured because the related risks and mitigating measures may not have been (fully) identified.
Completeness of the data used by the model
Data validation is one of the key activities of the Data, System & Process pillar in the presented validation methodology. The most important data validation challenge identified by the responders concerns validating the completeness of the data. Data validation activities start with detailing the IT landscape and identifying the systems that are relevant as a (potential) source for the transaction monitoring model. Depending on the size (and legacy of) the organization, the IT landscape can be complex which can severely complicate the validation. Furthermore, in order to validate the data, often (large) amounts of data have to be obtained and analysed in order to verify the completeness of the data flowing into the model.
Statistical methods used for tuning
Setting thresholds for the transaction monitoring scenarios is a valuable tool that institutions can use to tailor the transaction monitoring to their specific risks and risk appetite. Using data and statistical methods to determine appropriate thresholds can help institutions to substantiate the thresholds they have chosen. As part of the presented methodology, opportunities for tuning are identified under the activities in the pillar Ongoing & Effective Challenge. Only a third of the respondents indicated that their organization uses a statistical method to determine the thresholds. Furthermore, approximately 40% indicated that the thresholds are set based on expert judgment.
Looking back on the webinar, the survey and the topics brought up by the audience, it is clear that (improving) transaction monitoring models is high on the agenda for many institutions.