You Need a BSA Software Model Validation!     

Since the beginning of crime, there has been a need to hide the ill-gotten gains of criminal activity.  Early bad guys held their loot in caves.  Later, treasure chests provided a means of hiding criminal wealth.   However, despite the form that ancient loot took, the goal was and has always been to reduce loot to cash to currency so that it can be used in exchange for other goods and services.   The need to take illicit assets or money and hide its source is known commonly as “money laundering”.  Criminals of all sorts engage in money laundering and have become exceedingly sophisticated in their pursuit of hiding the sources and uses of their money.  

Because the “bad guys” continue to evolve, the history of the Bank Secrecy Act and Anti-Money Laundering laws (“BSA”) is one of ongoing change.  The laws that make money laundering illegal can be traced back to the Bank Secrecy Act of 1970.   Since the time the BSA was passed, there have been seven major legislative changes to the overall legislative scheme that covers this area.  Probably the most famous of these is the Uniting and Strengthening America by Providing Appropriate Tools to Restrict, Intercept and Obstruct Terrorism Act of 2001.  Of course this is more popularly known as the Patriot Act.  

As technology has changed, so have the goals of many of the criminals that want to launder money.  In addition to drug dealers, there are terrorists and persons that engage in human trafficking; all of whom are developing ways to hide their cash. 

Each of the changes in BSA/AML laws were designed to improve the overall monitoring of cash and cash equivalent transactions.  As the regulations changed, the expectations of the regulatory bodies evolved.  Today, no self-respecting financial institution would consider operating without a full BSA/AML compliance program.   Moreover, it is not feasible to try to get away with a manual system for tracking and aggregating the transactions of customers.   A sound BSA/AML program includes software that helps staff aggregate and monitor transactions of its customers.  

Along with the expectation that your financial institution will obtain BSA/AML monitoring software is also the requirement that a model validation be performed.  

 The Source of the Data Validation Requirement 

The OCC and the Federal Reserve issued guidance in 2011 that was called Supervisory Guidance on Model Risk Management”[1] .  This guidance was first thought to deal with only the financial models such as those that are used for projecting interest rate risk or the allocation of the allowance for loan losses.   However, a more complete review of the information included in the guidance has produced increased expectations in the area of BSA/AML. 

In relevant part, the guidance states that a model is defined as follows: 

“For the purposes of this document, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. Models meeting this definition might be used for analyzing business strategies, informing business decisions, identifying and measuring risks, valuing exposures, instruments or positions, conducting stress testing, assessing adequacy of capital, managing client assets, measuring compliance with internal limits, maintaining the formal control apparatus of the bank, or meeting financial or regulatory reporting requirements and issuing public disclosures. The definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature”[2]

When one reads this definition of a model, it is clear that BSA/AML monitoring software is included.  The guidance is directed toward the idea that modeling software cannot be a panacea when it comes to compliance.  Models can only be effective when they are part of a complete compliance program in any area.  In the area of BSA/AML compliance, this is especially true.  

The model guidance points out that there are several areas of risk that are associated with the use of models at a financial institution.  Many of these risks apply to BSA/AML monitoring software.   When the areas of risk are simplified, the two main concerns for BSA software are:

1.       The data that is being collected and loaded into the monitoring software is inaccurate: and

2.       The data that is being collected is insufficient to properly mitigate risk.  

 

Data Validation

To address the first of the above enumerated risks, all banks should perform a data validation. [3]   This process is basically what it sounds like.  The data validation is the process of making sure that the information in your monitoring software is being accurately and completely loaded from your core system.   While this portion of the guidance may be straight forward, there are few points to remember when preparing to perform an appropriate data validation.  

Know Thy Software

It is important to know the type of software that you are using.  Generally, there about five types of monitoring software that are currently popular and on the market.  It is important to know which type of software your institution is using so that you can determine whether the appropriate data is being pulled.  The five types of software are:

·         Risk based- These are systems that incorporates various factors such as NAICIS codes, zip codes, volume and frequency to predict higher risk customers

·         Rule based- system that compares transactions to scenarios that mimic suspicious activity

·         Behavior based- These systems establish a base line for a customer and track activity that exceeds the baseline

·         Intelligent systems- This software is based on decision trees that follow data that has indicated suspicious activity

·         Combination- These systems incorporate two or more of the above into the software.  

Regardless of the type of system that you are using, it is important to recognize how your system works so that you know the data points that should be recognized.   For example, if you are using behavior based software, it is important to recognize what information the software needs to know to establish the baseline for a particular customer. 

Software-Know thy User

It is critical that all of the transaction codes that your bank uses are being properly loaded into, and recognized by the software that you are using.   Each bank has its own unique set of transaction codes that have been established to identify transactions that are conducted.  The software vendor cannot know all of the idiosyncrasies of your bank and it is therefore incumbent on your compliance staff to ensure that the transaction codes are being properly loaded and recognized by your monitoring software. 

Compare to the Core

Many banks we have worked with use the data validation information that is provided by the vendor.  However, it is important to remember that this validation will simply tell you what happened to the information that you gave to the vendor.  If there are other errors in logic or misunderstandings about what information should be captured, this will not appear in the vendors’ validation.  We recommend that the data validation should be completed by comparing the software information to core data information.  

 Ongoing Validation

Many banks and vendors believe that once a data validation has been done, there is little need to do another one.  If everything checks out and all data is being loaded properly, what is the problem? have you ever logged onto your computer and found that everything had changed, even though you did not do anything different?  BSA software is the same.  Even though you may not have consciously made changes to the software or to the processes, things change for various reasons.  Because change is constant, it is a best practice to test data validity on a regular basis.    Consider this; if you do find a problem, it will be necessary to go back to the last data validation to determine the extent of the problem.  The longer you have waited, the bigger the problem!

The Known Knowns

Finally, a data validation would not be complete without considering what the data actually does and does not display.  For example, one weakness of many software monitoring programs is the inability to closely monitor transfer transactions.  Suppose a customer cashes a check and gives the proceeds to another customer who deposits the cash.  It is important to be able to determine how this information would be captured by the software.  In the alternative, if the information is not captured by the software, what provisions have been made to monitor such a transaction?

 

 

 

 

Model Validation

It is not enough to simply test whether or not the data in your BSA/AML software has been properly mapped.  You must also determine that the software is doing what the bank needs it to do to monitor suspicious activity.  

The OCC guidance points out that the use of models in any banking environment must fit within a risk framework.  This framework has essentially four elements:

·         Business and regulatory alignment – the model must fit the bank’s risk profile and regulatory requirements

·         Project management – a proper and appropriate implementation is an ongoing project that is dynamic as the bank’s operation

·         Enabling Technology – The use of the technology should facilitate the bank’s ability to meet its regulatory requirements

·         Supporting documentation – As a best practice, documentation of the rational for using the model should be maintained. 

For BSA/AML, monitoring software, the risk framework means that regulators expect financial institutions to know how its software works as well as the “blind spots”   for transactions that may not be completely covered by the way the software operates.   The expectations are that your staff will use monitoring software as a tool that is constantly being sharpened and improved.  The model validation process is the means to ensure that the software is improving.  

Its 11pm- do you know what your software is doing?   

The first consideration in completing a model validation is to determine just which type of software you are using and how it works; in other words the conceptual  framework of the software.    

It is important to document that the institution is aware of which type of software it has.   Moreover, it is important to document how this software’s concept aligns with the risk profile.   The model has to be one that has the ability to discern the transactions that your BSA assessment has identified as higher risk.  Ultimately, the expectation is that you will be able to document what it is that your software is monitoring and how it is keeping track of your risky transactions.   You should be able to document just how alerts are created and what they mean.  Are you getting a report when a customer sends a wire for the first time?  How about the first time a wire goes to a foreign country?  Does the software have the ability to track and aggregate ALL transactions associated with one customer and her affiliates?   

Along the same lines, it is critical that a gap analysis is performed to determine where the system has “blind spots”.  Ultimately, the conceptual framework of the software that you are using must match the products, customers and transactions of your bank.  

Practice Makes Better (Never Perfect)

Many institutions are shocked to find that they have been criticized for keeping the default settings of the software.  We are being conservative, the logic goes, by keeping the settings of our software as wide and open as possible.  However, this is most certainly not a best practice.  The use of models is supposed to be a tool that enhances the ability of the BSA staff to determine suspicious activity and act on that information as is necessary.  Default settings do not reflect the risk profile of an individual firm and often lead to a large number of alerts.  When alerts are generated for transactions that are clearly ordinary or not at all suspicious, the output from the software becomes ineffective as the BSA staff spends hours on superfluous data.   The model has to be trained to know the risk profile of the institution and to look at data that is outside of that profile.  Therefore, as a part of a model validation, it is necessary to review the rules or settings in place and to adjust to optimize the software.   This process should be based upon both the experience of the BSA staff as well as by doing comparative testing.  An effective means of optimization and tuning is to use test several accounts.  Using core data, a staff member can ensure that all transactions that should be considered suspect are being noticed by the software with alerts. 

Who’s Minding the Store?

The guidance from the OCC and the Federal Reserve also makes clear that a critical component of model risk management is output analysis.  The best data in the world will be very ineffective in the hands of a staff member who does not know how to interpret it.  Moreover, the staff that receives and analyzes data must also have the ability to act on their conclusions.    The output of monitoring software should be reported to senior management on a regular basis along with information about the actions taken in response to the data.   It is not simply enough to show that the software is creating alerts that result in a Suspicious Activity Report (SAR).  The expectation is that at some point when a customer has received multiple SARs, the data will be reported to senior management and a decision should be made whether or not to continue the relationship with the customer. 

Governance (It ALWAYS Comes Down to This!)

Ultimately, model validation comes down to the overall governance being practiced at a bank.  Models are only as effective as the structure in which they are used.   The guidance notes that there has to be governance structure that surrounds the use of monitoring software.  This structure should include:

·         Senior Management and Board Involvement – through regular reporting to the Board or a committee of the Board

·         Policies and Procedures – which should be updated on an annual basis

·         Roles and Responsibilities – the staff who are responsible for reading and interpreting the data should be designated by the Board. 

·         Enterprise Risk Management and Reporting – the software must be dynamic and should change along with the risk profile of the bank.

·         Independent Audit and Testing- The overall model’s effectiveness should be reviewed and tested regularly by an independent party.

 

 

PLEASE JOINS US FOR OUR FREE 15-MINUTE WEBINAR “ARE BSA SOFTWARE INDEPENDENT REVIEWS NECESSARY” THIS THURSDAY APRIL 21, 2016 AT 10AM PST.  PLEASE GO TO WWW.VCM4YOU.COM TO REGISTER.

---

[1] See OCC 2011-12;  Federal Reserve SR 11-7

[2] Ibid 

[3] The guidance clearly applies to ALL banks.