BNY Mellon Software Glitch: Cost of IT Control Failure

In the previous post on the BNY Mellon's technology woes, we explored what the company did right as well as the overall need for independent evaluation of the technology that runs the Information Age. In this post, we explore the costs and consequences of the breach.

One of the challenges for putting in controls around information integrity is that it is a hard sell: what's really the value of accurate information? This is in contrast to something like information security where it is also hard sell, but much easier. The reason? When an information security breach occurs, it is largely to access something of value that can be monetized. The Poneman Institute puts this cost at approximately $174 per record.

Consequently, it is easier for someone to go to the CEO/CFO and explain how tightening controls around information security will protect the company's bottom line. Furthermore, information security breaches are something that has entered the mass consciousness within the business community: SunGard was quick to reassure everyone that the issue affecting BNY Mellon's accounting software was NOT attributable to "any external or unauthorised systems access".

When making the business case for controls over information, it can be challenging to show how the control will lead to savings in terms of "decision failure", i.e. the cost of making the wrong decision due to unreliable information. Let's face it: most companies are willing take big risks on their information by continuing to rely on spreadsheets that have an error rate of 88%. Furthermore, as highlighted by this Protiviti study, internal auditors understand the information integrity challenges but are not getting the funding to tackle them.

So the incident at BNY Mellon is rare occurrence where something that is mis-priced can actually lead to costs. As noted in the Wall Street Journal:

"A software glitch this week at fund administrator Bank of New York Mellon Corp. caused difficulties in pricing many mutual funds and exchange-traded funds, prompting some fund sponsors to publish lists of funds whose stated asset values were erroneous.

What can you do if one of your funds is on the list, meaning you may have overpaid for shares?

Reach out to your fund company and ask for a refund. They don’t have to give you one but firms may do so because of their often long-term relationships—ones they want to keep—with investors, analysts said."

The other costs include:

• Direct costs: As noted in the WSJ, "BNY Mellon was working on a separate contingency plan, mobilizing more than 100 accountants to manually calculate the values of thousands of fund securities". The company also required SunGard to come in and work at their premises to fix the issue. If the company had to pay extra for this work to be completed, it would add up quickly. 
• Loss of business due to damaged reputation: Experiencing these types of issues exposes the company to the loss of business due to their reputation being damaged. 
• Regulatory scrutiny: Yes, BNY Mellon is under regulatory review. As noted in the Boston Globe and Yahoo Finance, BNY Mellon's technical problems have attracted regulatory scrutiny of Massachusetts's Secretary of State William F. Galvin who has sent letters of inquiry to  the company to investigate what happened. Regarding the incident he said: "In the warp-speed of trading these days computer problems can happen...But the fallout that seems only to affect large financial institutions can hit the average investor looking at his and her retirement money". 

Of course we won't know the full cost until, the regulatory probe finishes and the publish their findings or the cost was material and this shows up in the financial statements. Regardless, organizations should be proactive in ensuring that sufficient technology controls are in place and that these types of risk are controlled. 

BNY Mellon Software Glitch: Time to make SysTrust mandatory?

As was widely reported in the business press, BNY Mellon experienced a technical glitch that affected its ability to price mutual funds accurately. Based on the press release from one of the affected funds, the problems started on Monday August 24th, where one of BNY Mellon's system "InvestOne" managed by SunGard was pricing about 800 mutual funds inaccurately.

So what was the cause of this fiasco?

According to CNN, "BNY Mellon outage occurred after a SunGard accounting system it uses became "corrupted" following an upgrade. A back-up also failed."

Normally, this type of thing will force the party experiencing the breach intense scrutiny over what went wrong. However, as I went through the timeline posted by the company, I found (reading between the lines) that they did a number of things right, such as:

• Incident Management Communication Plan: One of the aspects of incident management is communicating to the public and making them aware of the issue. As it can be seen, the company posted details on estimates as to when the system would be ready as well as what was the source of the delay when they didn't meet their estimated deadline. 
• Contingency planning: BNY Mellon was able to get the Net Asset Values (NAVs) "using a secondary and more manual process" to the clients without the production system. According to WSJ, this involved "mobilizing more than 100 accountants to manually calculate the values of thousands of fund securities". The secondary procedure was not perfect as small differences were noted between the fund and, as reported the WSJ, this could expose the bank to "potential liability from the resolution of trades that may have been made during the pricing fog".  
• Effective Service Level Agreement (SLA) with SunGard: Based on the close coordination with SunGard, it appears that they were able to get SunGard to eventually rebuild the system "from scratch". 

That being said, there is always room for improvement. When I was reflecting on this, I speculated that this was another case of inadequate testing of the system upgrade. However, according to SunGard, this was not the case. As they noted on their website:

"The issue appears to have been caused by an unforeseen complication resulting from an operating system change performed by SunGard on Saturday, August 22nd. This maintenance was successfully performed in a test environment, per our standard operating procedure, and then replicated in SunGard’s U.S. production environment for BNY Mellon. This change had also been previously implemented, without any issues, in other InvestOne environments. Unfortunately, in the process of applying this change to the SunGard production environment of InvestOne supporting BNY Mellon’s U.S. fund accounting clients, that environment became corrupted. Additionally, the back-up environment hosted by SunGard, supporting BNY Mellon’s U.S. fund accounting clients, was concurrently corrupted, thus impeding automatic failover. Because of the unusual nature of the event, we are confident this was an isolated incident due to the physical/logical system environment and not an application issue with InvestOne itself."

Given my background as a CA, CPA and CISA, I have always thought it is an odd contradiction that we expect infrastructure (road, dams, bridges, etc.) to be certified by engineers to be in working order (key word is expect, as John Oliver notes in the video below, this is not exactly up to snuff!), but do not have the same expectations for the technology that runs the Information Age.

And that's where I have always proposed that it is necessary to have a framework like SysTrust (now SOC2 and SOC3) in place that requires companies to ensure that their systems are reliable: secure, available, and able to process information without messing it up.

Based on the experience between SunGard and BNY Mellon, I think it actually proves the case. Although companies, like SunGard, likely have such controls in place it is beneficial to others to have a second set of eyes on those controls, ensuring that they are in place, are designed effectively and are operating effectively. The reason is that with such mandatory audits in place, it will allow for the circulation of best practices through such audits. This occurs in the financial auditing world through "management letter points".

One other area that we should explore is the total impact of this error, as it will give insights into the "total impact of failed IT controls". This will be the topic of the next blogpost.