Why are US farmers facing a disaster?
Is it due to locusts? No.
It's due to inadequate IT disaster recovery planning.
As reported in the Wall Street Journal, the US Immigration Department is unable to issue visas to temporary workers due to a system failure. Specifically:
"“The system that helps perform necessary security checks has suffered hardware failure,” said Niles Cole, a State Department spokesman. “Until it is repaired, no visas can be issued.” He said technicians are working around the clock to resolve the issue but couldn’t offer a timeline for when the system would be back in action.
Specifically, a central database isn't receiving biometric information from U.S. consulates world-wide, he said. Biometric data, including fingerprints, are used for security screening of applicants."
And the losses are mounting daily. Over 200 workers are sitting at the Mexican-US border waiting to be processed by system so they can get into the US and help harvest the crops. The article reported that farmers are losing between $500,000 to $1,000,000 per day because the fruits are spoiling.
Reading this article I had the following questions
Why isn't there a hot site?
Given the importance of the technology, why don't they have the ability to swap to a new piece of hardware instantaneously?
Was the security information backed up and why is there no manual work around?
If it's digital information, why isn't there a manual work around to transmit the information and circumvent the faulty hardware? The data could be manually uploaded to the central database.
Was a proper risk assessment done? When a disaster recovery plan (DRP) is created for a system, the organization must determine the Recovery Time Objective (RTO) that determines how quickly a system will be stored after failure. Google, for example, has an RTO of zero. To determine what the RTO is there needs to be an assessment of the impact of such a failure. In this case when setting the RTO did the risk management professional include the fact that this system was critical in supporting the visa program H2-A for temporary farm workers? It should be noted that the US farmers association had paid into this program and now they are suffering losses of over $500,000. This will also reduce the amount of tourist visas issued potentially resulting in lost tourist dollars to the US.
The lesson we can learn from this is to ensure that we understand what business processes a system supports and understand the impact to those business processes should the system go down.
Showing posts with label backup. Show all posts
Showing posts with label backup. Show all posts
Wednesday, June 17, 2015
Sunday, July 8, 2012
Electrical and cloud outages: Is it time to bring both on premise?
Amazon experienced an outage that affected a number of companies that rely on their cloud service. The company informed its users that its service went down due to the power outage stating:
"On June 29, 2012 at about 8:33 PM PDT, one of the Availability Zones (AZ) in our US-EAST-1 Region experienced a power issue. While we were able to restore access to a vast majority of RDS DB Instances that were impacted by this event, some Single-AZ DB Instances in the affected AZ experienced storage inconsistency issues and access could not be restored despite our recovery efforts. These affected DB Instances have been moved into the “failed ” state."
This notice was actually taken from CodeGuard (a start-up that takes snapshots of websites enabling owners to undo unwanted changes) who was one of the companies affected by the outage.
As can be expected, many will use this as an opportunity to illustrates the danger of moving from on premise to the cloud. A parallel argument would be to highlight the dangers of drawing on electricity from the central grid. One would argue one is more reliant on power than on computing - so why not bring electricity "back" on premise? This is an absurd argument, but that is exactly the point. Companies, as pointed out by Nicholas Carr in the Big Switch, used to produce their own electricity, but eventually moved to rely on the grid for power. Today hardly anyone produces their own power, but has backup generators in place to provide power should grid go down. And that's the right question to ask: why was there inadequate backup power at Amazon? In other words, society has decided to live with the fact that electricity is delivered centrally - but has built in controls to manage issues that may arise.
Instead of viewing this as a black mark against cloud computing, it is important to view this discussion in the context of risk. Charles Babcock, InformationWeek published a good article on the reaction to the Outage. He noted that some are leaving AWS in reaction to the service. Specifically, Whatsyourprice.com (an online dating service) is moving to a hosted solution - away from the cloud. However, he also mentions, Okta (an identity management service) that was unaffected by the outage because they designed their application to be fault tolerant.
In other words companies need to focus on whether the benefits of cloud computing outweigh its risks. Cloud provide pay-as-you-go computing - giving companies who have uneven workloads the ability to buy compute resources when they need it. It also give start ups, like CodeGuard, a chance to get their offerings into the market. Here,here and here are the follow-up posts to their outage - they were able to get back online and they are sticking with Amazon. And this should not be a surprise to anyone. Technology startups can leverage the pay-as-you-go model of cloud computing to conserve their capital and instead focus on getting their offering out. For example, the founder of Animoto, points out they went from 50 to 80 compute to 3,500 instances over three-days (they were signing up 25,000 new users per hour at the peak) when their app went viral. So companies will hopefully use the cloud outage to highlight the need for good design and appropriate controls instead of an excuse to stick to the status quo of on-premise computing.
"On June 29, 2012 at about 8:33 PM PDT, one of the Availability Zones (AZ) in our US-EAST-1 Region experienced a power issue. While we were able to restore access to a vast majority of RDS DB Instances that were impacted by this event, some Single-AZ DB Instances in the affected AZ experienced storage inconsistency issues and access could not be restored despite our recovery efforts. These affected DB Instances have been moved into the “failed ” state."
This notice was actually taken from CodeGuard (a start-up that takes snapshots of websites enabling owners to undo unwanted changes) who was one of the companies affected by the outage.
As can be expected, many will use this as an opportunity to illustrates the danger of moving from on premise to the cloud. A parallel argument would be to highlight the dangers of drawing on electricity from the central grid. One would argue one is more reliant on power than on computing - so why not bring electricity "back" on premise? This is an absurd argument, but that is exactly the point. Companies, as pointed out by Nicholas Carr in the Big Switch, used to produce their own electricity, but eventually moved to rely on the grid for power. Today hardly anyone produces their own power, but has backup generators in place to provide power should grid go down. And that's the right question to ask: why was there inadequate backup power at Amazon? In other words, society has decided to live with the fact that electricity is delivered centrally - but has built in controls to manage issues that may arise.
Instead of viewing this as a black mark against cloud computing, it is important to view this discussion in the context of risk. Charles Babcock, InformationWeek published a good article on the reaction to the Outage. He noted that some are leaving AWS in reaction to the service. Specifically, Whatsyourprice.com (an online dating service) is moving to a hosted solution - away from the cloud. However, he also mentions, Okta (an identity management service) that was unaffected by the outage because they designed their application to be fault tolerant.
In other words companies need to focus on whether the benefits of cloud computing outweigh its risks. Cloud provide pay-as-you-go computing - giving companies who have uneven workloads the ability to buy compute resources when they need it. It also give start ups, like CodeGuard, a chance to get their offerings into the market. Here,here and here are the follow-up posts to their outage - they were able to get back online and they are sticking with Amazon. And this should not be a surprise to anyone. Technology startups can leverage the pay-as-you-go model of cloud computing to conserve their capital and instead focus on getting their offering out. For example, the founder of Animoto, points out they went from 50 to 80 compute to 3,500 instances over three-days (they were signing up 25,000 new users per hour at the peak) when their app went viral. So companies will hopefully use the cloud outage to highlight the need for good design and appropriate controls instead of an excuse to stick to the status quo of on-premise computing.
Subscribe to:
Posts (Atom)