DRP BCP Best Practices Defined
Here are some Disaster Recovery Business Continuity best practices
- Keep your primary backup disaster recovery business continuity data in house – this is your first line of defense, a quick fix in case something goes wrong.
- Analyze your critical systems and their subsystems – Identify applications that are critical to your business and the data and systems that these systems depend on.
- Your backup policy has to include some incremental systems and snapshots to be able to handle when single files or select data is lost.
- Think carefully about your backup solution – if your requirement is very granular – like restoring a damaged mailbox or maybe even a single mail, then your solution has to be selected keeping this in mind.
- Long term backup in a public cloud is a great option – it is not expensive, availability is good however be aware of security considerations.
- Run the backup from the cloud directly – if your backup data is kept in the cloud, then running a recovery directly from the cloud is easy. Y
- Test everything – test your backup solutions regularly till you are confident that they will work when required. The need to recover may occur at the middle of the night or when you are far away from your on premise systems. A backup solution in the cloud is a great solution to ensure business continuity
IT Job Opportunities
IT job opportunities – The U.S. Bureau of Labor Statistics (BLS) reported from January 2009 through December 2011, 6.1 million workers were displaced from jobs they had held for at least 3 years. The data shows that 26.7 of those individuals remain unemployed and another 17.4% have left the labor pool all together. That adds up tp 44.1 % of these long term employees continuing to be out of work and does not include any factor for those who are underemployed. IT job opportunities for long term unemployed remain poor.
This was less than the 6.9 million for the survey period covering January 2007 to December 2009. In January 2012, 56 percent of workers displaced from 2009-11 were reemployed.
Janco talked with a dozen IT Professionals who had been employed in IT for over 3 years with the companies and had been terminated. One of the common refrains was, after being out of the IT job market for several months many of these professionals felt that their skills has decayed and that that employers did not view them in a positive light because of the lenght of time they had been out of work. Some of these individuals were able to find “part-time” and “contractor” however they found the job market remains tight for them. Others have left the IT Market all together and had started alternate careers.
Long Term Employees Employment
Business continuity and disaster recovery planning took a real hit in the recession that started in 2008. First many companies reduced the number and intensity of testing the plans. Second many firms cut staff in support area like communications and do not have sufficient staff in place to provide the level of information that is required when an event occurs. Last but not least the rapid evolution of social networks and BYOD has added a level of complexity that did not exist before.
In addition to a DR / BC plan CIOs and operation executives need to have an Incident Communication in place that not only provides what to say but how to use technology like social media to get the message out.
Business Continuity has been a stepchild in this recession
The specific objective of this incident communication plan is to define who will provide key communications during a crisis and the content, recipients, schedule, method of delivery, frequency and priority of the communication. By outlining communications in advance, ENTERPRISE
- Protect the effect of a crisis on employees, associates, suppliers and customers,
- Reduce the impact of bad publicity, maintain customer service, bolster relations with vendors and
- Addresses the concerns of other key stakeholders
Here is a great video that you can use to create a presentation to explain why plan and how your plan should take shape.
Top ten by Janco – Wildfire caused and increase business interruption risk as the number of companies that are located in business parks located in the outskirts of population centers increases. This year scores of fires sparked by high temperatures, severe drought conditions and strong winds have blanketed the western part of the US, including Utah, California, Washington, Montana, Oregon, Idaho, Nevada and Arizona, making this fire season one of the worst in history for this area. According to the National Interagency Fire Center (NIFC), over 43,000 individual wild fires burned a record 6.8 million acres cross the West since January. The wildfire season continues through early number and prediction of over 7.5 million acres is now being made by some forecasters.
Janco’s Top 10 to minimize wild fire disaster planning risks
What can businesses do reduce the risk to properties? Janco’s guidance, supported by NIFC recommendations is:
- Have a clear area of at least 100 yards around the business park.
- Keep lawns hydrated and maintained. Dry grass and shrubs are fuel for wildfire.
- Landscape with native and less-flammable plants. When landscaping, choose slow-growing, carefully placed shrubs and trees so the area can be more easily maintained.
- Create a ‘fire-free’ area within ten feet of the property, using non-flammable landscaping materials such as rocks, pavers and/or high-moisture content annuals and perennials.
- Have no tall vegetation immediately adjacent to structures.
- Clear leaves and other debris from gutters, eaves, porches and decks. This helps prevent embers from igniting the property.
- Remove dead vegetation from around the property, especially within 50 feet of the premises.
- Remove flammable materials from within 50 feet of the property’s foundation and outbuildings.
- If you have trees on your property, prune so the lowest branches are 6 to 10 feet from the ground and none overhang the structure.
- Don’t let debris and lawn cuttings linger. Dispose of these items quickly to reduce fuel for fire.
Best of Breed solutions for disaster recovery and business continuity has four key components:
- High Availability – Best of breed requires service that have high availability. The service may go down but will recover quickly enough that the employees and clients are not significantly impacted, if they notice at all. High availability helps mask or minimize the effects of the failure and makes it less of an issue for those who consume that IT Service.
Best of Breed for World Class Organizations
- Fault Tolerance – Best of breed requires that A system have a high fault tolerance when the entire system will not fail even if a critical IT service component is compromised. This is achieved through solutions with redundant hardware and software so that the slack is immediately picked up by the secondary system.
- Continuous Operation – Planned downtime is a common occurrence for many if not all IT services. Recent innovations in virtualization allow IT teams to perform maintenance on systems without downtime. IT Services that meet this requirement are considered to be in continuous operation.
- Continuous Availability – This is the ultimate goal of DR and BC teams. This means that the service achieves 100% availability by avoiding both planned and unplanned downtime. This is achieved through a combination of Disaster Recovery and business continuity solutions such as those defined in Janco’s Disaster Recovery and Business Continuity Template.
Disaster Recovery — What are the major misconceptions when a disaster occurs with IT systems? Can your systems can not support your company’s day-to-day operations?
The major misconception is that a backup recovery plan is all that you need. At Janco Associates that is not enough. We have found that most companies are really not prepared. Files can be restored but it does no good if they do have facilities for their staffs.
Victor Janulaitis, the CEO of Janco Associates, was responsible for the creation of the Disaster Recovery Plan that Merrill Lynch implemented on 911. ML lost less than one minute of transactions with the plan that was created under his direction.
Disaster Recovery Audit
A core process that he identified was a Disaster Plan Audit. This Disaster Recovery / Business Continuity Audit program identifies control objectives that are meet by the audit program. There are 36 specific items that the audit covers in the 13 page audit program. Included are references to specific Janco products that directly address the areas the audit covers. This program can be used as standalone audit program or in concert with the following Janco offerings:
- Disaster Recovery / Business Continuity Template
- Security Manual Template
- Security Audit Program Template
- Business and IT Impact Questionnaire
- IT Service Management for Service Oriented Architecture
- Metrics for the Internet and Information Technology
Here is a great video that another company has produced that describes what some of the major misconceptions are in disaster recovery and business continuity planning. These thoughts are the same as Janco’s and the video is well worth watching.
As people continue to forecast an improvement in the economy, the latest BLS data shows there still are 8 states with unemployment number that are higher than 9%.
In the long run that is not good news for IT pros. First, companies in those regions do not have good prospects so they typically will not hire or invest in new systems. Second, those who have jobs in high unemployment areas are less likely to move to new companies because of uncertainty about the economy reducing the number of opening created by voluntary moves.
The states with the highest unemployment numbers are:
- Nevada – 12.0%
- Rhode Island – 10.8%
- California – 10.7%
- New Jersey – 9.8%
- North Carolina – 9.6%
- South Carolina – 9.6%
- Georgia – 9.3%
- New York – 9.1%
Add to that the influx of undocumented worker in California, Nevada, New York and New Jersey the employment picture looks like anything but positive. When will this turn around? If you listen to many economists, there is a new trend that they say is happening and this may be the norm for the next several years.
Paper disaster recovery and business continuity plans are difficult to keep up to date and be available for the recovery process. One solution that we have found that works is to the plan in the cloud. You can synchronize it (with tools like dropbox.com) via smartphones and tablets.
Disaster recovery and business continuity planning is now accepted as basics requirement for every business and organization. It is widely accepted that a detailed business continuity plan should not only exist, but should be up to date. It should reflect the actual on-going needs of the business activity or function.
Disaster Recovery Plans need to be readily available
The Disaster Recovery and Business Continuity Template that we have is very well suited for that type of implementation.
We found a video by one of my former employers (Deloitte Touche) whom I created thier published SDM with was the basis for their offering.
This is a great video on physical security as well as the the software security. This is a great primer which all CIOs and Data Center managers should consider. Now only does it address the physical security issues, disaster recovery, it also addresses how Google implements and disposes of new server devices.
“Disaster Recovery and business continuity are all about being ready for everything. The question that every IT manager and CIO has to answer every day is what should they complete today If they knew a business interruption was going to happen in the next 12 hours.”
Cascading problems are not things that most companies want to talk about but are disaster recovery business continuity risks
We have one client, who wants to remain nameless, on a Friday evening he thought the had a hardware problem. The weekend staff proceeded to connect that device into the network to diagnose the issue and a virus was released. Then they transmitted that virus to one of their largest suppliers. When it was all said and done they spent well over $500,000 to isolate the virus, restore the files, and make their supplier whole. They were just lucky this happened over the week-end so it did not impact as many people. Interestingly if this problem had surfaced a few hours earlier their regular staff would have diagnosed the problem off-line and it would not have gotten away from them.
Disasters and events that impact business continuity vary widely in more than duration. As you design your plan, consider the probability of threats that are:
- Chronicled — events that have occurred (Power outages, earthquakes, hurricanes)
- Human — events likely from carelessness, malicious intent, fatigue, or lack of training
- Geographical — events likely as a result of the location of your business (floods, storms, lightning strikes, earthquakes, typhoons, tsunamis)
- Localized — events due to system malfunctions (assembly line failures, computer crashes, sprinkler activations, chemical spills)
- Planned — scheduled events (software upgrades, system tests, facility moves) that go awry
Janco’s own list of top 10 disasters that CIOs and business managers need to plan for are:
- Weather related events like floods, tornadoes, hurricanes, forest/brush fires, and sand storms
- Facility fires
- Water pipe breaks in facility
- Fiber or communications line are cut – loss of network
- Power failures – Outage or sporadic service
- Human error like a redundant systems failure that goes unnoticed and hinders the recovery operation
- Security breach hacking and or malicious code
- Data corruption and loss – not only from physical device or network failure but also from application and user error
- Cascading system failure
Janco believes that a prepared, and well-rehearsed team address the issues associates with a major and minor business interruption much quicker than companies who have no plan and no preparedness.
Information Management Magazine and Insurance Networking News both report that there was significant growth in the Health Care field in the number of IT jobs available. Much of this is due to the requirement that all medical records (EHR) are required to be mechanized and new compliance requirements for the Affordable Health Care Act (aka Obama-care).
It is estimated that the Health Care IT spending will increase by up to 25% in the next two years. Spending last year for Health Care software was was close to $7 billion and is expected to grow by over $1 Billion in the next year. Much of that spending will be in the “small practice” physician and “small hospitals”. The question is how protected will they be from business interruptions and security attacks.
Do have any comments on this?
Modular data centers have come a long way.
Modular data centers have been in use by the government since the 1960s. Now they are commercially available for companies of all sizes. Here is a video that provides a great description of some of the options available to CIOs and Data Center Operations managers.
This is a far cry from the first ones that I saw in the late 60’s. The US Army had them with an IBM 1401, a workstation, communications gear, a generator, air conditioning, and a satchel charge to blow it up if the position was over run. They unit was flow in by helicopter for artillery calculations.
Disaster Recovery plans that depend on outsourcers face significant additional risk
What if your were in Florida and the Hurricane season was in full swing and your provider decided to go out of business. Would you have the time to move to a new provider and test your solution before you need to execute your plan?
For example, earlier in the year Google decided to close its Message Continuity service. Google gave most clients a reasonable timescale to find an alternative supplier. This allowed existing Message Continuity contracts to run until their contacts expired. What if that was the communication solution you had selected for communicating with your staff? Would you be able to implement a new one on time.
Another example was the news that Doyenz, the US-based supplier of rCloud, a service which offers disaster recovery for physical and virtual servers, had decided to pull the plug on its UK operations. Clients were given not weeks or months but days to respond and to find a new supplier.
CIOs and IT managers all need to consider all of the possibilities and have alternative solutions in place and tested.
FEMA conference videos which discuss tools and services available in the disaster and business continuity processes.