Emergency power: How to improve your firm's disaster recovery

Four things you will learn from this Masterclass:

1. How to determine your firm’s preparedness for a major outage

2. How to decide which systems and services to prioritise

3. How to manage the chaos of emotions during an outage

4. How to continually improve your business recovery time

 

Managing an IT department on a day-to-day basis is challenging enough, but adding a major outage to your day while managing your department is always going to make for an interesting day.

In 2011, hardware failures and power outages were the main reasons for invoking business continuity services in the UK, according to research by Phoenix. This shouldn’t come as a surprise, given how heavily businesses now rely on IT infrastructure. Indeed, 100 per cent availability is now no longer viewed as a ‘nice to have’, but rather as a business ‘must have’. No matter how unrealistic this may seem, it is what most law firms expect.

Obviously, during an outage, there are a number of challenges you will face. To determine if you are fully prepared to manage a major outage, consider the following questions and see how confidently you can answer ‘yes’ to each of them:

• Can you name your top ten (Tier 1) systems?

• Have you agreed the recovery priority order internally?

• Do you have a management structure that enables prompt decision making?

• Do you have documented disaster-recovery strategic plans in place?

• Can you access your plans remotely?

These are the most basic of questions that you should be asking yourself when considering your disaster recovery preparedness. If you cannot give a confident ‘yes’ to all of the above, your challenges will be many and varied on a daily basis. And, during a major outage, they will multiply rapidly.

Before the outage

Before the outage happens (and it will happen), review your business continuity capabilities and seek to understand them as best you can.

To be better prepared for any event that may impact your firm, you should begin by determining what is important to the business and which systems and services support it. Start with the firm’s vision and goals to determine what the businesses approach/attitude will be during an outage; build your recovery strategy from there.

Understanding how business processes achieve business goals will help you to determine the impact on the firm if you lose certain systems and services. You will then be able to identify which are your firm’s critical resources and Tier 1 systems.

During the business impact analysis, you will determine just what the cost of a systems outage will be and therefore begin to have a list of critical systems and services that need to be protected. The cost is linked to financial, reputational and operational costs incurred as a result of an outage.

Outages come in two forms: slow burning and sudden. Thankfully, sudden outages are reasonably rare, but they do happen.

Power outages can be classed as slow burners or sudden events, as they depend upon the protection that has been put in place for your firm’s critical services. You may have some time to shut down non-essential services, or a power outage may happen very suddenly if you haven’t invested in an uninterruptible power supply or generator for your computer room.

If only you know what your business continuity strategy is and it’s not aligned to the business objectives, you will find yourself trying to explain why you’re doing certain things when a major outage is happening around you.

When a major outage occurs, it’s a tense time and therefore stress levels will undoubtedly be high. Clearly, an outage is not the best time to discuss roles and responsibilities, so establish clear accountabilities and ensure they are documented and understood internally well before an outage occurs.

During the outage

Despite what partners may believe about their crisis management abilities, many will dramatically fail to keep their composure during a major outage/crisis involving their business; leading is something that does not come naturally.

I have personally witnessed very senior partners become angry and abusive during a major outage because, quite simply, they are out of their comfort zones. Anger, confusion, blame and self-preservation are common characteristics seen when a crisis occurs.

Confusingly, you may notice some people will become introverted, quiet or refuse to see the size and scale of the incident which is unfolding before them. This isn’t always the case, however; I have also seen some extraordinary acts of leadership. These tend to be played out by those people who are not trying to protect their own ‘empire’ but are truly trying to achieve a common goal.

So, the challenge you face during a major outage is to remove emotions from what is happening and to assess the incident with all of the information and resources available.

Unfortunately, but predictably, you may find yourself fighting a culture of ‘empires’, where individual needs and personalities start to take over and it becomes a battle of ‘he who shouts loudest’.

In order to combat this situation, you must have a good crisis management framework in place, where someone will take the lead and ensure everyone has equal opportunity to voice their concerns about the recovery steps being taken.

This is in addition to the work that IT will be doing to fix the problem. It needs to be understood that, while it may be an IT outage, every area of the business may be impacted and therefore everyone has a part to play. IT are all too often asked to explain ‘what has happened?’ and ‘why has this happened?’ while the outage is still happening, when they instead should be focusing on restoring key systems and services.

After the outage

The final challenge takes place after the outage has happened and the firm is returning to normal.

Finding the time to conduct a post-incident review (PIR) and a root-cause analysis (RCA) will ensure lessons can be learnt about how well the business responded to the event (the PIR) and will help the business to understand what is needed to prevent the outage from occurring again in future (the RCA).

The difficulty here is ensuring the business is honest about how well it reacted and taking the time to learn lessons and improve processes. In a fast-paced business, this will always be a hard.

 

---

Improving your firm’s response to a major outage

• Designate the role of business continuity (BC) to someonewho a) has the time to do it right and b) has the experience/knowledge/personality to do it right.

• Conduct a business impact analysis (BIA) to determine the criticality of a service/system to your business.

• Conduct a risk assessment (RA) to understand the threats and vulnerabilities which exist in your business.

• Build BC strategies based on the BIA and RA. Consider how you will minimise the risk of an incident occurring and, if it does occur, how you will recover these systems to another location.

• Build a communication plan. Consider how you will speak to your core team and to the business in general. Will you use social media sites? Will you have an ‘employee only’ section on your website?

• Create a computer incident response team (CIRT)within your IT department (these are the people who know each core system and service).

• Develop a business-wide crisis/incident management team, which the CIRT will feed into.

• Develop a BC plan for critical departments/functions. Ask each function “what will you be doing while IT are fixing the problem?” and ask departmental heads to document their answers in a BC plan.

• Develop a map of your firm’s systems and processes –where they physically sit and what information is held on them. Then rank the criticality of the system, based upon the BIA conducted earlier.

• Develop documented recovery plans which outline how you will recover key components of the service you provide (infrastructure, telephony, email, etc).

• Tell the business about the plans. From the top of the business down, you should have a communication strategy to ensure the correct people know their roles and responsibilities in a crisis.

• Exercise your recovery plans – but don’t simply think this is an IT issue. You should ensure the business is aware of the exercise planned (for the year) and include communication exercises, table-top exercises and technical recovery exercises. Combined, they should demonstrate that recovery can and will work.

---

 

Improving responsiveness

During a major outage, there will always be problems to overcome, but preparing more fully for them will improve your firm’s response to them.

The major challenge faced by IT is to find ways to make the business take responsibility for its services and to understand that IT is a service provider. The business must determine and understand what the recovery priorities are and then communicate them internally so that the IT team can focus on high-priority tasks in a crisis.

Breaking down silos and ‘them and us’ cultures is something all firms should be working towards and, unfortunately, in the legal sector this culture is prevalent. But, by taking their partners through the business continuity management lifecycle, IT teams and firm leaders can improve their business recovery time.

Gary Hibberd advises firms on business continuity management and is a director at information security services firm The Agenci (www.theagenci.com)

Endnote

1. For further information, see The Business Continuity Management Toolkit, Gary Hibberd, Ark Group, 2009