Liberating problems: Creating an automated SaaS-based solution

Key takeaway points:

1. Start with some markers and a whiteboard. Map out at a conceptual level what needs to occur.

2. Detail your desired end result.

3. Identify all the known systems that need to be involved in the automated workflow.

4. Identify key dependencies such as security, permissions, checking the equipment is on and functioning, and how long various tasks will or must take.

5. Start building your first solution. There will be at least two, as the first one will illuminate anything missed in the above steps.

 

Liberation is not a word usually associated with an impending and unavoidable problem. That is exactly what occurred, however, when we realised it would be impossible for us to use traditional methods to pull off the largest and most complex project we had ever faced.

We recognised that, unless we did things very differently, we would fail. We had no choice but to try something new. And with that, and a bit of effort, we accomplished more than we ever thought possible. Along the way, we saved a lot of time and money, generated system documentation and developed a skillset that would allow us to do many complex things safely and reliably.

Our efforts resulted in Morrison & Foerster winning the International Legal Technology Association’s award for most innovative law firm of the year.

The iceberg

Morrison & Foerster has 2,250 people in 17 offices worldwide. Our challenge was to upgrade each individual to Windows 7 and Office 2010 in an efficient, safe and repeatable way.

By itself, this does not sound overly imposing. But, as we spent time quantifying all the migration intricacies, dependencies and requirements, we came to realise that the Windows 7/Office 2010 upgrade was the tip of the iceberg. We would also have to successfully tackle many related many related projects in the same timeframe.

After a thorough review, we identified three key problems.

1. Upgrading many core systems. Many of our core systems would need to be upgraded so that they could interact with Windows 7/Office 2010.

2. Avoiding unnecessary capital expenditure. Our existing email system was based on Exchange 2003. To upgrade to Windows 7/Office 2010, we needed a new Exchange 2010 environment comprised of strong servers, high performance storage and backups.

We had two choices. We could leave our Exchange environment decentralised in the local offices and spend a tremendous amount adding to their existing infrastructure. Or, we could bring our regional data centres online and, within them, create the Exchange 2010 environments, consolidate people into their nearest regional data centre as part of their upgrade, offer much better performance, get disaster recovery options as a by-product and spend a lot less money.

3. Migrating many users concurrently. After all our research and investigation, we created a 37-page document which listed all the possible steps that might need to occur to upgrade someone. This list addressed the unique requirements of certain applications, permission issues, standardisation efforts, system dependencies and people’s data.

We could not safely upgrade one person if we had to follow a 37-page script.

Also, the radical difference between the Office 2010 interface and its 2003 predecessor required an extensive amount of training before, during and after the upgrade. The sheer size of people’s mailboxes further limited the amount of those who could be upgraded concurrently.

With those three problems clearly identified, we knew it would not be possible to quickly upgrade 2,250 people successfully using our traditional methods. Initial estimates to upgrade everyone were ‘years’. We had no choice but to consider what was once viewed as impossible: doing things very differently and automating the majority of the work.

Creating solutions

For problem one – upgrading many core systems – we identified all that needed to be touched, lined them up in order of priority and dependency and, one by one, completed their upgrade. This took a year.

Along the way, we decided on a key change in philosophy. Instead of upgrading a core system to the minimum version and configuration necessary, we decided to bring these applications as far forward as possible. This took more time, but it was the right strategic decision.

Solving problem two – avoiding unnecessary capital expenditure – would also take over a year. The solution was to act upon a strategic decision to consolidate our infrastructure and data into regional data centres. In addition to spending wisely, we modernised our infrastructure and greatly increased its performance.

This project took a year and a half to complete and included our new and optimised Exchange 2010 environments. As our data centres came online, we were able to start the Windows 7/Office 2010 upgrade in that region. As a result, we could retire old equipment and systems, simplify our environment, standardise our infrastructure and create a highly-resilient disaster recovery capability.

The solution to problem three – migrating many users concurrently – took the greatest amount of time, group talent and experimentation. We needed to automate our upgrade processes, but did not have an in-house option that could meet our technical, operational and communication needs.

We did, however, have ServiceNow, a software-as-a-service (SaaS) solution for our trouble ticketing system, and it did have the potential – with a notable amount of effort on our part – to do what we needed it to do. It had the ability to execute a large number of discrete tasks at precise moments, report on progress and generate new tasks depending on the state of progress.

As an SaaS solution, we subscribe to use it and are not required to provide either the infrastructure or its software. The system also runs well across the world. We did not have to worry about its maintenance, which allowed us to focus 100 per cent of our time and attention on creating the dependent technical, visual and communication processes.

As ServiceNow was our best and only choice, we started to learn what it could do, how to do it and what new challenges it would uncover. We created a preferred order of operations detailing how each individual would be upgraded.

Our initial whiteboard and spreadsheets quickly became a comprehensive automated process. The process allowed us to:

• identify a person and when we wanted him to be upgraded;

• determine where that person’s data resided and where it would go;

• remove him from the past and accurately place him into the right spots in our new systems;

• create a master calendar;

• assign tasks to people and roles; and

• send clear messages to the person being migrated.

Our SaaS-based solution could also execute our internally-created PowerShell scripts to enable different systems to talk to each other, set the right security permission levels and provide status updates. In effect, we turned a 37-page to-do list into an automated routine that would reliably perform every step of the process.

Lastly, it allowed us to quickly make a change whenever we encountered a problem or an opportunity to make our processes better. From that point forward, everyone would benefit from the improvement. Sometimes making a noticeable improvement only took a few minutes.

Battle against time

Time was our single biggest challenge. We did not have the luxury of enough time to research, investigate, experiment, plan, build, retire or train. We embraced the only solution available to us, which was to proceed with all possible haste. Our people were pushed far beyond their comfort zones. In many cases, we were truly learning as we went.

Due to the size of our users’ email files, we migrated people overnight and welcomed them to the future the next day. We could handle approximately 15 people per night per region – also a reasonable number for managing training and the resulting floor support over the following days.

Our users’ emails presented a tremendous logistical challenge. We ran one series of commands to export mail from the old archive into the new format, place the data in a designated spot and report successful completion back to the SaaS solution.

Next, another series of commands would import the data into the new Exchange 2010 environment. Once complete, it too would report its success, triggering downstream processes, including user communication and scheduling training. The size of certain accounts required additional coordination, as they were too large to seamlessly migrate overnight.

Along the way, we adjusted to the departures of a few key people. We also chose to upgrade a pivotal application twice. Internet Explorer 9 was released after we had successfully upgraded to Internet Explorer 8. We adhered to our strategy of advancing as far forward as possible and decided to make the upgrade to Internet Explorer 9 part of our rollout.

Steering the ship

We learned our most profound lesson after we acknowledged we had a problem and made clear any option would be considered. We were no longer restrained by fears of failing, encountering the unknown or making a mistake. Removing those restraints allowed us to sprint ahead with purpose and to enjoy our work at the same time.

We also learned that we have to keep moving. We now have methods in place to continuously improve our IT service offerings and equipment and thus avoid the need for massive and sweeping upgrades. The result is superior service to our firm.

We learned that over-communication within the IT department and with the people being upgraded does not hurt, but it also cannot prevent people from becoming very uncomfortable with so many things changing at once.

When a person was upgraded, he would immediately experience a large number of changes. At the same time, our support staff immediately received a large number of new questions. We therefore learned the value of delivering a steady stream of consistent information to all.

Measuring success

We accomplished all of our goals earlier than we forecast in the budget, while saving $750,000 (£470,000) because we completed the entire project using internally-generated automation routines via an externally-supported solution and avoided additional costs along the way.

And, by leveraging regional data centres and not building out from our local offices, we avoided a capital expenditure of at least $1m.

Inspired by our success, we have created a new team focused exclusively on furthering our automation efforts via SaaS. The team’s broad mission is to:

• identify areas with complex but repeatable processes;

• automate duplicative work to help departments do more work with existing resources; and

• improve operational efficiencies to avoid filling headcount vacancies.

Another broad goal is to further incorporate our operations, processes and system auditing into our SaaS-based solution. Many of our operational tasks and processes are repeatable and can be automated, freeing up our internal talent to spend their time on more strategic projects. This automation also has the cumulative effect of allowing us to work with fewer systems, improve our efficiency and reduce costs. The time invested will ultimately benefit the whole firm.

 

---