The 5 Restoration Phases of a Secure and Dependable System

We all want our systems to be secure and dependable, indeed the two topics are interlinked. Dependability requires high availability management, which has several aspects to it. We can try to achieve Fault Avoidance, with fault prevention and fault removal, but this isn't actually possible in all cases. For example, hard disk drives will have physical wear out due to moving parts, power supplies do not run indefinitely, etc. Therefore, we move towards Fault Acceptance. Fault acceptance relies on fault forecasting, to try to determine the most likely causes of faults, and fault tolerance to enable the system to continue functioning in the event of a fault. With fault tolerance we build redundancy into the system so that faults do not result in system failures. However, there are times when even our most fault tolerant systems will fail. What do we do then? Well, obviously we need to recover as quickly as possible.

The 5 restoration phases of a system are as follows:

1. Diagnostic Phase - find the fault, diagnose the problem and determine the appropriate course of action to recover
2. Procurement Phase - identify, locate, transport and physically assemble replacement hardware, software and backup media
3. Base Provisioning Phase - configure the system hardware and install the base Operating System (OS)
4. Restoration Phase - restore the entire system from the backup media, including system files and user data
5. Verification Phase - verify the correct functionality of the entire system as well as the integrity of the user data

It can sometimes be quite hard to diagnose the actual root cause of a fault, as certain faults will sometimes show up in confusing ways. A few months ago I was investigating a problem with a machine that I was told was an OS problem - "Windows keeps blue screening with errors; bl***y Microsoft!" However, on further investigation, it had absolutely nothing to do with the OS and therefore Microsoft. Actually, there was a memory fault. One bank of RAM was faulty and was causing so many errors that the OS couldn't recover. Simply changing the pair of RAM modules sorted the problem out and the machine has been running reliably since. The problem with this type of fault is that it manifests in such a way as to look like a different fault.

The procurement phase can be tricky, as it takes a long time to get components delivered. This is where fault forecasting come in. If we know the most likely faults, then we can keep stock of those hardware components and make sure that any system recovery media is at hand. Obviously, we need backup media to be stored off site as well, but we will need copies on site for quick restoration when the building hasn't suffered damage. Of course, many hardware vendors will offer Service Level Agreements (SLA) as to how quickly they can replace or repair your hardware, but some things you may want to deal with in-house as it will still be quicker than the standard 4 hour fix (or longer).

The final three phases are all centred around restoration of data from backup media. This brings about the point that you must backup your system, not just your data. How long will it take you to install the OS from scratch, install all the additional services and make all the configuration changes? This will take too long. If you have backed up your system state, then this can be restored onto a base OS very much more quickly and without making mistakes or omissions. Another aspect to think about is how have you backed your system up? You need to choose a backup scenario that suits the amount of data and the speed at which you need to recover. Remember that a server with 200GB of data backed up onto a DLT tape drive that supports an average transfer rate of 5MBps, will take over 11 hours to restore, assuming that you restore from the full backup only. Any incremental backups taken since the last full backup will also have to be restored. OK, we can go faster than this, even with tape backup, but the solution needs to fit your system and a live backup solution may be required. This is where virtualisation of servers can help dramatically. Virtualisation is more often sold as 'green IT', being more efficient and cheaper. However, a major benefit is the ability to snapshot running machines and redeploy them in seconds. If one hardware box fails, you can migrate the virtual machine onto another box until the first is fixed. This can take just a few seconds or minutes, depending on the architecture of your solution.

Availability is inversely proportional to the total downtime in the period covered and is usually expressed as a percentage, e.g. 99.9% availability, which equates to around 8 hours 45 minutes downtime per annum. The downtime is the sum of all outages in that period. Therefore, we need to decrease the frequency and length of those outages. The frequency is reduced by building fault tolerant systems and the length by having good restoration policies and practices. It is vital that IT staff know the restoration policy and have practiced it. You need to set out a clear timeline of what has to happen during restoration. Certain systems and services will need to be restored first. Your database server may be the most important server to you, but without the network, DNS, DHCP and directory services no other machines will be able to connect to your server anyway and it may rely on some of those services during start up. Also, don't leave it until you are having to restore a failed system to see if the policy works. You must practice and test the policy to make sure that it does.

InfoSecurity Europe 2009

Well, a couple of weeks ago I went to InfoSecurity Europe 2009 at Earl's Court, as I do every year. If you've never been, but are at all interested in network and information security or are looking for vendors, then I highly recommend visiting. To my surprise, and for the first time I can remember, Microsoft wasn't there. Apart from that, however, all the usual players were there and, to be honest, it was all much the same as before. There was no new emerging technology or hot topic, just new developments of old technologies. A couple of years ago we had the hot topic of 'social engineering' and 'securing the user'; ok, we all knew that the users were our weak link and phrases like "we spend all our time securing the first 2000 miles and forget about the last 2 feet", and "our network would be totally secure, reliable and fully functional if we didn't let users login", have always been commonplace, but there were new mass threats, new education programmes and new tools at the disposal of net admins to deal with them.

However, this year was different; nothing really sprang out. OK, cloud computing and Software-as-a-Service (SaaS) has expanded and matured, but other than that we see the same products and services as before. What amazes me is how some vendors and speakers can get it so wrong and don't appear to understand the actual level of security offered or operating environment in which their products will be used. I'm not going to list actual vendors here, but how can an encryption solution for mobile users that doesn't encrypt the data at rest be viable? Encrypting network traffic is commendable, but not the only safeguard required. What if they now lose their laptop, mobile or pen drive? A secure USB pen drive vendor admitted, when questioned, that files were decrypted into the C:\Temp folder while in use, then deleted after encrypting for storage on the drive again. They couldn't tell me if this was a secure delete or a simple removal of the pointer in the file allocation table as normal. Regardless of the deletion process, however, how many applications can read and write to that folder? What's to stop me from writing a very small bit of code to monitor that folder every few seconds and take copies? This might be secure enough if it is on a corporate machine, but why not just store the file on it then and not bother with the drive?

Similarly, email security is always a problematic area, with almost no solution fitting the technology properly. The big advantage of email, and the reason we all use it, is that it is independent of firmware - i.e. it doesn't matter what hardware, OS or email app we use, it still works. Unfortunately, security was never built in to email, so every email is like sending an electronic postcard. However, I would argue that a solution that only allows you to send encrypted email within your organisation is of limited value. What about all your customers and partners? There are also still solutions that store your files on their servers and send a link to the recipient. Why trust your files to them? I asked several vendors how they deal with password transfer, only to be told that they don't. "It's up to the user how they tell the recipient what the password is," was a common reply. We know users are unreliable, why leave it to them? I had to transfer some confidential files to someone via email recently (the only transport method they would accept), only to be told that they didn't want the hassle of decrypting it, so could I send them in plaintext. Having got over that hurdle, they wouldn't give me a mobile phone number so that I could transfer the passphrase via a call or SMS (as people seem to need them written down), asking me to email it to them. I know there are technologies out there to solve these problems, but they aren't without any problems.

This is turning into a bit of a rant and I'm getting off topic, so back to InfoSec. I was pleased to see that there were some voices of caution out there about the wholesale adoption of virtualisation without considering the security implications. One that springs to mind is Steve Moyle who has produced 10 points to consider implementing on his blog here. Virtualisation is a good technology for a number of reasons, but it does bring in new security threats and it must be implemented with these in mind and secured accordingly. I do also think that cloud computing and SaaS could be very important to SMEs (Small to Medium-sized Enterprises) who don't have the in-house expertise or large budget. They can still have large enterprise-level configurability and security, without the overheads. Finally, people seem to be taking information governance and user education seriously rather than just paying lip-service to them. In all, the show was encouraging, but many vendors are not quite there yet, which only goes to highlight that the majority of organisations are not taking the new threat-landscape seriously enough and countermeasures must be lacking.