Web Hosting Security Policy & Guidelines

I have seen so many websites hosted and developed insecurely that I have often thought I should write a guide of sorts for those wanting to commission a new website. Now I have have actually been asked to develop a web hosting security policy and a set of guidelines to give to project managers for dissemination to developers and hosting providers. So, I thought I would share some of my advice here.

Before I do, though, I have to answer why we need this policy in the first place? There are many types of attack on websites, but these can be broadly categorised as follows: Denial of Service (DoS), Defacement and Data Breaches/Information Stealing. Data breaches and defacements hurt businesses' reputations and customer confidence as well as having direct financial impacts.

But surely any hosting provider or solution developer will have these standards in place, yes? Well, in my experience the answer is no. It is true that they are mostly common sense and most providers will conform to many of my recommendations, but I have yet to find one that, by default, conforms to them all.

Site Categorisation

There are several different categories of hosting and several different ways to categorise sites, with different requirements. However, in my opinion, sites should be categorised based on the information that they contain and the level of interaction allowed. Sites should then be logically and physically separated into their categories.

Sites can be categorised as brochure sites if they have static content or do not collect information. These sites can then further be categorised into public or private depending on whether the data that they contain is public or not. Sites within these categories may be co-hosted with other sites in the same category, but the two categories should be segregated.

Sites can be classed as data collection apps if they collect sensitive or personally identifiable information (PII) from the user. Sites within this category should be hosted on their own servers with no co-hosting and be segregated from all other sites. The data must be stored on separate segregated database servers that are secured and firewalled off.

Finally, any site with even more sensitive data on it or company secrets should be hosted internally if you have the expertise in house.

Hosted Environment

The following list is an example of the requirements for secure web hosting. It is not necessarily complete, but if you do not have the following then you may have issues in the future. All websites and web applications must:

• be hosted on a dedicated environment - the hosting machine may be virtual or physical, but must not be shared with any 3rd parties. Multiple websites and applications from the same company may be hosted on the same machines according to the categories above
• have DDoS protection in place
• have AV running and configured properly on the server along with appropriate responses and reporting
• be hosted behind a Web Application Firewall (WAF) to protect against common attacks, plus allow the ability to configure it for specific services
• be hosted on security hardened Operating Systems (OS) and services to an agreed build standard
• be subject to regular and timely patching of the OS and services
• be subject to regular security testing and patching of any Content Management System (CMS) in a timely manner if used
• be subject to active monitoring and logging by the provider for security breaches and reporting to/from the organisation
• have formal incident management processes for both identifying and responding to incidents
• not be co-hosted with additional public services beyond HTTP/HTTPS (e.g. no public FTP)
• not allow DNS Zone Transfers
• use proper public verified SSL certificates - with a preference for Extended Validation (EV) certificates
• ensure that management services and ports are on different IP addresses and domain names preferably, but must not be available through the normal login or visible on the website
• ensure that administrative interfaces and services are restricted to certain IP addresses at least, but make use of client-side certificates or two factor authentication (2FA) if possible
• ensure staging servers are available for test and development, which must not be shared with live sites and should be securely wiped at the end of testing as soon as the site is deployed live
• ensure staging and test environments are not available on the public Internet or, if there is no alternative, they must be devoid of branding and sensitive information in all ways and restricted as above
• be built on a tiered architecture, or at least the database (DB) server must not sit on the same server as the web front end, must not be accessible from the Internet and must be securely segregated from the front end
• use encrypted storage for all sensitive information, (e.g. passwords and sensitive information)

Hosting Services

It is up to the hosting provider and third party developers, but should be backed up by specific contractual clauses, to ensure that:

• the site is backed up regularly off site in a secure location using encrypted media where the keys are stored separately from the media and able to be restored in a reasonable time frame with a suitable rotation and retention policy
• hardware and media that has reached the end of its life is securely destroyed
• all sites are made available for pentesting prior to going live and at regular intervals
• all vulnerabilities considered of medium risk and above should be remediated prior to go-live
• all sites are available for on-going regular automated Vulnerability Assessments
• domain names, code and SSL certificates are registered to the company and not a third party
• there are agreed processes for identifying approved personnel to authorise changes
• change management processes that track all changes are in place along with rollback and test plans
• capacity and bandwidth are actively managed and monitored
• all management actions are accountable (unique accounts allocated to individuals)
• all management should be through secure ingress from trusted locations
• egress filtering should be in place to block all non-legitimate traffic

System Recovery with Comodo's Time Machine

Comodo's Time Machine is a software application that runs on your Windows machine and periodically (either manually or automatically) takes snapshots of your system. You are then able to roll back to any of these snapshots in the future. Indeed you can jump backwards and forwards in the tree and new branches appear as you make changes to the system.

The idea behind it is that if you suffer any problems with corrupted software, malware, etc., then you can roll back to a known good state and start again. You can lock snapshots so that they don't get deleted and then clear out the ones that you don't want to keep any more. This is quite important, especially if you take automatic snapshots. You have to remember that every change made to the computer (i.e. every time you run it or change a file) the changes are stored. When a new snapshot is created, if you change a file you will have a new version on your system as well as the old one. Due to this, it requires a fair amount of space on your system. However, the upsides are fairly obvious.

I have been using it quite a lot recently on test boxes while performing testing of security software against various malware and other attacks. It enabled me to perform a test, roll back to the pre-test state and perform it again or try another attack from a fresh system. It greatly reduced the testing time for certain attacks as I wasn't having to deal with an imaging server, etc. For the normal user, however, this does mean that if you get infected with malware or something else goes wrong with your system, you can very quickly and easily roll back to a previous state and carry on working.

There are a few issues to keep in mind though. Firstly, as I've already mentioned, the space required can be quite large if you keep taking snapshots and don't clear previous ones off the system. Secondly, if you roll back your system, you won't have access to any new files or software that you have put on the system - you will need to roll forward again to get at these. Finally, I did have one or two occasions where the restore failed. When I say the restore failed, I mean one snapshot failed so that I couldn't boot into it. At the boot stage I had to select another snapshot to boot from. I could always find a snapshot the did work, but it is slightly worrying that there were occasions when the one I wanted wouldn't boot. This could be due to the fact that I was installing various service packs, updates and malware onto the system and switching between them many times, but it is still worth noting that you will require a full system backup and you must backup all your data regularly.

Of course there are other products out there that do the same thing and some reviewers say that they are better (e.g. Acronis). However, I found Comodo's Time Machine very easy to use and it is free. I'm not necessarily endorsing Comodo's product; I'm saying that this type of software is worth a look for keeping your systems running.

The 5 Restoration Phases of a Secure and Dependable System

We all want our systems to be secure and dependable, indeed the two topics are interlinked. Dependability requires high availability management, which has several aspects to it. We can try to achieve Fault Avoidance, with fault prevention and fault removal, but this isn't actually possible in all cases. For example, hard disk drives will have physical wear out due to moving parts, power supplies do not run indefinitely, etc. Therefore, we move towards Fault Acceptance. Fault acceptance relies on fault forecasting, to try to determine the most likely causes of faults, and fault tolerance to enable the system to continue functioning in the event of a fault. With fault tolerance we build redundancy into the system so that faults do not result in system failures. However, there are times when even our most fault tolerant systems will fail. What do we do then? Well, obviously we need to recover as quickly as possible.

The 5 restoration phases of a system are as follows:

1. Diagnostic Phase - find the fault, diagnose the problem and determine the appropriate course of action to recover
2. Procurement Phase - identify, locate, transport and physically assemble replacement hardware, software and backup media
3. Base Provisioning Phase - configure the system hardware and install the base Operating System (OS)
4. Restoration Phase - restore the entire system from the backup media, including system files and user data
5. Verification Phase - verify the correct functionality of the entire system as well as the integrity of the user data

It can sometimes be quite hard to diagnose the actual root cause of a fault, as certain faults will sometimes show up in confusing ways. A few months ago I was investigating a problem with a machine that I was told was an OS problem - "Windows keeps blue screening with errors; bl***y Microsoft!" However, on further investigation, it had absolutely nothing to do with the OS and therefore Microsoft. Actually, there was a memory fault. One bank of RAM was faulty and was causing so many errors that the OS couldn't recover. Simply changing the pair of RAM modules sorted the problem out and the machine has been running reliably since. The problem with this type of fault is that it manifests in such a way as to look like a different fault.

The procurement phase can be tricky, as it takes a long time to get components delivered. This is where fault forecasting come in. If we know the most likely faults, then we can keep stock of those hardware components and make sure that any system recovery media is at hand. Obviously, we need backup media to be stored off site as well, but we will need copies on site for quick restoration when the building hasn't suffered damage. Of course, many hardware vendors will offer Service Level Agreements (SLA) as to how quickly they can replace or repair your hardware, but some things you may want to deal with in-house as it will still be quicker than the standard 4 hour fix (or longer).

The final three phases are all centred around restoration of data from backup media. This brings about the point that you must backup your system, not just your data. How long will it take you to install the OS from scratch, install all the additional services and make all the configuration changes? This will take too long. If you have backed up your system state, then this can be restored onto a base OS very much more quickly and without making mistakes or omissions. Another aspect to think about is how have you backed your system up? You need to choose a backup scenario that suits the amount of data and the speed at which you need to recover. Remember that a server with 200GB of data backed up onto a DLT tape drive that supports an average transfer rate of 5MBps, will take over 11 hours to restore, assuming that you restore from the full backup only. Any incremental backups taken since the last full backup will also have to be restored. OK, we can go faster than this, even with tape backup, but the solution needs to fit your system and a live backup solution may be required. This is where virtualisation of servers can help dramatically. Virtualisation is more often sold as 'green IT', being more efficient and cheaper. However, a major benefit is the ability to snapshot running machines and redeploy them in seconds. If one hardware box fails, you can migrate the virtual machine onto another box until the first is fixed. This can take just a few seconds or minutes, depending on the architecture of your solution.

Availability is inversely proportional to the total downtime in the period covered and is usually expressed as a percentage, e.g. 99.9% availability, which equates to around 8 hours 45 minutes downtime per annum. The downtime is the sum of all outages in that period. Therefore, we need to decrease the frequency and length of those outages. The frequency is reduced by building fault tolerant systems and the length by having good restoration policies and practices. It is vital that IT staff know the restoration policy and have practiced it. You need to set out a clear timeline of what has to happen during restoration. Certain systems and services will need to be restored first. Your database server may be the most important server to you, but without the network, DNS, DHCP and directory services no other machines will be able to connect to your server anyway and it may rely on some of those services during start up. Also, don't leave it until you are having to restore a failed system to see if the policy works. You must practice and test the policy to make sure that it does.

Clustering Technologies

I have been asked to give a very brief overview of the clustering technologies that we can utilise for high availability. We are, therefore, going to ignore high power computational clustering, as this is about more power rather than redundancy. The two main techniques that we use are a shared-resource cluster (usually some kind of disk array) and a network load balancing cluster, which does exactly what it says on the tin! We'll deal with each of these in turn here, but they can be used together to provide a complete solution.

The goals of Server Clusters are to share computing load over several systems whilst maintaining transparency for system administrators. There is also transparency for users, in that a user has no idea which node in the cluster they have connected to, or indeed that they are connected to a cluster at all as it will appear as one machine. If a component fails users will suffer degraded performance only, but do not lose connectivity to the service. If more power is required at a later date, it is easy to add more components to the cluster, thereby sharing the load across more systems.

Three principle features of clusters:

• Availability - continued service in the event of the failure of a component
• Scalability - new components can be added to compensate for increased load
• Simplified management - administer the group of systems & applications as one unit

Shared-Resource Clustering

The Shared Resource Cluster is used for Line-of-business applications, e.g. database, messaging, file/print servers. Behind this technology is a physical shared resource, such as a direct attached storage device. Services then fail over from one node to another in the cluster, ensuring high availability of core, critical services. A rolling upgrade process enables the addition or upgrading of resources, which, when coupled with the high availability, ensures that line-of-business applications are online when needed. This technology removes the physical server as the single point of failure.

This type of cluster doesn't give us huge scalability like the Network Load Balancing cluster, as each node 'owns' the required storage at any one time and the others don't. Imagine that you are running a DataBase Management System (DBMS) on the cluster, then only one node can run the DBMS at any one time and all access to the DB is via that node. However, if it fails, then another node will 'seamlessly' take over. Now, in order for this to happen quickly and smoothly all nodes in the cluster will run the DBMS 'minimized', i.e. they will load it into memory, but not service any requests. Then, in the event of a failure, they can start responding very quickly after detecting the failure. The important thing to remember is that they are not all running the DBMS and servicing requests at the same time. Also, there is no replication to worry about, as the actual data is saved on the shared storage array that all nodes have access to. It is like ripping the hard disk out of one machine and putting it in the other. Of course, after a failover, we can failback the service when the node has been fixed.

The logical view of a shared resource cluster is shown here for a 2-node cluster running 4 services (actually, these are a bit made up as you wouldn't run a web server on this, you'd run that on an NLB cluster). Physically, there are two nodes (server boxes) in the cluster and we are running 2 services or virtual servers on each (the blue balls). This is not to be confused with virtualization with VMWare or Hyper-V, we aren't virtualizing whole servers, just services. However, these services are exposed as separate machines to the network.

The diagram shows the client view of the cluster, which has two physical nodes (CNode1.mycompany.com and CNode2.mycompany.com), which aren't directly accessed by clients, only for IT support, configuration and cluster services. However, each 'Virtual Server' is advertised in the DNS with a name and IP address, e.g. C_VS3.mycompany.com with an IP address of 10.1.1.6. Therefore, clients wanting to connect to Exchange will connect to C_VS3 on 10.1.1.6, which is usually running on CNode2 on 10.1.1.3. How does this happen? Well, a machine can have multiple IP addresses assigned to each network card. So CNode2 has a dedicated IP address of 10.1.1.3 as well as the IP addresses of the 'Virtual Servers' 10.1.1.6 and 10.1.1.7. If this node fails, then CNode1 will take over and these IP addresses will be assigned to it. In this way, the name and IP address of a service doesn't change even in the event of the failure of the node you were connected to.

Network Load Balancing Cluster

A Network Load Balancing Cluster (NLB) is used literally for load balancing network traffic and processing load across multiple machines, e.g. Web server farm, remote desktops, VPN & streaming media. This cluster type gives high availability and good scalability. Additional nodes can easily be added to the cluster to expand it and if any single node fails the cluster automatically detects the failure and redistributes the extra load, presenting a seamless service to users. This is achieved by load balancing all incoming IP traffic. Some of the benefits include the ability to scale Web applications by quickly and incrementally adding additional servers in a rolling-upgrade whilst ensuring Web sites are always online. The important distinction between this and the shared resource cluster is that all the nodes are running the same service with the same data at the same time, e.g. web servers with the same website on their local storage or a common network location. There are several different solutions to NLB, each with advantages and disadvantages. The most common are: RRDNS, central dispatcher and Negotiated Statistical Mapping.

The simplest form of NLB is to use Round-Robin Domain Name Service (RRDNS), which, as the name suggests, simply issues IP addresses from a list in a round-robin fashion. For example, if you have four web servers (web1.mycompany.com ... web4.mycompany.com) all serving the same website, then you can enter the Alias www.mycompany.com into your DNS pointing at each of the four nodes. When a client queries your DNS for the IP address of www.mycompany.com then they will be given the first node in the list, web1.mycompany.com. The next query to the DNS will result in the IP address of web2.mycompany.com being given out, etc. This has the advantages of being cheap and easy. Also, you don't need any special equipment or nodes that are aware of clustering technologies. However, the disadvantages are that the load is not distributed fairly and there is no detection of failed nodes. Imagine if every fourth query required very heavy processing, and the rest were simple GET requests. One node would get hammered and the others would sit around spinning their wheels. Also, if one node fails, the DNS server will still send every fourth query to that node as it doesn't know.

The second, and in a lot of ways most sophisticated, is the central dispatcher. This relies on a central device to receive all incoming requests and distribute them out among a set of nodes. This central device does not do the processing itself, it is merely a control to distribute work fairly and to healthy nodes only. The IP address of the central dispatcher is all that is advertised, but responses will come from the nodes directly. The advantages of this are that the dispatcher knows the capabilities of each node, so can distribute requests proportionately, and it knows the current workload of the nodes through querying. So, nodes won't get swamped, as the dispatcher will re-distribute the workload. It also means that you can run different services on different subsets of nodes. The disadvantages of this are cost and a single-point of failure - if your central dispatcher goes down, then the whole cluster is offline. Of course, you can have warm and hot-standby central dispatchers, but this can get very expensive.

The final method to look at here is negotiated statistical mapping. In this scenario, there is no central dispatcher or single-point of failure. The nodes all negotiate what load they will take and answer requests based on a statistical view of requests. Each node in the cluster will have two IP addresses, one dedicated address for it alone and the common cluster IP address. It is this common address that is advertised to clients requesting connection. In this way, all nodes in the cluster will receive all the packets for the cluster. The node that this request is mapped to will respond and all the others will discard the packet. Mapping can be done by individual IP addresses, subnets, etc. If one node fails, then the cluster will renegotiate the mappings and converge to a new model excluding the failed node - this happens within a couple of seconds. Similarly, when adding a new node, re-convergence is triggered and traffic will be distributed to the new node as well. The advantages of this are that it can be cheap to implement, as you have standard server hardware and there is only specific software to configure and control the cluster, and it has no single point of failure. However, there are disadvantages, namely that one node could get hammered again, as the nodes don't know what requests will actually come in as this is all based on statistical models. Also, it is more difficult to have subsets of nodes for different services.

Conclusions

There are two types of clusters to consider for high availability, namely the Shared Resource Cluster and the Network Load Balancing Cluster, which can be used separately or together to provide a complete clustered solution. The selection of which to deploy is based on the requirements of your service. An example is an e-commerce website, which would usually employ both technologies together in a complete solution - the web server farm at the front end will run a NLB cluster, whilst the backend database that they all access for live data will run on a Shared Resource cluster. Whose technology and implementation you choose will depend on budget, platform and feature requirements.

How Reliable is RAID?

We all know that when we want a highly available and reliable server we install a RAID solution, but how reliable actually is that? Well, obviously, you can work it out quite simply as we will see below, but before you do, you have to know what sort of RAID are you talking about, as some can be less reliable than a single disk. The most common types are RAID 0, 1 and 5. We will look at the reliability of each using real disks for the calculations, but before we do, let's recap on what the most common RAID types are.

Common Types of RAID

RAID 0 is the Stripe set, which consists of 2 or more disks with data written in equal sized blocks to each of the disks. This is a fast way of reading and writing data to disk, but it gives you no redundancy at all. In fact, RAID 0 is actually less reliable than a single disk, as all the disks are in series from a reliability point of view. If you lose one disk in the array, you've lost the whole thing. RAID 0 is used purely to speed up disk access. If you have two disks your access will approach twice as fast; three disks are nearly three times as fast, etc. Actually, you don't quite achieve these performance gains due to overheads, etc.

RAID 5, on the other hand is the stripe set with parity, which can cope with one disk failing. You need a minimum of 3 disks to deploy RAID 5, and you will lose the capacity of one, that is to say that a 3-disk RAID 5 array using 0.5TB disks will have a capacity of 1TB, whereas the RAID 0 solution would have 1.5TB. Similarly, if we had a 5-disk RAID 5 array with the same disks as before then we would have 2TB at our disposal. The reason for this is that data is written in equal sized blocks to each of the disks as with RAID 0, but we write a parity block to one of the disks in each stripe. The diagram below explains. The parity is simply the bit wise Exclusive-OR of all the other blocks. It then becomes obvious that we can regenerate any one disk if it should fail by XORing all the remaining disks together. However, if we lose more than one disk, we've lost the lot again. RAID 5 arrays are also faster than single disk solutions, as we can read from and write to several disks at once, but due to the parity data, it will be slower than RAID 0.

Finally, we have RAID 1, which is the mirror set and consists of two identical disks. All data is written to both disks, so from a reliability point of view they are in parallel and the array is accessible with only one disk working. This means that mirroring two 500GB disks will only give 500GB of storage. You can read from both at once, but as you have to write to both at the same time this is slightly slower than a single disk due to the overheads. However, we can again cope with one disk failure. The power of this technique comes into its own when we start combining RAID arrays, e.g. we can mirror two RAID 5 arrays to cope with total failure of one array and a single disk failure in the other.

Reliability of Disks

In order to calculate the reliability and availability of our RAID array we need to know how reliable our disks are. Manufacturers can quote this in one of two ways: Failure Rate or Mean Time To Failure (MTTF). These figures can sometimes seem misleading, so we'll look at how they're related and what they actually mean in terms of reliability of your disk arrays. Actually, the MTTF and Failure Rate are related via a simple calculation, as the annualised failure rate is usually quoted, which is simply the percentage of disks that will fail in a particular year given the MTTF of the drives.

If we take an actual drive as an example, the Seagate Barracuda 500GB 32MB Cache ST3500320AS, it has a stated MTTF of 750,000 hours or around 85.6 years. This doesn't mean that the drive will actually last that long, it means that on average in 750,000 disk drive hours you will get one failure. So, if we have 1000 disks in our data centre then we will on average suffer a failure every 750 hours or one disk will fail every month, on average (the 'on average' is important, as you could suffer three disk failures this month and none for the next two, for example). These figures are arrived at by the manufacturers in exactly that way. They will run a set of disks (usually under heavy load and probably at high temperature) for a set time and look at how many failed. For example, if they ran 2,000 disks for a month and had 2 failures, then they would get (2000 x 744)/2 = 744,000 hours (assuming a 31-day month as 31 x 24=744).

So how does this relate to the annualised failure rate? Quite simply, if we have one disk failure every 750,000 hours what percentage fail in one year? The first step is to work out how many years 750,000 hours is, so we have 750,000/(24 x 365.25), which is approximately 85.6 years. To get the annualised failure rate we take the inverse, i.e. 1/85.6, which gives a failure rate of nearly 1.2%. Of course, you have to remember that these figures do not take into account any batch failures, i.e. a fault in manufacturing causing a whole batch of disks to be faulty or less reliable.

Calculating the Reliability of RAID

We can now use the annualised failure rate of the disks from the previous section to calculate the reliability of a RAID array. We will look at several scenarios to see how reliable, or not, the common types of RAID are. We will start with a 3-disk RAID 0 solution. Each disk has an annualised failure rate of 1.2% or a probability of failing of 0.012, which gives us a probability of 0.988 that the drive will still be running at the end of the year. Now a RAID 0 array has all the disks in series, i.e. all the disks must be working for the array to work. If any one disk fails then we have lost the whole array. Therefore, we have the probability of reliability:

So, we can see from this that a RAID 0 array might be faster than a single disk, but it is less reliable. OK, we knew that RAID 0 gave us no redundancy, so what if we look at RAID 1 or RAID 5? Which one of these is more reliable? Let's look at a 2-disk RAID 1 array (it can only really be 2-disk remember). In this case the drives are in parallel, so it will only fail if both drives fail; it will still work if either one or the other or both drives are still working. Therefore, we have the probability of reliability:

RAID 1 is clearly much more reliable than RAID 0 and a single disk solution. Now we'll look at a 3-disk RAID 5 solution. The complication here is that it isn't simply in parallel or series, the array will keep working in the event of any one disk failure or no failures. The easiest way to look at this is to add up all the probabilities of the situations where it is still running. A table makes it easy to see.

You can see from the table that we are interested in the first three rows, i.e. when the array is still working. We can now simply add these up to get the overall reliability for the array as follows:

So, we can see that this is much more reliable than a single disk solution, but less reliable than RAID 1. Indeed, the more disks you have in a RAID 5 array, the less reliable it becomes. In fact, a 5-disk RAID 5 array will have a reliability of around 0.99859. At what point does RAID 5 become less reliable than a single disk? If you work it out, it turns out that in this case a 14-disk RAID 5 array has almost the identical reliability to a single disk. Of course, you have much more storage and faster access, but no more reliability.

What if we were to combine these arrays, e.g. mirror a RAID 0 stripe set? Well, it's simply a matter of combining the reliabilities of each RAID 0 array in parallel. The reliability of a 3-disk RAID 0 array was approximately 0.964. If we now put this figure into the RAID 1 calculation above instead of the 0.988 disk reliability, this will give us a reliability of approximately 0.9987, more reliable than a 5-disk RAID 5 array. Of course, if we mirror a 3-disk RAID 5 array, then we would get a reliability of approximately 0.99999982 - very reliable.

The important thing here is to be able to interpret what manufacturers are saying and predict failures. To be able to do this you have to know how to calculate the reliability of your system. If you have 100 servers with 5-disk RAID 5 arrays in them, how many system failures will you get in a 5-year operating life cycle and how many disks will you need to replace? Disk replacement is simple, you have 500 disks in total with a MTTF of 750,000 hours, or one failure every 1500 hours, which is one failure every 62.5 days. If the life span of your data centre is 5 years, then you will get around 29 disk failures, but how many system failures? Well, the reliability of each system is 0.99859, which implies that the annualised failure rate for the system is 1 - 0.99859 = 0.00141. If this is the percentage of failures per year, then the MTTF will be 1/0.00141 which is approximately 709.22 years. We have 100 such systems, so we will get a failure in around 7 years 33 days. This means that we might well get 1 system failure during the 5-year life span, but we may not get any.