What do you do when your company's systems go down, and you cannot get them back up?

That's poor planning and you shouldn't be in charge of leading a IT department you if don't know how to perform a risk assessment. If you are in charge of leading you prepare your department so that there is never any down time. I have never been part of a solution to where we couldn't restore service within the hour. It's 2019 with virtualization and cloud services that are offered no and days you shouldn't ever been down for longer than an hour. Can you image your the IT Director of a bank and your systems go down and you cant restore? 

First we diagnose the problem. Is the problem hardware, electrical or is it a software or update problem.  Once we figure out the problem then we start coming up with a solution.  Every issue is different so you just go to the basic trouble shooting skill and try and keep it simple.  Was there a new piece of equipment installed recently or was there a new software package or update installed recently and just work backward.  If its a total hardware issue, we replace the part and move on.  

We support a multitude of customers , from hosted servers, to hosted phone systems, and hosted internet for several as well. So when our network goes down, so do theirs. We have structured for fail over on most situations but as the IT world knows you cant plan for everything. When the critical happens I rely heavy on my team, together we work diligently to isolate the root cause of the issue and fix it. As well as to make sure we have a contingency plan in the event the same issues arises again.

I have all of my clients backed up in the cloud and have had to use the cloud to do a recovery. I keep two, Azure & Carbonite. It was quick, reliable and worked like it was supposed to. I also train people I work with (a client's employees usually) in critical recovery methodologies & practices so when the system goes down or fails completely they will know what to do. Or at least start until I arrive at the locale.   

This is where your disaster recovery plan comes into action...every business should have a disaster recovery plan in place so that when things go down you are ready to act 

First and foremost, hopefully you have a plan in place BEFORE you need to use it.  When you talk about systems, what really are you talking about?  That word could encompass a lot of different things such as computers (servers and desktops), infrastructure (internet and networking equipment), phones, etc.  I will try to address several of the key components we deal with here and how we handle them.

Internet
If internet is absolutely mission critical, you would have to have a router with ISP fail-over.  If your main source of internet goes out, the secondary source kicks in.  For example, if you have a router that uses either fiber or coax as your main provider and that provider experiences an outage, the secondary ISP configuration would kick in.  For our business clients where internet is mission critical, we use a router that supports 4G fail-over.  If the main internet source goes down, the 4G service becomes active keeping their internet alive.  Speeds may be an issue, but at least they have internet.  All other routing/router functions and rules still apply.  

Phones
In today's modern business landscape, there really is no reason NOT to use a quality VOIP service, whether it is hosted or on premises.  When picking a VOIP provider, make sure that they offer call continuity and system redundancy.  These features automatically forward calls in real time to designated mobile phone numbers in the event your internet service fails (see above), which helps to ensure that no VOIP phone call is ever lost.  At Alpha Geeks we provide a hosted VOIP solution to our clients for $27.95 per month per line, hardware (phones) included, that offers not only call continuity and system redundancy, but over 150 other features as well.

Servers
Servers are probably the biggest mission critical item within an organizations "systems" that have the highest rate of failure, or at least the highest chance of failure as compared to the other "systems" above.  There are many different options when it comes to servers these days, and there is no "one-size fits all" solution.  At their core however, all servers have hard drives, which like us, have a 100% mortality rate.  It is not a matter of if they will die, but when.  

Regardless of if you use a physical, hosted, or virtual server, the ability to restore that server in the event of failure is key.  In business time is money.  The longer your server is down, the more money you lose.  For our business clients we utilize a backup solution that takes snapshots of the server at intervals our clients deem appropriate for their businesses.  These snapshots are stored locally on a device, and also mirrored offsite.  In the event that a server goes down, their entire server can be recreated from these snapshots and launched in a virtual environment in about 15 minutes.  All of the endpoints "check-in" to the virtual server as if it were the real thing and business continues.  When the original servers gets fixed or replaced, any data that has changed on the virtual server gets exported to the primary server.  

Misc
Other "systems" can include physical computers, software programs, and network infrastructure such as routers, switches, or wifi access points to name a few. 

For software issues, if a piece of software that a business we manage has a maintenance contract, we require our customers to keep that contract active.  We will work with the software vendors tech support team to fix the issues, but ultimately it is the software vendors responsibility to ensure that their software is working properly (that is what the maintenance contract is for).  

Physical computers, like individual desktops or laptops are backed up either with the same image based backup system we use for servers, or just data only.  When it comes to determining what backup system to use, the key question to ask is "How long can you go without your data?"  If it needs to be available always, then the imaging software is the solution.  If it can wait a day or two, then data only restoration to a new computer is the way to go.  

All other miscellaneous hardware issues are on a case by case basis.  We keep extra networking equipment at our shop and can have it replaced same day if needed.  Some of our bigger clients keep extra equipment stored at their facilities.  Amazon is also a great source of picking up things really quickly if you live in a market that offers same or next day service.  

The above mentioned practices are for any size business.  We have clients as small as one person, one computer up to multi-state companies with many employees.  Size does not matter, disaster prep does.  Remember, those who fail to plan plan to fail.  

First of all, I have to start by stating that every business should have a disaster recovery plan in place for such an emergency.  If you don't, do it now before a disaster strikes.  Make sure you have all your business data backed up securely on local storage, such as a file server, or NAS (Network Area Storage).  Then ensure all of that same critical data is backed up off site, such as cloud storage.  The local storage is quick to recover data from accidentally deleted files, or a damaged or stolen PC by recovering the data to a new computer very rapidly.  The remote back up is intended for use if your local storage fails, or in the case of a natural disaster where your entire site is destroyed.  The insurance will rebuild your building and assets, but what about your customer files, and company data.  That's what the remote storage is for.  

Your network should also have redundancy or a back up plan for your critical systems if a portion of your network goes down.  So if you have these backups and redundant networks in place, you should not experience any interruption to your business network ideally.  Redundancy is expensive, and many times difficult to design to cover all scenarios, but having a plan in place will relieve some of that stress when that "Situation" arises.  Then your biggest decision may be how long do you attempt to recover the failed system before enabling your "backup or disaster recovery plan".  That decision can be made logically by the estimated amount of down time your company experiences, with the goal of always being the minimum amount of impact time as possible to your business. 

Back to the question: So if you don't have a disaster recovery plan, your only choice is to work on the system until you get it recovered, and call in all the experts you need to accomplish the goal.  Because the longer you delay, the more impactful it will be to your business. 

  

Well, it's what I do for work. I have backups, and I have backup systems. If my main systems go down, I spin up the backup systems and if possible copy the latest data from the failed system to the backup system, or restore last nights backup to the backup system.

We get a lot of inquiries from business owners who have the same concerns. We recommend focusing on preventative measures instead of being reactionary. If the cost of your system being down, for a certain amount of time, out ways the cost of having a company or internal team manage it for you it's best to go with the latter. 

Hope this helps! Feel free to message us if you have any other questions! 

Call us KCS Computer xxx-xxx-xxxx.