~ This event isn't fiction. Here follows the description of these events as they unfolded~
Paul had been working for this client for 15 years. He built him all of his sites!

Paul even wrote one of the Ecommerce platforms in PHP. Can you imagine? From scratch!— A la mano. He set up the servers, the synchronization scripts with the warehouses to manage the stock. He knows the business by heart, has all the settings in his head. *sigh* What would we do without Paul, and that's why we pay him so well.  

But this morning, end of September 2018, Paul died. Boom. 40 years old. Heart attack. -As if it wasn't tragic enough the main server failed 2 days later. No sound, no image. Just a ping.
  

…. Shit ….

Luckily, a few months ago, this client had asked us to give Paul a hand on some Prestashop. We didn't have root administration access, but just limited developer access. Basically we were completely screwed.

On the phone I tell the client : Don't worry bro, I'm putting Pavel and Artem on this.
You have no idea how much damage 2 senior Ukrainian sys-admins can inflict on a badly kept up server in very little time. (all within the confines of the Law of course). I'm used to it now, I've been working with my team for 11 years now, but to see so much experience going away at fixing a problem that's so complex amazes me every time.

  • Within 12 hours all the sites were online on a new server with reinforced security, we had root access and the client was in business again.
  • Within 24 hours,  we had done a full audit and created technical documentation for all the sites, with a working plan of what we thought would be indispensable for the operation of the business.

Normalisation and continuity of business in case of a major process failure.

Our work doesn't end there. In accordance with the client, we take the following measures:

  • We transferred all the sites made with proprietary code to well known e-commerce platforms: Magento/ Prestashop.
  • We set up the infrastructure of secured servers in order to insure the development of the business: (dev, Staging, Producction)Nous mettons en place une infrastructure serveurs sécurisée afin d’assurer le développement du business : (dev, Staging, Production).
  • We open up our project management system to the client, so he has access to all the documentation and history of work done.
The lesson seems hard to learn and some times heartless, but your online business shouldn't depend on Paul or John but on a well organized system to be ready for the worst and especially avoid hostage situations from programmers or sys admins who think they know everything and would make you believe they are indispensable for your business to continue functionning.

They are not.