As you already know, we practice self-hosting. At first, because you’re geeks and love it, setting up a network and configuring services. It’s always gratifying to see the result. Not to mention the independence and digital comfort that it brings us.

Then, self-accommodation allows us to stay in touch with the field. Contrary to many consultants, who quickly got through their technical mourning, we do not see how we could effectively advise our clients, or do legal IT (criminal or civil), in short, remain credible, without concrete experience of the subject and of its real issues.

And this week, fate had decided to give us some experience …

While we were having breakfast, we hear a noise, have we started a laundry? No, not this morning. However, the noise is insistent and evokes water … We open the doors and we discover 2cm (0.8 in) of water evenly distributed … the time to cut the general valve, the geyser stops and we will be able to manage the problem.

React

After shutting off the water supply to the general valve, the situation does not get any worse, it’s great, but we are only at the beginning …

First in our priority list, the servers. With 2 cm of water, they don’t risk much. The bay has a sufficiently high clearance and the cables do not touch the water. To be on the safe side (and to free up space), we will still stop them (properly) then unplug them and move them to the back kitchen, under the bay.

To easily recognize them, we used network cables of a different color. The thing is, we never documented that and, in a rush, we didn’t write it down either … For AD it’s easy, just one take and you know where it’s going. For proxmox, there are two … After an ip addr show, we locate the card that supports vlans and we can connect the last two cables.

By the time the servers start up, we check that the Internet and the NAS are accessible and we can move on. Call the insurance and the plumber and make them understand that, no, with children, you cannot wait until Monday without running water (5 days). Clear the area of everything lying on the ground and bail out the water.

The plumbers accommodated us the next day at noon, between other clients who had made an appointment. We can take showers again (for the toilets, we drew water from the pool).

Breathe and correct

Fortunately, we had no damage. Physically, only a few empty boxes are screwed up. Computer, everything works as before. Well almost…

We first noticed a few network failures: inexplicable latencies on certain sites, cuts when the mibox reads a movie on the NAS … in fact, it was the IPv6 of the routers which had not been fully recovered alone. It was enough to restart the master for it to reconfigure correctly, then restart the slave later.

Some ancillary services were not available, the time to realize it, to connect to proxmox and see that it is normal:

• We forgot to configure the autostart of some,
• We forgot to remove the virtual DVD from some and since the corresponding file had been deleted from proxmox, the machine could not start.

These two corrections made, we are back online 🎉.

And now ?

There are still a few small details, like relocating the servers to their place because there, they are not air-conditioned, which forces you to leave a door open and the noise will get on the nerves. Apart from a few minor issues, everything returned to normal very quickly (the network outage did not last 15 minutes and the water outage was corrected in 30 hours).

Next time we speak with a client, we can add “Little Flood” to our achievements:

• Blown fuse due to Raclette ,
• Dead Internet box due to lightning strike,
• Power cut due to backhoe,
• Break in the telephone cable due to snow,
• Various FAI failures (unknown causes),
• Small flooding due to leakage.

A little originality with water doesn’t hurt. For the fire, we prefer to wait but we will tell you.