Flawless design, but...

We design may of our remote sites with lots of redundancy and think of many failure scenarios that we can recover from with minimal impact.

Today, one of the sites experienced a brief power failure. Something our designs definitely can operate through without any issues. The one thing that even the best design needs to consider.

A local technician that left the generator in bypass mode during his last maintenance. It's a mechanical handle with no way to query the state/position of the handle. In this mode, the generator does not provide power when plant/grid power is removed.

Something so simple would have made it a non-event, but we ended up spending hours getting equipment back to normal operation.