Solving Problems Together

This evening I was at the office until 10pm solving a problem. We had two very important pieces of hardware in our data center behave strangely and that caused some serious problems with our environment. At first we didn't realize the extent of the problems and at 5pm I thought I would be stuck at the office for another hour. It turned out to be a bit longer than that.

Initially I was busy watching two other operations engineers bring systems back online. I could look at my monitoring dashboard which is nothing more than a list of descriptions with a number next to each line of text. I could see services starting back up and my massive data warehouse escaped without any issues. Then I was asked to check out a number of much smaller databases and I realized the extent of the damage to our system.

I didn't really notice time passing as a number of us all worked together solving the technological issues. Some guys hung around even though they could have gone home. Why? Because they would have wanted us to stick around if the roles were reversed. Eventually some really did have to leave because of family commitments and nobody faulted them for it. In the end, there were three of us on the phone solving the last of the issues and we all hung out until we were sure everything was working correctly.

Now I won't claim that my coworkers are perfect and that everything is all roses and sunshine where I work. However it was comforting to have a great team of engineers sticking around to make sure everything got put back to normal. That helped make staying late a lot easier. Thanks guys.