January 30, 2020

Elders,

Happy Thursday! Can you believe that we are transitioning from January to February already? I must be getting old(er). This past Sunday we had a great Quorum discussion. Aric talked about Elder Andersen’s conference message about “Fruit”. The discussion centered around Lehi’s dream. It is such an important part of the gospel and has meanings on so many different levels that it is good to get to talk about it and rehash it often. It was a great tie-in to our Come Follow Me lessons as well. Thank you Aric and each of you for the messages, comments, and contributions.

Just when I think that things are going to smooth out at work and get to a semblance of normal we fell into a trap of over-confidence. All was well at work. Life is good… The “fun” started with a developer doing something – ostensibly running unit tests. Something went terribly awry with the tests and started truncating database tables in our production system. Soon our internal system was totally non-functional and the alarms from the production teams were rolling in. We were able to stop the damage before it got too bad, but it was already bad enough. Truncate commands (as opposed to deletes) are irreversible. Delete commands log what they are doing and can be painfully rolled back. Truncates are far more evil – the data is just gone forever.

As the various teams assembled and the protocols went into action, we immediately went to our back-ups. Maybe we could restore quickly and only lose a few hours or maybe a day’s worth of work. It wasn’t to be. The backup script was reporting success, but wasn’t actually doing anything. The only backup copy that we could find was from mid-December. We set our sights on recovering a month’s worth of data and asked for the restore to take place into a separate database. Meanwhile, we schemed about what our next steps were. Several hours later we asked for a status update and the delayed response served as a harbinger of what was to come.

Evidently, someone decided to fix the backup scripts in the middle of the outage. They did and when the UTC time rolled over to midnight the backup did it’s magic. Part of the script was to delete any old backups… Yep, the one backup we had was suddenly deleted as well. We searched in vain to find any backup anywhere. We finally realized that we were on our own with no backup at all…

Fast forward a week. We are back up and running in only a slightly degraded state. There are still problems and there is about a week’s worth of data that we’ll never be able to recover. Many teams have spent many sleepless nights and performed heroic acts to get us to where we are today. I appreciate all the hard work.

It could have been avoided. Sure, problems happen. Any one or more of the failures could have happened but the sequence and timing brought the whole world crumbling down.

How does this relate to you and to me? We “thought” that everything was well and that we were safe and well-protected. We thought there were safeguards and processes in place to defend ourselves against human foibles. What we found is that we were sitting on a house of cards. This is the state that the adversary wants us to be in too. He wants us to believe that “all is well” and that “everything is fine”. He wants us to stop doing the little things, stop checking in on our friends and neighbors, stop evaluating our actions and decisions – until we get to a place where we think we are so safe that “it cannot happen to me”. At that point, one small thing can topple our whole house of cards.

I hope and pray that we can all examine ourselves and not just make a cursory look at where we are or assume everything is good. If we can really look and fortify ourselves and our families, we can put ourselves into a much stronger and safer position in our lives. Don’t wait for disaster to happen. Prepare and review, fix and strengthen, become better and improve in all the facets of your life. Then, when disaster strikes (and it will) we will be in a position to overcome. God will help you. He loves you and will magnify all of your efforts to improve.

One of our quorum members could use some help. Brother Von Gunten is in recovery at the Legacy Rehab Center 3200 W 5400 S. He would really enjoy some visitors. If you get a chance to swing by and talk with him, he would love it. He is in room 210.

We have a great quorum of men and I enjoy being together and learning from each of you. We have many perspectives and insights that come out that I had never before considered. Thanks for putting up with me and my ramblings. I love being a part of our quorum. 

Please reach out if I can be of any assistance.

Thanks,

President Oldroyd
801-573-6828