Azure Outage Causing Problems

I am trying to get a lot of tasks done today and am being thwarted by Microsoft's Azure outage. This is not a knock against Azure as I didn't think that many services used it. It is only during an outage that you realize how much you use that service and how it affects you.

I received an e-mail from my employer letting me know that the Azure outage made it impossible to contact our travel department by phone. They suggested using e-mail should we need to talk with someone. Having taken a few trips for work, I know that e-mail is a great alternative as they are quick to respond. I didn't worry about it and kept about my daily tasks.

Next my wife informed me that Costco might be a great place to get refills for a consumable item we need to replace which is filters for our furnace because it is running more with the cold weather. I didn't want to hop in my car and so I tried to log onto the Costco website. In white letters on a blue background it says that some customers are experiencing site performance problems. While some of the site works, not everything does. For instance I can't set my local store correctly. Even though I enter my zip code, it only shows me stores in the state of Washington. Eventually they will figure it out. Of course I do have the option of running down there in my car should it be an emergency.

Several days ago I received a letter from a previous employer saying that my employee information got hacked and they are providing me with free identity protection from Cyberscout. While it makes sense to be wary of a company you have not heard about, they are legitimate and owned by TransUnion, one of the major credit bureaus. In the course of setting up my account, they need to send a code to my phone for multi-factor authentication. While the phone number I provided worked once, it didn't send the second code. To finish setting up the account I had to call their service center.

Sometimes computers greatly simplify our lives. Other times, such as in the case of downed systems, they really slow us down. Fortunately we have old backup technologies like e-mail, phones and driving to a local store to help us get stuff done.  

Buy, Rent, or Borrow Computing Power for Machine Learning

I am going through an interesting exercise at work. There are a number of us going through a Reinforcement Learning book and we all want to play with the code described in the various chapters. While I have the luxury of having a Linux machine sitting under my desk, others are not so fortunate. One of my coworkers has a rather large Linux box with a lot of computing power and so several people are running the exercises there. Unfortunately it can only handle about 3 really large jobs concurrently and so we are looking at other options for getting compute power for our Machine Learning exercises.

We tried the first solution by buying a large computer but there are a number of problems associated with that. I explained the first being that we are limited to about 3 people using it at the same time. the resources are fixed and don't scale well. The second problem is we have to maintain that computer. During one particularly large job, the computer stopped running at 4am and we have no idea why. We believe it might have been a hardware failure because the logs just suddenly stopped recording anything. It would be nice to have someone monitoring the computer 24 hours a day but that is not possible. Especially for a simple learning exercise.

The next option is to rent space on a cloud service such as Amazon (AWS), Google (GCE), or Microsoft (Azure). We do not require graphics processing units (GPUs) and so we can get enough computing power for all our experiments for around $650/month. We will take 2 to 3 months to read the book and would require about $2,000. That is significantly less than the price we paid for buying the computer mentioned previously. Furthermore the hardware will scale nicely. If we want to run more experiments, we increase the number of servers we rent. When we don't need them any more, we shut them down and don't pay to keep them running.

Finally there is the option of borrowing computing power. There are a number of other groups within the company that have spare compute cycles we could use for our learning exercises. This is the ideal solution if we only factor in cost. However the reality is that someone could be kicked off the hardware when other higher-priority tasks need to run instead or the problem of spreading out experiments evenly across the company.

We will probably end up renting servers from one of the public cloud companies as it seems to strike the balance between being cost effective and least troublesome. Your situation may be different and it is always worth considering all 3 options.