AWS Documentation is Awful

This past week I had a coworker ask me to help with some Amazon Web Services (AWS) administration. I needed to go in and check the last time a handful of users had logged into the system. I had been shown how to do this in the past but didn't do a good job of taking notes. I didn't worry about it as I figured I could just use an Internet search engine to help me get the information I needed. That didn't work at all.

AWS is such a large system that there are half a dozen ways to get the information I needed. Furthermore the documentation kept saying to use a tool and write code to query log files. From my training at the beginning of the year, I knew there is an easy way to point-and-click my way to success. The thought of writing code felt like building a car before going to the grocery store to buy some eggs.

Fortunately I remembered some of what I had been taught and got roughly to the right place. Using information sent to me by my coworker I was able to stumble around for bit and get the information I needed. She sent me a screenshot and it provided the missing clues I needed.

I have a notebook that I keep for work. I write down notes from meetings and helpful hints. Naturally I wrote down all the steps to repeat how to get the information I needed. Logic follows that if I have had to get the information once, there is a good chance I will need it again. Next time I don't want to waste half an afternoon trying to figure it out.

90 Minutes Saved Me A Week In Las Vegas

This morning I started work a little earlier than usual as I attended a summary of Amazon's recent announcements from the AWS re:Invent conference held recently in Las Vegas. When I first started my career, I loved traveling and going to conferences in other cities. Then I spent a lot of time in planes to the point I now prefer to stay home, if possible. So any time I can have the host company of a conference summarize all of the important points in a 90-minute video conference, I am extremely happy.

AWS made quite a few announcements and they distilled them down into 3 different topics for the call this morning. I won't run through everything I learned as there are 2 announcements that really stood out. The first topic dealt with artificial intelligence. About a year ago, ChatGPT started getting a lot of attention. Naturally AWS is incorporating this technology in their product offerings. They also addressed how they are keeping the data used to train their large-language model (LLM) called Q, separate from your company's proprietary data. That is very important as you don't want someone outside your company asking Q for sensitive information that he/she shouldn't be able to access.

The second announcement I found interesting is using Q to generate database queries. I learned structured-query language (SQL) back in 1986 and thought it to be very natural-language like. As databases have grown in size and contain data in a variety of different tables, those queries have become less and less like natural language. You can now ask Q questions about your own data and as long as it has access to it, will show what you are looking for without knowing how to write the correct SQL. I will be interested to see how Q works in actual practice. I don't worry about it taking my job creating complex queries. Instead I see myself being able to do much more with its help.

My call this morning condensed a week-long conference into 90 minutes. I then pulled out 2 announcements that are important to me. Now I can go back to our AWS sales representative and get the in-depth information I need. To me, that is a step forward towards greater efficiency. For those that don't get to travel often, you may prefer the week in Las Vegas.

New Year's Resolutions

This morning I headed into work on my bike and then to the company gym where I usually start my day. Keeping with my early-rising habit I developed over the Christmas break I found myself there earlier than usual. I expected the gym to be packed with those seeking to lose a bit of weight gained over the Holidays. Nope, it felt really empty. Perhaps the idea of making and then breaking resolutions has gotten to the point where the new trend is to not make them in the first place. I hope I am wrong about that as the new year is always a great time to reflect on how we can improve ourselves.

On Sunday my wife asked if I had set any goals or resolutions and I had to confess that I have not made the effort yet. I sort of have a few ideas that carry forward from previous years like skiing more and dropping a bit of weight. So far I have started the year off right as I am down a few pounds which is a freaking miracle considering all of the feasting most people do during the holidays. I have also come back to work with a sore body from skiing so much over the break.

In this vein of looking at ways of improving myself over the next 12 months, I think I would like to learn a new technical skill or two. I have a very strong understanding of relational database systems and have even dabbled with other database technologies. Therefore those are not good areas to expand. Instead I should look at where my weaknesses are and set some goals to improve those parts of my professional skill set. I have used both Amazon's (AWS) and Google's (GCP) cloud platforms but could definitely improve there. Our company has a learning portal and should probably leverage it first and then fill in any shortcomings with a local conference.

Another area I would like to focus some effort is improving my artificial intelligence (AI) knowledge. Last year I ran through a Reinforcement Learning book and learned a lot. The book wasn't all that great but I did enjoy the code segments and doing the examples. There are a number of new AI tools emerging and I will try to find one that catches my interest and do something with it.

Finally I want to do more with gaming technology. I have dabbled with the Unity game engine but think I could learn a lot by learning Unreal as well. There are so many samples and tutorials that it would be good to spend an extra hour on Tuesday and Wednesday evenings hanging out at the office and building something interesting.

While it is nice to create a list of improvements, be sure to write them down. Someone once said that a goal not written is just a wish. If you want the wish to come true, start by writing it down and then coming up with a plan. I have the starting part down, now I should follow through and make a proper plan for each of these resolutions. Good luck on your own New Year's Resolutions.

Buy, Rent, or Borrow Computing Power for Machine Learning

I am going through an interesting exercise at work. There are a number of us going through a Reinforcement Learning book and we all want to play with the code described in the various chapters. While I have the luxury of having a Linux machine sitting under my desk, others are not so fortunate. One of my coworkers has a rather large Linux box with a lot of computing power and so several people are running the exercises there. Unfortunately it can only handle about 3 really large jobs concurrently and so we are looking at other options for getting compute power for our Machine Learning exercises.

We tried the first solution by buying a large computer but there are a number of problems associated with that. I explained the first being that we are limited to about 3 people using it at the same time. the resources are fixed and don't scale well. The second problem is we have to maintain that computer. During one particularly large job, the computer stopped running at 4am and we have no idea why. We believe it might have been a hardware failure because the logs just suddenly stopped recording anything. It would be nice to have someone monitoring the computer 24 hours a day but that is not possible. Especially for a simple learning exercise.

The next option is to rent space on a cloud service such as Amazon (AWS), Google (GCE), or Microsoft (Azure). We do not require graphics processing units (GPUs) and so we can get enough computing power for all our experiments for around $650/month. We will take 2 to 3 months to read the book and would require about $2,000. That is significantly less than the price we paid for buying the computer mentioned previously. Furthermore the hardware will scale nicely. If we want to run more experiments, we increase the number of servers we rent. When we don't need them any more, we shut them down and don't pay to keep them running.

Finally there is the option of borrowing computing power. There are a number of other groups within the company that have spare compute cycles we could use for our learning exercises. This is the ideal solution if we only factor in cost. However the reality is that someone could be kicked off the hardware when other higher-priority tasks need to run instead or the problem of spreading out experiments evenly across the company.

We will probably end up renting servers from one of the public cloud companies as it seems to strike the balance between being cost effective and least troublesome. Your situation may be different and it is always worth considering all 3 options.

Software Development Trends

Today I had an experience at work that underscored the evolution of software development. When I was in school working on programming assignments for various classes, there really wasn't a need to use a formal methodology. I looked at the problem, sketched out my algorithm, sat down and coded it. Life was pretty easy but so were the assignments, relative to the projects I am involved with now.

I got out into the software development field and had a team that kept up on all of the latest software development methods and we were one of the leaders in implementing the capability and maturity model (CMM). It worked for us and we were able to create some pretty amazing computer software that was on time and under budget.

Fast forward a few years and I found myself doing consulting for various organizations. As I was a lone coder working on projects, there really wasn't a need to completely embrace CMM. Instead I took what felt good and coupled it with what worked in college. Once again life was easy but my projects were rather small and not very complex.

Eventually I found myself working for a very small software organization in Southern California and was introduced to agile software development. My first thought was that "agile" must have been developed by 3M because you use a lot of sticky notes. It felt like the next evolution of software development and seemed fairly efficient.

Then "the cloud" comes along and the next bit of evolution is needed so DevOps is born. DevOps doesn't replace "agile" but becomes coupled with it. In addition to creating software, a group of software developers gets tasked with writing tools to deploy the software. This isn't a large task for small and simple programs. However bigger software projects requiring large numbers of computing nodes or instances running on such services as Amazon Web Services (AWS) or Goole's Compute Engine (GCE) require a lot more care to ensure the code gets deployed correctly.

In my meeting today we discussed DevOps and whether it should be done by a specific team. The group consensus is that each developer should have a hand in the deployment process and understand DevOps. I wouldn't say we are ready to disband the DevOps team yet as each project needs at least one person responsible for architecting the entire deployment system. However this is a shift from how we have done things in the past and seems to be the next step in the evolution of software development. Of course, now that I have written this down it is obsolete. Tomorrow there will be yet another twist.