Putting on the Pith Helmet

 

 

Early in my career I heard the story of a software developer that had the unique practice of putting on a pith helmet whenever forced to debug his own code. I'm not sure how true the story is but it evokes the image of an early explorer trudging through the jungles of a distant continent. In my current role, I am often asked to search for system problems and help resolve them.

Last Friday I received a cry for help and immediately looked into the issue. My manager didn't like how I responded as he wanted me to find the source of the problem before bothering the group that reported it. I saw it differently as I wanted them to know that we immediately started looking at the issue and wondered if they had any more information they could provide to help diagnose the issue. Ultimately the only thing I could do was log a bug with another group in the company and have them research the root cause of the issue.

We told the group that reported the issue that we would work to get the issue resolved by Monday. Such is the luxury of working for a world-wide company like Sony. I can go enjoy my weekend and hope another team working in Japan or India has enough time to find the issue and resolve it before Monday even starts in the United States. Unfortunately sometimes an issue can be more difficult to troubleshoot and so it is important to get as much information as possible on Friday so we don't have any back-and-forth with questions.

Fortunately one of our excellent engineers saw the problem on Saturday and worked through the weekend to get it fixed. The root cause boiled down to "too much data." As this means the problem may arise again we have a two-pronged approach to solving it: The first is to monitor for larger-than-normal data sets. That will alert the operations team before the system users get a hint that something has gone wrong. The second is to modify the architecture to handle larger volumes of data.

This morning I came in and saw everything that had taken place over the weekend and could report back to the system user who noticed the problem. I could also share the resolutions that we plan to implement so it doesn't happen again. The user confirmed the problem fixed and I closed the bug appropriately.

While I like the idea of wearing a pith helmet to signify that I am trudging into the jungles of code, I am glad I have not adopted the practice. First, I work out of my home office so nobody would have any idea of the significance other than my wife who would just laugh at me. Second, I would have gotten a lot of strange looks and questions wearing the unique hat through the weekend. 

Repeatability

Have you heard the story of a guy that traded a paperclip for a house? It didn't happen that simply. He traded the paperclip for something else and continued trading until he eventually had a much more expensive item. When I think about what he did, I imagine he took advantage of having something that someone else needed to the point where they would trade something more valuable for the less valuable item he had. I also doubt the process he followed is very repeatable. While someone could take some of the ideas, trades for different items would need to be substituted. Furthermore, I doubt many would have his same success.

This leads me to my topic of interest today and that is repeatability. When software engineers create a program, they require a team of quality assurance (QA) engineers to test their code to look for problems. The worst kinds of problems to find are non-repeatable ones. If the program runs 9 times out of 10, it becomes the QA engineer's responsibility to figure out the exact repeatable steps to find the one time it doesn't work.

One area where this becomes increasingly difficult is when using a program that creates random numbers. The whole point of random numbers is to make things unpredictable. This is why most random number programs have two calls. The primary call provides a random number. To make sure it is repeatable, it will always provide numbers in the same order. If you ask for a single digit random number, it might first provide a 3 and the next time you call the function it returns a 5. Stopping the program and restarting it give you the same numbers in order. This is very helpful when trying to locate software bugs. The important thing to remember is to also use the second call or randomize function. This tells the program to not use the same order but instead to mix it up.

Why is this important? Several years ago I noticed that the software provided by the governing body for ski racing forgot to include the randomize function in their software. My son attended a race at Jackson Hole and had yet to complete a Super-G race. He got lumped in with all the other first-timers. The weekend had 3 individual races where all the first-timers should have been given random starting orders within their group. Starting near the front has its advantages and so one would think my son would have his start position changed randomly. Nope the 10 to 15 racers all started in the same order for each of the 3 independent races. The developer of the software never randomized the start order as he or she should have. I tried to protest but ski coaches and race organizers don't understand the nuances of software development and accepted as fact that the coding had been done correctly.

When looking for software bugs, repeatability is important. It makes it a lot easier to find problems and fix them. Just remember to add in that randomize function at the end. Otherwise you are not really creating random numbers and that is one area hackers exploit to break into systems.

The Art of Debugging

I mentioned in a previous post that I am in the process of creating a prototype web application using React. Unfortunately I am also learning how to debug my React code. I created a base version of the application using the tutorial I mentioned in my previous post. Once I got all of that working, I started expanding.

React takes the code you write and compiles the code so that it runs quickly. This compile step is where I find a lot of errors. I have a console window open that spits out a bunch of important information each time I save a file using my interactive development environment (IDE). Sometimes I will get errors and other times I get warnings. While the code is supposed to run with warnings, it generally means something is wrong and is best to fix it. Otherwise your application may not behave exactly as you thought it should.

The worst problems I have encountered are runtime errors. That means my code compiles just fine but the application doesn't look right or doesn't render at all. The first such problem that took me a while to solve involved the HTML break tag. I had a picture and wanted to have a caption underneath it. To make everything look clean, I laid out everything in a table. To get the layout perfect, I used a test HTML page and then just transferred everything to React when I was happy. I didn't know that React treats the break tag a little different and required a slash in the tag. Instead of using <br>, I had to use <br />. That looks pretty simple, right? Well it broke the whole application. Instead of seeing a funny-looking table, I didn't see anything. In order to find the offending tag, I had to copy each part of the HTML table into React and discover when things stopped working.

Today I ran into another problem. I have a button component that looks very integrated with the entire site. It is something I copied from the tutorial mentioned earlier. The guy giving the tutorial didn't bother to do anything special with the button other than to have it jump to a single page. I have put a number of those buttons all over the place but no matter which button you click on, it goes to the same page. Today I wanted to connect one of the buttons to a different page. I looked through the code and figured out a way to make it happen. Unfortunately when I added the code, it broke everything. Not a single page would load. It turns out that by adding the destination page as a parameter to the button, I broke every instance of that button. Seeing as I had a button on the heading of every page and on the footer, that should have been a sign to add the destination page as a parameter to all of the buttons in the application. Once I did that, everything started working again.

The Internet has been very helpful in solving a number of problems. I tried to embed a YouTube video into my application only to get an error from my browser saying that it can't show the video as a security precaution. A quick search showed that I needed to add a library to React and then things started working great.

I'm sure there will be many more problems I run across, but so far I have been able to solve them in a timely fashion. If I feel I am spending too much time on an issue, I put it on hold and work on something else. Sometimes I will go for a walk and the solution will come to me. Ultimately there are a number of tricks to debugging computer programs and it doesn't matter if you are coding in Python or React. I suppose the most important trick is to not get frustrated. I'm still working on that one.

Learning React

If you ever find yourself looking for a new job, you will soon discover that you need something other than job hunting to keep you busy. I have started a project to build a graphically pleasing website that is also database driven. I am amazed at how much happier I am now that I have results I can see from my day's efforts. I also spend an hour or two looking for the perfect job, but it isn't all encompassing like it has been.

Right now I am focusing on building the front end or the user experience (UX) part of the website. I did a bunch of research and discovered that all of the cool kids are using React. I found a great tutorial on YouTube that is forming the basis of what I want to build. Of course there are a number of other graphical elements I want to add that are not covered, but that is what Internet searches are for. Those searches also help with figuring out why my version of the project isn't working. The tutorial I found was created over a year ago and the React libraries have changed since then. When something doesn't work, I just have to query the Internet and I have a fix.

Last night I worked on the project for hours and didn't notice the time passing. This morning I awoke early and eager to continue my journey of learning. This is in sharp contrast to the depression I felt after getting one more rejection from a job where I expected to receive an offer. With the general shutting down that is happening at companies due to the Christmas break, I am glad I have something that energizes me and keeps me excited. At the end of the break, I hope to have a fully functioning demo that I can show.

You Need to See It

I am currently working on a project for my job that is not going well. Part of the problem is that I was not given explicit instructions on what people wanted. Instead I got instructions to make something better. I sat down and asked questions about specific improvements that people wanted to see. Then I started putting things together so I could solicit feedback.

The first pass went well and people liked the idea of what they saw. That is actually the wrong response. I really needed people to tell me I was headed in the wrong direction or how they thought my ideas could be improved upon. Now we are getting close to the end of the project and I am being asked, "That is all?" Fortunately I still have some time to get things back on track.

Several years ago, I wrote about "Agile Development." The idea is to show something every couple of weeks and constantly ask for feedback. This experience underscores the importance of constantly showing updates. Even though that is what I thought I was doing, I didn't do it well enough as it is only now that stakeholders are coming to me with new ideas about the project.

I don't think the Agile process should be limited to just software development. It could be applied to any job or chore. Think about teaching your children to do dishes or mow the lawn. You don't send them out unsupervised for hours at a time. Instead you periodically check on the work to make sure it is being done correctly. Early in the process is the best time for feedback as the longer you wait, the more fixing needs to be done. Now for my project I just need to figure out how to create the largest flash in the shortest amount of time.

Revisiting Agile Development

A few years ago I wrote about Agile Development. Earlier this week I had an experienced that reinforced one of its principles: quick iterations. I have a project and have been given six months to complete it. I have another developer helping me and so we are meeting often to work on the deliverables. I thought we had a good start to our plan and sat down with my boss to go over some details.

My plan included installing some new software and migrating hundreds of people to the new system. I knew it would be a tough sell but that is how I planned to incorporate one of the most important goals of the project: world-wide access for all teams. My boss suggested using existing software and making small tweaks to bring teams on-board gradually. Rather than trying to get everyone onto the new software in a big push, we will make minor adjustments to the software that teams in one geographical location are already using. Then we will add another team midway through the schedule. Rather than trying to get the third and final group onboarded to new software that they are not familiar with, we will just copy data from our system to theirs and vice versa on a nightly basis.

My plan is much more comprehensive but runs the risk of being impossible. There is a chance that we will get to the end and not have anything working. This will extend the project for an unknown amount of time and could end up being a never-ending project. The new plan is incremental and allows us to have something to show every two weeks, which is our sprint length. If we are late on any of our deliverables, stakeholders will see the delays as they happen and be able to plan accordingly.

This weeks experience reminded me that even though I know software development principles, there are many ways to put them into practice and some are better than others. I need to remember not to focus on one goal at the expense of being able to get something done.

Relearning C++

I have a project at work that requires some C++ coding. For those that don't know, C++ (pronounced see-plus-plus) is a computer programming language. I learned it back in college but then never really did much with it. I spent part of my professional career writing C programs but it is slightly different and so I have been spending some of my spare time going through some old C++ books I have. Those that compare C to C++ have been most helpful.

It is fun to go back and relearn something I used to enjoy. While some software developers hate C++, I actually appreciate the language and prefer it to many others including Java. As I had a background in the language, it is much easier to pick it up again.

I had hoped when I started the project that I would simply be able to look at some example source code and remember everything I had forgotten. Unfortunately that is not the case. It has been helpful to run through my book and play with the examples.

Learning technology no longer requires a book to go through. You can use any number of tutorials and explanations found on websites. Once you get a basic understanding of the technology you are trying to learn, you may opt to supplement your learning with a book, but it is not necessary. The necessary thing to do when learning (or relearning) something new is to go through the examples. After all, we learn by doing.

Software Quality vs. Getting it Done

Yesterday I had to be at our San Francisco office for some meetings and arrived shortly before lunch. My office in San Mateo has a cafeteria and so I didn't know what my lunch options might be in that office. Sure I could have walked around and found something quite easily but I wanted to catch up with an old work associate and so I sent him a lunch invitation e-mail. He had the time and so we headed to his favorite place and caught up.

We didn't go any place fancy and I appreciated that. A simple meal is all I needed. We got to talk about the differences between the two offices that belong to the same company. Even though he is doing similar things as to when we worked together, there is a different feel to it. When we worked together, my friend felt like he could do the job correctly and fine-tune the software he created. In the new environment, he feels like there is more of a push to get something out and working. It doesn't matter if it is not optimal, it just needs to work. He mentioned that he appreciated being able to create quality software and not just quantity.

If you talk to a software developer, his or her code is never really finished. It just gets good enough to use. Therefore a healthy balance needs to be struck when creating software. Programs need to be created and they need to work. They also need to be put out in a finite amount of time. So if you see a video game release date moved to a further date, you can be sure the development team is wrestling with getting the product into your hands as a consumer but also making sure it meets minimum quality requirements.

Software Development Trends

Today I had an experience at work that underscored the evolution of software development. When I was in school working on programming assignments for various classes, there really wasn't a need to use a formal methodology. I looked at the problem, sketched out my algorithm, sat down and coded it. Life was pretty easy but so were the assignments, relative to the projects I am involved with now.

I got out into the software development field and had a team that kept up on all of the latest software development methods and we were one of the leaders in implementing the capability and maturity model (CMM). It worked for us and we were able to create some pretty amazing computer software that was on time and under budget.

Fast forward a few years and I found myself doing consulting for various organizations. As I was a lone coder working on projects, there really wasn't a need to completely embrace CMM. Instead I took what felt good and coupled it with what worked in college. Once again life was easy but my projects were rather small and not very complex.

Eventually I found myself working for a very small software organization in Southern California and was introduced to agile software development. My first thought was that "agile" must have been developed by 3M because you use a lot of sticky notes. It felt like the next evolution of software development and seemed fairly efficient.

Then "the cloud" comes along and the next bit of evolution is needed so DevOps is born. DevOps doesn't replace "agile" but becomes coupled with it. In addition to creating software, a group of software developers gets tasked with writing tools to deploy the software. This isn't a large task for small and simple programs. However bigger software projects requiring large numbers of computing nodes or instances running on such services as Amazon Web Services (AWS) or Goole's Compute Engine (GCE) require a lot more care to ensure the code gets deployed correctly.

In my meeting today we discussed DevOps and whether it should be done by a specific team. The group consensus is that each developer should have a hand in the deployment process and understand DevOps. I wouldn't say we are ready to disband the DevOps team yet as each project needs at least one person responsible for architecting the entire deployment system. However this is a shift from how we have done things in the past and seems to be the next step in the evolution of software development. Of course, now that I have written this down it is obsolete. Tomorrow there will be yet another twist.

Video Game Algorithms

Recently I have felt the need to do a bit more software development and so this evening I pulled out the code for a game I have been writing. Yes it is taking me a long time to write this game but it is the journey I am more interested in than the destination. I need to create an algorithm that allows a bad guy to hunt down the player. It looks so easy to do graphically but is much more difficult as I try to think out the steps so I can translate them into computer code.

I threw together something simple and then discovered that it doesn't work very well. When the player gets close to the edge of the screen, the program gets stuck in an infinite loop. I know the cause and so I am having to create a more complex algorithm. Not only do I have to check if the bad guy is close enough to the player to see him, I also have to check to make sure I don't try to run off the board or through an object on that is supposed to be a barrier.

When I learned how to write computer programs in high school, my teacher taught me to pull out a scratch sheet of paper and step through the algorithm development process. So that is what I find myself doing this evening. I have a notepad with X's (the player) and O's (the bad guy). Then there are arrows showing player movement with the bad guy chasing him down. I am making progress but not nearly as quickly as I had hoped. I do have to admit it is an interesting algorithm I am trying to develop though. One that can be used for a number of video game situations. Unfortunately I had to wake up too early this morning to catch my flight from home to the Bay Area and so I am finding it hard to concentrate. Perhaps I will have more luck tomorrow.

Satisfaction in Coding

A couple of days ago I was adding an entry to this blog when I noticed that I had "Assembly Language" as one of my labels. It briefly reminded me of some assembly language programming I had to do while in college. For those that don't know, assembly language is one step above the ones and zeros that computers understand. Nobody wants to remember that 00000001 is the command to move the contents of register A to register B and so the designers of the Intel 8088 microprocessor created the MOVAB command. MOVAB is much easier to understand for us humans. Very few people program in Assembly Language as it can be rather tedious. However those programs are small and run incredibly fast.

Thinking back to my college classes reminded me how much fun coding is. When I first started at my current job, I got to do quite a bit of SQL coding. Now I do less and less as I am managing a bunch of developers. Every once and a while, I will get to do something technical and I feel a great sense of satisfaction.

Last night I got back to my boat early and had time to play around with a bit of coding. It felt good to create few lines of Python and watch them do something interesting. Once again, I am back on the boat early and plan to continue with my sample coding. I'm not planning to write the next version of Tetris or a better game than Angry Birds, but that doesn't matter. Coding allows me to build something. It allows me to create something. Writing software allows me to turn ideas into reality with virtually no cost other than my time. That alone is its own reward.

Following Directions

This morning at work I helped perform an upgrade of some software I have helped create. It is made up of a bunch of small programs that have evolved over time. Each program is quite simple and very easy to understand. Together they form a complex system that runs on over a dozen different servers and process large amounts of data.

The upgrade seemed to be running smoothly and I was able to do my portion of the deployment without any issues. When we went to start several of the programs, we saw some errors. That led to everyone gathering around a collection of screens, trying to troubleshoot the problem as we were under a tight deadline to get the system back up and running.

As all of my stuff was working correctly, I acted as a messenger to one of our operations engineers who was at our data center in another part of the country. It would have been nice to talk to him on the phone but was impossible because of all the noise where he was. We had to resort to texting. The whole team worked together and we quickly diagnosed the problem. It turns out that the engineer doing most of the upgrade missed a critical step. Once we figured that out, everything came up nicely.

How did this critical step get missed? We had a list of instructions with every individual process outlined for the upgrade. The engineer was meticulously checking off each task as it was completed. After two pages of instructions, he got a little careless and checked twice after only completing one step. It was an easy mistake to make but shows the importance of carefully following directions. Not doing so in a car can get you lost. Not doing so with a bunch of computer servers can really screw things up.

Agile Development

There are a number of methods that software development teams can use to help manage themselves. Years ago, teams would create detailed specifications over a period of months and then spend years coding only to discover they had built the wrong product. In 2001 a bunch of successful software developers met at a ski resort in Utah and created the Agile Manifesto. This then spurred a new process where developers would work on small pieces of the product with short release cycles so they could get constant feedback on what they were building. Then product users could provide instant feedback to ensure the development team is on the right track.

My current software development team uses Agile development and so the company hired an instructor to come in and teach a 2-day refresher course. It was interesting to spend a couple of days going through the training material. There were other teams new to the process taking part in the course and it was fun to see their reaction. Some were eager to embrace the method while others were less than eager to change their current development practices.

I have to say that there is one problem with Agile for me. I have a specific skill that requires me to work with a number of different teams. So while there is  about 4 hours a week of meeting overhead for the rest of the team members, I end up with about 12 hours a week of meetings. Considering there are only 40 hours of work during the work week, that means I only have 28 to get useful stuff done. So when someone gets all energetic about Agile, I have to bite my tongue to keep from grumbling too loudly.

I am hoping that the guys that created the Agile Manifesto will show up at Snowbird again, where they originally drafted the short statement, and come up with a way to keep me from going to so many meetings. I'd even provide a couple of lift tickets.

Proper Software Development

I am working on several software development projects. Some for work and others for my own personal edification. One of my projects has recently come under some deadlines and so I have been forced to take one or two short cuts. This generally means that proper software development methods get thrown out the window in favor of getting something done quickly. You might be able to get away with it for a day or two, but eventually it will come back to bite you.

Having been in the software business for the past 26 years, I have learned when it is best to avoid certain short-cuts. This saved me today. Even though I was told to move a piece of code into production, I did so long enough to run a single test. Then I backed out the new code and put the old working code in place until the results of my test could be properly verified.

The new piece of code seems to be working correctly. To me, it works exactly the way I wrote it and the way the user asked for it. However, sometimes those looking at the results don't like what they asked for and request more changes. If I had left the new code in place, it would have meant a lot more work should any changes be requested. Guess what? There are always more changes and I saved myself quite a few hours of work.

Hammers and Tools

A few years ago everyone started using the phrase, "If your only tool is a hammer, you tend to treat all your problems like nails." It made a lot of sense and people started re-evaluating what tools they were using for any type of job, not just home improvement projects.

At around this same time, my boss where I was working coined the phrase, "You can't build a skyscraper with a hammer." I was working for a company that had a really expensive software development tool that let small groups of programmers build really complex applications in a relatively short amount of time. The hammer and skyscraper metaphor made a lot of sense and opened several doors for the company.

Anyone who has ever worked on a seemingly simple home improvement project knows that the right tool can help keep the job simple. Not having the right tool can make it unbearably complex.

This past Friday I was working on a software project and was trying to build a skyscraper with a hammer. This meant I was going to be stuck writing a lot of code and I didn't ever think I was going to finish. Then I stumbled across several tools that made quick work of the project. Something that I expected to take a week took less than half an hour to complete and I could move on to other tasks.

So as you are working on your computer and seem to be running into a brick wall, remember to check your proverbial toolbox and see if you don't have a better tool to help with your task. If that doesn't work, you can always borrow a brick out of the wall you seem to be running into and smash your computer to bits. While it may not accomplish much, it can be very therapeutic.

A New Project

This week I started a new software project. I'm not sure it will make me much money but it is good to be coding again. It is a short project and so I am hoping to have it completed within a week.

The exciting thing about my new endeavor is that it involves doing something I have not yet done. Normally my software projects are made up of pieces of things I have already put into other programs. If it needs to save data, I have an example of code I used on my previous project. Do I need to validate any of that data? If so, then I have a piece of code I used two projects ago. Sometimes software development is just like putting together pieces of Legos. It can also be just as fun.

The new piece that I am working on is a relatively new feature found in web-based applications. This means that I have to go the original RFC and figure out how to implement the feature on my own. Now you may be asking yourself, "What the heck is an RFC?" RFC is an acronym that stands for "Request For Comments."

When the Internet was in its infancy, the technologists involved exercised a bit of humility and democracy. Instead of coming up with a list of "must do's" they decided they would use a community process to determine the best way to implement technological features. Someone working on a problem would throw out a "Request For Comments" with his or her solution to the problem. Others working on the Internet were then free to suggest alternative methods. If nobody had any meaningful comments, then the RFC was adopted as a defacto specification.

One thing about computers and technology is that they continue to evolve. Even someone such as myself, with almost a quarter of a century working with computers, needs to learn new things. Now if I can just figure out how to do this one little thing, my project will be close to finished.