Performance Improvement

One of the fundamental lessons I learned in my days as an Electrical Engineering student is the order that performance improvements should be addressed. Always work on the largest problem first. This is really counterintuitive as people will throw out minor improvements because they are easily seen and understood. Unfortunately the improvements may be so minor that they are imperceptible. Let's look at some examples.

Recently I drove a car from San Mateo, California to Salt Lake City, Utah. This is a great example of performance as you always want to minimize the amount of time spent in the car. There are a number of variables you can play with to decrease the amount of time the trip takes. First off is which route you will take. Interstate 80 is the fastest route but is sometimes closed because of snow. You also have how fast you go. You can drive the speed limit or try to go above it. That introduces a few more variables like gas mileage and the risk of getting pulled over to receive a warning or speeding citation. The faster you drive, the more you will have to stop for gas. Don't forget that you may require additional stops to use the restroom or get food.

Thinking of the driving example, most people will point out that you can save time by combining food and restroom breaks with fuel stops. During my recent trip, I stopped for an hour to sleep and that could have been avoided by leaving earlier in the day. Leaving earlier would have also helped with the gridlock traffic I encountered leaving the Bay Area. Ultimately though, that all had very little effect on the overall drive. The best performance gains on a long trip like that are going to be solved by changing the speed of the trip from 80 miles per hour, the legal speed limit for most of the trip in Nevada and Utah, to 500 miles per hour which is how fast you can make the trip in a commercial plane. That is how I got to San Mateo in the first place and if I didn't have to bring a lot of stuff they don't allow on airplanes, it is how I would have gone home.

Another example of performance improvement I saw came in the form of a National Geographic article. It described how to save water while doing dishes. In many parts of the world, water is a scarce resource. In other parts it falls from the sky in large quantities. The article encouraged everyone to avoid rinsing your dishes before placing them in the dishwasher. No offense National Geographic but if you have a dishwasher, there are significantly better ways to save water. Taking a shorter shower is a good start. Better yet is to water your lawn only when necessary. Not rinsing your dishes saves about 2 gallons of water. Taking a shorter shower will save more than the 2 gallons of water. Watering your lawn only when necessary will save hundreds of gallons of water. In other words, stop worrying about 2 gallons and worry about 100 gallons. Once everyone gets that down, you can worry about smaller amounts.

So what does this have to do with computers? A lot actually. If you find yourself looking to improve performance, look at the system as a whole. Break down the tasks and their various times to complete. If task A takes minutes and task B takes seconds, focus on task A. Don't worry about B as it doesn't really matter until you can get task A into the seconds range.

Performance Tuning in the Cloud

I have been spending the last couple of days working on performance tuning for our big project at work. We are moving the software from ancient hardware and migrating it into our own private cloud. What is the difference? With the old system, we had a fixed number of servers and had to manually spread pieces of the application across them. With our own private cloud, we have a number of servers that we make look like a single machine. Then we divide up that machine so that each piece of the application looks like it is running on its own server. One thing that we are discovering is that performance problems can be much more difficult to solve.

Yesterday we ran into an interesting problem that we spent today figuring out. In the video gaming world, people don't like it when the game crashes. To combat this issue, we run redundant systems and so there are always two of everything. The problem we came across was that one server was processing more data than the other, even though they were configured identically. Everyone looking at the problem suggested going through both configurations line by line. We did and verified the two systems were set up identically. Then we all sat around scratching our heads wondering what to look at next. We tried twisting some nobs and pushing various buttons only to have more questions after looking at the performance numbers.

It turns out that the process of taking a bunch of machines and combining them together to form this computing cloud obfuscates what is happening on the raw hardware underneath. To figure out what is really taking place, one has to look there. We are still in the midst of improving performance but now have a much better handle of what is going on. Amazingly this new system already is running circles around the old one, we just want it to run faster.

The Observer Effect

I have run into a performance problem with one of my databases at work. We are constantly adding new records to the database and the system runs fine. When we try to augment the normal load with some extra data, the system slows to a crawl and I have been trying to figure out why.

I have a tool that will help me monitor performance. Every 2 seconds, it queries the program sending data into the database and asks how many packets of data are waiting to be saved. When the system is keeping up, this number is 0. When it is running slow, the number grows. Yesterday I got tied up with other things and turned my monitoring program off. I wasn't really paying close attention, but the performance seemed to increase because I wasn't watching.

This morning I wanted to see if there was a true performance increase or if it just seemed that way. I started loading data and made a note of the time. Yesterday it took almost exactly 2 hours to complete. This morning it took 1 hour and 40 minutes. I thought that was interesting and so I ran the test again. Sure enough, the second test completed in 1 hour and 50 minutes. While my monitoring of the system didn't slow things down too much, it did have an effect.

I guess you could say that the old cliche about a watched pot never boiling can be true. In Physics, this is called the Observer Effect, where merely observing something changes what is being observed.

A New Job

After several years of consulting, I have finally taken a job with a new company which happens to be based in Los Angeles. This means I am spending most of my time in sunny California. I have to say the weather is a lot warmer than back in Utah. The downside is that it is really cutting into my skiing.

My new job is as the database expert and administrator for a development team. We are using PostgreSQL and so I was a natural fit for the position. With any new project, there is some low-hanging fruit that is easy to implement and this job is no different.

PostgreSQL is a great open-source database and there are a lot of tools to make it better. One thing you can do to help increase performance is connection pooling with the help of pgbouncer or pgpool. Connection pooling is like keeping a phone line open to your best friend and never hanging up. If you have something to say, just talk. There is no delay while you wait for your friend to answer the ringing phone.

We decided to try pgbouncer because it has slightly better performance than pgpool (but doesn't do nearly as much). It took only fifteen minutes to install and we are already seeing huge performance improvements with our application. I like it when you can spend a few minutes and do something that has such a huge impact. The only problem is keeping it up.