Why Can't Stuff Just Work?

This morning I received a work e-mail indicating that something is broken with a system over which I am responsible. I quickly went to work troubleshooting the issue. I used AI to help me comb through our system documentation and found a number of possible leads on how to solve the issue. I threw it out to the people reporting the problem only to discover our documentation is incorrect. I had to go back to square one.

I'm trying to solve the issue at work when I get a text from my wife who can't get the new toner cartridges to work at the office. That shouldn't really bother me as I'm not paid to help troubleshoot her office computer problems. Unfortunately we have the same printer at home and I just bought the same toner cartridges for ours. While I don't need them to work now, I will in the future. Furthermore today is the 31st day after I received them from Amazon which means I am past the 30-day return window.

Now two problems to solve in the same day shouldn't be a problem. Any mother with a 4-year old knows that is the first 10 minutes of every day. However I try to ensure all of my technologies stay working and don't break down. Discovering you need to print something important only to have new toner cartridges not recognized might lead to panic and expletives. Fortunately my wife discovered that problem early.

To address the issue at work, I gathered more information and looked at possible solutions. Then I made sure not to commit to anything on the behalf of others. If I can fix something, then I can commit. If the solution involves others, I can't speak for their time. This keeps others from committing my time. It looks like we will need to have network engineers involved in the solution and so I need to make my notes available and let others solve the problem.

For the printer, I exchanged a number of messages with my wife. She was unable to get her printer working but succeeded in returning her work toner cartridges to Amazon. So I gave them a call. I explained the process she went through and how I have the same printer and would rather return unopened product which might be resold to someone that can use them. The Amazon customer service agent looked up the details of my purchase and immediately agreed to allow me to return them for a full refund.

Now the only problem is that my printer is almost out of toner and I need to order replacement cartridges. HP makes my printer and they have gotten wise to Chinese manufacturers selling compatible toner for half the price. To combat it, they require a special chip in new cartridges and regularly update printer firmware to reject any toner than their own. If you want to ensure your printer will continue to work, you have to buy new cartridges from HP. Yes they are more expensive but at least they should work, right? I sure hope so cause it is really expensive.

Troubleshooting My 3D Printer

One of my tasks for this Christmas break is to figure out a problem I am having with my 3D printer. My youngest son and I sat down today to figure it out. Fortunately we succeeded and now have a better working device.

To troubleshoot the machine, we looked at any changes we had made to the printer. The first is a glass print bed and I highly recommend one as it makes a huge difference in print quality. I didn't think that could be the cause and so we looked at the only other change we made: changing the print nozzle. My son found a great deal on different sized print nozzles on Amazon and ordered them. For some reason he put one of the new nozzles on the printer after trying some of the different sizes. It turns out the new nozzle is not exactly the size advertised. It is supposed to be 0.4 mm in diameter but is slightly smaller. That makes it hard for the print material to bond to lower layers as it gets laid down. We were able to measure the nozzle diameter using a tool included with the printer. Replacing it with a correct-sized nozzle solved the problem and now our prints are working nicely. We measured all of the new nozzles and have identified the 0.4 mm nozzles that really are 0.4 mm.

Since discovering the problem, we have printed a number of small parts and all of them are turning out well. So if you have a 3D printer and suddenly the print material doesn't adhere to the lower layers of your print, the nozzle size could be the problem. Beware of low-cost 3D printer parts found on Amazon. They may not meet the tight tolerance requirements.

The Internet is Down

This week I have changed my regular routine and am flying to California on Tuesday instead of Monday. That is because I will be flying home on Friday so I can attend my company's winter hackathon. That means I spent Monday working from home instead of being at the office. Things were going well and I got a lot done until all of a sudden the Internet stopped working. I had a virtual private network (VPN) into my office and it immediately told me about the problem. Then I checked on my personal computer, which is not connected to the VPN and it had a problem too.

Having my home Internet down really cut into my immediate productivity. Rather than worry about troubleshooting the problem, I just turned my work phone into an Internet hotspot and kept working. Unfortunately that meant the rest of the house still couldn't use the Internet but I was the only one home and so it didn't matter. When my wife got home from work, I let her know that we had a problem and she remarked that we had a Comcast truck in the neighborhood and he must be working on it.

I headed up to dinner and we still didn't have an Internet connection. At this point I felt an obligation to troubleshoot the problem as it was irresponsible to head to California without fixing the issue. The first thing I checked was to make sure our signal booster had power. Not all Internet connections require a signal boosters but ours does (thanks Comcast). A tripped circuit breaker has caused problems in the past. Then I reset the cable modem. That didn't change anything. Ultimately I just needed to wait for the Comcast technician to finish his work as that really turned out to be the problem.

Sometimes there is nothing that you can do when your Internet is not working. You should be proactive about solving the problem though. One more thing I could have done is to call my Internet service provider and asked about an outage in the area. If you do end up calling and a technician comes out, don't be surprised if you get charged for the service call should it be determined that you are at fault.

Fixing a 24-Year-Old Air Conditioner

Yesterday afternoon my wife texted me and asked me to call her. I couldn't because of a meeting and so she sent me another message saying the air conditioner in our house broke. When I could finally call her, I had her run through some troubleshooting steps to see if we couldn't isolate the problem. Ultimately nothing we tried would bring the AC back on. As this is the really hot part of the year in Utah, cooling the house is very important.

I left the office and rode back to my boat trying to figure out what the problem could be. I thought about all of the steps I had my wife try and figured that we either lost the blower motor, the capacitor for the blower motor, or the computer that controls the blower motor. Based on some issues we had a few weeks ago, I figured it was a 95% chance the computer died. Using the power of a smartphone with a camera, my wife helped me figure out the part number for the old computer and find a replacement part on the Internet. The only problem is that I couldn't guarantee the arrival before the end of the weekend when I would have to catch a flight from Utah back to the Bay Area.

This morning I had my wife call around and see if she could find a replacement part locally. She did and it cost $175 while the same part can be purchased for $75 on the Internet. When it is 95 degrees outside, $100 is a small price to pay for being able to cool the house immediately and so I had her pick up the part.

This evening I boarded a plane back to Utah and my wife picked me up at the airport. We came home and I began working on fixing the AC. One would think it would be as simple as labeling all of the dozen or so wires and plugging them back into the new computer. Nope, the new board had a completely different configuration and only a few of the wires had the same labels.

I went to my desktop computer and looked up part numbers on the Internet. My old AC computer is part number ST9120G2008 and my replacement is ST9120U1011. I searched for a bit but couldn't find instructions on how to rewire the computer but I did find one clue: white wires are neutral. I found all of the neutral connections on the computer and started there. Next I found labels on the the old computer and matched them with the new one. That left only five unknown wires. Fortunately the instruction manual came with a very nice wiring diagram. I just had to recall all of my Electrical Engineering homework from a long time ago and I matched the remaining wires.

While carpenters like to measure twice and cut once, electrical engineers like to recheck their work and so I did. Then I checked it again. Once you let the magic smoke out of electrical components, they don't work very well after. I didn't want to have to spend another $175 for a wiring mistake. Once I convinced myself everything worked, I did a quick test. The blower motor started right up. I then tested the AC and it is now working to bring my house down from an unbearable 85 degrees to a more manageable 76.

I am not an HVAC specialist and normally I would call a repairman for such a job. In fact I have done that in the middle of winter when the heater stopped working. The problem turned out to be a $4 light switch. I still had to pay $120 for the service call. I figure at that rate, I can spend a bit of my own time and try to figure it out. Furthermore buying a new part through the repairman would run me about $300. I figure I saved myself around $245 doing the work myself. I also learned a lot in the process.

If you find yourself in a similar situation but have the slightest concern that you didn't correctly diagnose the problem, then call the repairman. If you think the Internet will help guide you through the fix, then give it a shot. You might just save yourself some serious cash.

A Tale of Two Projects

This past weekend I spent time digging a giant hole in my front yard looking for a water main break. Fortunately I found the problem on my first hole as there were two likely places the leak could have been. Once I found the problem but before fixing it, my son decided he would try and resolve a problem with his car. In the process he learned that you never start another project until one is complete. As the water was off in the house, he couldn't wash his hands after working on the car until the first problem was completely solved. He also managed to break both his car and my other son's car in the process and so at the end of the afternoon, we went from having one project to three.

My son's problem was actually fairly simple. In the process of replacing the cylinder head on his 1996 Jeep Cherokee, we broke one of the temperature sensors that screws into the the engine. My son wanted to replace it with a new one and so he went to the auto part store. He immediately broke that sensor in the process of trying to put it in. Naturally that was the only one the store had. Fortunately he was able to bring it back, get his money back, find another sensor at a different auto parts store, and break the new sensor. Another trip to the second auto part store and he got another one. Yes, the temperature sensor is incredibly brittle and breaks easily. One would think they would make them a bit more resilient.

The 1996 Jeep Cherokee actually has two engine temperature sensors and the one my son kept breaking is the one that sends information to the temperature gauge on the dashboard. Unfortunately the new one told my son that his engine was overheating and so my son couldn't go out with friends that evening. He was pretty bummed. When we popped the hood and checked the engine, it didn't seem any hotter than normal and didn't have the usual signs of the radiator boiling over. So we needed to determine if the engine was really overheating or if the temperature gauge was bad.

I'm a firm believer in the saying that "A man with one watch knows what time it is, a man with two isn't quite sure." In 1997, the engineers at Jeep decided the second temperature sensor in the Jeep Cherokee was superfluous and removed it. Fortunately for us, our job was to tap into the second sensor on my son's Cherokee and see if it was giving the same high-temperature results. All we needed to do was to tap into the car's OBD-II computer. Fortunately my son has a code reader that is a little more sophisticated than mine and it reads various parameters that include the engine temperature. The only catch is that the car has to throw a check-engine code first. My son tried unplugging various sensors only to have the engine keep running without any problems. He even tried unplugging one of the spark plugs with no luck. Finally he pulled the plug on one of the fuel injectors and the car threw a code immediately. He checked the code and the engine parameters and discovered that the engine was in its optimal operating temperature. That means the gauge sensor was wrong and his car really wasn't overheating. We now know why the engineers at Jeep took out the separate gauge sensor and just use one now.

The weekend is over and the hole in my front yard is now filled in. My son's Jeeps is running fine and the erroneous temperature gauge seems to be fixing itself. My son has also fixed the problem that arose on my other son's car. It took a couple of days, but things seem to be back to normal. At least until the next weekend.

Solving Problems Together

This evening I was at the office until 10pm solving a problem. We had two very important pieces of hardware in our data center behave strangely and that caused some serious problems with our environment. At first we didn't realize the extent of the problems and at 5pm I thought I would be stuck at the office for another hour. It turned out to be a bit longer than that.

Initially I was busy watching two other operations engineers bring systems back online. I could look at my monitoring dashboard which is nothing more than a list of descriptions with a number next to each line of text. I could see services starting back up and my massive data warehouse escaped without any issues. Then I was asked to check out a number of much smaller databases and I realized the extent of the damage to our system.

I didn't really notice time passing as a number of us all worked together solving the technological issues. Some guys hung around even though they could have gone home. Why? Because they would have wanted us to stick around if the roles were reversed. Eventually some really did have to leave because of family commitments and nobody faulted them for it. In the end, there were three of us on the phone solving the last of the issues and we all hung out until we were sure everything was working correctly.

Now I won't claim that my coworkers are perfect and that everything is all roses and sunshine where I work. However it was comforting to have a great team of engineers sticking around to make sure everything got put back to normal. That helped make staying late a lot easier. Thanks guys.

An Interesting Network Problem

Last week I had a bad case of the stomach flu. Normally that is the type of illness that is done in a single day. Not the strain I had. It was with me for a good 5 days. When it came time to take my normal flight back to the office on Monday, I wasn't able. Instead I stayed at home in Utah and rested in bed. Eventually I had to get out of bed and read e-mail.

I tried to use my Linux desktop to read e-mail, but couldn't get the computer to connect to the Internet. Normally it only takes me turning it on for a connection to happen. When that wasn't sufficient, I tried to reset my cable modem, followed by power cycling the network hub. That didn't work. The rest of the computers in the house were all connecting to the Internet just fine and so I resigned myself to the fact that I needed a new networking card.

I wanted to try one more thing before heading to the store to get a new network card. I unplugged the computer and put it up on my work bench. I pulled off the cover and checked all of the connections. I was going to vacuum out the dust, but there wasn't any. I put the computer back together and booted it up. Amazingly it worked. It turns out that the real problem was that even though the computer was turned off, the motherboard was still getting power. Simply unplugging it after turning off the computer was sufficient. I must remember that the next time there is a network issue with one of my computers.

Effective Troubleshooting

Yesterday and today I spent several hours hunting down a computer problem in a production system at work. I did a pretty good job of narrowing down the problem to a piece of software. I changed the configuration file, upgraded to a newer version, and tried a number of other solutions only to have the problem continue.

My coworker and I were able to get the system limping along so we could sleep on the problem. We both woke up this morning with a similar idea on how to locate the true cause of the issue. It turns out that the real culprit was a database table was larger than it should be. The system was designed for this table to have a few hundred rows. Instead it had over 5 million. There was a clean-up script that wasn't running.

The first half of solving this problem meant I had to manually clean out the table. Trying to use an automated system would have brought the database machine crashing down. Once that was done, we had to get the clean-up script rewritten. That took a good part of the day, but we got it working. The total problem solution took a few hours to come up with. I just wish we had done a better job troubleshooting last night.

Linux Fixes Another Computer

One of my friends has been having computer problems and was about to go out and buy another computer. Rather than spending a lot of money, I had him bring the computer over so I could see if I couldn't fix the problem. Yesterday he dropped it off on the way to baseball practice when he briefly explained the issues he was having.

Windows XP wasn't recognizing the network card that was built into the computer. He went down to Best Buy where he talked with one of their "Geeks." He didn't have the computer with him and so the Geek asked several diagnostic questions. The conclusion was that the computer needed a new network card and so the Geek quickly found one for a really great price.

My friend brought the card home and installed it. He turned on the machine and installed the drivers for the new card. The new network card didn't work either. This means that the computer could not be used to browse the Internet which rendered it slightly more useful than a boat anchor (as a sailor, computers make really lousy boat anchors). So now it was my turn.

The first thing I did was go back to square one. I removed the new card the Geek had sold my friend and confirmed that Windows XP could not see the built-in network hardware. Then I put in the new card and went into the computer's BIOS to turn off the built-in network card. On his computer, it was as simple as hitting the "F1" key during the boot sequence and going to the "Advanced" menu option. Having two network cards in a computer is possible but sometimes they like to step on eachother's toes. It is advisable to have only one network card active when you are trying to debug problems like this.

Once again the network card was not recognized by Windows XP. This was a huge red flag. How do you know if the machine is bad when the the measuring device isn't working properly. In this case, the measuring device was the Windows XP operating system. I tried a simple experiment. I booted Linux from a CD. When you boot from a CD, it doesn't install anything on the hard drive and so you can test your machine's hardware without overwriting anything.

The CD takes a little bit longer than a hard drive to boot and so I went and got myself a sandwich while the computer thought about starting. Once it was up, I opened a browser window and could get to any number of websites just fine. This confirmed that it was Windows XP that needed help and not the hardware.

I told my friend that he had two options: He could reload Windows XP, which would probably fix everything, or he could install Linux. He picked up the machine this morning and will be installing Linux. I think that is a good decision.