I’ve been asked many times why I care about IT security. It started in earnest for me way back in 2000 when somebody invaded my house. I first published this story in the February, 2001 edition of Enterprise Linux Magazine.
International Terrorism in Minnesota
I’ve written extensively in this column about a small Linux DNS server I run. Imagine my surprise a few weeks ago when I found my system launching a denial of service attack against the Government of Brazil. That set a chain of events in motion every bit as traumatic for me as the recent Presidential election was for everyone else.
It all started when I tried to access my email. For some reason, the response time was unbelievably slow. About that time, my wife complained she couldn’t get to the Martha Stewart Web site, or anywhere else on the Internet, and what did I do to the computers this time?
I started investigating and found my house LAN was indeed running very slow. I looked at my hubs and found port 4 on one hub going nuts. This was the port leading to my DNS server. The ps –ax command showed me the following process:
ping -s 65000 -f nn.nn.nn.nn (I won’t share the target IP address.)
My DNS server was sending 65,000 byte packets as fast as it possibly could to a system across the Internet. When I killed the process, performance went back to normal.
A feeling of dread came over me and my adrenaline started pumping. Then I got mad as I realized some jerk broke into my DNS server and set up this attack. Fortunately for the Internet, I don’t have enough bandwidth for anyone significant to seriously care about. Unfortunately for me, this jerk found out where I am and how to break in to my network. I felt violated, angry, and afraid all at the same time, especially when I thought about all the data I have squirreled away in various directories on computers all over my network. I wanted to find this jerk and strangle him or her, but I didn’t have the tools to even know where to begin.
So I called my friends at Mission Critical Linux for help. I explained the situation and we all agreed that somebody had compromised my system. I learned a lot about network break-ins that day. I learned that BIND 8.2.2-P5, the version of DNS bundled with Red Hat Linux 6.1, has “hundreds” of security vulnerabilities, and that Red Hat keeps a list of bug fixes and updates on its web site. I should have periodically checked for these updates.
I learned to shut down services such as sendmail, telnet, and ftp because they serve no useful purpose on this machine. Sendmail uses its own process while the inetd process controls ftp, telnet, and others. These commands ensure they won’t start at boot time:
/sbin/chkconfig –level 345 sendmail off
/sbin/chkconfig –level 345 inet off.
That’s when I remembered that telnet had been behaving strangely. When I tried to connect via telnet, it wouldn’t echo anything and lately would just tell me the process was ending.
The support person laughed and told me I’d been suckered by the oldest trick in the book. Somebody probably replaced the real telnet with a fake version designed to steal passwords for later transmission to the bad guys. The system had definitely been compromised.
The technical recommendation: Wipe the hard drive and rebuild the system from scratch. The next recommendation: Call the FBI immediately because the IP address my system attacked belongs to the Brazilian National Government, and I could face legal trouble if I didn’t report it.
As soon as we hung up, I called the Minneapolis FBI office and asked for somebody who deals with computer crime. The receptionist sent me to a lady. The conversation went like this:
Greg: “Hi – I need to report a computer crime. Somebody broke into my DNS server and launched a denial of service attack against the government of Brazil.”
FBI Lady: “Wait a minute. Did you say D-E-S server?”
Greg: “No, a DNS server.”
FBI Lady: “Oh – D – N – S, OK. What did they do to your computer?”
Greg: “Somebody tried to use my computer to attack a computer that evidently belongs to the Brazilian Government.”
FBI Lady: “OK, . . ., who did it? Do you have their address?”
Greg: “No. See, a DNS server translates names to addresses on the Internet. One of my computers is a DNS server and somebody out there on the Internet tried to use my computer to attack this other computer in Brazil.”
FBI Lady: “OK, but we need to know who did it. We need a name or address or some way to find this person.”
Greg: “Well, I was kind of hoping you guys could help me figure that out.”
FBI Lady: “There’s not much we can do if we don’t know who broke into your computer. Don’t you have any idea how to find this person?”
Greg: “I wish. See, the Internet is a whole bunch of computers all around the world and they’re all connected to each other. Somebody on one of those computers found my computer and made it do this attack. Since all these computers are connected to the Internet, we don’t know if the attacker is next door or across the world someplace. But maybe they left some clues inside my computer to help track them down.”
FBI Lady: “OK, let me get your phone number and somebody will call you back.”
Greg: (after giving my phone number) “Any idea when I’ll hear from somebody?”
FBI Lady: “No. They’re all pretty busy, ya know.”
I made that call on Tuesday, Nov. 11, 2000 at roughly 1 PM central time. I called again at 4:30 PM the same day. As of this writing on December 15, 2000, I still haven’t heard back from the FBI. I don’t mean to complain, but I was hoping the FBI would be sharper than that.
I’ll share how I rebuilt my DNS server and a list of helpful books in a future column.
I realized later, I made a mistake on my dates in the article. Nov. 11, 2000 was a Saturday. I know I called the FBI on a Tuesday, so the correct date would have been either Nov. 7 or Nov. 14. To this day, I have no idea how I came up with Nov. 11 for a date in the original article. But this key detail gave me an insight into how the FBI works.
My phone rang one morning in Feb. 2001, a few days after the article ran. It was a manager in the Minneapolis FBI office and he wanted to troubleshoot. I thanked him for the call, but said I could not afford to shut down my life and wait three months for a callback from law enforcement. I had long ago wiped and rebuilt that system.
That’s when he went into CYA mode. He said that since I called on a Saturday (remember, I really called on a Tuesday) I must have connected to a weekend operator. That was why they had no record that I had ever called. Yeah. Uh-huh. My tax dollars at w0rk.
Lesson learned – law enforcement is of little or no value in data breach scenarios. Over the next several years, I would learn that lesson a few more times.
Here is why everyone should care about incidents like this. Somebody exploited a flaw in one of my public facing systems to invade my house and use me as a drone in their attack against a third party. Although nobody physically tramped through my house, the net result was the same–I was violated. And I was on my own to fix it. How many times since have we heard variations on that story?
If you’re running a business and somebody violates your company IT systems, the odds are slim that anybody from law enforcement will help you. If you’re an individual consumer, the odds are even slimmer. So read books like “Bullseye Breach” to educate yourself on how these violations happen, read earlier posts in this blog, and keep an eye on future posts for ideas to reduce your attack surface.
If you bury your head in the sand, don’t be surprised when somebody kicks your exposed rear-end.