I’ve been asked many times why I care about IT security.  It started in earnest for me way back in 2000 when somebody invaded my house.  I first published this story in the February, 2001 edition of Enterprise Linux Magazine.

---

International Terrorism in Minnesota

I’ve written extensively in this column about a small Linux DNS server I run.  Imagine my surprise a few weeks ago when I found my system launching a denial of service attack against the Government of Brazil.  That set a chain of events in motion every bit as traumatic for me as the recent Presidential election was for everyone else.

It all started when I tried to access my email.  For some reason, the response time was unbelievably slow.  About that time, my wife complained she couldn’t get to the Martha Stewart Web site, or anywhere else on the Internet, and what did I do to the computers this time?

I started investigating and found my house LAN was indeed running very slow.  I looked at my hubs and found port 4 on one hub going nuts.  This was the port leading to my DNS server.  The ps –ax command showed me the following process:

ping -s 65000 -f nn.nn.nn.nn (I won’t share the target IP address.)

My DNS server was sending 65,000 byte packets as fast as it possibly could to a system across the Internet.  When I killed the process, performance went back to normal.

A feeling of dread came over me and my adrenaline started pumping.  Then I got mad as I realized some jerk broke into my DNS server and set up this attack.  Fortunately for the Internet, I don’t have enough bandwidth for anyone significant to seriously care about.  Unfortunately for me, this jerk found out where I am and how to break in to my network.  I felt violated, angry, and afraid all at the same time, especially when I thought about all the data I have squirreled away in various directories on computers all over my network.  I wanted to find this jerk and strangle him or her, but I didn’t have the tools to even know where to begin.

So I called my friends at Mission Critical Linux for help.  I explained the situation and we all agreed that somebody had compromised my system.  I learned a lot about network break-ins that day.  I learned that BIND 8.2.2-P5, the version of DNS bundled with Red Hat Linux 6.1, has “hundreds” of security vulnerabilities, and that Red Hat keeps a list of bug fixes and updates on its web site.  I should have periodically checked for these updates.

I learned to shut down services such as sendmail, telnet, and ftp because they serve no useful purpose on this machine.  Sendmail uses its own process while the inetd process controls ftp, telnet, and others.  These commands ensure they won’t start at boot time:

/sbin/chkconfig –level 345 sendmail off
/sbin/chkconfig –level 345 inet off.

That’s when I remembered that telnet had been behaving strangely.  When I tried to connect via telnet, it wouldn’t echo anything and lately would just tell me the process was ending.

The support person laughed and told me I’d been suckered by the oldest trick in the book.  Somebody probably replaced the real telnet with a fake version designed to steal passwords for later transmission to the bad guys.  The system had definitely been compromised.

The technical recommendation:  Wipe the hard drive and rebuild the system from scratch.  The next recommendation:  Call the FBI immediately because the IP address my system attacked belongs to the Brazilian National Government, and I could face legal trouble if I didn’t report it.

As soon as we hung up, I called the Minneapolis FBI office and asked for somebody who deals with computer crime.  The receptionist sent me to a lady.  The conversation went like this:

Greg:  “Hi – I need to report a computer crime.  Somebody broke into my DNS server and launched a denial of service attack against the government of Brazil.”

FBI Lady:  “Wait a minute.  Did you say D-E-S server?”

Greg:  “No, a DNS server.”

FBI Lady:  “Oh – D – N – S, OK.  What did they do to your computer?”

Greg:  “Somebody tried to use my computer to attack a computer that evidently belongs to the Brazilian Government.”

FBI Lady:  “OK, . . ., who did it?  Do you have their address?”

Greg:  “No.  See, a DNS server translates names to addresses on the Internet.  One of my computers is a DNS server and somebody out there on the Internet tried to use my computer to attack this other computer in Brazil.”

FBI Lady:  “OK, but we need to know who did it.  We need a name or address or some way to find this person.”

Greg:  “Well, I was kind of hoping you guys could help me figure that out.”

FBI Lady:  “There’s not much we can do if we don’t know who broke into your computer.  Don’t you have any idea how to find this person?”

Greg:  “I wish.  See, the Internet is a whole bunch of computers all around the world and they’re all connected to each other.  Somebody on one of those computers found my computer and made it do this attack.  Since all these computers are connected to the Internet, we don’t know if the attacker is next door or across the world someplace.  But maybe they left some clues inside my computer to help track them down.”

FBI Lady:  “OK, let me get your phone number and somebody will call you back.”

Greg:  (after giving my phone number)  “Any idea when I’ll hear from somebody?”

FBI Lady:  “No.  They’re all pretty busy, ya know.”

Greg:  “Thanks.”

I made that call on Tuesday, Nov. 11, 2000 at roughly 1 PM central time.  I called again at 4:30 PM the same day.  As of this writing on December 15, 2000, I still haven’t heard back from the FBI.  I don’t mean to complain, but I was hoping the FBI would be sharper than that.

I’ll share how I rebuilt my DNS server and a list of helpful books in a future column.

---

I realized later, I made a mistake on my dates in the article.  Nov. 11, 2000 was a Saturday.  I know I called the FBI on a Tuesday, so the correct date would have been either Nov. 7 or Nov. 14.  To this day, I have no idea how I came up with Nov. 11 for a date in the original article.  But this key detail gave me an insight into how the FBI works.

My phone rang one morning in Feb. 2001, a few days after the article ran.  It was a manager in the Minneapolis FBI office and he wanted to troubleshoot.  I thanked him for the call, but said I could not afford to shut down my life and wait three months for a callback from law enforcement.  I had long ago wiped and rebuilt that system.

That’s when he went into CYA mode.  He said that since I called on a Saturday (remember, I really called on a Tuesday) I must have connected to a weekend operator.  That was why they had no record that I had ever called.  Yeah.  Uh-huh.  My tax dollars at w0rk.

Lesson learned – law enforcement is of little or no value in data breach scenarios.  Over the next several years, I would learn that lesson a few more times.

Here is why everyone should care about incidents like this.  Somebody exploited a flaw in one of my public facing systems to invade my house and use me as a drone in their attack against a third party.  Although nobody physically tramped through my house, the net result was the same–I was violated.  And I was on my own to fix it.  How many times since have we heard variations on that story?

If you’re running a business and somebody violates your company IT systems, the odds are slim that anybody from law enforcement will help you.  If you’re an individual consumer, the odds are even slimmer.  So read books like “Bullseye Breach” to educate yourself on how these violations happen, read earlier posts in this blog, and keep an eye on future posts for ideas to reduce your attack surface.

If you bury your head in the sand, don’t be surprised when somebody kicks your exposed rear-end.

Here are a few FAQs (frequently asked questions) about Internet security.  I should have put this together a long time ago.

Q: I don’t keep national security secrets inside my computer or cell phone. Aren’t all these so-called security products the real scam?

A: You probably don’t have any secrets anyone cares about.  But the game is not to steal your secrets.  The real game is to make you an unwitting drone in a scheme to steal somebody else’s secrets.  You spent money for your computer equipment and you spend money every month for Internet and cell phone service.  If you don’t care about somebody using you for criminal projects, then don’t protect yourself.  You are either part of the solution or part of the problem.

Q: Why don’t all those lonely teenage hackers get a life?  And why are the most powerful companies in the world at the mercy of a few evil computer genius hackers?

A: These are the wrong questions to ask.  The image of a lonely teenage boy in his bedroom stealing national security secrets for fun might play well in Hollywood, but it’s not real. So are the images of an evil computer genius threatening to destroy the world by guessing the secret password and typing a few commands, and the good guy genius who saves the world in the nick of time. Most of the bad activity these days comes from organized criminal organizations or nation-states, not any single individual. Those powerful companies are vulnerable because the people charged with keeping them safe did not do their jobs.

Q: If there are no evil computer genius hackers, then why do we see almost daily reports of cyber breaches?

A: I didn’t say there are no evil geniuses, only that the Hollywood images are wrong. There are plenty of evil geniuses in the world, but they are only a small part of an entire global criminal industry.  Just like legitimate industry, the shadowy Internet criminal industry has venture capitalists, inventors, markets, tech support services, and specialists for every conceivable discipline.

Q: Why are we all such sitting ducks on the Internet and why doesn’t somebody do something about it?

A: Just like humans developed an overwhelming advantage over other animals on our planet by developing language, bad guys currently have an advantage over good guys because bad guys collaborate better than good guys.  Business and government can erase that advantage by bringing security practices out into the open and giving them more than lip service.  We can influence policy by educating ourselves and using our market power to support organizations with good security policies.

Q: Is it true that my Internet connected baby monitor can destroy the Internet?

A: No, not by itself.  But combined with millions of other poorly designed IoT (Internet of Things) products, it can wreak plenty of havoc.  When you buy Internet connected devices, such as baby monitors, DVRs, security cameras, door locks, thermostats, ovens, you name it, make sure they have a mechanism for updates in the field.  Make sure you don’t use factory default passwords and make sure they don’t have default passwords or other back doors permanently baked into the hardware.  And put them all behind a credible firewall.

Q: Speaking of firewalls, since all my stuff is behind a firewall, doesn’t that mean I’m safe?

A: No.  Firewalls are one part of a bigger picture.  They stop unsolicited traffic.  Firewalls are worthless when you invite the traffic in.  That’s why it’s important to be careful about what websites you visit and avoid opening email attachments.  And that’s why you need antivirus software, even if nobody has a perfect antivirus solution.

Q: Today’s high tech is boring and complicated.  Why can’t they just make this stuff simple and usable?

A: They is really us.  Spend more time with security, where technology and psychology meet and the results are fascinating.

Q: Where can I find an entertaining story about how major data breaches play out?

A: One great perk about my own blog: I get to plant great lead-in questions.  Here is a shameless plug for my first book, “Bullseye Breach,” an educational book about data breaches disguised as a thriller novel about how the Russian mob penetrates Minneapolis retailer, Bullseye Stores, and steals 40 million customer credit card numbers.  Here is a two minute silent video about how that attack unfolds.

And stay in touch for information about book #2 coming soon.  This time, a nation-state really does mount an attack.  And the stakes are much higher than credit card fraud.

By now, everyone knows about yesterday’s FBI announcement about the Hillary Clinton email server investigation. James Comey’s words, “extremely careless” were widely quoted. As expected, the Trump camp responded with much sound and fury, signifying nothing. And the Hillary camp responded by claiming vindication. Both camps are wrong. What a surprise.

I downloaded and read a copy of the transcript and listened to a recording of the whole announcement today. Read this paragraph from the Comey statement:

“I have so far used the singular term, ’email server,’ in describing the referral that began our investigation. It turns out to have been more complicated than that. Secretary Clinton used several different servers and administrators of those servers during her four years at the State Department, and used numerous mobile devices to view and send email on that personal domain. As new servers and equipment were employed, older servers were taken out of service, stored, and decommissioned in various ways. Piecing all of that back together — to gain as full an understanding as possible of the ways in which personal email was used for government work—has been a painstaking undertaking, requiring thousands of hours of effort.”

I said earlier that if I were in Hillary’s shoes back in 2009, I might have put in my own email server too. I haven’t heard anything to change my mind, especially given what we’ve learned recently about government data breaches.

The email server isn’t the issue. The real issue is respect. Why does somebody use several different servers and administrators over four years? As somebody who delivers server administration services, I can think of only one reason – she was either an unreasonably demanding customer or she hired amatuers willing to work cheap.

Good email administrators are professionals and the former Secretary of State should have respected the professionals she hired for this purpose – not switched them out like changing clothes. I would love to talk to a few of the people she brought in and then got rid of. Were they professionals that she treated badly or were they amatuers who didn’t know what they were doing? Either answer is bad for Hillary.

What about Trump? He continues to make a fool of himself and too many Americans are too willing to follow him off a cliff.

For the first time in my life, I’m faced with two awful choices for President. Maybe a 3rd alternative with a credible chance of winning will come along.

It’s good to know that at least my family listens to my constant Internet safety lectures.  I wish more business leaders would do more than talk about taking security seriously.  I am under constant cyber-attack.  Every single day, more than 100 phishing emails hit my inbox.  Some are clever.  One cussed me out for sending a bogus invoice, conveniently attached to the message.  Another cussed me out for not paying an invoice, also attached.  Many claim to come from UPS or USPS or Amazon with news that the package I was expecting had a delivery problem.  Open the attachment for details.  My “Bullseye Breach” book website regularly comes under attack, most recently from a Russian IP Address.  Since “Bullseye Breach” is a book about how Russians steal forty million customer credit cards from a large retailer named Bullseye Stores, I guess the only surprise is that it took the Russians so long to attack it.

So when my daughter came to me with strange cell phone behavior, I knew it had to be another attack.

She was trading text messages with another mom to set up a play-date for my grandson.  The other mom offered to have my grandson over to her house to play with her son, and my daughter offered to stay and help.  Boys can be rambunctious when they get together.  This was one of my daughter’s messages, quoting with permission:

“Sounds good.  I am cool with staying and hanging out if you want.  I just don’t want you to feel like overwhelmed or anything.”

The other mom responded and they continued their text conversation.  I still don’t see the appeal of text messages as a primary form of communication.  Those teeny tiny keys and auto-correct drive me nuts.  If we’re both tapping little buttons on a phone, why not just talk to each other?  Maybe it’s a generational thing.

In the middle of her conversation with the other mom, two identical text messages from two different unknown local phone numbers came in.  The messages were, “who dis?” followed by forwards of my daughter’s messages to the other mom.

Shocked and afraid, my daughter asked me to help figure out how somebody invaded her phone.  Why was somebody stalking her from two different phone numbers and taunting her with her own text messages?  How did some lowlife intercept her text messages and play them back for her?  What did they want?

I was curious myself.

Looking over the conversations, the texters knew my daughter’s name and the date and time she planned to meet the other mom.  But we knew nothing about the texters.  Time to put on my tough guy dad hat.  I texted one of the numbers with “who are you and what do you want?” and was about to try to identify the other number and call the Police, when her phone rang from the first number. My daughter looked at me and handed me the phone.

“Hello,” I said in the strongest dad voice I could muster.  (It’s not the weapons you bring to the fight that count, it’s what the other guy thinks you bring to the fight that counts.)

To my surprise, the caller was a woman and she was just as mystified as my daughter.  She said she received a text message about staying and hanging out from this number, but had no idea what that meant or what was going on.  She knew my daughter’s name because my daughter used it in another message in the conversation thread.  The mystery was, how did this unrelated third party end up with a copy of part of my daughter’s half of a conversation with the other mom?

Curious, we called the other number.  That was also a woman, but she thought my daughter was a guy sending inappropriate advances.  What does “hang out” really mean anyway?  We had a long talk and cleared it up.

Apparently, my daughter’s cell carrier, T-Mobile, had a text message routing problem and sent copies of text messages to unintended phone numbers that night.  Imagine receiving a text message about hanging out and don’t feel overwhelmed with no other context from a strange phone number.  But this time, there was no cyber-attack, no stalkers, no perverts.  Just a T-Mobile tech glitch with suspicion layered on top.

Lessons?  Yup, a few.

• It’s not always a cyber-attack or an evil cyber-stalker.  Sometimes it’s a tech glitch.
• If you want to share intimate messages with somebody, best to do it by voice or face to face.  Text messages can be mis-routed.  I saw it first-hand.
• And “who dis” must be a common text greeting.  I need to learn a new language.