What made me care about IT security and why you should too

I’ve been asked many times why I care about IT security.  It started in earnest for me way back in 2000 when somebody invaded my house.  I first published this story in the February, 2001 edition of Enterprise Linux Magazine.


International Terrorism in Minnesota

I’ve written extensively in this column about a small Linux DNS server I run.  Imagine my surprise a few weeks ago when I found my system launching a denial of service attack against the Government of Brazil.  That set a chain of events in motion every bit as traumatic for me as the recent Presidential election was for everyone else.

It all started when I tried to access my email.  For some reason, the response time was unbelievably slow.  About that time, my wife complained she couldn’t get to the Martha Stewart Web site, or anywhere else on the Internet, and what did I do to the computers this time?

I started investigating and found my house LAN was indeed running very slow.  I looked at my hubs and found port 4 on one hub going nuts.  This was the port leading to my DNS server.  The ps –ax command showed me the following process:

ping -s 65000 -f nn.nn.nn.nn (I won’t share the target IP address.)

My DNS server was sending 65,000 byte packets as fast as it possibly could to a system across the Internet.  When I killed the process, performance went back to normal.

A feeling of dread came over me and my adrenaline started pumping.  Then I got mad as I realized some jerk broke into my DNS server and set up this attack.  Fortunately for the Internet, I don’t have enough bandwidth for anyone significant to seriously care about.  Unfortunately for me, this jerk found out where I am and how to break in to my network.  I felt violated, angry, and afraid all at the same time, especially when I thought about all the data I have squirreled away in various directories on computers all over my network.  I wanted to find this jerk and strangle him or her, but I didn’t have the tools to even know where to begin.

So I called my friends at Mission Critical Linux for help.  I explained the situation and we all agreed that somebody had compromised my system.  I learned a lot about network break-ins that day.  I learned that BIND 8.2.2-P5, the version of DNS bundled with Red Hat Linux 6.1, has “hundreds” of security vulnerabilities, and that Red Hat keeps a list of bug fixes and updates on its web site.  I should have periodically checked for these updates.

I learned to shut down services such as sendmail, telnet, and ftp because they serve no useful purpose on this machine.  Sendmail uses its own process while the inetd process controls ftp, telnet, and others.  These commands ensure they won’t start at boot time:

/sbin/chkconfig –level 345 sendmail off
/sbin/chkconfig –level 345 inet off.

That’s when I remembered that telnet had been behaving strangely.  When I tried to connect via telnet, it wouldn’t echo anything and lately would just tell me the process was ending.

The support person laughed and told me I’d been suckered by the oldest trick in the book.  Somebody probably replaced the real telnet with a fake version designed to steal passwords for later transmission to the bad guys.  The system had definitely been compromised.

The technical recommendation:  Wipe the hard drive and rebuild the system from scratch.  The next recommendation:  Call the FBI immediately because the IP address my system attacked belongs to the Brazilian National Government, and I could face legal trouble if I didn’t report it.

As soon as we hung up, I called the Minneapolis FBI office and asked for somebody who deals with computer crime.  The receptionist sent me to a lady.  The conversation went like this:

Greg:  “Hi – I need to report a computer crime.  Somebody broke into my DNS server and launched a denial of service attack against the government of Brazil.”

FBI Lady:  “Wait a minute.  Did you say D-E-S server?”

Greg:  “No, a DNS server.”

FBI Lady:  “Oh – D – N – S, OK.  What did they do to your computer?”

Greg:  “Somebody tried to use my computer to attack a computer that evidently belongs to the Brazilian Government.”

FBI Lady:  “OK, . . ., who did it?  Do you have their address?”

Greg:  “No.  See, a DNS server translates names to addresses on the Internet.  One of my computers is a DNS server and somebody out there on the Internet tried to use my computer to attack this other computer in Brazil.”

FBI Lady:  “OK, but we need to know who did it.  We need a name or address or some way to find this person.”

Greg:  “Well, I was kind of hoping you guys could help me figure that out.”

FBI Lady:  “There’s not much we can do if we don’t know who broke into your computer.  Don’t you have any idea how to find this person?”

Greg:  “I wish.  See, the Internet is a whole bunch of computers all around the world and they’re all connected to each other.  Somebody on one of those computers found my computer and made it do this attack.  Since all these computers are connected to the Internet, we don’t know if the attacker is next door or across the world someplace.  But maybe they left some clues inside my computer to help track them down.”

FBI Lady:  “OK, let me get your phone number and somebody will call you back.”

Greg:  (after giving my phone number)  “Any idea when I’ll hear from somebody?”

FBI Lady:  “No.  They’re all pretty busy, ya know.”

Greg:  “Thanks.”

I made that call on Tuesday, Nov. 11, 2000 at roughly 1 PM central time.  I called again at 4:30 PM the same day.  As of this writing on December 15, 2000, I still haven’t heard back from the FBI.  I don’t mean to complain, but I was hoping the FBI would be sharper than that.

I’ll share how I rebuilt my DNS server and a list of helpful books in a future column.


I realized later, I made a mistake on my dates in the article.  Nov. 11, 2000 was a Saturday.  I know I called the FBI on a Tuesday, so the correct date would have been either Nov. 7 or Nov. 14.  To this day, I have no idea how I came up with Nov. 11 for a date in the original article.  But this key detail gave me an insight into how the FBI works.

My phone rang one morning in Feb. 2001, a few days after the article ran.  It was a manager in the Minneapolis FBI office and he wanted to troubleshoot.  I thanked him for the call, but said I could not afford to shut down my life and wait three months for a callback from law enforcement.  I had long ago wiped and rebuilt that system.

That’s when he went into CYA mode.  He said that since I called on a Saturday (remember, I really called on a Tuesday) I must have connected to a weekend operator.  That was why they had no record that I had ever called.  Yeah.  Uh-huh.  My tax dollars at w0rk.

Lesson learned – law enforcement is of little or no value in data breach scenarios.  Over the next several years, I would learn that lesson a few more times.

Here is why everyone should care about incidents like this.  Somebody exploited a flaw in one of my public facing systems to invade my house and use me as a drone in their attack against a third party.  Although nobody physically tramped through my house, the net result was the same–I was violated.  And I was on my own to fix it.  How many times since have we heard variations on that story?

If you’re running a business and somebody violates your company IT systems, the odds are slim that anybody from law enforcement will help you.  If you’re an individual consumer, the odds are even slimmer.  So read books like “Bullseye Breach” to educate yourself on how these violations happen, read earlier posts in this blog, and keep an eye on future posts for ideas to reduce your attack surface.

If you bury your head in the sand, don’t be surprised when somebody kicks your exposed rear-end.

A few security FAQs

Here are a few FAQs (frequently asked questions) about Internet security.  I should have put this together a long time ago.

Q: I don’t keep national security secrets inside my computer or cell phone. Aren’t all these so-called security products the real scam?

A: You probably don’t have any secrets anyone cares about.  But the game is not to steal your secrets.  The real game is to make you an unwitting drone in a scheme to steal somebody else’s secrets.  You spent money for your computer equipment and you spend money every month for Internet and cell phone service.  If you don’t care about somebody using you for criminal projects, then don’t protect yourself.  You are either part of the solution or part of the problem.

Q: Why don’t all those lonely teenage hackers get a life?  And why are the most powerful companies in the world at the mercy of a few evil computer genius hackers?

A: These are the wrong questions to ask.  The image of a lonely teenage boy in his bedroom stealing national security secrets for fun might play well in Hollywood, but it’s not real. So are the images of an evil computer genius threatening to destroy the world by guessing the secret password and typing a few commands, and the good guy genius who saves the world in the nick of time. Most of the bad activity these days comes from organized criminal organizations or nation-states, not any single individual. Those powerful companies are vulnerable because the people charged with keeping them safe did not do their jobs.

Q: If there are no evil computer genius hackers, then why do we see almost daily reports of cyber breaches?

A: I didn’t say there are no evil geniuses, only that the Hollywood images are wrong. There are plenty of evil geniuses in the world, but they are only a small part of an entire global criminal industry.  Just like legitimate industry, the shadowy Internet criminal industry has venture capitalists, inventors, markets, tech support services, and specialists for every conceivable discipline.

Q: Why are we all such sitting ducks on the Internet and why doesn’t somebody do something about it?

A: Just like humans developed an overwhelming advantage over other animals on our planet by developing language, bad guys currently have an advantage over good guys because bad guys collaborate better than good guys.  Business and government can erase that advantage by bringing security practices out into the open and giving them more than lip service.  We can influence policy by educating ourselves and using our market power to support organizations with good security policies.

Q: Is it true that my Internet connected baby monitor can destroy the Internet?

A: No, not by itself.  But combined with millions of other poorly designed IoT (Internet of Things) products, it can wreak plenty of havoc.  When you buy Internet connected devices, such as baby monitors, DVRs, security cameras, door locks, thermostats, ovens, you name it, make sure they have a mechanism for updates in the field.  Make sure you don’t use factory default passwords and make sure they don’t have default passwords or other back doors permanently baked into the hardware.  And put them all behind a credible firewall.

Q: Speaking of firewalls, since all my stuff is behind a firewall, doesn’t that mean I’m safe?

A: No.  Firewalls are one part of a bigger picture.  They stop unsolicited traffic.  Firewalls are worthless when you invite the traffic in.  That’s why it’s important to be careful about what websites you visit and avoid opening email attachments.  And that’s why you need antivirus software, even if nobody has a perfect antivirus solution.

Q: Today’s high tech is boring and complicated.  Why can’t they just make this stuff simple and usable?

A: They is really us.  Spend more time with security, where technology and psychology meet and the results are fascinating.

Q: Where can I find an entertaining story about how major data breaches play out?

A: One great perk about my own blog: I get to plant great lead-in questions.  Here is a shameless plug for my first book, “Bullseye Breach,” an educational book about data breaches disguised as a thriller novel about how the Russian mob penetrates Minneapolis retailer, Bullseye Stores, and steals 40 million customer credit card numbers.  Here is a two minute silent video about how that attack unfolds.

And stay in touch for information about book #2 coming soon.  This time, a nation-state really does mount an attack.  And the stakes are much higher than credit card fraud.

Our political leaders set a sorry security example

I am constantly amazed by how much cyber-security effects our 21st century lives every day, and by how clueless our leaders on both sides of the political isle are about all of it.

Let’s start with Hillary and the Democrats.  I’ll dump on Trump and the Republicans in a minute.

First up is Hillary’s email server.  I’ve said over the years that I have no problem with Hillary running her own email server.  And, given what we’ve since learned about US Government security with stories like the OPM breach, I might have run my own email server if I were in her position.  One difference – I know more about running an email server than Hillary.

Whether or not what she did is criminal is still being argued, but we all learned she was, at minimum, wildly careless handling sensitive information.  A United States Secretary of State should know better.  Her reaction?  Double-down on ignorance.  Check out this piece from The Daily Beast here.  Another link to the embedded Youtube video here.  At around the 1:05 mark, the reporter asks Hillary about wiping her email server.  Her reply – “You mean, like with a cloth or something?”  Arrogant, ignorant, and proud of it.  A dangerous combination.  The FBI report came out this summer (2016).  I posted thoughts about FBI Director Comey’s announcement here.

Check out FBI Director Comey’s announcement, where he describes how an army of FBI professionals needed a year to painstakingly comb through that server hard drive to recover thousands of deleted messages.  Why were they deleted?  Only one explanation holds up: Hillary must have ordered her email administrator to uninstall Microsoft Exchange and delete the datastore, but nobody wiped the deallocated space.  A rookie mistake?  Or a bungled coverup?  How much would an enemy of the United States pay for a copy of the discarded hard drive from the Secretary of State’s email server?  So, yeah, wildly reckless is a charitable characterization.

Although there is no evidence Hillary’s email server was ever penetrated, apparently the Russians did penetrate the Democrats’ email server. And now the whole world sees a daily barrage of  embarrassing, private messages, courtesy Wikileaks.  And in the process, we’ve now legitimized Wikileaks, even though its leader is currently holed up in the Ecuadorian Embassy to block extradition for sexual assault.  Full disclosure here – I have personal experience with Wikileaks.  Here are details.

And that leads to Donald Trump, chief Wikileaks legitimizer.  The Donald, maybe our next President, who fires apprentices for making weenie excuses for failure.  So how did Trump Industries handle its data breach last year, when it exposed thousands of its own customers to credit card fraud?

Like virtually every other company these days, we have been alerted to potential suspicious credit card activity and are in the midst of a thorough investigation to determine whether it involves any of our properties,” the statement reads. “We are committed to safeguarding all guests’ personal information and will continue to do so vigilantly.”

I added the italics for emphasis because it was a weenie excuse.  Read the July, 2015 krebsonsecurity.com story here, and the Krebs followup October, 2015 story here.

It gets worse.  Krebs reported a second data breach in April 2016.  Article here.

That’s right.  Anyone who stayed in a Trump hotel through most of 2014, 2015, and early 2016 should consider calling their bank and requesting a new credit card.

And now, the ultimate in irony.  “We’re so obsolete in cyber,” Trump told The New York Times. “We’re the ones that sort of were very much involved with the creation, but we’re so obsolete.”

Donald said that in March, 2016.  Now it’s October, 2016 and we all recently learned how right Donald was.  Although not in the way he intended.

The news broke on Monday, Oct. 17 when security researcher, Kevin Beaumont, did some simple probes of publicly available data and found that the Trump organization uses Windows 2003 with Exchange 2003 as its email server.  Here is a ZDNet article with details.  Here is a Vice News article with more.

IT professionals’ jaws should be dropping right now.  For the uninitiated, as of October, 2016, Windows 2003 really is 13, count ’em, 13 years old.  Which means today’s 7th graders weren’t born yet when Windows 2003 first became available.  Microsoft no longer supports Windows 2003 and no longer issues security updates.  Which means the Trump public facing email server is the Internet equivalent of a large rob me sign taped to the front doors of all Trump properties.  Which may explain why criminals were able to so easily steal thousands of customer credit card numbers from Trump Industries, not once, but twice.

And it gets worse.  Trump’s response is nonsense.

“The Trump Organization deploys best in class firewall and anti-vulnerability technology with constant 24/7 monitoring. Our infrastructure is vast and leverages multiple platforms which are consistently monitored and upgraded using current cyber security best practices.”

Defending the choice to continue operating a hopelessly obsolete email server because it’s behind a firewall is like changing the car oil to compensate for bad tires.  The Trump response demonstrates an amazing lack of basic understanding about what firewalls do – and don’t do.

I wonder if Trump will still be a Wikileaks supporter when his private emails start showing up in newspaper headlines?

And finally, we learn that Republicans and Democrats do share some common ground in this divisive election year.  They’ve both been breached.  The Democrats lost emails and the Republicans lost credit card numbers.  Anyone who purchased anything from the Republicans between March 2016 and the first week of October should contact their bank and ask for a new credit card.  Details here.

If you’re a political candidate or an organization decision maker, listen up.  Based on what I’ve seen, you probably don’t know nearly as much as you think you know about cyber-security.  So accept my shameless book plug and consider buying a copy of “Bullseye Breach,” right here.  You’ll be entertained and you’ll learn how this stuff really works and what you can do to stop it.

I’m also looking for an agent and publishing partner for book #2, where a nation-state really does attack the United States.  More news on that as it gets closer to publication.

Hillary and respect for IT and her email server

By now, everyone knows about yesterday’s FBI announcement about the Hillary Clinton email server investigation. James Comey’s words, “extremely careless” were widely quoted. As expected, the Trump camp responded with much sound and fury, signifying nothing. And the Hillary camp responded by claiming vindication. Both camps are wrong. What a surprise.

I downloaded and read a copy of the transcript and listened to a recording of the whole announcement today. Read this paragraph from the Comey statement:

“I have so far used the singular term, ’email server,’ in describing the referral that began our investigation. It turns out to have been more complicated than that. Secretary Clinton used several different servers and administrators of those servers during her four years at the State Department, and used numerous mobile devices to view and send email on that personal domain. As new servers and equipment were employed, older servers were taken out of service, stored, and decommissioned in various ways. Piecing all of that back together — to gain as full an understanding as possible of the ways in which personal email was used for government work—has been a painstaking undertaking, requiring thousands of hours of effort.”

I said earlier that if I were in Hillary’s shoes back in 2009, I might have put in my own email server too. I haven’t heard anything to change my mind, especially given what we’ve learned recently about government data breaches.

The email server isn’t the issue. The real issue is respect. Why does somebody use several different servers and administrators over four years? As somebody who delivers server administration services, I can think of only one reason – she was either an unreasonably demanding customer or she hired amatuers willing to work cheap.

Good email administrators are professionals and the former Secretary of State should have respected the professionals she hired for this purpose – not switched them out like changing clothes. I would love to talk to a few of the people she brought in and then got rid of. Were they professionals that she treated badly or were they amatuers who didn’t know what they were doing? Either answer is bad for Hillary.

What about Trump? He continues to make a fool of himself and too many Americans are too willing to follow him off a cliff.

For the first time in my life, I’m faced with two awful choices for President. Maybe a 3rd alternative with a credible chance of winning will come along.

Sometimes it’s not a cyber-attack

It’s good to know that at least my family listens to my constant Internet safety lectures.  I wish more business leaders would do more than talk about taking security seriously.  I am under constant cyber-attack.  Every single day, more than 100 phishing emails hit my inbox.  Some are clever.  One cussed me out for sending a bogus invoice, conveniently attached to the message.  Another cussed me out for not paying an invoice, also attached.  Many claim to come from UPS or USPS or Amazon with news that the package I was expecting had a delivery problem.  Open the attachment for details.  My “Bullseye Breach” book website regularly comes under attack, most recently from a Russian IP Address.  Since “Bullseye Breach” is a book about how Russians steal forty million customer credit cards from a large retailer named Bullseye Stores, I guess the only surprise is that it took the Russians so long to attack it.

So when my daughter came to me with strange cell phone behavior, I knew it had to be another attack.

She was trading text messages with another mom to set up a play-date for my grandson.  The other mom offered to have my grandson over to her house to play with her son, and my daughter offered to stay and help.  Boys can be rambunctious when they get together.  This was one of my daughter’s messages, quoting with permission:

“Sounds good.  I am cool with staying and hanging out if you want.  I just don’t want you to feel like overwhelmed or anything.”

The other mom responded and they continued their text conversation.  I still don’t see the appeal of text messages as a primary form of communication.  Those teeny tiny keys and auto-correct drive me nuts.  If we’re both tapping little buttons on a phone, why not just talk to each other?  Maybe it’s a generational thing.

In the middle of her conversation with the other mom, two identical text messages from two different unknown local phone numbers came in.  The messages were, “who dis?” followed by forwards of my daughter’s messages to the other mom.

Shocked and afraid, my daughter asked me to help figure out how somebody invaded her phone.  Why was somebody stalking her from two different phone numbers and taunting her with her own text messages?  How did some lowlife intercept her text messages and play them back for her?  What did they want?

I was curious myself.

Looking over the conversations, the texters knew my daughter’s name and the date and time she planned to meet the other mom.  But we knew nothing about the texters.  Time to put on my tough guy dad hat.  I texted one of the numbers with “who are you and what do you want?” and was about to try to identify the other number and call the Police, when her phone rang from the first number. My daughter looked at me and handed me the phone.

“Hello,” I said in the strongest dad voice I could muster.  (It’s not the weapons you bring to the fight that count, it’s what the other guy thinks you bring to the fight that counts.)

To my surprise, the caller was a woman and she was just as mystified as my daughter.  She said she received a text message about staying and hanging out from this number, but had no idea what that meant or what was going on.  She knew my daughter’s name because my daughter used it in another message in the conversation thread.  The mystery was, how did this unrelated third party end up with a copy of part of my daughter’s half of a conversation with the other mom?

Curious, we called the other number.  That was also a woman, but she thought my daughter was a guy sending inappropriate advances.  What does “hang out” really mean anyway?  We had a long talk and cleared it up.

Apparently, my daughter’s cell carrier, T-Mobile, had a text message routing problem and sent copies of text messages to unintended phone numbers that night.  Imagine receiving a text message about hanging out and don’t feel overwhelmed with no other context from a strange phone number.  But this time, there was no cyber-attack, no stalkers, no perverts.  Just a T-Mobile tech glitch with suspicion layered on top.

Lessons?  Yup, a few.

  • It’s not always a cyber-attack or an evil cyber-stalker.  Sometimes it’s a tech glitch.
  • If you want to share intimate messages with somebody, best to do it by voice or face to face.  Text messages can be mis-routed.  I saw it first-hand.
  • And “who dis” must be a common text greeting.  I need to learn a new language.

We all need Apple to win the FBI encryption dispute

In December, 2015, two terrorists in San Bernardino, California, committed a horrific and gruesome crime when they murdered 14 coworkers and seriously injured 22 others.  Law enforcement caught up with these murderers a few hours later and they died in a shootout.  Good riddance.  If you commit an act of terrorism, you deserve the harshest consequences society can offer.

But this blog post isn’t about terrorism.  It’s about the aftermath these terrorists left behind in an encrypted Apple iPhone 5c.  Three months later, the FBI is unable to break into that phone to examine its contents.

The phone belongs to the San Bernardino County Department of Public Health, where one of the murderers was an employee, and is now in FBI custody.  The phone’s contents are encrypted and the phone may be set to brick itself after a small number of penetration attempts. Apple itself has no way to access it.  Here is a blog post with details.

The FBI wants Apple to engineer a special firmware update for this specific phone to allow the FBI to bypass the phone’s security and look at its contents.   The FBI secured a court order compelling Apple to cooperate.  Here is a PDF with the FBI motion.  Here is a PDF with the court order.  Apple CEO, Tim Cook, expressed Apple’s opposition to the order in an open letter, published on the Apple website.  Here is a PDF copy in case the website link goes bad.

And now the fight is on.  It’s the long awaited clash of privacy rights versus counter terrorism.  And although I question the value of anything stored inside the specific phone at the center of this fight, the big picture stakes could literally be life and death.

Naturally, politicians are weighing in.  In this article Donald Trump called Apple “disgraceful.”  Trump also said, “We should force them to do it. We should do whatever we have to do.”

And in a USA Today Opinion piece, Senator Richard Burr, R-NC, Chairman of the Senate Select Committee on Intelligence, said, “The newest Apple operating systems allow device access only to users — even Apple itself can’t get in. Murderers, pedophiles, drug dealers and the others are already using this technology to cover their tracks.”  Here is a PDF of Senator Burr’s article in case the link goes stale.

On a personal level, if my family or friends were victimized by a terrorist attack, I would do everything in my power to gather information to bring the attackers to justice, and if encryption got in my way, I would bust whatever heads I needed to break it, legal or not.  I would not care about bigger policy issues.

But when I look dispassionately at the bigger picture, I am forced to conclude Apple is right and the FBI is wrong.  And the longer I look at the issue, the stronger my convictions become.

Tim Cook framed the Apple arguments around privacy and trust and a slippery slope to tyranny.  And his arguments are persuasive and right on.  But the arguments so far on both sides miss a larger point – the perceived trade-off pitting privacy against law enforcement is not the most important issue.  More important than any trade-off, weakening encryption, even to break into this one phone, hurts the fight against terrorism more than it helps and the government has no business trying to compel companies to break the security of their products.

Apple could break into this one phone and maybe the government might uncover a few names.  Maybe.  But at what long term cost?  Senator Burr, if Apple loses this fight, then murderers, pedophiles, drug dealers, and others will simply find another encryption tactic to cover their tracks.  If the government wins this skirmish with Apple, we will all pay a long term price in the more important war against crime and terrorism.

In a Feb. 19, 2016 interview on CBS This Morning, Assistant New York City Police Commissioner, John Miller, took Apple to task by asking Apple how many victims in Paris and San Bernardino were Apple customers.  Miller is right to frame the debate in life and death terms.  So my question for Miller, if I ever get a chance to ask is, how many more people will die if law enforcement forces tech companies to weaken encryption?  How does it make sense to cripple the good guys when the bad guys won’t follow the rules?

Like it or not,  strong encryption is here to stay.  It’s a fundamental part of 21st century society.  We can no more roll back encryption than we can replace cars with horses and buggies.

Don’t believe me.  Just use recent history as a guide.

For many years, the open source IPSEC community refused to accept contributions from US citizens because of the threat of US Government regulation.  So encryption technology continued to progress, just without input by the United States.   If we return to that broken way of thinking, we will blind the United States when dealing with our enemies.  Not only will we not be able to decipher encrypted communications, we may not even know they’re going on.

I spent 16 months writing and publishing “Bullseye Breach,” an educational book disguised as an international fiction story about how Russian criminals steal 40 million credit card numbers from a large US retailer named Bullseye Stores.   No amount of government regulation inhibiting or regulating encryption would have helped in any real-world breach scenarios, and the arguments suggesting the government act as a safe storage location for encryption keys has more holes than Swiss Cheese.  Just ask any victim of the recent OPM breach about the safety of US Government servers.  If people who apply for security clearances can’t trust the United States Government with private information, why should the general public trust the Government with millions, perhaps billions of encryption keys?

That’s why Apple must win this fight. To stop a first step down a slippery slope and keep the playing field level between the good guys and bad guys, so the good guys have a chance to fight back.  Crippling encryption cripples the good guys.  It delivers exactly the opposite result the government and all of us want.

One final note:  I am now a Red Hat employee.  For people unfamiliar with the tech industry, Red Hat is the preeminent open source software company and is rocking the IT industry.  The opinion expressed here is mine, and may not reflect what the leaders at Red Hat think.  But I’m right.

How a printer inkjet cartridge clobbered one of the world’s largest retailers

My wife, Tina’s smartphone gave us three good years of life.  But with crashes and hangs and a battery that would not stay charged, it was time for a new phone.  I know I can buy phones and supplies directly from my carrier, but  every time I compare equipment prices, Sams Club is less expensive.  And that’s what led us to our local Sams Club store a few days ago.

Charles in the cell phone department helped us select a phone and set it up.  All went smoothly.  Just one more thing to do – print and sign the paperwork.  Charles clicked the “Print” button on his POS system and that started a chain of events that destroyed this store’s ability to sell phones and tablets.   But it would take us more than two hours to come to that conclusion.

When the paperwork would not print, Charles checked the Lexmark CS510de printer at this station and found it needed a new black inkjet cartridge.  Charles found one, but it didn’t fit.  He looked all over the store, but there were none.  Why not just send the paperwork print job to another printer in the store?  Because this POS system was locked down and nobody had permissions to set up different printers on it.  The POS system and this network connected printer were mated.

Time for plan B – just swap printers with another station.  Bring in a different printer and assign it the same IP Address as the original printer.  Two roadblocks with this.  First, this was the only Lexmark CS510de printer in the store.  But there were other Lexmark printers.  Maybe one of them was close enough that the driver would be compatible.  So there was a way around the first roadblock.

Navigating through the menus on the printer touch panel, we ran into the next roadblock.  We needed to look up the IP Address on the original printer so we could configure the replacement.  Unfortunately, we quickly found that this printer prohibits looking up any network parameters when it’s not ready.  And it can only be made ready when it has good inkjet cartridges.  No inkjet cartridge, no IP Address.

Insert my editorial question here.  If anyone from Lexmark reads this post, what were you guys thinking and who made the boneheaded decision to prohibit accessing any network parameters when the printer is offline?

With no way to look up the IP Address for the original printer from either the printer touch panel or the locked down POS system, it was time for plan C – find an inkjet cartridge for this printer.  Somewhere.  Surely it was in stock at a store in the area. Right?  Charles jumped in his car and tried the nearby stores and came up empty.  With no replacement inkjet cartridges and no ability to look up the IP Address of the original printer, there was no way to fix the original printer or set up any replacement to mimic it.

Why not just call the corporate IT Department?  Surely they had tables of which IP Addresses belonged to what devices.  But by now it was evening Central time on Friday and nobody was available in Corporate IT to take a call.

Somewhere during all this, the department manager was also involved.  And two other department members.  And me, tinkering with printer settings.  The problem consumed three employees and a customer for two hours.

A brand new Apple phone was available for sale starting Saturday, but this department could not sell anything with a broken printer.  So now the problem was bigger than just my paperwork.  We tried to fix it.  But sometimes the obstacles are insurmountable.  The manager gave me a $50 gift card for my trouble and apologized.

What could Sams Club have done differently to stay out of this predicament?  Having a few spare inkjet cartridges is the obvious answer.  But more important, who designed a business system that depended on finicky inkjet cartridges that can fail at any time?

The system should have had redundancy built in with every component.  A spare printer, a spare POS system with the ability to print to a default printer and alternates, and a standard printer model at all stations throughout the store.  Why use an inkjet printer in a high volume printing station?  Why not a real laser printer?  And why use a printer that prohibits looking up its IP Address?  Where was the corporate IT Department?  Sams Club stores open at 7 AM on the US East coast and close at 9 PM on the US West Coast.   Why was nobody in IT available during prime time Central timezone shopping hours who could address the situation?

I wonder how much in sales this store lost, especially on the day of a major Apple announcement, because of a depleted printer inkjet cartridge?

The fallout continues on the OPM data breach

It seems the Chinese plundered the United States Office of Personnel Management (OPM) at will for at least a year.  Here is my original blog post about this nightmare.

If you’re a CEO of a major organization and you still think Internet security is abstract and doesn’t apply to you, I hope you have a nice retirement package set aside.  Don’t believe me?  Just watch the ongoing revelations about the OPM data breach.  The news just keeps getting worse.  The latest tally is 22 million people.  It’s the biggest and maybe the worst data breach in US Government history and it cost Director Katherine Archuleta her job.  I imagine a few more heads will roll over the next few days.  Here is a link to a NY Times article with details.

Want to see one of the best examples of government CYA in action?  Take a look at this press release from a company named Cytech.  PDF here in case the original link goes bad.  Apparently, a Cytech April 2015 demo uncovered a set of unknown processes on some Windows systems.  I’m guessing they were Windows systems – none of the reports overtly mention it.  Cytech worked with OPM to chase down those processes and the rest is history.

But wait – a sales demo uncovering the worst data breach in US Government history makes high government officials look bad.  Spokespeople to the rescue.  Here is a Fortune article with the response from OPM spokesman Sam Schumach.  PDF here in case the link goes bad.  I’ll quote Sam’s first sentence:

“The cyber intrusion announced last week affecting personnel records for approximately 4 million current and former federal employees was discovered through enhanced monitoring and detection systems that OPM implemented as part of an aggressive effort in recent months to strengthen our cybersecurity capabilities. …”

You can read the rest in the Fortune article.

Pause for a minute.  Beyond CYA posturing, what are the real-world consequences of this debacle?  Well, for one thing, personal information for everyone who applied for a US Government security clearance since 2000 is now compromised.  If you applied for a US Government clearance and you contacted somebody in a hostile country who helped the United States, it’s likely the Chinese learned about it back in 2014.  Do I need to connect any more dots?  Still think IT security is abstract and doesn’t apply to you?  Real, flesh and blood people who wanted to help the good guys may have died because the United States Federal Government only paid lip service to taking your security seriously.

Now back to CYA posturing.

I’m not sure I would want to be in Cytech CEO Ben Cotton’s shoes right now.   Imagine this scenario.  A large government agency invites your company to do a sales demo for your flagship product.  You spend days, weeks, maybe months and a fortune in investor private sector money preparing.  You put it all at risk because that’s what we do in the private sector.

And it seems to pay off when you unexpectedly uncover a huge mess.  And then you help remediate the problem because it’s the right thing to do.   Word leaks out, speculation runs rampant, and you feel forced to do a press release in response because everyone is naming your company anyway.  But now the people running the agency that invited you in look bad and they put out their own statements contradicting you.  What are the odds you’ll earn a sale from your hard work?  No good deed goes unpunished.

And there’s more.

After the news about the breach came out, OPM offered free credit monitoring for victims.  The questionable value of this free credit monitoring is well documented, and once the monitoring period ends, then what?  But forget about that – how did OPM notify victims?  By sending an email with a “click here” link.  To millions of Federal employees.

Why is that significant?  Because that’s how phishing schemes operate.  “Dear customer.  We at your bank found an irregularity.  Please click here to make it all better.”  Bla bla bla.  Except the email didn’t come from your bank, it came from a con artist on the other side of the planet who wants to plunder any information in your computer.  It’s one of the oldest and most well known con-jobs on the Internet.  And people still fall for it.  See my blog post, “How to spot a phishy email,” for more.

So guess what?  Almost immediately after OPM sent its “click here” email, scammers and spammers duplicated it and sent identical emails with their own “click here” links pointing to their own shady websites.  Take a look at these articles, here and here.  Talk about rubbing salt in the wound.

Now take a look at this link.  It’s the National Institute of Standards’ cybersecurity framework.   That’s right.  The United States Federal Government literally wrote the book on cybersecurity.  And keeps it updated.  It’s a shame the leaders at the United States Federal Government HR office apparently didn’t read it.

Finally, if you’re mystified and curious how these breaches happen at the grass roots, and if you’re not, you should be, take a look at my new book, Bullseye Breach.  Here is a link.  It’s a story about how a fictional large Minneapolis retailer named Bullseye Stores loses 40 million credit card numbers to some Russian crooks.  I used fiction as a vehicle because the world already has enough how-to books that nobody reads.  So I used fiction and a compelling story to hopefully keep your attention.  Every CEO should read this book – it might save you from putting out a press release explaining how you take security seriously after a major breach.

 

 

Here we go again – another sensational data breach

Here is yet another data breach headline, published yesterday (July 1, 2015) by Brian Krebs.  Here is the link to the article.

Who is the latest victim?  None other than The Donald himself.  It seems the banks uncovered a trail of credit card fraud leading right back to Trump Hotel properties.  This one has apparently been going on since Feb. 2015.

We’re early in the cycle of this latest sensational data breach, but they all follow the same pattern.  Watch for it with this one.  Here’s how they unfold.

  1. Lax or dysfunctional management ignores all the warnings about potential IT security problems.  Those techies – all they want to do is spend money on tech toys.  We sell hammers or hotel rooms or clothes.  Or we’re a Government HR department.  Or we make movies.  We’re not a tech company.
  2. A sensational news story hits the wires.  Millions of credit card numbers stolen!  Personal information stolen by the Chinese!  Fortune 500 company brought to its knees!
  3. The CEO or other leader of the breached organization puts out a press release.  “We take our customers’ privacy seriously.”  The press release includes a generous offer of worthless free credit monitoring for potential victims for a year.
  4. PR teams gear up as leaders in the breached organization fill the airwaves with excuses and all the important steps they’re taking to mitigate this breach.  They use words like “sophisticated” and “criminal syndicate” or “nation state” to describe the attackers.
  5. Columnists and bloggers express outrage.  (That’s what I’m doing right now.)
  6. Lots of people share commentary about how awful this all is and the poor state of our security.  But nobody shares any specifics about conditions leading up to the breach, how the bad guys penetrated the victim organization, or the get-well steps.   (I saw one exception to this in a KrebsOnSecurity.com post about the Sally Beauty breach.)
  7. Embarrassed Boards of Directors and other VIPs outdo themselves with knee-jerk reactions as they pour a fortune into closing the barn door after the horses have already escaped.
  8. Sometimes, a major news magazine does an in-depth story about the personalities involved at the victim company a few months later.
  9. The story eventually fades away and the public is left to believe that breached companies are helpless victims of sophisticated criminal syndicates or nation-state sponsored terrorists.  There’s nothing anyone could have done about it.

Don’t believe this crap for even one second.  Every single sensational data breach we’ve read about was preventable.  Every single one.

Want to fix the problem instead of putting out CYA press releases?  Here’s what needs to happen – and it doesn’t cost a fortune.

First, a tactical step:  Improve the topology.  Put the most valuable systems behind an internal firewall with a white list and log access to it.  Notify the right people if the systems holding that critical data try to communicate outside the white list.  I have something to gain here because I build firewalls based on open source technology – re-branded as software defined perimeters because the concept of “firewall” is rapidly becoming obsolete.  My SDP appliances can compete with any system from any manufacturer and win.  Here is some more information.

Second is vigilance.  When we peel back the onion layers on these breaches, we find too many people asleep at the switch.  Or nobody minding the store.  Pick your metaphor.  The Chinese run rampant through the US Office of Personnel Management network and nobody notices traffic flying to China?  What’s up with that?  The North Koreans run rampant through Sony Pictures and nobody notices?  Let’s call this what it is – carelessness from the people who should know better.

And that leads to the third step:  Openness.  This is counter-intuitive, but organizations should publish what they do for security.  This doesn’t mean give away passwords and encryption keys.  But publish their standards and methods.  In detail.  Present at conferences, do media interviews, and open up to community scrutiny.  This is a departure from traditional large organization operating procedure and I can already hear the screams of agony:  “If we tell the world how we do security, then everyone will know and it will be worthless!”

I answer that with a question: “Given recent sensational data breach headlines, how’s the current operating procedure working out?”  Right now, only the bad guys know the relevant details and they’re plundering us.  So level the playing field.  Open it up.  The surviving encryption methods are all open and well-known.  And hardened because they’ve passed a gauntlet of public scrutiny.  Business and government should take a lesson.

Do those three things and IT security will naturally gain the attention it needs at the top levels of business and government and appropriate investments will follow.

Finally – want to read a fiction novel with a realistic story about how a sensational data breach unfolds?  Check out my new book, Bullseye Breach.

The Chinese may now have personal information on 4 Million US Government employees

Yet another sensational data breach headline – not even shocking anymore.  Yawn.  But listening to the story on the radio on the way home last night after being slaughtered in softball again, I started thinking.  And I dug a little deeper into the story when I got home.  I was shocked.

The systems penetrated belong to the US Government Office of Personnel Management.  Yep, that’s the United States Federal Government Human Resources Department.  It holds personal information for everyone who works for the US Federal government.  It’s the agency that hands out security clearances.  Think about this.  Let it sink in.

The Chinese broke into the system that US Government investigators use to store information about background checks for people who want security clearances.  That’s right.  If you applied to the US Government for a security clearance, it’s a good bet the Chinese know a lot about you now.  Which means you’ll probably be the target of some finely crafted spear phishing campaigns for the next several years.

And that’s only one system out of 47 operated by the Office of Personnel Management (OPM).  It’s not the only one the Chinese penetrated.

Update:  According to this Washington Post article, (PDF here in case the link breaks) the Chinese breached the system managing sensitive information about Federal employees applying for security clearances in March 2014.  The latest OPM breach targeted a different data center housed at the Interior Department.

Update June 12, 2015:  The original reports were bad.  Now it’s even worse.  It now seems the Chinese have detailed information on every US Federal employee.  14 million, not 4 million.  And people may die directly because of this breach. But even now, we don’t know the extent of the damage.  This article from Wired Magazine sums it up nicely.

Reactions from high government officials were typical. They all take the problem seriously.  Bla bla bla.  According to the Wall Street Journal:

“We take very seriously our responsibility to secure the information stored in our systems, and in coordination with our agency partners, our experienced team is constantly identifying opportunities to further protect the data with which we are entrusted,” said Katherine Archuleta, director of the Office of Personnel Management.

Here’s another one, from the New York Times:

“The threat that we face is ever-evolving,” said Josh Earnest, the White House press secretary. “We understand that there is this persistent risk out there. We take this very seriously.”

This one from the same Washington Post article is my favorite:

“Protecting our federal employee data from malicious cyber incidents is of the highest priority at OPM,” Director Katherine Archuleta said in a statement.

Do I really need to ask the question?  Katherine, if it’s such a high priority then why didn’t you address the problem?

As I mentioned in a blog post way back in Feb. 2014, about dealing with disclosures, we’ve heard lots of noise about this breach but very little useful information.  Here’s what we do know.  I want to thank David E. Sanger, lead author of the New York Times article, “U.S. Was Warned of System Open to Cyberattacks,” for sending me a link the 2014 Federal Information Security Management Act Audit report.  In case that link breaks, here is a PDF.

We know the Chinese penetrated the OPM in fall 2014 and stole at least 4 million records over the next six months.  That’s it. As usual, nobody I can find is forthcoming with details.

The report from the Office of Inspector General (OIG) gives us some clues.  Apparently, the various program offices that owned major computer systems each had their own designated security officers (DSO) until FY 2011.  The DSOs were not security professionals and they had other jobs, which means security was a bolted on afterthought.  In FY2012, OPM started centralizing the security function.  But by 2014, only 17 of the agency’s 47 major systems operated under this tighter structure.

All 47 major systems are supposed to undergo a comprehensive assessment every three years that attests that a system’s security controls meet the security requirements of that system.  It’s a rigorous certification process called Authorization.  Here’s what the report said:

“However, of the 21 OPM systems due for Authorization in FY 2014, 11 were not completed on time and are currently operating without a valid Authorization (re-Authorization is required every three years for major information systems). The drastic increase in the number of systems operating without a valid Authorization is alarming, and represents a systemic issue of inadequate planning by OPM programming offices to authorize the information systems that they own.”

Remote access also had problems.  Apparently the VPN vendor OPM uses claims the ability to terminate VPN sessions after an idle timeout.  But the idle timeout doesn’t work and the vendor won’t supply a patch to fix it.

Identity management was also weak.  Although OPM requires multi-factor authentication to enter the network, none of the application systems do.  So if a Chinese bad guy penetrates the network, he apparently has free reign to everything in it once inside.  And since OPM had no inventory of what systems it owned or where they were or their use, OPM had no way to know the Chinese were plundering their data.

It adds up to a gigantic mess.  And an embarrassment, which probably explains why nobody wants to talk about details.

Wonderful.  So what can a small IT contractor from Minnesota offer the multi trillion dollar United States Federal Government to address this problem?  Here are some suggestions from an outsider who wrote a book about data breaches.

Three attributes will keep our systems safe.  Sharing, diligence, and topology.

Sharing drives it all.  So first and foremost – move from a culture of hierarchy, secrecy, and “need to know” to a culture of openness, especially around security.  What does that even mean?  For an answer, check out the new book by Red Hat CEO Jim Whitehurst, “The Open Organization,” published by the Harvard Business Review.

The Chinese, and probably others, penetrate our systems because a government culture of secrecy and “need to know” keeps our teams isolated and inhibits collaboration and incentives for excellence.  It’s a traditional approach to a new problem the defies tradition.  I’ll bet the Chinese collaborate with each other, and probably also with the North Koreans.

Instead of a closed approach, adopt an open approach.  Publish source code, build communities around each of those 47 systems, and share them with the world.  To protect it better, share how it all works with the world.

And when breaches happen, don’t tell us how you take security seriously.  You’re supposed to take security seriously.  It’s your job.  Tell us what happened and what steps you’re taking to fix the problem.  Instead of hiding behind press releases, engage with your community.

And use open source tools for all your security.  All of it.  Firewalls, VPN systems, IDS/IPS (Intrusion detection/Intrusion prevention systems), traffic analyzers, everything.  Breaches occur with open source software, just like proprietary software, but when they happen, the open source community fixes them quickly. Why? Because the developers’ names are on the headers and they care about their reputations.  You won’t need to wait years for a VPN patch in the open source world.

Openness doesn’t mean granting access to everyone.  Openness means building communities around the software systems OPM uses and accepting patches and development from the community.  Community members are compensated with recognition and opportunities for paid engagements.  OPM is rewarded with hardened, peer reviewed software driven by some of the smartest people on the planet.

When teams move away from hierarchy to an open culture, diligence and topology will follow.  There is no substitute for diligence and no technology to provide it. Teach everyone to be diligent and practice it often with drills.  Reward the cleverest phishing scheme or simulated attack and reward the cleverest defense.

And topology – put layers of security in front of key databases.  Put in appropriate access and authorization controls for key databases to ensure personal information stays personal.  Consider physically segregating these database systems from the general network and setting up a whitelist for their interactions with the world.

None of this proposed culture shift needs to cost a fortune.  And in fact, in this era of doing more with less, might save taxpayer money by igniting passion at the grass roots of the OPM IT staff.

Am I proposing radical change to a government that resists change?   Yup.  So why do it?  I’ll answer that question with my own question – given the recent headlines and your own Inspector General audit reports from the past several years, how’s the current method working out?