I’ve been in the IT industry my entire adult life, so sometimes I use words and just assume everyone thinks they mean the same thing I think they mean.  I was recently challenged with the word, “redundancy.”

“What does that even mean?” asked my friend.

“It means you have more than one.”

“So what?”

“So if one breaks, you can use the other one.”

“Yeah, everyone knows that, but what does it mean with IT stuff?”

Seems simple enough to me, but as I think about it, maybe it’s not so simple.  And analyzing how things can fail and how to mitigate it is downright complex.

Redundancy is almost everywhere in the IT world.  Almost, because it’s not generally found in user computers or cell phones, which explains why most people don’t think about it and why these systems break so often.  In the back room, nearly all modern servers have at least some redundant components, especially around storage.  IT people are all too familiar with the acronym, RAID, which stands for Redundant Array of Independent Disks.  Depending on the configuration, RAID sets can tolerate one and sometimes two disk failures and still continue operating.  But not always.  I lived through one such failure and documented it in a blog post here.

Some people use RAID as a substitute for good backups.  The reasoning goes like this:  “Since we have redundant hard drives, we’re still covered if a hard drive dies, so we should be OK.”  It’s a shame people don’t think this through.  Forget about the risk of a second disk failure for a minute.  What happens if somebody accidentally deletes or messes up a critical data file?  What happens if a Cryptolocker type virus sweeps through and scrambles everyone’s files?  What happens if the disk controller in front of that RAID set fails?

Redundancy is only one component in keeping the overall system available.  It’s not a universal cure-all. There will never be a substitute for good backups.

Virtual environments have redundancy all over the place.  A virtual machine is software pretending to be hardware, so it’s not married to any particular piece of hardware.  So if the physical host dies, the virtual machine can run on another host.  I have a whole discussion about highly available clusters and virtual environments here.

With the advent of the cloud, doesn’t the whole discussion about server redundancy become obsolete?  Well, yeah, sort of.  But not really.  It just moves somewhere else.  Presumably all good cloud service providers have a well thought out redundancy plan, even including redundant data centers and replicated virtual machines, so no failure or natural disaster can cripple their customers.

With the advent of the cloud, another area where redundancy will become vital is the boundary between the customer premise and the Internet.  I have a short video illustrating the concept here.

I build systems I like to call SDP appliances.  SDP – Software Defined Perimeter, meaning with the advent of cloud services, company network perimeters won’t really be perimeters any more.  Instead, they’ll be sets of software directing traffic to/from various cloud services to/from the internal network.

Redundancy takes two forms here.  First is the ability to juggle multiple Internet feeds, so when the primary feed goes offline, the company can route via the backup feed. Think of two on-ramps to the Interstate highway system, so when one ramp has problems, cars can still get on with the other ramp.

The other area is redundant SDP appliances. The freeway metaphor doesn’t work here. Instead, think of a gateway, or a door though which all traffic passes to/from the Internet.  All gateways, including Infrasupport SDP appliances, use hardware, and all hardware will eventually fail.  So the Infrasupport SDP appliances can be configured in pairs, such that a backup system watches the primary. If the primary fails, the backup assumes the primary role. Once back online, the old primary assumes a backup role.

Deciding when to assume the primary role is also complicated.  Too timid and the customer has no connection to the cloud.  Too aggressive and a disastrous condition where both appliances “think” they’re primary can come up.  After months of tinkering, here is how my SDP appliances do it.  The logic is, well, you’ll see…

If the backup appliance cannot see the primary appliance in the private heartbeat network, and cannot see the primary in the  internal network, and cannot see the primary in the external Internet network, but can see the Internet, then and only then assume the primary role.

It took months to test and battle-harden that logic and by now I have several in production.  It works and it’s really cool to watch.  That’s redundancy done right.  If you want to find out more, just contact me right here.

I have a friend, let’s call her Mandy.  Mandy is an identity theft victim.  Mandy is not her real name because this is a private story and she wants to maintain her privacy.  She’s willing to share it, anonymously, because she read “Bullseye Breach” and she knows what I do for a living.  She’s hopeful that her story might help others in a similar situation.

For anyone who still thinks the law enforcement bureaucracy will help you when you’ve been violated in this manner, Mandy’s story will change your mind.  And hopefully this deeply personal story will help persuade you that IT security is important and you need to take it seriously.

I am privileged to post Mandy’s story, in her own words.

#####

Living in a nice neighborhood can give you a false sense of security. Maybe you know most of your neighbors and don’t think twice about leaving your windows open all day to let in cool air.  Maybe you don’t even lock your doors at night.

I’ve never been that trusting. I grew up in a South Florida neighborhood where it seemed like we were receiving flyers on a weekly basis about break-ins.

They left an impression on me. Once out on my own, I always made sure my doors and windows were locked, but turns out that didn’t matter.

On the morning of Nov. 7, 2005, someone pried open a locked window and got into my home anyway. My husband and I returned from work around the same time that evening to find our home ransacked.

The thief or thieves must have spent a long time inside because everything, and I do mean everything, that was both portable and valuable was gone. Every room in the house had been gone through.

Missing were thousands of dollars worth of electronics, including a laptop computer that contained personal information and a video camera with precious video of my son inside; all of our checkbooks and bills that had been written out but not yet sent; a set of extra keys to our house and one of the cars; and the coin collection I had been building since I was a kid.

May sound hard to believe, but it wouldn’t have been so bad if that was all that had disappeared. What’s ten times more devastating is the fact that my family also fell victim that day to what has become the number one crime in America — identity theft.

Like so many people I know, we had our social security cards and birth certificates in a fire box under the bed. The thief found the key to the box in my underwear drawer and cleaned it out.

I feel stupid for having left the key in such an obvious place, but my husband has convinced me that if they hadn’t found the key, the thieves would have just taken the whole box anyway. I should have hidden it better.

We spent all of November and December worrying about how our information was going to be used, but nothing bad happened. Then the other shoe dropped the night of January 11th.

Because of the fraud alert we put up on our credit reports after the break-in, someone from Dell Computer called our house around 10 o’clock that night. He said he had J. on the other line and was calling to confirm his identity.

My husband was not the man on the line with Dell. We were being violated again.

After hanging up with Dell, we ran our credit report and found out that a few days earlier, someone had tried to secure a home mortgage in our name.

When I got to work the next morning, I looked up our client contact at one of the credit bureaus, called her up and started asking a lot of questions. She couldn’t answer all of them, so she put me in touch with Kevin Barrows, the former FBI agent who is credited with busting up one of the country’s largest identity theft operations in 2002.

He told me, “Because you put the fraud alert up and filed a police report, you will not be liable for anything the identity thief does; but at the same time, you do need to get his inquiries and the false addresses he gave off your credit report as quickly as possible.”

That night, I embarked on another round of letter writing. The next morning it was off to the post office again.

Early on in the process, I had read an article that recommended all communications with the credit bureaus be sent certified with return-receipt. I’ve spent close to $100 sending letters that way so far.

That’s in addition to the thousands of dollars spent installing an alarm system, fixing our broken window, replacing a damaged sliding glass door; rekeying our house and car; replacing stolen documents; etc. Some, but not all of our losses, were covered by insurance.

Just when we thought we had the situation under control, my husband and I started getting calls from credit card companies calling to confirm our identity because of the fraud alert on our accounts. One after another… I lost count around 30… We would tell the people on the other line that no we did not authorize the opening of an account.

Right away after the calls started coming in, I pulled our credit reports again and found mention of multiple inquiries made by creditors we had never heard of, plus a mysterious address in Illinois added to both mine and my husband’s accounts. I called the police department in that city to report that someone at that address was fraudulently using my address to try and establish credit.

Believe it or not, the detective I spoke with actually told me they had received similar reports from others about that exact address, but there was nothing they could do because it was a federal crime. I was referred to the Post Master General, I presume because the thieves wanted to get credit cards fraudulently sent to them through the mail.

The person I spoke with took down my information and referred me to the FBI. The agent I spoke with at the FBI told me there are too many cases like mine for them to pursue all of them. They referred me back to the local police dept in the jurisdiction where the theft happened. My hometown police department basically said, “Sorry, there is nothing we can do about a crime being committed across state lines.”

I am sharing my story in hopes that I can help make the recovery process easier for someone else.

Here are the steps I’ve taken since the day of the break-in:

1. Called the police to file a report. (This is a critical step. You will need that report in order to get extended fraud alerts issued).
2. Called the credit bureaus. (Work your way through the automated menus until you find the option to get a fraud alert issued. Experian, Equifax and TransUnion are required to share information with each other, but to give yourself peace of mind, contact all three anyway. I did.)Equifax: 1-800-525-6285; www.equifax.com; P.O. Box 740241, Atlanta, GA 30374-0241

   Experian: 1-888-EXPERIAN (397-3742); www.experian.com; P.O. Box 9532, Allen, TX 75013

   TransUnion: 1-800-680-7289; www.transunion.com; Fraud Victim Assistance Division, P.O. Box 6790, Fullerton, CA 92834-6790

3.  Called the banks to get all of my accounts frozen immediately after discovering the theft. Went into the branches I do business with the morning after the break-in to get new account numbers issued; and also secured a safe deposit box to store personal information in from now on.
4. Cancelled all of my credit cards. The thieves only made off with the two they found in the fire box, but I have no way of knowing if they went through my files to get other numbers too.
5. Called all my creditors to see which ones had received payment on my accounts. Sent new checks with a letter of explanation for the lack of a stub to the others.
6. Had my mail stopped so the thief couldn’t return to the house and steal our mail. Went to the post office daily for over a month until I was able to find, purchase and install a secure mailbox.
7. Went to the Department of Motor vehicles to get new driver’s licenses issued with new numbers. We have no way of knowing if the thieves came across our old numbers when they went through our file cabinet.
8. Went to the Social Security office to request new copies of our cards.
9. Filed a complaint with the Federal Trade Commission (FTC), which shares information about identity theft with law enforcement agencies across the country.You can file a complaint with the FTC using the online complaint form at www.ftc.gov; or call the FTC’s Identity Theft Hotline, toll-free: (877) ID-THEFT (438-4338); or write Identity Theft Clearinghouse, Federal Trade Commission, 600 Pennsylvania Avenue, NW, Washington, DC 20580.
10. Sent letters to the Department of Vital Statistics in the three states in which our family members were born to get new certified birth certificates. Also had to get a new copy of our marriage certificate.
11. Once things settled down, called a few alarm companies, took bids, then hired one to install a home burglar alarm for us.
12. After receiving confirmation of the initial fraud alerts from the three credit bureaus in the mail, sent in letters requesting a 7-year extended alert along with a copy of my police report.
13. Signed up for 3-in-1 credit monitoring so I’ll know instantly the next time someone fraudulently applies for credit in our name.

#####

If anyone reading this wants to contact Mandy, just contact me and I’ll work on setting it up.

I was in a Barnes and Noble bookstore a few days go, pitching my new book, “Bullseye Breach,” to one of the folks working behind the counter.  I know all the big decisions are always made at corporate headquarters, but nobody invited me to corporate headquarters and I have to start somewhere.  So I started at this store.

While pitching for all I was worth, a lady who said she works at the Target Corporation Credit Department here in the Twin Cities walked up to the counter.  Many have suggested I patterned my fiction story in “Bullseye Breach” after the real world Target breach – I’ll leave that for readers to judge.  I had a copy of my book with me and she seemed interested.  Which helped my ego tremendously.  Those million book sales start with the first one.

We talked for a while and she said, “It’s a shame we’re all so vulnerable.  No matter how big you are, no matter how much you’re loved in the community, no matter how much good you do, a group of crooks can still break in over the Internet and do this to you.”

That triggered a diatribe from me about believing press releases and people who should have known better not doing their jobs.  I said lots of other things, most of it politically incorrect.  To my surprise, she thanked me for being passionate about this topic and even insisted on buying the copy of my book I had with me on the spot.  I walked away dumbfounded and grateful.

That encounter put a whole series of thoughts in motion.  Since I insisted that organizations can protect themselves, that being a victim to cybercrime is not inevitable, what would I do if somebody actually invited me to corporate headquarters to provide advice and counsel to the CIO?

So here is the advice I would offer.

First is topology.  Retailers, isolate your Point of Sale systems from the rest of your network and keep a whitelist for where they can interact.  This is a shameless plug, but this is my blog so I can get away with it.  Infrasupport builds firewalls using open source tools that can do this job nicely.  Here is some information.

Set up automation to notify the right people if those POS systems try to interact with anything outside that whitelist.  Other industries may have similar issues, but retail POS systems are special because untrained store clerks interact with them and they interact with payment processors across the Internet.  Their interactions with the internal network and the rest of the world need to be strictly regulated and monitored.  If the topology had been right, and the right people heeded the warnings, none of the sensational data breach headlines we’ve read about recently would have happened.

That leads to diligence.  No matter what technology is in place, there is no substitute for human diligence.  People are and always will be the last and best line of defense against attack.  Train end users to stay away from the wrong websites and not to fall prey to phishing schemes.  Run drills.  Do probes.  Test often and discuss results.

But even with the best diligence and awareness training and drills, a company with 1000 employees means 1000 potential attack vectors.  Inbound spam filtering and outbound web filtering can help, but sooner or later, somebody will visit the wrong website or click on the wrong email attachment.  That’s why the right people need to pay attention to the inevitable warning signs and take action when warranted.

Which leads to sharing.  This is counter-intuitive, but the best way to defend against attack is to share how all the defenses work.  In detail.

This comment to a Brian Krebs blog post deconstructing the 2014 Sally Beauty breach is a great example.  It was a gutsy call for Blake Curlovic to publicly share the detailed information about this breach, both in the Krebs article and in subsequent comments, and the information he shared will be invaluable to future IT Departments fighting bad guys.

In cryptography, the algorithms are public.  Everyone knows them.  That’s why we have strong cryptography today – the surviving algorithms have all been peer and public reviewed, attacked, and strengthened.  CIOs should operate similarly.  Openly discuss security measures, expose them to public and peer review, conduct public post mortem incident reviews, publish the results, and adjust the methods where necessary.

Bad guys are already reviewing, discussing, and probing security in the shadows.  Bad guys have a whole supply chain dedicated to improving their ability to plunder, complete with discussion forums and specialists in all sorts of dark endeavors.  The bad guys have unlimited time and creativity and the good guys are out gunned and out manned.

Against such an adversary, what CIO in their right mind would want to stand alone?

This doesn’t mean CIOs should call press conferences to brag about the latest security tool.  But CIOs should be visible at conferences and should contribute keynotes and other presentations in a running dialog to help continuously improve the state of the art.  They should also be engaged in online forums discussing and refining the latest ideas.  And when it makes sense to appear in front of the written and TV press, they should take the lead and use the forum to educate the public.

Smart good guys should join forces out in the open for the common good.  Contribute to and profit from a thriving marketplace of good ideas and everyone wins.

The short answer is, no.

The pro argument says law enforcement needs this tool to fight crime and terrorism, and we can build appropriate safeguards into any law to prevent abuse.  The con arguments point out the danger in granting more power to the government, suggesting that safeguards have limited value.

I’ve read through the pros and cons and concluded it’s a bad idea to grant the government power to access encrypted communications.   Nobody wants to give terrorists and other bad guys a free ride – but as many have pointed out elsewhere, bad guys will find their own ways to do encryption regardless of any US law.  So if we pass a law essentially crippling encryption technology in the United States, we hurt the good guys and help the bad guys.  Tell me how this makes any sense.  We’re all better off with a level playing field.

With a law granting the government this power, even loaded with safeguards, what’s to stop corrupt individuals from abusing it? Attempted abuses of power are already easy to find. There was a case in Minnesota a few years ago when male law enforcement professionals looked up driver’s license records for a few female troopers, politicians, and news media celebrities.  In another case, the IRS as an institution put up roadblocks to make it unnecessarily difficult for some nonprofit groups to gain tax exempt status because individuals in positions of authority apparently disapproved of these groups.  So if we grant the government even more power, imagine the possibilities for abuse and tyranny on a massive scale. It would be 1984 in the 21st century.

Some have advocated an approach combining new technologies with court approval as a safeguard against such tyranny.  The ideas essentially come down to inventing an electronic lock-box to hold everyone’s decryption keys.  Law enforcement can access the lock-box only with appropriate court orders.  The idea sounds nice, but it’s short-sighted and foolish.  Does anyone seriously believe a determined group of bad guys would have any trouble coming up with an attack against such a lock box?  Does anyone seriously want to trust our cryptographic keys with the same government that brought us healthcare.gov and sensational headlines around NSA break-ins?

But my opinion is not worth the disk space to store it. Don’t believe me? Just look at what happened to US cloud providers shortly after the Snowden revelations. Look at what happened to RSA’s credibility after the stories about RSA and the government being in cahoots started circulating.  Now imagine what would happen to confidence in the entire United States data grid if such a law were to pass.

Why would anyone trust any service provider with anything important if the government can access all of it? My private information is mine, and I choose who sees it. Not the government. And I promise you, if I have information I care enough about to keep private, I’ll find a way to safeguard it regardless of any law.

Carrie Cordero and Marc Zwillinger recently wrote a point/counterpoint article on this topic in the Wall Street Journal, here.  In case that link breaks in the future, I saved a PDF here.

There are other ways to fight back against the bad guys besides granting tyrannical power to the government.  I wrote an education book about IT security, disguised as an international fiction thriller titled, “Bullseye Breach.” Take a look at the website, right here.

Tam Henderson was a Christian missionary with roots in Minnesota, but deeper roots in a war ravaged Vietnamese orphanage.  Eternally grateful to the American parents who adopted and raised him and taught him to love Jesus, he dedicated his adult life to sharing the Gospel with Vietnamese children and their parents.

And that was why he found himself sweating on this spring day in the jungle heat of a village near Cam Ranh Bay, Vietnam.  It had been a long and fruitful day, filled with happy kids and preaching and singing, and he was eager to share pictures and video with his own aging parents back in Minnesota.  He would upload these later.  But right now, it was time to wind down and enjoy a late evening snack.  If only he could find some ice.  Tam chuckled to himself – at least I’m not knee deep in snow anymore!

Tam did not leave behind all his Minnesota roots.  He loved baseball and his Minnesota Twins and tried to catch an occasional game whenever he had some time and could connect to an Internet streaming service.  It was spring training, the eternal season of hope for all major league baseball teams, and Tam was curious about the new, young starting pitchers the Twins had acquired in the off season.  After an embarrassing season last year and the ribbing he endured from colleagues and friends stateside, anything would be an improvement.

He opened his laptop and connected to a satellite Internet service and visited www.espn .com to catch the latest updates and spring training scores.  News about his beloved Minnesota Twins was sparse that day, but an ad on the website caught his eye.  An online Internet company was offering a spring training special for softballs, bats, and gloves.

“How do these guys know I like baseball?” he thought.  “And why are they tempting me with ads for softball equipment in Vietnam? “

What Tam – and most people – did not know is, the ESPN website did not send the ad to Tam’s laptop.  ESPN sold space on the screen displaying its website to another company, which delivered the ad to Tam’s screen based on a carefully crafted profile of all the websites Tam visited over the past several months, stored in a directory deep in Tam’s laptop.  Similar to traditional television, but more sophisticated, this is how ESPN and other websites are able to offer web content for free to viewers – by also delivering ads from other websites, and the companies hosting those websites pay for screen exposure.  Anyone visiting the espn.com website, or any number of other advertising supported websites, also visits several other unnamed advertiser websites.

“This could be interesting”, Tam thought, as ideas started to form.  He had a few hundred dollars available.  What if he could equip, say, 20 kids with softball equipment and teach them the game?  These kids could teach other kids and softball could become a Christian outreach.  Baseball as a sport was becoming popular in Vietnam, why not bring a version of it right here, to this mission?  Who knows – if it takes off, maybe this could be a legacy.  He chuckled again at the thought – “ Pastor Tam Henderson, who tried to teach the Gospel, but left softball instead.“

But God is in control and maybe that’s why the ad appeared and caught his eye.  Nothing to lose by checking it out.  He hovered his computer mouse cursor over the ad and noticed the URL string at the bottom of his web browser window.  “How do those programmers understand all those symbols?  I think they put all that in just to confuse us mere mortals.”

He clicked on the ad and waited for the details to come up.  After about 30 seconds, but what seemed like several minutes, he started to grow impatient.  Give it a little bit longer.  Maybe the clouds are interfering with the satellite feed.  Finally, after what seemed like an intolerably long wait, the details behind the equipment ad came up.  “Lord, please forgive my impatience.  I know you’re in control of everything.  If it’s Your will, I would like to order this equipment and find a way to ship it here, to Vietnam.  Please give me the means to do so and kids willing to learn the game of softball and have fun.  Amen“

A few thousand miles west, in a basement in Tehran, Iran, a shady botnet master named Bahir Mustafa knew exactly what all those symbols at the bottom of Tam’s web browser window meant, because he wrote the scripts containing them.  And Tam Henderson, from a jungle in Vietnam, tenuously connected to the Internet via an unreliable satellite link, was about to execute them.  The programmers who developed the website for the sporting goods company that contracted with ESPN to display the ad on Tam’s workstation worked for a temporary staffing firm in the Philippine Islands.  With tight timetables and little money, they managed to produce a usable website barely in time for sales on spring sports.

But they took some development shortcuts.  One shortcut was leaving the site open to a cross site scripting (XSS) attack.  XSS attacks can be complex, but the idea is, when Pastor Tam clicks on a link from one website, that website returns an invisible script instructing the browser on Pastor Tam’s workstation to run a script on another, unrelated website.  Bahir Mustafa managed to create an account for himself on the sporting goods website.  He used his credentials to insert code in the appropriate “click here” field to first run a script on Mustafa’s website, before visiting the sporting goods website.

Tam noticed the script took an unusually long time to run.  He attributed the problem to his lack of patience or maybe satellite issues.  But the satellite signal was perfect on this day.  Otherwise, Mustafa’s malicious download may not have run to completion on Tam’s laptop.  When the download finally finished, another dot lit up in Bahir Mustafa’s global heat map of compromised computers as Tam Henderson’s laptop, from a jungle village in Vietnam, became a drone soldier in a hidden war controlled by a shadowy botnet master in Iran, all because of a careless programming mistake from a programming team in the Philippines, contracted by a US sporting goods manufacturer.

Tam eventually ordered the softball equipment and had a great time teaching the basics of the game to his kids in Vietnam.  He collected hundreds of pictures and videos and put it all together for a Christmas presentation to his home church later that year.  But every time he connected his laptop to the Internet, he noticed a significant slowdown.

Jerry Barkley was a church member at Tam’s home church and filled a role as the unofficial IT support staff.  Church employees thought Jerry was slightly eccentric, but he was friends with everyone and they all used his expertise to tune up or fix their computers.  When Tam connected his laptop to the church network, Jerry noticed an immediate slowdown in Internet access for everyone else at the church.  Curious, Jerry used a variety of tools on the open source firewall system he built for the church and traced the problem to Tam’s laptop.  He found Tam’s laptop saturated the Internet connection with a brute force password attack against a large bank website, with occasional packets to a website somewhere in Iran.

With one week remaining before Tam had to return to Vietnam, Tam put his laptop in Jerry’s hands and Jerry found and removed a mysterious piece of malware.  It was not easy to find and it took several days and late nights to locate and remove it.  But with one day left before Tam had to return to Vietnam, Jerry returned Tam’s laptop, now free from malicious software, with some advice on how to keep it that way.

“Tam, this was a nasty one and it wasn’t easy to find.  The next one might even be tougher to get rid of.”

“How did it get there?”

“Nobody knows – it could have come from anywhere.  Do you go out on the Internet a lot?”

“No, not really.  Sometimes I look up sports scores, stuff like that.  I don’t have a lot of time to spend on the Internet.”

“Well, sometimes those websites can get compromised.  Listen, get a credible antivirus program.  Not the chintzy consumer stuff, but some real antivirus software and put it on this laptop.  Keep the signatures up to date.”

“The signatures?”

“Yes.  All the antivirus programs work by keeping signatures of known viruses.  The bad guys cook one up, the good guys find out, they issue an update.  It’s an arms race.  So make sure you have up to date signatures.  Sometimes they update hourly.”

“Wow!”

“Yup, wow is right.  Antivirus software is not perfect.  It can only find malware it knows about.  I tried a few antivirus programs on your laptop and they all scanned clean.  None of the automated tools I threw at it found the problem.  But every time I connected it to my DMZ network, it blasted traffic to this site.  That’s why it took me so long to find it.  It was a needle in a haystack.  It was buried with a bunch of other Kernel drivers that load at boot time.  Whoever did this knew what they were doing.”

“DMZ what?”

“Don’t worry about it.”

“So what do I do?”

“There is no perfect solution.  But if you suspect something is wrong, let’s say it starts to run unusually slowly or it starts just generally acting badly, try a system restore.”

“What’s that?  I have a ton of files I need to keep.  I can’t afford to wipe it all out.”

“I know – and that’s not what a system restore does.  Every time you install some new software or do an update, the system should save a copy of its old self.  Not your user files and stuff like that, system state stuff.  What programs are installed, what’s your computer name, how does it do networking, things like that.  So one tactic is, when something goes bad, try restoring the system state to a point before the time when things went bad.  All your pictures and videos and documents stay the same – it’s just the system information around all that content that goes back to its earlier state.”

“This sounds tricky.”

“It’s not bad.  And you’re out there with nobody else around, so you might have to tackle it.  Or call me and I’ll walk you through it if you get in trouble.  And think about putting in one of my firewalls at your church over there.  I have all kinds of diagnostics that can help track down this kind of stuff.  That way, if you suspect something is wrong, bring it back to your Vietnam church and connect it behind your firewall and I can look at the traffic in and out.”

“Thanks.”

“You’re welcome.”

One dot disappeared from Bahir Mustafa’s global heat map display that day, leaving thousands more remaining.  How that malicious program came to reside on Tam’s laptop remained a mystery to all Tam’s friends and colleagues.

But not to Bahir Mustafa.  A Ukrainian mobster paid $10,000 to deploy his special program onto the computers in Bahir’s botnet.  The program tried combinations of letters and numbers in a brute force password guessing attack against banking websites, looking for credentials for a few Hollywood celebrities.  With thousands of rented drone personal computers around the world each running a portion of the attack, a few were bound to find pay dirt.  Sensational headlines saturated the tabloids a few weeks later, but the headlines all missed how Bahir’s customer used his stolen $millions: to buy weapons for rebels in eastern Ukraine.

Bahir Mustafa and others like him are part of a vast underground value chain, complete with sophisticated, automated systems to constantly probe for vulnerable computers.  Don’t be a victim.  Don’t become an unwitting drone in somebody’s crime scheme.

If you liked this short story, you’ll love my new book, Bullseye Breach.  Check it out, here.

And if you’re concerned you might have a problem with malicious software, don’t hesitate to contact us.

If you’re part of an IT department or a help desk, feel free to share this story with your end users.  Especially the ones who have trouble believing IT security is important.  This story is fiction – I made it up – but it’s realistic.  Enjoy.

************************

Abby Kramer was a third year student at a Bible college in Colorado. A pastor’s daughter, she liked to socialize online with friends from all over the world and kept a large library of pictures and videos from friends in her Facebook account. After a hard day of classes and studying, she allowed herself a few minutes each evening before bed to watch a new video or laugh at a few pictures and comment on posts from her online friends. The dialog with friends was always refreshing and no matter what frustrations the day brought, these few minutes always brightened her mood before bed.

She was shocked when she woke up one Saturday morning after a difficult mid semester week filled with tests to find this email waiting in her inbox:

From: Facebook [mailto:update+hiehdzge@facebookmail.com]
Sent: Saturday, March 16, 2013 4:16 AM
To: akramer@cobible.org
Subject: You requested a new Facebook password

Hello,

You recently asked to reset your Facebook password.
Click here to change your password.
Didn’t request this change?
If you didn’t request a new password, let us know immediately.

Change Password

This message was sent to akramer@cobible.org at your request.
Facebook, Inc., Attention: Department 415, PO Box 10005, Palo Alto, CA 94303

It was a shame Abby never looked at the email header. If she had, she would have noticed it originated in Florida and routed through a relay server in China. It came from a character who called himself “Duceml.” It didn’t come from Facebook.

But Abby didn’t know or care about how to look up any of that.

First alarmed that somebody tried to change her password, then relieved that Facebook had the wisdom to put in this email safety mechanism, Abby quickly clicked the “Change Password” link, which took her to what looked like a Facebook password change screen. Obviously, somebody had her password. She would change it and make sure nobody ever knew it this time. If Angie Gilroy ever saw what she said about Angie’s brother to Donna Gustafson, it would be awful.

A few seconds after filling in the old password and new password boxes, she found herself looking at the Facebook login screen. She was dying to know if Angie Gilroy found out what Abby said about Angie’s brother and what she had to say about it, so she decided to log in and check. When she saw a popup box with “Invalid username and/or password,” she tried again with her old password. Curious – her old password still worked. Didn’t she just change it? Annoyed, she went through the password change process again. This time it worked.

It was time for breakfast in Abby’s dormitory, and a school dance was coming up that night and Abby quickly forgot about her Facebook scare.

But a Russian FTP server did not forget. FTP – file transfer protocol (or program) – is one of the oldest programs on the Internet. Millions of people use FTP every day to upload and download files to and from websites. And criminals use FTP to surreptitiously upload and download information to and from computers owned by naive users.

Had Abby looked more closely at that first Change Password screen, she would have noticed it said, www.facebrook.com.ru. It was a website in Russia designed to look like Facebook. But Abby didn’t look closely. Instead, she entered her old and new password and waited several seconds as that fake website scooped it all up and redirected her computer to the real Facebook website. And even though she changed her Facebook password, she used the same email address and password for the bank account she shared with her parents to cover college expenses.

An anonymous criminal somewhere in Russia eagerly monitored the growing list of Facebook usernames and passwords accumulating in his FTP server. He would try these credentials against a list of retailers and banks and no doubt find a few matches. It would be tedious trying variations of user akramer@cobible.org with password either, “IheartJ3sus” or “i@msav3d” against thousands of banking websites, but that’s why people write software – to handle tedious tasks. And a program could do the job in a few minutes.

He smiled when he found a match at a large bank website and looked up the bank balance – more than $1000 US dollars. After posting the credentials for sale on an underground website, somebody in the US named Matt1117 bought them for $750, paid into an anonymous escrow account. The transaction was routine. Just one drop in an ocean of transactions every day.

Two weeks later, Tamara Kramer, Abby’s mother, waited in the checkout line in the local grocery store. She wanted to surprise her starving college daughter with some ramen noodles and other snacks. When she swiped her debit card from the shared checking account with her daughter, the cashier politely told her it was declined. Surely there must be some mistake? She swiped it again and was declined again. As people queued up in line, Tamara called her bank. What was going on? After waiting on hold for more than 15 minutes, she finally connected with an agent named Nancy with a thick Indian accent who tried to be helpful. The language barrier was difficult to overcome, but Nancy eventually told Tamara that her bank account was over drafted.

“What? How can this be? I deposited $1000 in that account 3 weeks ago and haven’t bought anything since then.”

“Ma’am Tamara, it says here you spent $1232.55 at an online electronics store last week.”

“I did not! … Unless Abby did. Thank you, I will talk to my daughter.”

Embarrassed, Tamara paid for her groceries with a credit card and apologized to the cashier and everyone waiting in the growing line. She called Abby and left a message. Abby returned the call several hours later and felt the wrath of a mother betrayed. Abby tearfully assured her mother she did no such thing. Tamara called the bank, disputed the bill pay and closed the checking account. She had to visit the local branch of her bank to open a new account, and contacted everyone with checks from the old bank account that had not yet cleared. Over the next two weeks, Tamara managed to reimburse everyone to whom she or Abby had written checks by scavenging money from savings and delaying other bills. The bank fraud department investigated and after 3 months, filed an insurance claim and reimbursed Tamara for the stolen money, less Tamara’s $50 liability. The bank called appropriate law enforcement agencies about the matter, which took the reports and filed them away with thousands of similar reports.

Nobody tried to recover or even locate the stolen money. But a teenager named Kenny enjoyed the new game console he bought on Craigslist from somebody named Matt1117.

****************************

If you liked the story about Abby Kramer, you’ll love the book titled, “Bullseye Breach.”  Here is a link to a teaser.

I just finished my own disaster recovery operation.  There are still a few loose ends but the major work is done.  I’m fully operational.

Saturday, Dec. 20, 2014 was a bad day.  It could have been worse.  Much worse.  It could have shut my company down forever.  But it didn’t – I recovered, but it wasn’t easy and I made some mistakes.  Here’s a summary of what I did right preparing for it, what I did wrong, how I recovered, and lessons learned.  Hopefully others can learn from my experience.

My business critical systems include a file/print server, an email server, and now my web server.  I also operate an OpenVPN server that lets me connect back here when I’m on the road, but I don’t need it up and running every day.

My file/print server has everything inside it.  My Quickbooks data, copies of every proposal I’ve done, how-to documentation, my “Bullseye Breach” book I’ve been working on for the past year, marketing stuff, copies of customer firewall scripts, thousands of pictures and videos, and the list goes on.  My email server is my conduit to the world.  By now, there are more than 20,000 email messages tucked away in various folders and hundreds of customer notes.  When customers call with questions and I look like a genius with an immediate answer, those notes are my secret sauce.  Without those two servers, I’m not able to operate.  There’s too much information about too much technology with too many customers to keep it all in my head.

And then my web server.  Infrasupport has had different web sites over the years, but none were worth much and I never put significant effort into any of them.  I finally got serious in early 2013 when I committed to Red Hat I would have a decent website up and running within 3 weeks.  I wasn’t sure how I would get that done, and it took me more like 2 months to learn enough about WordPress to put something together, but I finally got a nice website up and running.  And I’ve gradually added content, including this blog post right now.  The idea was – and still is – the website would become a repository of how-to information and business experience potential customers could use as a tool.  It builds credibility and hopefully a few will call and use me for some projects.  I’ve sent links to “How to spot a phishy email” and other articles to dozens of potential customers by now.

Somewhere over the past 22 months, my website also became a business critical system.  But I didn’t realize it until after my disaster.  That cost me significant sleep.  But I’m getting ahead of myself.

All those virtual machines live inside a RHEV (Red Hat Enterprise Virtualization) environment.  One physical system holds all the storage for all the virtual machines, and then a couple of other low cost host systems provide CPU power.  This is not ideal.  The proper way to do this is put the storage in a SAN or something with redundancy.  But, like all customers, I operate on a limited budget, so I took the risk of putting all my eggs in this basket.  I made the choice and given the cost constraints I need to live with, I would make the same choice again.

I have a large removable hard drive inside a PC next to this environment and I use Windows Server Backup every night to back up my servers to directories on this hard drive.  And I have a script that rotates the saveset names and keeps 5 backup copies of each server.

Ideally, I should also find a way to keep backups offsite in case my house burns down.  I’m still working on that.  Budget constraints again.  For now – hopefully I’ll be home if a fire breaks out and I can grab that PC with all my backups and bring it outside.  And hopefully, no Minnesota tornado or other natural disaster will destroy my house.  I chose to live with that risk, but I’m open to revisiting that choice if an opportunity presents itself.

So what happened?

The picture above summarizes it.  **But see the update added later at the end of this post.**  I walked downstairs around 5:30 PM on Saturday and was shocked to find not one, but two of my 750 GB drives showing amber lights, meaning something was wrong with them.  But I could still survive the failures. The RAID 5 array in the upper shelf with my virtual machine data had a hot spare, so it should have been able to stand up to two failing drives.  At this point only one upper shelf drive was offline, so it should have been rebuilding itself onto the hot spare.  The 750 GB drives in the bottom shelf with the system boot drive were mirrored, so that array should (and did) survive one drive failure.

I needed to do some hot-swapping before anything else went wrong.  I have some spare 750 GB drives, so I hot-swapped the failed drive in the upper shelf.  My plan was to let that RAID set rebuild, then swap the lower drive for the mirror set to rebuild.  And I would run diagnostics on the replaced drives to see what was wrong with them.

I bought the two 2 TB drives in slots 3 and 4 of the lower shelf a few months ago and set them up as a mirror set, but they were not in production yet.

Another note.  This turns out to be significant.  It seems my HP 750 GB hotswap drives have a firmware issue.  Firmware level HPG1 has a known issue where the drives declare themselves offline when they’re really fine.  The cure is to update the firmware to the latest version, HPG6.  I stumbled onto that problem a couple months ago when I brought in some additional 750 GB drives and they kept declaring themselves offline.  I updated all my additional drives, but did not update the drives already in place in the upper shelf – they had been running for 4+ years without a problem.  Don’t fix it if it ain’t broke.  This decision would bite me in a few minutes.

After swapping the drive, I hopped in the car to pick up some takeout food for the family.  I wouldn’t see my own dinner until around midnight.

I came back about 1/2 hour later and was shocked to find the drive in the upper shelf, slot 4 also showing an amber light.  And my storage server was hung.  So were all the virtual machines that depended on it.  Poof – just like that, I was offline.  Everything was dead.

In my fictional “Bullseye Breach” book, one of the characters gets physically sick when he realizes the consequences of a server issue in his company.  That’s how I felt.  My stomach churned, my hands started shaking and I felt dizzy.  Everything was dead.  No choice but to power cycle the system.  After cycling the power, that main system status light glowed red, meaning some kind of overall systemic failure.

That’s how fast an entire IT operation can change from smoothly running to a major mess.  And that’s why good IT people are freaks about redundancy – because nobody likes to experience what I went through Saturday night.

Faced with another ugly choice, I pulled the power plug on that server and cold booted it.  That cleared the red light and it booted.  The drive in upper shelf slot 4 declared itself good again – its problem was that old HPG1 firmware.  So now I had a bootable storage server, but the storage I cared about with all my virtual machine images was a worthless pile of scrambled electronic bits.

I tried every trick in the book to recover that upper shelf array.  Nothing worked, and deep down inside, I already knew it was toast.  Two drives failed.  The controller that was supposed to handle it also failed. **This sentence turns out to be wrong.  See the update added later at the end.**   And one of the two drives in the bottom mirror set was also dead.

Time to face facts.

Recovery

I hot swapped a replacement drive for the failed drive in the bottom shelf.  The failed drive already had the new firmware, so I ran a bunch of diagnostics against it on a different system.  The diagnostics suggested this drive really was bad.  Diagnostics also suggested the original drive in upper slot 2 was bad.   That explained the drive failures.  Why the controller forced me to pull the power plug after the multiple failures is anyone’s guess.

I put my  2 TB mirror set into production and built a brand new virtualization environment on it.  The backups for my file/print and email server virtual machines were good and I had both of those up and running by Sunday afternoon.

The website…

Well, that was a different story.  I never backed it up.  Not once.  Never bothered with it.  What a dork!

I had to rebuild the website from scratch.  To make matters worse, the WordPress theme I used is no longer easily available and no longer supported.  And it had some custom CSS commands to give it the exact look I wanted.  And it was all gone.

Fortunately for me, Brewster Kahle’s mom apparently recorded every news program she could get in front of from sometime in the 1970s until her death.  That inspired Brewster Kahle to build a website named web.archive.org.  I’ve never met Brewster, but I am deeply indebted to him.  His archive had copies of nearly all my web pages and pointers to some supporting pictures and videos.

Is my  website a critical business system?  After my Saturday disaster, an email came in Monday morning from a friend at Red Hat, with subject, “website down.”  If my friend at Red Hat was looking for it, so were others.  So, yeah, it’s critical.

I spent the next 3 days and nights rebuilding and by Christmas eve, Dec. 24, most of the content was back online.   Google’s caches and my memory helped rebuild the rest and by 6 AM Christmas morning, the website was fully functional again.  As of this writing, I am missing only one piece of content.  It was a screen shot supporting a blog post I wrote about the mess at healthcare.gov back in Oct. 2013.   That’s it.  That’s the only missing content.  One screen shot from an old, forgotten blog post.  And the new website has updated plugins for SEO and other functions, so it’s better than the old website.

My headache will no doubt go away soon and my hands don’t shake anymore.  I slept most of last night.  It felt good.

Lessons Learned

Backups are important.  Duh!  I don’t have automated website backups yet, but the pieces are in place and I’ll whip up some scripts soon.  In the meantime, I’ll backup the database and content by hand as soon as I post this blog entry.   And every time I change anything.  I never want to experience the last few days again.  And I don’t want to even think about what shape I would be in without good backups of my file/print and email servers.

Busy business owners should  periodically inventory their systems and update what’s critical and what can wait a few days when disaster strikes.  I messed up here.  I should have realized how important my website has become these past several months, especially since I’m using it to help promote my new book.  Fortunately for me, it’s a simple website and I was able to rebuild it by hand.  If it had been more complex, well, it scares me to think about it.

Finally – disasters come in many shapes.  They don’t have to be fires or tornadoes or terrorist attacks.  This disaster would have been a routine hardware failure in other circumstances and will never make even the back pages of any newspaper.

If this post is helpful and you want to discuss planning for your own business continuity, please contact us and I’ll be glad to talk to you.  I lived through a disaster.  You can too, especially if you plan ahead.

Update from early January, 2015 – I now have a script to automatically backup my website.  I tested a restore from bare virtual metal and it worked – I ended up with an identical website copy.  And I documented the details.

Update several weeks later. After examining the RAID set in that upper shelf in more detail, I found out it was not RAID 5 with a hot spare as I originally thought.  Instead, it was RAID 10, or mirroring with striping.  RAID 10 sets perform better than RAID 5 and can stand up to some cases of multiple drive failures, but if the wrong two drives fail, the whole array is dead.  That’s what happened in this case.  With poor quality 750 GB drives, this setup was an ugly scenario waiting to happen.

Merry Christmas.  Maybe now I can get some sleep.  Backups first this time.  Then sleep.

***********************

As of 4AM Dec. 24, the most important web pages should be back online. The submenu items and some of the blog entries are still missing.

***********************

Pardon the mess while I rebuild the Infrasupport website.  It was a casualty of a catastrophic hardware failure on Saturday night, 12/20/2014.  I’ll have it back up with updated versions of WordPress and the new Responsive Mobile theme from Cyberchimps in a few days.  Watch for minute by minute updates to menus and content while I recover it all and a new blog post with details once I get it back in shape again.  In the meantime, a hearty thanks to the folks at archive.org.  Until I’m fully recovered, you can find a copy of most of the content here.

– Greg Scott
Dec. 22, 2014

Updated Dec. 24, Dec. 25, 2014, and January 6, 2015.

By now, we’ve all read and digested the news about the December 2013 Target breach.  In the largest breach in history at that time and the first of many sensational headlines to come, somebody stole 40 million credit card numbers from Target POS (point of sale) systems.  We’ll probably never know details, but it doesn’t take a rocket scientist to connect the dots.   Russian criminals stole credentials from an HVAC contractor in Pennsylvania and used those to snoop around the Target IT network.   Why Target failed to isolate a vendor payment system and POS terminals from the rest of its internal network is one of many questions that may never be adequately answered in public.  The criminals eventually planted a memory scraping program onto thousands of Target POS systems and waited in Russia for 40 million credit card numbers to flow in.  And credit card numbers would still be flowing if the banks, liable for fraudulent charges, hadn’t caught on.  Who says crime doesn’t pay?

I parted with some money 2 years ago and bought myself a tablet.  My wife – the most un-high-tech person on this planet said, “Greg, you need to get one of those.  The world is changing and you need to know what’s going on.”  Just one many reasons why love her.