P.T. Barnum may be smiling in his grave today

­

I spent an awful day yesterday with Microsoft Office 2013 Home and Business Edition.  Full disclosure – my company is a Microsoft Registered Partner and this blog entry won’t make me any friends in Redmond.  But right now, I am frustrated beyond belief and I will have trouble sleeping until I put electronic pen to virtual paper.

After more than 20 years of Microsoft producing a product named Office, by now everyone knows what it includes – a spreadsheet named Excel, a word processing program named Word, an email client named Outlook, a presentation package named Powerpoint, a personal database product named Access, and a desktop publishing program named Publisher.  Different editions of Office include different combinations of packages and licensing and Microsoft mixes them up with each new version.  By now, Office is the de-facto standard for electronic document formats.

With Office 2013, Microsoft combined the audacity that comes with monopoly power with technological incompetence.  What possible rational reason could anyone give to force customers to create a unique login on the Microsoft website for every single retail copy of Office Home and Business?  If you own, say, 50 computers and you have 50 copies of Office Home and Business, you need 50 Microsoft logins to make it work.

Sheer insanity.  Or is it?  Microsoft is filled with competent engineers and savvy marketers.  Microsoft did this for a reason, and this is really a story about a 21st century shakedown scheme.  But it’s buried underneath a pile of technical jargon so very few will notice.

With Office 2013, Microsoft offers three licensing choices, called Volume licensing, retail licensing, and a subscription service named Office 365.  Office 365 is new, the rest have been around a long time.

Volume licenses come with lots of flexibility businesses care about.  Companies can deploy volume licenses any way they see fit.  A volume license for Microsoft Office Standard edition includes only Word, Excel, and Outlook and lists for roughly $370.  Microsoft Office Professional Plus includes all the Office packages and lists for roughly $500 per seat.

Retail licenses cost less, but are less flexible.  For example, Office Home and Business includes Excel, Outlook, Powerpoint, and Word – more packages than Office Standard, but with a lower price of around $220.  The Home and Business license is only good for one computer.  Once installed on any computer, that license is married to that computer forever.  If your PC dies and you need to reinstall Office Home and Business, you need permission from Microsoft.

So far, so good.  Here comes the audacious part.

Starting with Office 2013, Microsoft purposely made Office Home and Business a nightmare to install by adding an artificial impediment.  Microsoft now requires a unique login on its website for every single individual copy of Office 2013 Home and Business.  For each individual login, you must specify the name, phone number, address, email address, and other identifying information.  After setting up this login, you can download and install your individually tailored copy of Office 2013 Home and Business.  The download is roughly 2.2 gigabytes. Customers who use T1 Internet connections will need almost 4 hours per download and each installation now requires its own download. 50 installations means 50 downloads.

If anything goes wrong – a network hiccup during the download, a wrong answer to a question, anything – you’ll spend hours fiddling with registry entries and deleting files by hand because it won’t remove cleanly. I had 4 identical brand new computers and spent most of a day cleaning the remnants of a botched installation on one, with lots of telephone advice from Microsoft Customer Support about undocumented registry entries.

And finally comes the new offering, Office 365.  It’s a Microsoft hosted solution, meaning you connect to a website and work on your documents from there.  The cost is $99 per year or around $10 per month.   No installation hassles, quick and easy to set up, no up-front financial pain for end users.  Your documents live inside a Microsoft cloud, so they are accessible globally and you don’t need a server anymore. Naïve CFOs and Purchasing Departments will love it.

P. T. Barnum reportedly once said ”there’s a sucker born every minute” and he may be laughing in his grave at this modern massive con job. Why would Microsoft price its hosted offering so low relative to a locally installed copy of Office?  Why would Microsoft take such apparently boneheaded steps to artifically complicate installations of Office Home and Business?  And why would Microsoft spend $millions for the cloud capacity to store and manage millions and millions of new user accounts?

Only one answer makes sense – increased revenue.   How does spending $millions to host all this stuff generate revenue?

I can think of only one answer – and I promise, you won’t like it.  Microsoft wants to be the repository for all your personal and business content.  Office 365 will capture your documents, Outlook.com will capture your email, Lync will capture your video meetings.  If Microsoft can make your installation experience expensive and miserable when installing on your own computer, and make it hassle free and low cost when hosting in its cloud, many people will opt for the path of least resistance and put their documents in the Microsoft cloud.  Millions of Office 365 users will blindly trust Microsoft with their most private data because getting started is cheap and easy.

Once Microsoft captures all your content, marketers will pay Microsoft a holy fortune to slice, dice, and analyze your content.  You will provide raw material for marketers and you will pay Microsoft for the privilege.   But marketers will pay much more.  Marketing will be the real Microsoft revenue source – your $99 per year subscription is just a few giblets on the real gravy train.

What to do about it?  If you don’t care if an army of marketers digs deep into your content, trust Microsoft.  If you do care about privacy, maybe now is the time to start looking at alternatives.  Several are available, including Libre Office and other free and minimal cost offerings.  If enough people start adopting some of today’s great alternatives, maybe Microsoft’s monoply power can be tamed.  But if history is a good predictor, this probably won’t happen.

Big versus small is a lousy way to judge IT service companies

Big versus small may be one of the oldest debates in business.

Big companies are perceived as safe, stable, predictable, comfortable.  Small companies are the opposite; just prefix “un” to all those adjectives.  This debate is especially contentious in the IT industry, where everyone wants safe, stable, predictable, and comfortable.

Full disclosure – Infrasupport is currently a one person company. I’ve lost business simply because I’m a one person company and the victim of a negative stereotype.  Frustrating doesn’t begin to describe the feeling after pouring countless hours into crafting a quality solution to a customer problem, only to lose to an inferior solution at the end because of my company size.

So in this article, I want to examine some of the stereotypes around big companies to determine if they match reality.  Am I nuts?  Are big companies really safer partners?

At a macro level, here are the top 15 companies by revenue from 2007 :

  1. Wal-mart
  2. Exxon Mobile
  3. General Motors
  4. Chevron
  5. ConocoPhillips
  6. General Electric
  7. Ford Motor
  8. Citigroup
  9. Bank of America
  10. American International Group
  11. J.P. Morgan Chase
  12. Berkshire Hathaway
  13. Verizon Communications
  14. Hewlett-Packard
  15. IBM

Of those top 15 companies, at least 5 companies –  GM, Citigroup, Bank of America, AIG, J.P. Morgan either received a US Government bailout or ceased to exist during the great 2008 recession.  Do I really need to rehash the horror stories around American International Group and the other supposedly stable large financial powerhouse companies?

Over the past few years, Hewlett Packard fired a CEO who wanted to be a rock star, created a scandal when members of its own board of directors illegally spied on each other, fired another CEO in a sex scandal, spent $11 billion to buy a software company based on questionable accounting numbers, and shocked the IT industry by threatening to pull out of the PC marketplace after buying a failing tablet company.  This is stability?

Here are a few more recent big company horror stories:

  • Most of the entire US Airline industry filed for bankruptcy in the past few years.
  • The entire US Auto industry would have failed without a taxpayer bailout.
  • When the entire US Financial industry melted down, I bailed them out with taxes from my small company.
  • Top managers at companies such as Enron, Adelphia Communications, Qwest, and WorldCom are now retired and living in prison.  Most of these companies no longer exist, having been taken down by massive fraud.

The IT industry is littered with the remnants of once large and supposedly stable companies now in the ash pit of history.   Make a case to the 120,000 former employees of Digital Equipment Corporation about big company stability.   Other names, such as Burroughs, Sperry/Univac, NCR, Control Data, Honeywell, and Cray are now either long forgotten or skeletons of their former selves.  More recently, Sun Microsystems no longer exists, and now the entire PC segment is in turmoil.

Closer to home, during the dot com bust of 1999 – 2001, the IT Service market in the Minneapolis St. Paul area collapsed by roughly 50 percent.  Does anyone remember local companies such as Born Information Services, Ranier Technologies, All Systems Go, Benchmark Computer Learning, and a host of others?  In the late 1990s, these were among the leading IT consulting firms in the Twin Cities area, with combined revenue in the hundreds of millions.  Today, they are all long gone.  One firm, All Systems Go, sold to a national chain.  Within the first few years, everyone in the original All Systems Go either left or was forced out, leaving a legacy of unsatisfied customers and angry former employees behind.

This is stability?

The 2013 Twin Cities IT service industry is still tumultuous.  In at least one large firm, technicians hired two years ago are now the most senior employees in the company.

Despite the overwhelming body of evidence to the contrary, many customers still use company size as a major factor in finding an IT support partner.  The perception of a deep bench seems to offer an illusion of stability and this prejudice against small companies is deep and extraordinarily difficult to overcome.

Let’s put prejudice aside and start thinking rationally.  The fact is, nobody in the IT service industry can afford a deep bench.  People are expensive, and people without paying customers will quickly kill any service company.   If you’re a potential customer looking for IT service, and a deep bench is your primary selection criteria, you will likely be disappointed.

I propose using a more accurate set of criteria to evaluate IT service companies.  Look at the quality and longevity of the relationships between individual people who work for the customer and individual people who work for the IT service company.  The key to a successful IT support experience is the quality of these human relationships, not the size of the bench.

All IT support companies, including Infrasupport, can build nice looking websites and publish meaningless statistics about all our certifications.  Look past that to evaluate what is really important.

How?

Ask me about Infrasupport’s long standing relationship with customers.  Get to know Infrasupport by trying a small project as a test.  If you like it, try another bigger project.  If my company earns your trust, then reward Infrasupport with your business.  If Infrasupport fails to earn your trust, we can both learn from the experience and move on.

All I ask is for a level playing field.

How a gross IT security lapse hurt a US Senate campaign

­

This story is personal.   It is one of the best examples I’ve seen where poor IT security practices and the physical world collide and leave a trail of destruction.

Way back in 2006, I registered my name with the Norm Coleman for Senate campaign. Although the US Senate election was two years away, I felt kind of like an insider when the Coleman campaign sent me email updates.  Fortunately for me, I never gave the campaign a credit card number.

The 2008 Minnesota Senate election between Norm Coleman and Al Franken was too close to call.  There were recounts, court challenges, and recounts of recounted recounts.  Franken eventually won by a few dozen votes.

This is where it gets personal.

On March 10, 2009, I received this email, reproduced below with original spelling errors:

From: Wikileaks Press Office [mailto:press-office@wikileaks.org] Sent: Tuesday, March 10, 2009 9:29 PM To: undisclosed-recipients Subject: Norm Coleman leak

Senator Norm Coleman supporter / contributor list leaked.

Your name, address and other details appear on a membership list leaked to us from the Norm Coleman Senate campaign.

If you have contributed financially to the Coleman campaign there are additional details.

We understand that Norm Coleman became aware of the leak in January.

The information has been passed around out of public view.

We have sent you this note as a curtesy in case Norm Coleman has not contacted you previously.

We have not released the material yet, but may do so within the next few days.

In line with our policy of completely neturality for whistleblowers and political sources, the material will be treated impartially.  We support all those who engage in the struggle for political reform and wish you well.

For additional details, see: [Web links in the remainder of the email are no longer any good]

Apparently, my name and email address were now in the public domain because I filled out a web form on the Coleman for Senate website.  Not a big deal for me – I’m already on several spam lists anyway.  But information about all of Coleman’s online donors was also in the public domain, including credit card numbers and security codes.  This was a big deal.

Apparently, after the election and during one of the many recount challenges in January, 2009, the Coleman campaign decided to move its website.

Unfortunately, the campaign left a copy of its website content at the old hosting site, wide open for the whole world to see.  One of the files was an unencrypted spreadsheet listing donor contact information, credit card numbers, and security codes.  This is a wildly reckless violation of security best practices and PCI (Payment Card Industry) rules.  Credit card information should never be stored on the same system as a public facing website.  If the website is breached, the credit card information is also at risk.  This data should reside in a back end database server with carefully crafted access controls, putting another line of defense between this sensitive information and potential thieves.  And as a final line of defense, credit card information should always be encrypted, which at least makes it difficult for data thieves to exploit.

Organizations storing donor or customer sensitive information have an almost sacred duty to protect that information.  After all, these are the people  who fund and trust the organization.  With its amateur approach to security, the Coleman campaign demonstrated a reckless disrespect for its own donors’ trust and paid dearly for it.

Adria Richards, an IT consultant specializing in website security, found the old website content, took a screenshot of what she found, and posted the screenshot on her blog.  Here is the only remaining evidence I can find of Richards’ blog, and here is a PDF copy in case the web link goes bad.  The Minnesota Independent published an article on January 28 2009 about the incident.  Here is the article and here is a PDF copy.

While Richards’ detective work is admirable, she should have notified the Coleman campaign first, before publicizing the problem. Her failure to contact the campaign before publicizing her findings violated an ethical best practice.

Sometime between January 28 and early March, 2009, Wikileaks obtained a copy of the spreadsheet, and that led to the email I found in my inbox when I woke up the morning of March 10.  The public reaction came fast and furious.  Here is another Minnesota Independent article and PDF copy.  Here is a Computerworld article and PDF copy.  And here is a Minnesota Independent article and PDF copy with donor reactions. Predictably, donors were upset and at least one donor reported being victimized by credit card fraud.

For the next few days, the story saturated Minnesota TV and print media.  Although the Coleman campaign tried to defend itself in the press, it ended up with a major public relations black eye as the campaign alienated its own donors and supporters.

Coleman eventually lost the recount battle in one of the closest US Senate elections in United States history.  In early 2013, Coleman floated the idea of trying a rematch against Franken in the 2014 election. A few days later, Coleman announced he would not run in 2014.

I wonder how much Coleman’s poor IT security practices hurt his political career?  After studying this incident and Coleman’s bungled reaction, I know I don’t want Norm Coleman representing me in the US Senate or anywhere else.  I have a hunch many others feel the same way.

Computer Whodunit – a Computer Troubleshooting Detective Story

This story is a great example of characterizing a problem, getting closer and closer to a solution with each step, and why the process is so important.  The story flows like a detective novel, with Greg the gumshoe uncovering new clues with each new step, all leading to a surprising conclusion that generates more unexpected questions for subsequent episodes.

Opening scene

Like most detective stories, the day started innocently enough.

My friend and customer, Lynn, called with a common problem.  Her email was broken.   Many of my problem calls start with broken email because pretty much everyone uses email.  But sometimes problems are not what they seem and the path to a solution can take many twists and turns.  This was one of those times.

I built the IT network in Lynn’s office and I know its characteristics the same way Scotty knew the original Starship Enterprise.   I knew Lynn used Microsoft Outlook on her desktop, the server was named ehcserver1, and the server ran Microsoft Exchange.  The server is in the basement of the building and everyone connects over a series of Ethernet switches.   Time for a good problem description.

Greg: “What happens when you launch your Outlook program”

Lynn: “It just sits there for a while and then gives me an error message, something about the server.”

Greg: “When did it break?”

Lynn:  “It worked fine when I shut down yesterday, but when I came in this morning and turned on my computer, now it doesn’t work.  I promise, I didn’t change anything.”

I could push Lynn harder for more details, but this told me enough.  Her Outlook program was not able to find the Exchange Server.   And I know Lynn well enough to believe her when she tells me she did not change anything.  This suggested something out of her control must have changed.

The next logical step in characterizing the problem was to find out if the problem was specific to Lynn or more widespread.  Quickly polling a few people near Lynn, we discovered Bruce had the problem, but not Ayrica, Joe, or Mike.  Since at least one other user had the problem, this suggested the problem was not specific to any workstation setting.  The problem was something common to Bruce and Lynn, but nobody else.

Start Unraveling the Mystery

Experience suggests most email problems are really symptoms of a more general network or server issue.  Everyone reports email problems because email is the application they use most often.  But email depends on the overall network.  If the overall network is broken, email will also be broken.

To find out if the problem is specific to email or something deeper, try a different application and see how it behaves.

One rule about working with end users.  Always start with an easy test and then dig deeper as necessary.  People seem to appreciate it more that way.

Greg:  Let’s see if you can see other stuff on the network.  Click Start…Computer, try to open one of your network drive mappings and let’s see what happens.  What happens when you open, say, the V drive?

A network drive mapping is really a directory on the server.  The idea is, the desktop computer “thinks” it’s another hard drive, thus the drive letter, but really it’s a directory on the server.   This is far and away the most common use for servers in an office.

All IT support companies have their own style and I set up many of my customers with a “V” drive, accessible to everyone.  It’s a convenient place to test.   Why V?  Because V stands for eVeryone.   Why not use “E”?  Because some computers use “E” for a locally connected CD or DVD or USB card reader.  It’s generally easier to use high letters in the alphabet for network drive letter mappings and leave low letters for locally attached devices.

Here is a picture similar to what Lynn saw.  (The picture will open in a different tab on your browser.)  The red X on the network drive mappings does not necessarily mean they are offline.  The only test that generates anything meaningful – just double-click on the drive letter and observe what happens.  Either the contents or an error message will show up in a window.

When Lynn double-clicked on the V drive, she saw an error message.  So did Bruce.  Since another application depending on the server and network was broken, the problem was not specific to email, but instead something common to both email and viewing drive letter mappings on the server.  But only common to Lynn and Bruce.  Mike, Joe, and Ayrica were fine.

Whodunit?

Computer troubleshooting is often compared to a good mystery movie.  Uncover clues and follow them where they lead.  This one was starting to feel like a Hollywood whodunit.  Time for some more in depth tests.

I asked Lynn to launch an old-fashioned DOS command window and try a few commands.  In Windows 7, Click Start…All Programs…Accessories…Command Prompt.  In Windows 8, click the upper right corner of the display to launch the Start screen, click the Start icon, right-click anywhere, click apps in the lower right corner of the system tray on the bottom of the screen, find the Command Prompt, and double-click on it.  (How much money did Microsoft spend on this new, “improved” interface?)

I knew the server was named ehcserver1.  So in that Command Prompt window, I asked Lynn to type “ping ehcserver1″, press the enter key, and tell me what it said.  Here is a picture similar to what Lynn found.  Here is a picture similar to what Lynn should have found.

How was it possible that Lynn could not translate the name of her server?  Clearly, something was fundamentally wrong with the network.  But it only effected a few users.  The next step is a deeper diagnostic.  In that DOS command window, type

ipconfig/all

Here is a PDF file with a sample report and some annotations taken from a Windows 7 computer in the Infrasupport network.

The computers in Lynn’s network should all have IPv4 addresses that look like 192.168.10.nnn, where nnn is a number between 1 and 254.  The gateway should be 192.168.10.1, DNS Server 192.168.10.20.  I built this network; I know what these values should be.

Surprise plot twist

But in a surprise plot twist worthy of the best Hollywood has to offer, both Lynn and Bruce’s computers showed IPv4 Address, Gateway, DHCP Server, and DNS Server Addresses of 192.168 2.nnn.  Note the 2.nnn instead of 10.nnn.

No wonder Lynn and Bruce’s computers were broken.   They both had bogus IP Addresses that did not belong to this network.  This was stunning!

The only possible explanation:  Somebody introduced a rogue DHCP server into this network and it was competing with my real DHCP Server.

DHCP servers lease IP Addresses and other network parameters to computers in an office.  Although there are carefully crafted special cases, typically an office should have exactly one and only one DHCP Server.  If an office has multiple DHCP servers, it is not possible to predict which DHCP server will lease a computer its network parameters.  This means computers may appear to suddenly fail at random times, and for random lengths of time, as their old leases expire and a rogue DHCP server assigns them bogus new network parameters.

This was exactly the case here.  The rogue DHCP Server serviced both Lynn and Bruce’s computers, while the correct DHCP Server took care of Ayrica, Joe, and Mike.

The suspicious character with the shifty eyes did it – or did he?

Wonderful.  Problem identified.  Now, what to do about it?  See  part 2 for the exciting conclusion to the story.   And, as always, contact us if you need help with a computer  troubleshooting situation.

Computer Troubleshooting 101 – Characterize the Problem

Just like most IT professionals, I get computer troubleshooting questions all the time from customers, friends, and family. A few are, uhmm, well, memorable. For example, the one about email a while ago.  The conversation started out something like this:

Friend:  My email doesn’t work.

Greg:  (Trying to be helpful)  OK, what email program do you use?

Friend:  Huh?

Greg:  Well, you run a program on your computer to get to your email, right?

Friend:  No, I just click on “email”.  But now it doesn’t work. What’s wrong with it?

I don’t think we ever solved that problem.  And most IT people reading this, after they finish laughing at an all too familiar story, know why.  I didn’t have enough information to begin solving the problem, and my friend was unable or unwilling to provide it.

All IT people read articles with advice about communicating with “normal” people.   The articles usually scold us for speaking a language most people don’t understand.  Fair enough and guilty as charged.  But we have our “IT words” for a good reason, as do all other professions.  I’m not sure why we get picked on so mercilessly.  For you finance people – why is it OK to say “EBIT-DA”, but not OK for IT people to say, “DHCP server”?

This blog entry is a little different.  I’m an IT guy and I’m asking so-called  ”normal” people who do not speak IT as a natural language to stretch just a little bit.  If you can say non IT words like “EBIT-DA”, you can say some IT words too.  It won’t hurt, I promise.

Meet us in the middle for your own benefit.  We IT people are pretty good at solving problems – that’s why we’re IT people – but we need more than “it doesn’t work”.  If you want your problem solved,  we need more from you.  I’ve learned at the feet of some of the best in the business, and what follows are some great troubleshooting tips.

First, before solving the problem, we have to identify it.  We call this characterizing the problem.  The process is part science, part art form.

Here are some things you can give me to help you get back up and running again:

What exactly happens when it breaks?  What do you do and how does the computer respond?  Give me a sequence of events leading up to the problem.  Give me exact error messages, codes, and pictures of screen shots if possible.  Details are important because at least one of those details may be a significant clue.

Has the system ever worked as expected or has it always been broken?  If it worked earlier and is broken now, when did it break?  What changed between when it worked earlier and now when it’s broken?

“Nothing changed” is always the wrong answer.  If nothing changed, then the system would still behave the same as it did earlier.  My friend, Bruce had a cell phone email problem a while ago.  He insisted nothing chanaged and his email just stopped working for no reason.  We talked about it and ended up removing and adding the email account to his smartphone.  Email behaved properly after that, and then Bruce said, “Oh yeah – a big update for my phone came out a few days ago and my email broke right after that!”  My other friend, Bob was also in the room, and Bob said, “wow – that’s probably why my cell phone email stopped working too!”

That’s the power of characterizing the problem – sometimes it helps solve multiple problems.

If the system worked before and is broken now, something broke it.  That something may be subtle and difficult to find, and that’s why details are important.  So think back to everything that happened with your broken system around the time the problem started.  Put together a detailed sequence of events.  Write it all down if this helps.  If I had known about that cell phone software update with Bruce and Bob, we could have saved time and jumped immediately to the solution.

Is the problem reproducible at will, or does it only happen sometimes?  If reproducible at will, what are the steps to reproduce it?  And if only sometimes, what is different about when it works versus when it breaks?  One time, I had a Dell laptop that sometimes refused to connect to the office wireless network.  After hours of trial and error, we finally found a pattern – the problem happened when the laptop was running on battery power, but not on AC power.  This turned out to be a (questionable) feature and not a bug – somebody at Dell thought it was a good idea to conserve power by turning off the wireless adapter by default when running on battery power.  The cure – press a function key to turn it on.

The solutions to many problems seem obvious, but generally only after going through the exercise to find them.

Perhaps most important – compare and contrast how the system should behave versus how it actually behaves.  It’s your job to explain this clearly and in detail to an expert who cannot be as familiar with the history of the problem as you.

Answer these and similar questions and now we have a well defined problem.

Next comes finding a solution.  The process is also part science, part art form.  For the science part, we form a possible solution based on the problem definition, come up with a way to test it, then evaluate the results.  The process is usually iterative, sometimes tedious, and always slower than anyone wants.  For the art part, sometimes inspiration strikes and sometimes it’s right.  Check out this article for a great example of a troubleshooting scenario.  And watch this space for more articles about interesting troubleshooting scenarios as they come up.