I seem to encounter more than my share of tech problems that nobody else has ever seen before. I don’t know why, they just seem to find me. The good news is, I like to think this makes me stronger. If they don’t kill me first.
The saga getting this blog page up and running is a typical example.
Apparently, it all started about a year ago when somebody decided it would be better to setup the new systemd to start mysqld on Fedora using a private tmp directory instead of the system wide tmp directory.
Understanding the sentence above needs some background. Briefly – I am hosting this website on a Fedora 18 virtual machine. Fedora is a free, open source offering from a great company named Red Hat. Because Fedora is free, lots of tech enthusiasts use it and help debug it and provide feedback back to Red Hat. Red Hat incorporates the feedback and periodically releases another product with paid support subscriptions named Red Hat Enterprise Linux. The model works for everyone. I get a free platform, Red Hat gets a more solid paid offering. And I’m a Red Hat partner, so it’s good to use the products I help resell and support.
Mysqld (pronounced, “My-S-Q-L-D”) is part of the well known open source mysql database package. And systemd (pronounced “System-D”) is a new, sophisticated set of software to start Linux systems. Systemd is an improvement over the old way to do it. It’s rapidly maturing will soon become part of Red Hat Enterprise Linux.
The takeaway from all this is, it’s bleeding edge packaging and I am essentially a tester for this packaging. Sometimes that testing produces unexpected results.
Why is this important to me? Because I chose a package named WordPress to develop my new website and I’ve spent a significant portion of the past month of my life learning how to use it. The website and blog you’re reading right now is the fruit of my labor. WordPress depends on the mysql database hosted on my Fedora system, which, in turn, uses systemd to start itself.
Mysqld apparently writes temporary data to a temporary directory to perform its work. This could be a potential security issue if others have access to that same temporary directory. So about a year ago, somebody decided it would be a good idea to use systemd to “fool” mysqld into using a private temporary directory only available to mysqld. Take a look here for details. Unfortunately, apparently systemd removes these private temporary directories periodically and this breaks mysqld, but only after it has been successfully running for several days. Problems like this are maddening to identify and troubleshoot because the system passes all tests and then suddenly fails for no apparent reason.
Around the time I set up my first blog post, systemd apparently decided to clean out its temporary directories. This broke mysqld, which, in turn, broke WordPress, which, in turn, broke my blog entries and website menu construction. This triggered a flood of electronic correspondence in various support forums to find an answer. The problem was magnified because the WordPress theme I chose, named “Responsive” had an upgrade around the same time and the upgrade had some bugs.
Here and here are a couple of links with details. Here is another one. The one sentence summary – I haven’t slept much the past few days and I really need a shower.
I am deeply grateful and indebted to the people in this discussion thread who found the mysqld/systemd problem and to the support staff at Cyberchimps.com who seem to work the same weird late night hours as I do.
And now, this blog should finally be visible to the world.
Recent Comments