Category Archives: All things Unix

Raising Software: A Tale of Unruly Tools

One of my retirement hobbies has been shooting and editing videos, mostly of our bicycle adventures.  When I was young, back in the middle of the 20th century, one of my uncles had a movie camera and projector, and I learned to splice broken film and edit clips together with razor blades and transparent tape.  He moved away and I grew up, and the skill was filed away as just another data point in How The World Works.

Fast forward to the second decade of the 21st century, when I decided to strap a cheap digital camera to the handlebars of the bicycle.  Digital editing requires software, a much less messy and more forgiving (i.e., non-destructive to the original “film”) process.  Since we use Linux exclusively as our computing platform of choice, there were a number of choices for Open Source video editing projects, mostly attempts to clone commercial video editing software for Windows and Apple.

I looked at Cinelerra and Kdenlive, which are fairly complex tools with a steep learning curve, but settle on OpenShot, a simpler tool with a lot of attractive features and a cleaner, no frills user interface.  Openshot 1 was essentially a one-man project, a user interface written in Python with the Tk graphical toolkit, built on the Unix principle of lots of little programs working together, using the multimedia command-line editor ffmpeg and associated libraries, the Inkscape vector graphics editor, and the Blender animation tools and libraries.

Openshot 1.4.3

OpenShot made it possible to load up a project with multiple clips, arrange them on a timeline, trim the ends or snip out shorter clips, and add titles and audio tracks, like voice-over and musical scores, in multiple overlaid tracks, and turn video and audio on and off by track and clip.   For several years, this worked fine.  However, success is not always a good thing, and Openshot suffered from it.  Not in the usual riches-to-rags story of an entrepreneur or rock star who descends into excess and loses his way, but in the attempt to seek wider appeal.

OpenShot was originally a purely Linux product, as mentioned.  To port the project to Windows, it was necessary, for a project with limited manpower resources, to keep a common code base.  Openshot attempts to keep a smoothly flowing user interface through parallel processing, using OpenMP.  The Windows philosophy is based on a single-user, single-task model rather than the multi-user, multi-tasking model of Unix.  When Windows evolved into a multi-tasking system, it used the pre-emptive model, which is relatively inefficient for the pipelined processing flow in the Unix cooperative model.  So, Windows applications tend to be monolithic, with all resources loaded in one huge process.  Parallel processing in Windows monolithic applications is accomplished largely through threads, rather the inter-process communication protocols.  I’ve programmed with threads in Linux, which tends to be tricky at best, and takes a thorough knowledge of parallel processing and memory management to do successfully.

The move to Windows-compatible architecture necessitated rewriting a lot of the Unix-specific standard library code in C++, which introduces the danger of memory-management issues. Openshot began to get buggy, with newer versions crashing often.  The developers claim it is the fault of unstable system libraries, but I’m not buying that explanation.  Since the user interface was also getting a major overhaul, work on version 2 meant that no more bug fixes were forthcoming for the now-crippled version 1.4.3.  Alas, initial releases of Version 2, with the back end still largely 1.4 code base, was also prone to crashing as well as presenting an unfamiliar user experience.

Openshot 2.4.1

So, we stayed with version 1.4.3 for a while longer, with short auto-save intervals.  Finally, with crashes and deadlocks rampant, we just had to try version 2 again.  Yes, the crashes had been largely fixed, but the new version was a monolithic package wrapped in a “launcher,”  (AppImage), apparently a type of container tool including all of the library dependencies, which rapidly ate up all available memory and much of the swap space, becoming so slow as to be indistinguishable from deadlock. Memory leaks come to mind when seeing this type of behavior.  On top of that, some of our favorite tools for controlling video blanking and audio muting by track were missing, to be restored by popular demand in a future revision.   Back to 1.4.3 .

Kdenlive

The other alternative, kdenlive, based on the Konquerer Desktop Environment (kde, not native to Ubuntu, thus necessitating loading the complete KDE library support suite), is yet another learning curve, with many editing feature differences and rendering options.  We did use this for one video, as the internal de-shaking algorithm is a bit more efficient than reprocessing the clips with command-line utilities, and we had a bad experience with a new camera mount that was sensitive to shaking.  Kdenlive also crashes from time to time, lending some credence to the Openshot claim that the system libraries are at fault.

But, I continue on, putting up with slow response, freezes, and crashes, because I’m familiar with the features I like, and it produces acceptable videos.   I may spend some time to learn kdenlive, but, hopefully, Openshot 2 will improve over time.  The other alternative is to try to build a native Ubuntu version from source, which is a daunting task, since most open source software has a very specific support software version dependencies.   Despite the woes with Openshot’s growing pains, it is still faster than writing command-line scripts to use the Linux multi-media-processing utilities ffmpeg or avconv to trim and assemble video clips and sound files.  I use those command-line tools to assemble time-lapse videos from my home-built security camera system, but that is much simpler.

Affordable Linux: A Quest

As most readers know, we here at Chaos Central are a Linux establishment for our primary computing.  We use Apple iPhones and iPads for mobile convenience and casual surfing, but depend on Linux for everything else–with the exception of what has become known at CC as “The Screaming Season,” where we are dependent on Windows to run Turbotax to render unto Caesar.

As much as we’d like to be on a 3 to 5-year cycle for computer upgrade/replacement, retirement, and before that, negotiated fee schedules, have reduced that to “when it breaks, MacGyver it.”  The result has been dismal.  When Judy’s desktop Linux machine died last year, having already received a CPU cooling fan transplant several years ago, salvaged from an old machine before it went to the recyclers, we pressed our old 2007-vintage laptop back into service, restoring her files from backup.  That machine had had a new hard drive and more memory installed during its last incarnation, but the extra memory had faded away, and the CPU not up to the demands of the 2016 version of Ubuntu Linux, so it was running slowly with the reduced-capability desktop.

Since the current crop of machines with pre-installed Linux are aimed at developers rather than standard office use, we couldn’t justify a new laptop from one of the few vendors who supply them.  So, as in the “bad old days,” we went shopping for a cheap Windows laptop on which we could install Linux.  The “cheap” machines are now essentially Microsoft’s answer to the iPad and Android tablets they see as main competition–a light-weight, small-screen device with as small (32GB) flash memory (EMMC) instead of a hard drive, and no optical drive.  Not to worry, I loaded a light-weight version of Linux (Lubuntu) on a 64GB flash drive and it became her new machine, with the added advantage of being portable.  The big disadvantage was the need to plug in the Linux memory stick to run it.  The biggest problem was the short life span of flash drives when used as the primary drive on a Unix-like system.   And, the system was incredibly slow.

Well, in only a few months, the flash drive expired.  No problem, I thought, just build another and restore.  But, as many others have found, the ability to install in the first place was a fluke: these little Windows tablets masquerading as a laptop by having an attached keyboard are finicky: try as I might, I could not get it to boot from the install memory stick again.  So, the little blue Dell machine has become the new Turbotax home, effectively retiring the $80 refurbished Vista->Windows7->Windows10 desktop we acquired a couple years ago and built up (another $40 video card) to run Win10 (badly).

So, Judy was back to the old laptop.  This time, we pressed her old 2010 HP Netbook back into service, even slower and more clunky than the older Compaq (which she also used during this evolution), running a light-weight version of Mint Linux.  The office was beginning to look like the computer labs of the early 2000s, where we pressed piles of recycled obsolete machines into experimental compute clusters to get 21st-century performance out of 20th-century machines, a distributed computing architecture conceived in the mid-20th century but not economically feasible until there was a huge supply of retired machines awaiting landfill space.

A comprehensive check of Linux-capable systems was not promising.  The custom-built laptops (our new criteria is portability to fit our mobile retirement life-style) are expensive and beyond our budget with no current paying clients.  My own laptop, which was high-end (and expensive) in its day (2010) is showing its age, with a too-small disk, weak battery and some screen burn-in.  But, the equivalent new models are in excess of $2000, difficult to justify on a pension without contract income to support it.  Nevertheless, it is only a matter of time before it wheezes its last, so we need to first find an affordable and long-term solution for Judy’s immediate need while keeping in mind the future budget hit for a developer machine.  The other option for affordable Linux-ready or pre-installed laptops are refurbished machines proven to be compatible.  The disadvantage there is the same facing my situation: the batteries are in mid-life, so long-term reliability is an issue,as well as mobile portability.  Older machines are also heavier than desired.

So, we finally settled on a fourth option: installing Linux on a Chromebook.  Chrome is a laptop operating system from Google, a variant of the Android system used on smart phones and tablets.  Since Chrome and Android are basically embedded Linux, it is relatively easy to add Linux to them as a container, using Crouton, a management system similar to the Docker system used to install and manage containers on native Linux machines.  I say relatively, but that assumes some heavy-duty skills in system administration, since Chrome is a locked system.  First, the system must be unlocked into Developer mode, which basically voids the warranty and security protections.  At least, Google does not provide support for a machine in this state.  The hacker community, then, is at the mercy of well-intentioned, but not always well-informed peers for advice.

I followed the available advice in the hacker forums into the corners that most others had painted themselves, finally figuring out an important missing step.  To use the Crouton system, the wily hacker must first use the official Google developer shell to set the root and administrator passwords.  Then, Crouton can access the administrative account to install the Linux container.  The less-helpful solutions offered essentially required reinstalling Chrome and starting over, which was not necessary at all, but the default obvious solution to developers and admins raised in the Age of Windows, where frequent rebooting and periodic reinstalling is considered a normal operational necessity.  Unix and Linux admins only reboot after installing a new kernel, which may be months or years, depending on the stability of the system and whether the management decides to incorporate the latest patches when they are released (many don’t, for fear it will break fragile applications, despite the security risk of not upgrading).

So, now Judy has a brand-new system that, hopefully, will be stable for a reasonable time.  The advantage of this system is that it is still Chrome, so she can use the user-friendly desktop apps available to Chrome, similar to the ones on Apple iOS, and switch to the Linux desktop with a simple key-combination to run the Linux applications we depend on.  A similar key-press combination switches back to Chrome.  And, the Chromebook, though large enough to have a full HD display, is light enough to be highly portable, with a very long battery life.  The minor concession to our ill-advised attempt to co-exist with Windows is that we now have a portable tax preparation machine.  During the rest of the year, we may turn it on once in a while to let the updates install and top off the battery, so that it won’t take three days to update next tax season.

Chromebook, running Chrome with Crouton in a browser window.

 

Chromebook, running Ubuntu Linux with LXDE, with a simple browser, and essential apps, including Fiberworks Bronze weaving software running under WINE. (a Windows emulator running in a Linux container under Chrome–boggles the mind, no?)

The Parkins Report: Events of 2017

As we move into the beginning of our ninth year of “retirement,” we are finally learning to take life as it comes, with minimal rush.  This includes being involved in activities that satisfy us, rather than from some sense of obligation or need (although there is still plenty of that to go around).

Travels

This year was again a year of travel. In January, we headed south the day before Inauguration Day.  The drought had broken in California: we drove in slushy snow in the north and rain in the central and southern parts of the state. The first week, we took Judy’s brother-in-law Ben from Anaheim to San Diego to visit her cousin Margaret, then headed east to New Mexico and west Texas: Las Cruces, El Paso, and Albuquerque, to visit Larye’s children, grandchildren, great-grandchildren, and great-great grandchild.  Then, it was back to California, via Flagstaff and Bakersfield, then through rain again to San Francisco for a week exploring the city before driving home.

While at home, we worked on our van conversion project, building a folding sleeping platform with room beside it for the bicycle. In April, we made a test run to Idaho, camping overnight to and from McCall, where we spent a week with our friends Gary and Char at a timeshare, getting in a couple of short bike rides despite the snow and wet of central Idaho. We toured the Painted Hills of central Oregon on the way back. While training for the summer bicycling season, we had a frame failure on our Bike Friday, prompting a trip to the factory in Eugene to have it repaired. That trip showed us the old van was not ready for our ambitious touring schedule, so it was back to the shop for some major repairs on that, too.

While our bike was in the shop, we dusted off our 31-year-old Santana tandem for a scheduled charity ride and ended up taking it to Victoria, Canada when we attended the Association of Northwest Weavers Guilds conference over the Canada Day weekend. After the conference, we rode parts of the local trails we missed in the spring of 2010.

At the end of July, we set off on Road Trip 2017, starting with a detour to Eugene to pick up our Bike Friday, then off to northern Idaho for another week with Gary and Char at their vacation home. We soon discovered that our old van had no working air conditioning, so we spent the next six weeks of summer heat reliving the nostalgic days of yesteryear when turning on the “factory air” meant cranking the side windows down.

From Idaho, we headed east, spending a week in western Montana, visiting relatives, some also visiting from Florida and New York, visiting friends in the Bitterroot, and checking out the new Experimental Aircraft Assoc. chapter hangar at the Missoula airport. Heading southeast through Wyoming, we got in some trail riding in Nebraska and a weekend in Lincoln to be there for the total solar eclipse on Monday. After a brief stop in southern Minnesota to drop off a family heirloom with cousin Cathy, we worked our way through Iowa, riding around Lake Okoboji in the northwest, then the High Bridge Trail north of Des Moines. We drove down the Des Moines River, posing for Grant Woods’ American Gothic painting before turning north up the Mississippi River at Keokuk.

At the Quad Cities, we bicycled along the Great River Trail in Moline, Illinois and up Duck Creek in Bettendorf/Davenport, Iowa. We continued up the Iowa side of the Mississippi, then along the Wisconsin/Illinois border and up to Middleton, to visit son Matt and family over the Labor Day weekend, getting in one family bike ride in the process.

Crossing over the Mississippi back in to Minnesota, we stopped in Shakopee to visit a newly found cousin on Larye’s maternal grandfather’s side of the family. We bypassed the traffic around the west side of Minneapolis and checked into a campground on the south end of the Paul Bunyan Trail to ride up the trail to Baxter. The next day, we met with more of Larye’s cousins for a weekend reunion in Baxter and nearby Motley, near where the clan’s great grandparents had homesteaded.
Following the reunion, we rode some more of the Paul Bunyan Trail, starting north of Brainerd where we had turned around two years ago. The next morning, we headed to North Dakota to spend a couple of days with Judy’s cousin Fred and his wife, Ann. Smoke from the fires in Montana made visibility poor, so we pushed on west toward home, bypassing a return stop with the Montana folks to get home after a long trip, with the rain coming in and snow starting in the mountains.

The last weekend in October, we went to Astoria, Oregon to camp at and ride the trails at Fort Stevens State Park, in perfect weather. Our riding was cut short by the first flat on the front tire, which has lasted through two back tires, nearly 6000 km (3600 miles) in six years. The casing is a bit thin in the grooves, and a tiny puncture in the thickest tread: we “retired” it to secondary spare status.

By the end of November, our wanderlust struck again, and we retreated to Long Beach for a few days on the beach, on the edge of winter, one of our favorite times, since the crowds of summer are long gone.  In their place, however, is cold rain.  We also finally got talked into upgrading our vacation club membership, despite uncertain financial future of our status as elderly poor.

A return trip to Vancouver, BC in December capped the touring season, with Char joining us this time, Gary stayed home with a sick pet.

Travel Hosts

Between our own tours, we host international bicycle tourists through the Warm Showers network. We had 14 in April and May, then restricted visitors to “by invitation only” while we were preparing for our summer tours, picking up two more, a weaver from New Zealand we met on Facebook and a 69-year-old world traveler from Australia we met at the Olympic Bakery near Spencer Lake and invited to drop by on his way through Shelton.  On our return in the fall, we took in six more tourists before the rainy season and cold weather.

Transitions

As the rainy and cooler weather arrived in mid-October, Delia, our feline companion for the past 17 years, lost her struggle with kidney disease, just short of her 21st birthday. She had come to us in Missoula in the spring of 2000, a 3-1/2-year old “pound kitty,” wary of people in general. Over the years, especially after the demise of our other pound kitty, Nicolaus, in February 2005, she warmed to us and spent many hours of lap time in front of the fire. She also came to enjoy the attention of the many bicycle tourists who passed our way. She saw us through four houses and spent a lot of time “vacationing” at Pampered Pets in Darby, Montana and Just Cats Hotel in Olympia, where she was a favorite guest over the last eight years. She had been in poor health for about a year, but rebounded in the spring and summer, her favorite times of the year.

We welcomed a new great-great-granddaughter, Bea, in August, who we have not yet met. Bea joins her brother, Hyperion, in our growing and dispersing family. Visiting family takes longer now that grandchildren and great-grandchildren are becoming adults with their own households and schedules. Judy made a trip back to her hometown, Sunnyside, Washington this fall, for a family gathering of cousins, many of whom she had not met or had not seen for many years: Larye had a weaving class scheduled, so did not attend.

Lifestyle

For the first time in more than a dozen years, we have television, the result of upgrading our Internet service, which came bundled with a TV offer. The set is installed in Judy’s upstairs craft studio, which we furnished with a thrift shop small sofa. However, only a few available programs have piqued our interest so far, so the space has become just another reading room in the evenings. Public radio, both broadcast and satellite, remain our primary source of news and entertainment, along with selected video clips on the Internet.We continue to regularly practice yoga at the local senior center (when we are in residence), and attend the Ruby Street Art Quilters group in Tumwater. Judy completed a project for an exhibit at a brew pub in Olympia, and Larye finally finished a 2012 class project quilt as a baby quilt for Bea. We also joined the Friends of the Shelton Timberland Library this year and spend one afternoon a week sorting and pricing donated books and restocking the sale shelves, from which the proceeds support youth programs at the library.

We are still active in both the Olympia and Tacoma Weavers Guilds, and Larye manages the web sites for both. We both attended classes at the conference in Victoria this summer, and Larye attended a class in Olympia this fall, but not much progress on projects during this year. Between our travel schedules and taking care of our ailing cat, there simply hasn’t been a lot of time to actual work on the hobby projects for which we belong to the many organizations.

Find our videos on YouTube: Larye’s YouTube Channel, or view a summary of our bike touring season below:


and on Vimeo: Larye’s Vimeo Channel

Home-grown Webcam Evolution

Good, fast, or cheap: pick any two. A few years ago, I decided to build a webcam, rather than buy one, which were about $100, plus whatever monthly service charge for hosting the link on the cloud. I’m not sure I beat the cost, quality, or speed, but it’s kept me actively managing the system. Instead of a plug-n-play wifi-enabled little module, I have a rats-nest of wires, USB hubs, USB external disk drives, Raspberry Pi with external camera on a ribbon cable, and, now, extension cords and 50-foot CAT5e cable. About once a year, I wear out the flash drive that the system runs from, so there is some on-going cost. Plus, much coding in Python and Bash, a distributed network system to process the video, cron jobs, an API key and code to get weather information and sunrise/sunset times to turn the camera on and off.
 
Meanwhile, the landscaping has grown up around the office window, so the camera sees mostly flowers and bees (left view). So, I moved it to the office closet, which was not so simple. 1) Being “cheap and fast,” the software wasn’t very “good,” so I had to modify the Python code to provide a way to restart the system during the day without losing all the footage: the system keeps a week’s worth of data, and erases last week’s when starting a new day. This also entailed generating images with a timestamp, rather than a simple index, as the camera software libraries start indexing at 1 each instance.
OK, that’s done, and the system retested, bugs fixed, etc., which ended up losing most of a couple day’s surveillance: “cheap” means not having a second system for development and test, and “fast” means not doing a proper code review before testing, which leaves “good” out of the equation.
Of course, nothing ever goes smoothly: after moving the computer/camera, the USB hub and disks into the closet, we weren’t getting communication with the processor.  So, drag everything out next to the desk so we could hook up a console (keyboard, mouse, and monitor) to the computer, retest with the original ethernet cable, then with the long one.  Everything worked, inexplicably, since nothing really changed except having the console hooked up.  Unhooked the console, and moved everything, still running, back into the closet, then adjust the camera  view, and we’re done–except for resetting the key agent so the computer could talk to the video processing computer.
Our program takes a photo every 10 seconds, updated to the web server, then assembles a timelapse video once an hour, showing one hour in 30 seconds.  After letting the revised program run for a couple hours, we checked the logs and directories: still showing last week’s video.  Aha.  The video compositor program needs a numerical sequence for the images in order to assemble a video: the timestamp doesn’t meet specification.  So, back to the drawing board, rewrite the Bash script on the video processing computer to renumber the files in a format the video assembly utility understands.  Success at last.  The system is now fully functional, but made a bit more complex by the simply addition of a restart ability.
The results can be viewed at http://www.parkins.org/webcam

So, not fast, not good, and not cheap, when you consider the effort put into a custom, one-of-a-kind system. But, it keeps me in practice coding and designing.  And, because it runs on Linux, I can keep the security patches current: many purchased plug-and-play “appliances” have their code burned in at time of manufacture, and may be designed around already obsolete and buggy software.  My little system has undergone several major upgrades of the Debian Linux distribution core system  (Linux kernel 4.9.35, patched 30 June 2017: latest release is 4.12) and gets regular security patches and bug fixes.  That’s even newer than my primary laptop (Kernel 3. 13.0, patched 26 June 2017).  Considering all the little Rasperry Pi machines scattered around the house, it may be prudent to work on configuring them for diskless boot, in order to preserve the flash memory chips on-board.

Not your plug-n-play webcam…

Beware the Wiley Hacker: a Cautionary Tale

Not an approved method of securing your network...
Not an approved method of securing your network…

A while back, we wrote about rebuilding a crashed Raspberry Pi system.  In the course of reinstalling the system (on a new chip–the old SD card that contains the operating system had “worn out”), we had made a fatal slip.  This system happens to be our gateway system, i.e., connected directly to the Internet to provide us access to our files and some web services while out of the office.  Unfortunately, this also provides the opportunity for the world-wide hacker community to try to break in.

Normally, we have safeguards in place, like restricting which network ports are open to the outside and which machines and accounts are allowed login access.  However, in our haste to get the new system up and running as quickly as possible, we connected the device to the Internet to download upgrades before the configuration was complete, meaning the system was exposed without full protection for several hours to several days.

Screenshot-larye@raspberrypi2: -var-log
One hour’s worth of break-in attempts by hackers. Note that attempts to access system accounts (root, pi) are denied because we don’t allow external logins for these accounts.  The accounts that are allowed are restricted to public-key authentication (basically, a 1700-character random password).  Attempts on this one-hour snapshot come from four different sources: Porto, Portugal; Shanghai and Baoding in China; and Tokyo, Japan (possibly hacked machine, as the name is spoofed).

Now, our security logs record, on a normal day, hundreds of break-in attempts (see screenshot above).  We aren’t the Democratic National Committee or Sony, just a small one-man semi-retired consulting business.  But, the hackers use automation: they don’t just seek out high-value targets, they scan the entire Internet, looking for any machine that isn’t fully protected.  If they can’t steal data or personal information, they will use your machine to hack other machines.  If you use Microsoft Windows, you are undoubtedly familiar with all sorts of malware, as there are many tens of thousands of viruses, trojan horses, adware, ransomware, and other malevolent software that invades, corrupts, and otherwise takes over or cripples your machine.  Unix systems are less susceptible to these common attacks, but, if an account can be compromised, or a bug in the login process exploited, eventually a persistent attacker can gain system privileges and install a ‘rootkit,’ a software package that replaces the common monitoring and logging software, redirecting calls through the rootkit, which hides its existence and activities from the reporting tools, even the directory listing utilities.

Once an attacker takes over a Unix or Linux machine, there is no limit to the damage they can do on the Internet, as Unix/Linux is the basis of most of the servers on the Internet, and can become as a SPAM server, web-spoofer, or hacker-bot itself.  (Microsoft Windows Server has the rest, nearly half, and they are even easier to break into.)  I began to suspect this might have happened when normal system functions failed to terminate or run correctly.  We have a lot of custom software built on this machine, which runs on a scheduler.  The machine got slower and slower, and it was apparent that the jobs run by the scheduler were never exiting, filling up the process table with jobs that weren’t doing anything, except taking up space, which is always at a premium.  Clearing out all the jobs, restarting the machine, and starting the processes manually worked–until about 2:00pm.  Very suspicious: a rootkit, once installed, can repair or re-install itself even if the administrator restores many of the co-opted command files by normal upgrades or by a conscious attempt to recover from the intrusion.

The main problem seemed to be with /bin/sh, the system shell, which is actually /bin/dash, a shared object.  Cron, the scheduler, uses dash to run the jobs, where the normal user login shell uses /bin/bash, a non-linkable executable shell with similar functionality.  A rootkit is generally constructed as a filter, wrapped around the co-opted commands, so it would be easier to link to the *real* /bin/dash in an undetectable manner from the filter program than it would to wrap /bin/bash.  In this case,  assuming an intrusion was the cause, something went wrong, rendering dash non-functional.  Perhaps the intrusion was not compiled for the ARM  processor used by Pi, though most of a rootkit would be scripted to be portable among different CPU architectures and Unix/Linux versions.

An analogy to the problem would be like finding out who let the horses out: it is easy to identify wolf or horse-thief tracks outside the barn when the door is barred, but, if you left it open and the horses have bolted, it is more difficult to find out what happened–the traces are covered. I did install some intrusion-detection software, but running it after the tracks are covered over is usual a futile effort.  However, there were enough questionable traces to warrant taking corrective action.  Besides, even if the problem had been caused by some inadvertent misconfiguration on my part (unlikely, considering the fact that the machine could be made to run for several hours before the problem reasserted itself), the solution was clear:  reinstall everything.

The first step is to backup the data, including configurations.  Now, this is not just an ordinary computer:  Raspbian, the Debian-based operating system distribution designed for the Raspberry Pi computer, comes with a simple desktop intended to introduce new users to Linux.  But, this machine doesn’t use the desktop and is not even connect to a monitor most of the time: it is an internet gateway, web server, and custom webcam driver, so has a lot of “extras,” both loaded from software repositories and written especially for this installation.  Backups are important, since much of the software only exists on this machine.  And, since we only have one camera, fail-over isn’t possible without physically moving the camera from one machine to another, not a trivial exercise, as the connection is on the motherboard rather than an external connector.

Now comes the glitch: Since the introduction of the Raspbian operating system, it has been based on Debian 7; but, since Debian 8 was recently released, a new version of Raspbian is also available.  So, the machine was rebuilt with Raspbian “Jessie”, replacing Raspbian “Wheezy” (the releases named after Toy Story characters rather than just the numbers–as with Apple OS/X, Debian releases tend to have names in addition to release numbers).  Installation on Raspberry Pi is not like other computers.  Since there is no external boot device, the operating system “live” image is loaded onto the SD card that serves as the boot device and operating system storage.  Initial configuration is best done without a network connection, since the startup password is preset and well-known.

So, avoiding that mistake (booting on a network with the default passwords, the single most preventable source of hacker intrusions), we booted with the network cable disconnected and a monitor and keyboard attached, changed the password and expanded the system to fill the SD card, set up the other user accounts, then shut down the system, removed the SD card, and mounted it on the backup server to finish transferring vital data, like the security keys and system security configurations.  In larger systems with a permanent internal boot drive, such “hacker-proof” installation is done on an isolated network, but, since the boot drive on a Pi is removable, it is easy enough to edit the configuration files on another system.

So, with the system fairly well hardened by securing the system accounts and user accounts, it was rebooted attached to the network and the system upgrades and extra software packages (like the web server) installed.  So far, so good.  But, since we upgraded the operating system, the server packages were also upgraded, most notably moving from the webserver, Apache, version 2.2 to version 2.4.  Apache has been the predominant web server software on the Internet for 20 years, so it is in a constant state of upgrade, for security and feature enhancements.  Between version 2.2 and 2.4, many changes to the structure of the configuration files  were made, so that not only did the site configuration need to be restored manually, but there was a fairly steep learning curve to identify the proper sequence and methodology by which to apply the changes.

Then, of course, were the additional Python modules needed to be installed to support the custom software, which involved downloading and compiling the latest versions of those, since Python 2 also upgraded from version 2.7.3 to 2.7.9 (we haven’t yet ported the applications to Python 3, which moved from version 3.2.3 to 3.4.2 between Debian 7 and Debian 8).  Finally, there were other tweaks, like comparing system configuration files to update group memberships for access to the camera hardware, loading the camera drivers, and setting file ownership and permissions for data and program files.

We could have saved most of this by sticking with Raspbian Wheezy, but eventually, support for older systems goes away, and the newer systems are usually more robust and faster: open source software evolves rapidly, with new minor releases every six months and new major releases every two years for most distributions, and an average life span of five years for maintenance of major releases and a year for minor releases.  As we said before, Linux is free, if your time is worth nothing.  The price of keeping current is constant maintenance.  Patch releases occur as they are available, with maintenance upgrades almost daily.

Finally, after a week of tweaking and fiddling, the webcam service is back up and running.  And, the security logs show break-in attempts every few minutes, from multiple sites all over the world (one from Portugal recently, others from unassigned addresses–ones with assigned addresses undoubtedly come from computers that have been compromised and used as hacking robots, as hackers don’t want to be traced back to their own computers, ever).

So, the moral of this post is:  don’t ever expose a stock, unmodified computer system directly on the Internet (which is difficult to do, when all upgrades, new software, etc is available only through download from the Internet–which should be accomplished only from behind a proven firewall).  But, you can set passwords and change system accounts before joining a network.  And, if your computer is hacked, take it to a professional, and don’t grumble about the cost or time it takes to restore it.  Pay for malware scanning software and keep your subscription up to date, as well as scheduling upgrades on a regular bases.  And, if you are a professional, don’t take shortcuts (i.e., install and configure off-net or behind a firewall), keep good backups, install intrusion-detection software early, and check for security upgrades daily.  Change the default passwords immediately, and create a new, weirdly named administrative user, and deny external logins for all administrative users.  Use two-factor authentication and public-key encryption on all authorized user accounts.  They are out there, and they are coming for your computer, even if you don’t have data worth stealing: they can use your computer to spread SPAM or steal data from someone else.