MythTV Tweaks

After using and tinkering with MythTV for over a year, a small pile of adjustments have stacked up that have been made to both the backend and the frontend that have improved performance. These can’t be listed as must have’s, but they definitely are good to have. Some of these may seem obvious, but not everyone sees things the same way.

1 Store Your Recordings on a Separate Drive (or drives)

The installation of mythtv recommends storing your recordings on a separate partition from the rest of the operating system. This is done to insulate the recordings files from the key operating system files, and, in general, it is a Best Practice. However, for optimal performance, this should be taken a step further: Set up a separate drive or drives to hold your recordings. This will separate all of the disk interaction between the OS and its related files from mythtv and its recordings. The result will be a smoother end user experience.

An internal SATA drive is probably the best and least expensive solution. An external SATA drive is good, too, but more expensive. A PATA drive would work well, but it is best to keep that drive on a different controller than the drive that your OS is on.

2 Reduce The Size Of Your Guide

By default, MythTV will collect 14 days of guide data to get information about future shows. This is actually great for searching for shows, and performing other user-related options, but it can wreak havoc on the system performance. A dead give-away is in th mythbackend.log file:

2007-12-23 09:31:54.099 Scheduled 871 items in 12.4 = 0.20 match + 12.17 place

If the scheduled items creeps up too high (typically greater than 700), then your system is working pretty heavily to schedule a lot of shows based upon the 14 days of guide data. This can cause the database to become rather large and slow down all database queries, which, in turn, slows down the end user experience.

A happy medium is reducing the number of days of guide data from 14 to 9.

3 Optimize the IVTV Paramaters

The default kernel parameters for the IVTV based cards do not take advantage of the buffers on the cards. This means that you will see a lot of logged error messages referring to “application not reading fast enough”. This could result in choppy recorded video which is a Bad Thing(tm).

options ivtv enc_yuv_buffers=8 enc_mpg_buffers=32 enc_vbi_buffers=8 enc_pcm_buffers=8

4 Know Your Hardware

One of the largest problems is that the underlying nature of MythTV is to take full use of the hardware it has at hand. One cannot help but try to squeeze everything (and more) out of their hardware. However, keep in mind that if you have a single processor frontend/backend system, you don’t want to be recording too many shows at any one time or running too many concurrent commercial flagging processes because the single processor just cannot handle it.

The most optimal set up is to have a fully networked frontend/backend system:

  • Have a frontend (client) system for each television in the house. It is probably best to have these as identical systems with minimal hardware requirements since they will only be acting as display systems.
    • NFS mounting of videos, mp3’s, photos, etc.
  • Have a backend (server) system stowed away in a closet. This will be the system that you will end up beefing up. A dual-processor system is recommended since it will be performing all of the work (recording, streaming to the frontend processes, commercial flagging, etc).
    • SATA storage for recordings
    • Separate storage for video, mp3s, photos, etc.
  • 100MB networking from the backend system to each of the frontend systems.

By having the same frontend systems, if you ever have a failure of a frontend, you can always easily shuffle them around while you repair or replace the failed components.

The same goes for remote controls. Although the temptation is to just reuse the multifunction remote you already have for your TV, cable box, Tivo, etc, it just isn’t worth the hassle. Simplicity in the setup will definitely improve everyone’s WAF (Wife Approval Factor). If each frontend has the same remote (although configured specifically for the equipment attached to that frontend), then, as a user (not the admin), you just have to learn the buttons on the one remote, and it applies to all of the frontends.

Grab yourself an inexpensive multi-function remote like the OneForAll (http://www.amazon.com/gp/product/B0000ALPBA) for each of your frontend systems. I was very hesitant at buying a remote I couldn’t test out since many multi-function remotes disable certain buttons based upon the mode you are in. However, this remote specifically has a PVR mode and you can move button functions from one mode to another.

5 Know What You Are Displaying To

The K.I.S.S. (Keep It Simple, Stupid!) Principle really helps when maintaining your MythTV systems. By having your frontend system as similar as possible, the level of administration needed on those system drops. In other words, if you tinker with one, you know you just have to make that same change on all of the others. Unfortunately, it is harder to control what types of televisions you are displaying to.

Remember: essentially, your expensive TV screen is just a simple computer monitor at this point. You are not using the internal tuner and you are not using all of the various input (DVD player, Antenna, Cable, etc) connections. You are just using the one input on the TV that is coming out from the computer. It is nice and simple.

IF it is possible, try to stick to the same kind of input, even though you will end up with different TV’s throughout your home. For example, if your TVs all have component input as the best quality input, then make sure that all of your frontend systems can output component video.

Try to avoid using any of the ‘PC’ inputs on TV’s. Component, HDMI, Composite, S-Video, and coaxial are all bound by TV broadcast standards. Once you start fiddling with the PC inputs, you have a lot of extra work to perform in order to get it to properly display the broadcast video on the screen. Stick with the standards, and you will be satisfied.

In my case, all of ours have HDMI connections on them. Some video HDMI as evil, but that’s another topic all together. I got away with purchasing some inexpensive DVI to HDMI cables (10ft, ~$10ea. on-line).

By keeping things similar or the same, the X configuration for each of the frontend systems will be the same.

6 Optimize the database

The MySQL database behind MythTV takes an enormous beating as shows are recorded, scanned for commercials, watched, bookmarked, and deleted, just to name a few causes. It does not take a very long time for the database to become more ‘fragmented’ and provide slower responses due to the increasing level of ‘fragmentation’. This has a direct impact on the user experience in a very negative way.

The solution to this is to periodically optimize the database. The problem with optimizing the database is that it is intensive and will only make things worse before it makes it better. So, it is much better to run this off-hours.

The easiest solution is to run a wrapper around mythfilldatabase which optimizes the database. But, because mythfilldatabase is run according to the grabber’s suggested schedule, this might end up running in the middle of the day.

What I found as the happy medium between the two was to craft the wrapper such that the database gets optimized only one day a week:

#!/bin/sh
# mythoptdb – Call this in mythfilldatabase’s place in
# mythtv-setup

if [ `date +%a` = “Sun” ]; then
optimize_db.sh
fi

/usr/bin/mythfilldatabase –remove-new-channels –quiet –max-days 9 $*

The “optimize_db.pl” script that comes as part of the contrib from MythTV has one shortcoming: it doesn’t always work properly. It worked for me for a long time, then started hanging on the optimization/repair when my database grew rather large. I ended up replacing it with optimize_db.sh which does the job more efficiently. Still, you must keep in mind that the database will be unavailable during the optimization/repair of all tables. The goal is to keep this (very) short.

7Understand the order of your inputs

Setting up MythTV can be very confusing: there are so many options in the mythtv-setup screens that is can be easy to either miss an important option or be unable to find that elusive option later on. Add in to the equation the fact that there are easily just as many (if not more) options in the frontend setup menus, it becomes almost overwhelming.

One item that took a couple of emails to the mailing list to figure out is the ordering of the tuners. When recording, the backend server process is set up to select the first available tuner, and when you go to watch live tv (which I almost never do now-a-days), it also is set up to select the first available tuner. Additionally, if you are watching live TV and mythtv needs to record something, you get the expected warning that it needs the tuner for that recording.

All of that sounds normal, and it is. However, you can change the settings to have mythtv’s backend select the tuners in reverse order so as to avoid conflict

8 High Definition recordings vs Standard Definition recordings

Standard Definition (SD) recordings, when displayed via MythTV on that nice screen just don’t look as good as that old 4:3 tube television. The problem there is that SD is broadcast as interlaced video, and the computer (the mythtv box) only displays in progressive mode (see this great description of the differences). That, and the quality of SD is so low, many of the artifacts that you didn’t notice previously now show up amplified because of the larger screen size, the better screen quality, and the MPEG2 encoding. However, SD has one advantage: it is small. Typical hour-long shows run about 300-500MB.

High Definition (HD) recordings are an entirely different beast, and they come in many different flavors. Aside from an awesome increase in picture quality, they all have one important factor in common: they are huge. A typical 1 hour recording can run between 3 and 6 gigabytes.

PLAN YOUR STORAGE ACCORDINGLY.

My experience is limited to only Over-The-Air (OTA) HD transmissions from the local affiliate stations, but I definitely recommend the HDHomeRun for an OTA setup. MythTV supports it right out of the box, it is easy to configure, and you can get HD recordings set up within minutes.

Obtaining HD recordings off of your cable/satellite box is an entirely different beast, and, unfortunately, I have not gone down that route just yet.

BE PREPARED TO ADMINISTER THIS SETUP

MythTV is still in a beta phase, and, although there are a lot of people who are using it, it still is not entirely ready for prime-time; the system isn’t made to run entirely on its own and it does need some tlc. I cannot stress this enough: don’t go cheap on your backend system’s motherboard. Make sure that it is a solid board with well supported chips. Some inexpensive, and tempting, motherboards use chipsets that work well as windows systems, but perform with numerous problems under Linux (some Jetway, PCChips, MSI, and some via-based motherboards) which will only give you major headaches.

This is not to say that you couldn’t have a successful system that you don’t need touch for weeks on end, but it is likely that things may creep up that you, and no one else, expected. Besides, you will want to tinker with it over time to fully optimize the system.

A great example is the impact of cpu frequency scaling on a MythTV backend: don’t do it. If you have multiple cards and a couple of drives (OS and video), cpuscaling can wreak havoc on the system: ivtv drivers have been known to stop responding (requiring the backend to be shut down and the drivers unloaded and reloaded) and active/busy SATA interfaces have been known to falsely believe that there are ext3 journal errors which results in the drive being put in to read-only mode.

With all of the above said, you can create an excellent home-wide DVR system using COTS (Common Off The Shelf equipment) that is DRM-free, completely customizable to your liking, and extremely versatile.

Openvpn + Windows = One Awesome VPN Setup (Quick and Easy!)

While at a previous employer, I came across an interesting article in the now-defunct SysAdmin magazine which covered, soup-to-nuts, setting up a rather basic OpenVPN server and NSIS-based Windows client package. It wasn’t long before I got hip deep in to customizing the Windows installer NSI script, and tweaking the server configuration. Eventually, I configured up a Solaris 8 system in our data center to act as a secondary openvpn server (using the “tun” package), and all of the adminstrators/dba’s were very happy with the results.

Nearly two years and two employers have passed by, and I find myself going through a similar setup. Unfortunately, i discovered that the backup of the files I took when I left my employer did not contain some of the key files, so I had to start back at square one. The article was a great starting point, but I was worried that, since the magazine was now defunct, the article might not be around for much longer (although, as a subscriber, I received a CDROM containing all of the issues).

The article was also rather basic, and not very security conscious. Additionally, some key steps (such as managing client certificates) were completely left off. I’ve attached the original article for posterity.

The Server

To start off, the server needs to be set up and configured. Some key items to keep in mind:

  • The standard OpenVPN port is 1194. However, there is nothing stopping you from running it at a different port.
  • Decide between a “bastion host” setup versus a “completely integrated setup” (I’ll cover these in a moment).

On an RPM based distro, install the openvpn package. In addition to the core openvpn components, it will also install a series of SSL/cert management tools in a directory at /usr/share/doc/openvpn-<version>/easy-rsa. Copy that entire directory to /etc/openvpn/easy-rsa. Now, cd to /etc/openvpn/easy-rsa, and take a look at the README file to familiarize yourself with the steps. The general breakdown is:

  • cd /etc/openvpn/easy-rsa
  • edit the “vars” file to your liking. The only things I changed were the KEY_SIZE to make it 2048, and the KEY_ values to reflect the company’s information. Save the file.
  • export KEY_CONFIG=/etc/openvpn/easy-rsa/openssl.cnf
  • export KEY_DIR=/etc/openvpn/easy-rsa/keys
  • . vars
  • ./clean-all
  • ./build-ca
  • ./build-dh
  • Build a key for the server: ./build-key myserver

Now that the certificate and keys have all been made for the server, there are a few additional steps to set up the directory structure and generate a server port key:

  • cd /etc/openvpn
  • ln -s /etc/openvpn/easy-rsa/keys keys
  • openvpn –genkey –secret port1194.key
  • useradd openvpn (if the account hasn’t been created already)
  • echo 1 > /proc/sys/net/ipv4/ip_forward

At this point, all of the security components are set up and actual configuration of openvpn is necessary. I’ve attached a modified version of the sample server configuration file, but the highlights of the file include forcing out DHCP entries, setting routes, communicating via UDP only (UDP is much more efficient than TCP in this situation, so use it if you can), and logging.

Openvpn can handle multiple port configurations as it will read the /etc/openvpn directory and look for all files ending in “.conf”. In most cases, you will only be working with one, but it is nice to know it supports multiple.

Once the configuration file is in place (openvpn Port Configuration File) in /etc/openvpn, start up openvpn via /etc/init.d/openvpn start and check the log files in /var/log/openvpn for information. The server is up, and now on to the clients.

The Clients

The poor-man’s way of doing things is to create one client certificate and hand it out to everyone. While this works, the down side is that it makes it next to impossible to seamlessly deactivate any one person’s access. The best practice is to create individual certificates for each user, and provide them to the individual users. Since the server has a copy of the user’s certificate, you can easily (re)move a certificate to disable a given user.

Server Side Configuration

Create a user certificate via:

  • cd /etc/openvpn/easy-rsa
  • . vars
  • export KEY_CONFIG=/etc/openvpn/easy-rsa/openssl.cnf
  • export KEY_DIR=/etc/openvpn/easy-rsa/keys
  • ./build-key clientname

At the end, there will be three files created:

  • clientname.crt
  • clientname.csr
  • clientname.key

Remember these. We’ll need two of them (the .crt and the .key) for later use when they get copied over to the client.

Client Side

The article has some good coverage of the steps necessary on building the Windows client via NSIS, and, personally, I think it (NSIS) is awesome. After installing NSIS and downloading the openvpn gui (from the link in the article), I was testing the installer in no time. Now, I did have to tweak the nsi file a number of times, just to get the default version working since it seems that the NSIS plugins have been updated to conflict with some settings in the default NSI. So, I modified that NSI to get the final version of the openvpn.nsi file that is fully branded, installs our ca.crt (which was copied over from /etc/openvpn/keys/ca.crt), sets up all of the start menu items, and installs our client configuration file.

With the installer package built and tested, it can now be supplied to the user, along with the clientname.key and clientname.crt files and instructions as to where to place those two files.

Bastion host versus Complete Integration

Getting the user(s) access to the network via VPN is a good chunk of the battle, but you really have to decide how you want to support this new VPN network. Right now, the only access they really have is to the server running the openvpn process. The other machines on your network do not know how to communicate back to this new network you created (remember, you’ve just turned this server in to a router).

A bastion host is a host that sits within your network that your VPN user(s) can log in to, and, from there, jump around to the rest of the network. It is essentially a terminal server for those Windows folks out there.

However, if you would rather provide direct access from the machines coming in via the VPN, you would have to either:

  • On each host on your network, add a route to your new vpn network with your openvpn server as the gateway, or
  • On each router in your network, add a route to your new vpn network with your openvpn server as the gateway.

Read more ›

Save Those BlackBerry SMS Messages

The party line from both RIM (makers of the Blackberry) and Verizon was that there was no way to move SMS messages from one Blackberry to another.  The provided software from both parties only allowed backing up the contacts, email, and any related data (photos, video, etc), but they both stood hard and fast stating that those were the limits.

Unfortunately, we had a user who absolutely needed to have the SMS messages copied from their 8830 to their new Blackberry Curve, and they wouldn’t give up the old phone until the messages were moved over.

Linux to the rescue!  Using the tool called barry, we managed to perform a full backup of the 8830 and successfully restore it to another 8830.  Unfortunately, we couldn’t restore it to the Curve because barry wouldn’t recognize the curve with the version of barry that we had (0.12).  Additionally, we couldn’t modify the backup file (a tar.gz file containing directories and binary files of the messages) in any way (not even a simple untar/uncompress and immediate tar and re-compress) since barry would immediately complain about the contents.

So.. here we had a backup, but couldn’t get it on to the curve.  Some browsing through the mailing list archives for barry turned up an obscure reference to the CVS repository for another fix, so we checked out the CVS repository, built and installed, and gave it a shot.

It worked to a certain level.. It transferred 78 of the 128 SMS messages, then hung.  A second attempt (after killing off barry) got to 108 messages before hanging, and the end user was happy with the 108 messages (they included the ones they wanted to save) and we left it at that.

Wiping the blackberry was easy:

  1. Go To Options
  2. Select Security Options
  3. Select General Settings
  4. Click the Menu key and select Wipe Handheld
  5. Click Continue and follow the prompt by entering the word blackberry

The unit will then wipe itself and it will be back to factory defaults.