nexus

Hardware fun

I'm a cheap bastard. The two computers that I own are both salvaged from an electronics dumping area at work. Both are old. PCI, AGP4 only.

But I love to load them up with lots of disk space. The machine I'm writing about now has one 500G PATA drive and one each of 1T, 1.5T and 2T SATA disks in it.

A couple of days ago, my brand new Western Digital Green started acting up on me. Considering the nature of my hardware, I actually suspected a bad port. After all, of the four SATA ports on the motherboard, I think two are bad. And of the two 2-port SATA cards laying around, I think some of those are bad, too. But the machine had been stable for a while and I got too distracted by other things to sit down and really work out what was bad and mark them as such.

Now, years ago I'd sworn off of WD drives. Never again, I said. I'm beginning to think I should have stuck with that mantra. But, I wasn't a fan of Seagate either and had lots of success with Maxtor over the years. Probably the opposite of everyone else in the universe. And now that Seagate has owned Maxtor for a while, I decided to get WD a try. And it worked, for about two months before it started dying. Currently it stays online as long as I don't write anymore content to it. So much for IDEs automatic remapping of bad sectors (smartctl seems to think it has plenty to remap to).

So, to try to isolate the fault. First thing I tried to do was boot from WD's tools CD. And I promptly ran into a problem. Not long ago my PATA DVD drive died. So I replaced it with a SATA DVD-RW. And, until now, I've never bothered trying to boot from it. Guess what doesn't work?

The CD burn was good, I booted the laptop using it (quickly complained that there were no WD drives attached). Trying Seagate tools, which I know I've booted this machine with before, no luck. Maybe it was never designed boot from a SATA CD/DVD. I gave up. Another annoying part about this was, I just took something like a dozen old CD drives into work and dropped them off in that dumping area. As I'm writing this, I think I may have one that is hiding out in the garage (don't know why, just remember thinking I should put that with the rest).

Instead, I went to Fry's and bought a new controller. Hoping that maybe a new controller would give me better luck. Hah! Do you know how are it is to find a 4-port SATA card that is for PCI and NOT PCIe? It took me a while to find them, and every single one of them had one of those infamous Fry's REDUCED PRICE stickers. Someone else has already tried them and returned them. Oh, boy. Well, at least it says Linux is supported on the package, and only $30. Let's give it a try.

No luck. Hangs on every boot identifying drives. I did a little research and found out that the ``new'' Sabrent SBT-SRD4 card I bought is known to ship will an ancient BIOS, and will hang any time a disk 1TB+ is attached to it. And I didn't have anything smaller! Turns out the shipping BIOS is dated from 2004 and the most recent SiL 3114 BIOS (chipset on the card) dates from 2008. Cool. Except I need to burn an ISO to boot to FreeDOS as I don't have any USB drives handy. Oh wait, I can't boot the machine from CD. Crap.

Oh, it turns out that Linux has a utility called flashrom, and it just so happens to support the SiL 3114 flash mechanism. Now to figure out how to use it. Turns out to be fairly easy.

Backup the old ROM:
# flashrom -p satasii -r backup.bin
# strings backup.bin | head -n 4
wSILICON IMAGE
SIMG
5.0.35
02-25-2004
# file backup.bin
backup.bin: BIOS (ia32) ROM Ext. (100*512)

After downloading the newer BIOS from www.siliconimage.com:
# strings b5500.bin | head -n 4
SILICON IMAGE
SIMG
5.5.00
04-22-2008
# file b5500.bin
b5500.bin: BIOS (ia32) ROM Ext. (120*512)

Looks promising.

# flashrom -p satasii -w b5500.bin

Went smoothly.

Rebooted. Good. Shutdown, reattached all of the big drives, booted and SUCCESS! All drives noticed and my LVM is back on line.

Now, for the final test, can I write to the WD drive?

No. Shit. Tomorrow night I'll dig out a USB drive, save off the contents, then hopefully make it bootable with the WD tools. Or swap out a drive from another machine. But for now, time for bed.
nexus

Migrate grub2 to new boot disk

I had a hard time finding simple instructions for this chore. Most of the hits referenced older versions of grub which no longer applied. These may not be the best instructions, but they worked for me.


On one of my Linux machines, my boot disk was failing, to the point where sometimes it wouldn't be seen by the BIOS on boot, but fortunately, still worked well enough that I was able to copy everything off of it before it totally died.

That disk has one big partition for the OS; all user data is on other disks.
I used UUID for everything in /etc/fstab and the grub configs. After all, each time I boot, all of my entries in /dev are different each time.

I actually moved to a smaller disk, but since this is only the OS, that's fine. It takes up less than 5G.



I attached the replacement drive and booted. In this case, it showed up as /dev/sdb.

fdisk /dev/sdb
# delete existing partitions
create primary partition 1 for 20G
create primary partition 2 for rest of disk
make partition 2 swap
make partition 1 bootable
write updates, exit fdisk

mke2fs /dev/sdb1
mkswap /dev/sdb2
mount /dev/sdb1 /mnt
cd /
find -xdev -depth | cpio -pdmv /mnt     # have lunch
umount /mnt
tune2fs -j /dev/sdb1    # copying into ext2 is faster than ext3
tune2fs -o journal_data_writeback /dev/sdb1    # saner option
mount /dev/sdb1 /mnt
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
chroot /mnt


At this point, my memory is a little fuzzy (fighting a cold while I was doing this and forgot to write down what I actually did that was successful)

grub-install --recheck /dev/sdb
grub-update


Maybe both commands were needed?


Get UUID for the filesystem:
file -s /dev/sdb1


Check /boot/grub/grub.cfg and make sure it's using the correct UUID and not the old one (this gave me some grief for a while).

Then update /etc/fstab to have / and swap use new UUIDs:
file -s /dev/sdb1 >> /etc/fstab
file -s /dev/sdb2 >> /etc/fstab
vi /etc/fstab   # edit in the obvious manner

exit   # to get out of the chroot
shutdown -h now 


Disconnect the old dying drive, turn machine back on

All worked.



Now, in theory it should have been possible to do this without the bind mounts for proc + dev (not even sure proc was necessary). It looks like this should have been all that was necessary:
grub-install --recheck --root-directory=/mnt sdb


But that didn't work for me. Not sure why.
nexus

Installing Galleon on Debian Linux

Galleon is a piece of software that interacts with TiVo's to transfer shows back and forth and provides applications that can be ran on the TiVo. It can be found at http://galleon.sf.net/. I primarily use it to archive shows off the TiVo before they expire.

I recently started having with my 2 year old installation of release 2.5.1 and figured it was time to update to 2.5.5. I did this as a new installation so that I could document anything that didn't work out of the box with their installation instructions, which are pretty straight forward.

I'm using Debian Testing, only a day or two out of date.

I think the only additional package that I needed is Java:

apt-get install sun-java6-jdk

download galleon (as a .zip) from galleon.sf.net
mkdir galleon
cd galleon
unzip ../galleon-2.5.5-linux.zip

as root from the same galleon directory:

make install

Initial server start:
/etc/init.d/galleon start

Monitor /var/log/galleon/wrapper.log, and should eventually see a line that says:
Galleon is ready.

Make a minor fix to the gui script. Add -Djava.net.preferIPv4Stack=true
right after java so the line reads:

java -Djava.net.preferIPv4Stack=true -cp $CLASSPATH ....

Execute the GUI:
/usr/share/galleon/bin/gui.sh

By default it tries to log to /var/log/galleon, so it typically throws a bunch of errors to the console that can be unsettling. I've not bothered to figure out how to redirect that, so I typically just run it as root.

The GUI actually talks to the server over RMI, so permissions aren't a problem. Probably not the most secure of protocols, so only run on trustworthy networks, I suppose.

Configure, per docs at SF.

The init script is provided by the wrapper.sf.net package, which is useful for wrapping any Java app that needs to act as a Unix server or NT Service. Galleon only ships with a Linux binary, but Wrapper supports others, so should be easy enough to roll your own if not already available on your platform.
nexus

gen-ripit-configs

When using ripit, one of the things I find myself doing over and over is generating config files for it. So I wrote a little tool to help out with that. Since it's really built around lsdvd, I decided to write this one in Python rather than Bash. It was easier to deal with the emitted Python dictionary than trying to write an XSLT for the XML output that it can do. And never mind trying to parse the output in Bash.

So this tool takes three arguments, -t, -s and -l. -t for the title, -s for the season (if it's a collection of TV shows), and -l for the lower limit of tracks you're interested in (currently defaults to 30 seconds, I like to rip the 45 second trailers for TV shows). Some DVDs have a bunch of outtakes or stuff that less than 30 seconds, so I finally made that a configurable value.

Just run gen-ripit-configs and a little bit later you end up with a new directory (the same name as the title embedded on the DVD) populated with a bunch of files named N.config for each track on the DVD. You can then go in and edit each one to taste. The format will be different depending on whether or not -s was present. I like the format to be Title-Season_NN-Episode_Name

It's only 73 lines, but probably not the best document Python in the world.

#!/usr/bin/python

import getopt
import os
import re
import sys

midentify = 'mplayer -vo null -ao null -frames 0 -identify dvd://%d'
sid_regex = re.compile('^subtitle \( sid \): (\d+) language: (\w+)')
shortopts = 'l:s:t:'
longopts = ['limit=', 'season=', 'title=']
opts, pargs = getopt.getopt(sys.argv[1:], shortopts, longopts)

title_format = 'title="%(title)s-EXTRA"'
title_params = {
  'limit': 30,
  'title': 'OOPS'
}

for arg, value in opts:
  if arg in ['-l', '--limit']:
    title_params['limit'] = int(value)
  elif arg in ['-s', '--season']:
    title_params['season'] = int(value)
  elif arg in ['-t', '--title']:
    title_params['title'] = value
  else:
    raise RuntimeError, 'Unknown argument: %s' % arg

if 'season' in title_params:
  title_format = 'title="%(title)s-Season_%(season)02d-NN-TITLE"'

input = os.popen('lsdvd -a -s -Oy')
a = input.read()

exec a

directory = lsdvd['title']
print 'using directory %s with limit %d' % (directory, title_params['limit'])
os.mkdir(directory)

for track in lsdvd['track']:
  if (track['length'] < title_params['limit']): continue

  ix = track['ix']
  vobsubs = []

  # Some DVDs have subtitles in more than one aspect ratio.  Currently
  # lsdvd and mplayer have differing opinions on how they should be
  # referenced.  Fortunately, it appears that mplayer and mencoder are
  # in sync.  So, rather than using lsdvd's notion of what subtitles to
  # use, we'll use mplayer's.
  if (len(track['subp']) > 0):
    (sin, sout, serr) = os.popen3(midentify % ix)
    for line in sout:
      result = sid_regex.match(line)
      if result:
        vobsubs.append('[%s]=%s' % (result.group(1), result.group(2)))
    sin.close()
    sout.close()
    serr.close()

  f = open('%s/%d.config' % (directory, ix), 'w')
  print >>f, 'track=%d' % ix
  audio = [(int(a['streamid'], 16), a['langcode']) for a in track['audio']]
  print >>f, 'audio=(',
  for a in audio:
    print >>f, '[%d]=%s ' % a,
  print >>f, ')'
  print >>f, 'vobsub=(%s)' % ' '.join(vobsubs)
  print >>f, title_format % title_params
  print >>f, '# length=%d' % track['length']
  f.close()
nexus

update for ripit

After seeing how I actually use ripit, I've made some mods to facilitate my actual use cases.

First, I almost always do something like:
for v in *.config; do ripit -r -c $v ; done

Later followed by:
for v in *.config; do ripit -e -c $v ; done

That way I can be encoding one set of rips while processing the next DVD.

So, now there is no more -c option. You have to pass at least -r or -e (if you leave off both, it doesn't do anything). Instead, you pass one or more config files on the command line and it loops over them.

Second, I use the script on two different systems now. The second system doesn't have a new of an mplayer installation, so I built my own, and gave it a slightly different name (well, mencoder actually). So I broke out the hardcoded mencoder string and made it a variable at the top of the script to point to whatever the name of mencoder it is (I point it to nexus-mencoder on the other machine). It's not a machine I can easily change the apt repository for, so I figured this was the easiest thing to do.

It's now up to 149 lines.

#!/bin/bash -eu

MENCODER=mencoder

usage() {
cat << \EOF
 $0 [--help] | [--rip] [--encode] config1 config2 ... configN
 $0 -h | -r -e config1 config2 ... configN

Where config file looks like:
track=2
audio=([128]=en es fr en)
vobsub=([0]=en en es fr fr en)
title="Hulk"
EOF
exit 0
}

error() {
  echo "ERROR:" "$@" 1>&2
  exit 1
}

init() {
  . $config
  chapters=$track.chapters.txt
  mkvcmd="mkvmerge --chapters $chapters -o $title.mkv --noaudio $track.avi"
}


rip_chapters() {
  dvdxchap -t $track /dev/dvd > $chapters.tmp
  mv -f $chapters.tmp $chapters
}

rip_video() {
  $MENCODER \
    -msglevel muxer=0:decvideo=3:cplayer=1 \
    -quiet \
    -of mpeg \
    -oac copy \
    -ovc copy \
    -noautosub \
    -o $track.mpg dvd://$track
}

rip_audio() {
  for index in ${!audio[*]}; do
    $MENCODER \
      -msglevel muxer=0:decvideo=3:cplayer=1 \
      -quiet \
      -of rawaudio \
      -ovc copy \
      -oac copy \
      -aid $index \
      -o $track.$index.avi dvd://$track
  done
}

rip_vobsubs() {
  count=0
  for index in ${!vobsub[*]}; do
    $MENCODER \
      -msglevel muxer=0:decvideo=3:cplayer=1 \
      -quiet \
      -vobsubout $track.vob \
      -vobsuboutindex $count \
      -sid $index \
      -oac copy \
      -ovc copy \
      -o /dev/null dvd://$track
    ((count++))
    : # noop because the above can trigger -e handling
  done
}

rip_cc() {
  ccextractor -srt -o $track.srt $track.mpg
}


rip() {
  rip_chapters
  rip_video
  rip_audio
  rip_vobsubs
  rip_cc
}

encode() {
  [ -s $track.srt ] && mkvcmd="$mkvcmd --language 0:en $track.srt"
  [ -s $track.vob.sub ] && mkvcmd="$mkvcmd $track.vob.idx"


  deint="-vf pullup,softskip"
  deint="-vf yadif"



  $MENCODER \
    -msglevel muxer=0:decvideo=3:cplayer=1 \
    -quiet \
    $deint \
    -oac copy \
    -ovc lavc \
    -lavcopts vcodec=mpeg4:vqscale=4:mbd=2:trell:mv0:v4mv \
    -noautosub \
    -o $track.avi $track.mpg

  for index in ${!audio[*]}; do
    [ -s $track.$index.avi ] || continue
    mkvcmd="$mkvcmd --language 0:${audio[$index]} $track.$index.avi"
  done

  # mkvmerge will return an error code on warning type items
  $mkvcmd || true
}

main() {
  init
  if [ "${rip:=no}" == "yes" ] ; then
    rip
  fi
  if [ "${encode:=no}" == "yes" ] ; then
    encode
  fi
}


name=$(basename $0)
OPTS=$(getopt -o -hre --long help,rip,encode -n $name -- "$@")
eval set -- ${OPTS}

configs=""

while : ; do
  case "$1" in
    -h|--help) usage ;;
    -r|--rip) rip=yes ; shift ;;
    -e|--encode) encode=yes ; shift ;;
    -c|--config) config=$2 ; shift 2 ;;
    --) shift ; break ;;
    *) configs="$configs $1"; shift ;;
  esac
done

for config in $configs; do
  main
done
nexus

I'm an idiot

I used a -z where I meant to use a -s

As a result, for the last three weeks or so, I've not been putting closed captions into the streams I've been pulling off of my Tivo.
  • Current Mood
    aggravated aggravated
nexus

ripit: Ripping DVD to MKV

For a long time now, I've been wanting to start ripping DVDs to MKVs. I really liked the MKV container format for the native support of multiple audio tracks and subtitles.

For a long time, though, I didn't do this because the then current tools didn't support the vobsub type of image-based subtitles that DVDs have, and I didn't want to deal with the process of running vobsubs through OCR software. Plus I wanted to keep whatever languages were actually on the DVD. However, all of that has pretty much matured now so that just about everything I want is supported. The only thing that I originally wanted to do that I currently can't do is support the DVD menu into an MKV, and after much thought, decided I didn't want to bother anyway.

Most of the existing tools that I could find, like dvd:rip, acidrip and so on, were too complicated for my needs. But, I did go through them and pull out the interesting bits and pieces that I've found useful, and put together my own script.

It's rough, but it works, for me. This isn't a pop-in-a-disc-and-press-a-button tool. It requires a bit of manual set up before hand, but after that, it works well.

I built this on a Debian/unstable system, though I don't see why it wouldn't work on just about any system where the tools are already ported. I don't give much in the way of configuration. It's designed to work for me on my system. But, because of that, I can keep it pretty simple.

The tools that the script actually uses:
dvdxchap
mencoder
mplayer
mkvmerge

I think that all of those are available on most systems with native ports. Additionally, I use a tool off of SourceForge called ccextractor. I use that to extract closed captions from some DVDs. I've found that some collections of TV shows on DVDs, instead of having English as an image based vobsub, they just keep the closed captions they used for the original airings. Makes sense, why redo the work? Stargate SG-1, Season 1 is like that, for instance. Anyway, I'm not sure how well packaged ccextractor is, but it's pretty simple code that should be fairly portable to any system. It comes with Win32 and Linux binaries, at least, perhaps more. Just drop it somewhere in your path. I also use this tool to extract captions from streams off my TiVo, which I need to post about sometime as well. But back to ripit.

The versions of mplayer/mencoder I use are from debian-multimedia. I use the YADIF deinterlacer that seems to not be in any version yet available from Debian natively. Choosing your own deinterlacer, and you may be able to make this work with a different version, though word on the net seems to be that YADIF is the best of the bunch. I tend to agree. This version I just hardcode it to always using YADIF, even when not necessary. I've not ripped any movies since I set that up, so I made need to make that a job specific option.

Another tool that I often use is lsdvd. I switch back and forth to using lsdvd and mplayer -identify to figure out what all is on the track, so that I can rip it. At some point, I suppose I should make it so that the tool could automatically use lsdvd to extract the info, but so far, I've not felt enough pain to do that.

So, how does it work?

Well, it assumes that dvds can be played using using [mplayer dvd://1] with no options. Generally, that means that /dev/dvd is a symlink to the appropriate subsystem. Actually dvdxchap uses that as the access point. If you can't do that, that's the first place you need to start to make this work locally for you.

Each rip/job requires a config file. I typically name the file 1.config, 2.config, etc, to match the track number. Running [ripit -h] will give an example one, but here's another example of something I'm doing right now:


track=1
audio=([128]=en)
vobsub=([0]=en)
title='Stargate_SG1-Season_03-18-Shades_of_Grey'


Nothing magic about the title, that's just the format that I use, borrowed from a common way of ripping CDs.

The audio and vobsub settings are basically KSH/BASH style sparse arrays. The audio starts at 128 and vobsub at 0. Again, [mplayer -identify] and [lsdvd -x] can be used to find those values. I typically set it up to pull in every language for both.

You can then run [ripit -c 1.config] and some time later, you end up with Stargate_SG1-Season_03-18-Shades_of_Grey.mkv.

Since I have two machines, and only one DVD drive, I actually usually do it in two parts: ripping then encoding.

So, I'll do something like:


for v in *.config; do ripit -r -c $v; done


And just rip all of the raw data into the current directory (just a few gigs). Then, on the various machines, I'll do:


ripit -e -c 1.config
ripit -e -c 2.config
ripit -e -c 5.config


Basically, I manually load balance across all of my CPUs/cores.

I don't bother rencoding the audio. The video takes up such a large portion that, in my opinion, it's not worth the hassle. No sense going from one lossy encoding to another on the audio.

On the video, I don't bother trying to target a specific bitrate or file size. I just want is somewhat smaller, so I basically go to an mpeg4 encoding at quality 4. It's a tad blocky, and if I was watching this stuff on a big screen TV, I'd probably go up the quality 3. But, for me, this is a good trade off. It's about 1/3 smaller. A typical 45 minute TV show is 500M.

Anyway, here's the script, all 130 lines of it:

#!/bin/bash -eu

usage() {
cat << \EOF
 $0 [--help] | [--rip] [--encode] [--config CONFIGFILE]
 $0 -h | -r -e -c CONFIGFILE

Where config file looks like:
track=2
audio=([128]=en es fr en)
vobsub=([0]=en en es fr fr en)
title="Hulk"
EOF
exit 0
}

error() {
  echo "ERROR:" "$@" 1>&2
  exit 1
}

init() {
  . $config
  chapters=$track.chapters.txt
  mkvcmd="mkvmerge --chapters $chapters -o $title.mkv --noaudio $track.avi"
  for index in ${!audio[*]}; do
    mkvcmd="$mkvcmd --language 0:${audio[$index]} $track.$index.avi"
  done
}


rip_chapters() {
  dvdxchap -t $track /dev/dvd > $chapters
}

rip_video() {
  mencoder \
    -quiet \
    -of mpeg \
    -oac copy \
    -ovc copy \
    -noautosub \
    -o $track.mpg dvd://$track
}

rip_audio() {
  for index in ${!audio[*]}; do
    mencoder -quiet -of rawaudio -ovc copy -oac copy -aid $index -o $track.$index.avi dvd://$track
  done
}

rip_vobsubs() {
  count=0
  for index in ${!vobsub[*]}; do
    mencoder \
      -quiet \
      -vobsubout $track.vob \
      -vobsuboutindex $count \
      -sid $index \
      -oac copy \
      -ovc copy \
      -o /dev/null dvd://$track
    ((count++))
  done
}

rip_cc() {
  ccextractor -srt -o $track.srt $track.mpg
}


rip() {
  rip_chapters
  rip_video
  rip_audio
  rip_vobsubs
  rip_cc
}

encode() {
  [ -s $track.srt ] && mkvcmd="$mkvcmd --language 0:en $track.srt"
  [ -s $track.vob.idx ] && mkvcmd="$mkvcmd $track.vob.idx"


  deint="-vf pullup,softskip"
  deint="-vf yadif"



  mencoder \
    -quiet \
    $deint \
    -oac copy \
    -ovc lavc \
    -lavcopts vcodec=mpeg4:vqscale=4:mbd=2:trell:mv0:v4mv \
    -noautosub \
    -o $track.avi $track.mpg

  $mkvcmd
}

main() {
  init
  [ "${rip:=no}" == "yes" ] && rip
  [ "${encode:=no}" == "yes" ] && encode
}


config=config
name=$(basename $0)
OPTS=$(getopt -o -hrec: --long help,rip,encode,config: -n $name -- "$@")
eval set -- ${OPTS}

if [ "$1" == "--" ]; then
  rip=yes
  encode=yes
fi

while : ; do
  case "$1" in
    -h|--help) usage ;;
    -r|--rip) rip=yes ; shift ;;
    -e|--encode) encode=yes ; shift ;;
    -c|--config) config=$2 ; shift 2 ;;
    --) shift ; break ;;
    *) error "internal error processing options" ;;
  esac
done

main
nexus

Less taxes!

I just got a letter from the County Assessor.

They've lowered the assessed value of my house. Cool!

Oh, wait... that means that the value of my house has dropped 8% in the 10 months I've owned it.

Crap... that means I now owe 99.9% of the value. So much for equity.
nexus

When good xterms go bad.

When using xterms, I often single/double/triple-click to initiate a selection, then extend the selection with a right mouse click.

This morning, when I came out to the office, I noticed it wasn't working any more. As soon as I released the left-mouse button, the selection would be un-highlighted, and impossible to extend. I could still paste what I had managed to select, just couldn't extend.

This was the second time that this had happened.

It turns out that is may be related to me running x2x to connect my two workstations. Sometime over the evening, x2x had died, and I'd simply restarted it.

When tracking down the selection problem, I noticed it was happening on both machines, no matter which mouse I used. So I killed x2x, and then selection started working properly on both machines. Restarted x2x, and no issues (yet).

I suppose that something got into a strange state, which is what caused x2x to die. I'm not sure if restarting x2x, or doing proper selections with x2x absent fixed things. But, for the moment, it seems to be working.