townx

Using Linux every day means that I often grapple with how to re-encode proprietary formats so that I can watch them on the computer of my choice. I also do some DVD ripping and creating new DVDs of home movies etc., for which the Linux command line tools work very nicely (more quickly, more consistently and in a more stable fashion than some of the GUIs).

So I've gathered a whole load of tips on encoding, ripping, burning, culled from dozens of forums, websites, manpages etc.. This, then, is the current state of my understanding on this topic, and hopefully distills many hours of pain into an easily-digestible format. It's not very well organised, but hopefully useful. I should mention that this stuff works on Ubuntu, but your mileage may vary. Here goes.

A note on tools

All of the tools I use are easily installable on Ubuntu, either from official repositories or universe/multiverse. You will also need to install the proprietary codecs if you want to work with them. Here's what I tend to use:

• vlc media player
• mplayer media player
• ffmpeg format transformer
• mencoder
• dvgrab
• dvdauthor
• dvdbackup
• growisofs
• mkisofs
• sox
• lame

mp4 to mpg

Use mencoder for this:

mencoder in.mp4 -ovc lavc -oac lavc -o out.mpg

Ripping from a dv camera with dvgrab

dvgrab --autosplit --timestamp --format jpeg

(The camera should be detected automatically, and I think this waits until it detects some input before the capture starts. I use a firewire cable to connect from the camera to the computer.) The files get named after timestamps coming from the film.

Encoding from .avi to .mpg

Use:

ffmpeg -i infile.avi outfile.mpg

(though this produces quite low quality output)

This produces better quality (bitrate = 800):

ffmpeg -i infile.avi -vcodec mpeg2video -acodec mp2 -b 800 outfile.mpeg

Ripping DVDs to hard disk

(See http://www.bunkus.org/dvdripping4linux/single/index.html for lots of good tips)

May need dvdcss decoder to be installed:

sudo /usr/share/doc/libdvdread3/examples/install-css.sh

Use dvdbackup to rip a DVD to hard disk.
See http://dvd-create.sourceforge.net/dvdbackup-readme.html for full instructions

# Get info. about DVD
dvdbackup -i /dev/cdrom -I

# Rip whole DVD
dvdbackup -M -i/dev/cdrom -o /media/usbdisk/dvdripping

# Rip main feature
dvdbackup -F -i /dev/cdrom -o /media/usbdisk/dvdripping

# Rip title set (in this case, title set 2)
dvdbackup -T 2 -i /dev/cdrom -o /media/usbdisk/dvdripping

# Rip title (here, rip title 1)
dvdbackup -t 1 -i /dev/cdrom -o /media/usbdisk/dvdripping

Direct DVD copying

This effectively copies the DVD's iso image to hard disk:

dd if=/dev/cdrom of=file.iso bs=2048

If the disk is encrypted, this might fail. In this case, it might be worth running this first:

dvdbackup -I -i /dev/cdrom

Then try dd again.

See http://gentoo-wiki.com/HOWTO_Backup_a_DVD#dd for more details

Note that this produces a mountable DVD image. However, it does not remove encryption, so you would still need to rip to individual VOBs using dvdbackup to get rid of that. However, it is possible to mount the iso and play it as if it were a DVD (see later): vlc is good for this.

Playing a partially ripped DVD

If you've ripped some of the content of a DVD (e.g. using dvdauthor -i /dev/cdrom -F), you can play the partial rip with:

mplayer dvd:// -dvd-device <ripped_dvd_directory>

(vlc might also be able to do this)

Encoding audio out of a VOB file (note that this drops the video altogether)

mplayer -vo null -vc null -ao pcm:file=outfile.wav infile.VOB

You could probably use ffmpeg for this, too.

Creating a DVD using dvdauthor when you don't want the whole disc

If you just have the main feature (a set of .VOB files) and you want to create a playable DVD from them.

Create a dvd.xml file in the top level directory of the ripped DVD (e.g. in VIDEO_TS)

<dvdauthor>
  <vmgm />
    <titleset>
      <titles>
        <pgc>
          <vob file="VTS_01_1.VOB" />
          <vob file="VTS_01_2.VOB" />
          <vob file="VTS_01_3.VOB" />
          <vob file="VTS_01_4.VOB" />
          <vob file="VTS_01_5.VOB" />
        </pgc>
      </titles>
    </titleset>
</dvdauthor>

(Adding another element for each VOB which you want included in your "movie". Note that these can equally well be your own mpg files.) You can also add titles etc.: see the man page for dvdauthor for more details. This works fine for me, though.

Then run it with:

dvdauthor -o <output directory for DVD structure> -x dvd.xml

Burning complete DVD structure to a new DVD

See http://www.linux.com/articles/53702

(I think you might need to have used dvdbackup -M (complete rip) for this to work, or have a DVD structure created using dvdauthor (see above))

growisofs -dvd-compat -Z /dev/cdrom -dvd-video <ripped dvd structure>

Creating an iso from a ripped DVD structure

If you've ripped the structure of a DVD (e.g. using dvdbackup) or created your own DVD structure (e.g. using dvdauthor), you can turn it into a single iso file with:

mkisofs -dvd-video <ripped dvd directory> | dd of=file.iso obs=32k seek=0

Mounting an iso filesystem so you can read it

mkdir mountpoint
sudo mount -o loop file.iso mountpoint

Once mounted, you can play the mounted iso (including its menu system) using vlc. For example, if we mounted it on the directory "mountpoint" we could do:

vlc dvd:///path/to/mountpoint

Encode VOB to mpg

If you ripped a VOB off a DVD but you want a smaller mpg:

ffmpeg -i VTS_01_1.VOB -vcodec mpeg2video -acodec mp2 -b 1000 sleeper1.mpg

Ripping RealPlayer streams

See https://help.ubuntu.com/community/HowToRipRealaudioStreamsToMp3 for full instructions

Short version:

vsound --timing --dspout --file=myfilename.wav realplay http://url.to.rip

Ruby script for converting from m4a (iTunes format?) to mp3

This is a shell script for converting files from m4a to mp3. It uses mplayer and lame behind the scenes. sox is supposed to do this, but I can never work out how to install proprietary codecs for it. This doesn't retain tags, unfortunately. Sneetchalizer will probably also work.

#!/opt/lampp/bin/ruby
# only works if the script is in the directory with the m4a files
prefix = "new_file_prefix"
Dir['*.m4a'].each do |f|
  new_filename = prefix + File.basename(f).gsub(' ', '_').slice(3..-5).downcase
  wav = new_filename + ".wav"
  mp3 = new_filename + ".mp3"
  `mplayer -ao pcm:file='#{wav}' '#{f}'`
  `lame -h -b192 #{wav} #{mp3}`
  `rm #{wav}`
end

Ruby script for converting from ogg to mp3

Uses ogg123. This script loses all the tags, though. Sneetchalizer would do this, too.

#!/opt/lampp/bin/ruby

filename = ARGV[0]

if filename.nil? or !(/\.ogg$/ =~ filename)
  puts "I don't think that's an ogg file, mister"
  exit
end

base_filename = filename.chomp(".ogg")
mp3_filename = base_filename + ".mp3"

`ogg123 -d au -f - #{filename} | lame - #{mp3_filename}`

mp3 to ogg

Bit basic, this, but you get the idea:

mpg321 -s input.mp3 | oggenc -r -a "artist" -t "title" -b 100 -M 140 -o output.ogg -

ogg to wav

sox is OK for this, as neither codec is proprietary:

sox input.ogg output.wav

I recently got a Dell Latitude D820 at work, and spent about 10 days pulling my hair out using Windows XP on it. Once I'd run out of hair, and after clearing it with the technical support people and my bosses, I persuaded them to let me install Linux (though they insisted on me keeping a small Windows boot partition).

Anyway, I've been using it successfully for the last 6 weeks or so, and it's been working great. Today, I worked from home for the first time, and wanted to get a few other things working:

• I wanted to be able to move the machine around the house (wireless).
• I wanted to get the touchpad working properly (i.e. scroll when I slide my finger down the side, and turn off taps being translated to mouse clicks).
• It would be nice to get suspend to RAM working, so I can close the lid when I'm having my lunch and save some electricity.

In summary, I managed to get this lot working fine, and would say this machine is a nice one for running Linux. I don't like it quite as much as my old Thinkpad Z61t (keyboard isn't quite so nice), though I've never managed to get the wireless working on that.

Wireless

The wireless worked fine, but for some reason the network applet just refused to allow me to connect. Eventually, I managed to get it working by hard-coding the IP address into the network config., and setting up the router to assign a static IP to the wireless card. But there was no need to install any drivers or anything: all of that was just working out of the box.

Touchpad

The touchpad was another matter, and took me about half an hour to figure out. In the end, after having read about 6 different articles on installing various shades of Linux on this model of laptop, I came across this extremely useful article, which enabled me to fix it definitively.

Eventually, I had to make two edits to /etc/X11/xorg.conf (as root). First, I added a line to the ServerLayout configuration element:

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen "Default Screen"
        Inputdevice     "Generic Keyboard"
        Inputdevice     "Configured Mouse"
        # HERE'S MY EDIT
        Inputdevice     "Touchpad"      "AlwaysCore"
        Inputdevice     "stylus"        "SendCoreEvents"
        Inputdevice     "cursor"        "SendCoreEvents"
        Inputdevice     "eraser"        "SendCoreEvents"
EndSection

Then I added a new section for the touchpad itself:

Section "InputDevice"
         Driver "synaptics"
         Identifier "Touchpad"
         Option "Device" "/dev/input/event3"
         Option "Protocol" "event"
         Option "LeftEdge" "130"
         Option "RightEdge" "840"
         Option "TopEdge" "130"
         Option "BottomEdge" "640"
         Option "FingerLow" "7"
         Option "FingerHigh" "8"
         Option "MaxTapTime" "0"
         Option "MaxTapMove" "0"
         Option "EmulateMidButtonTime" "75"
         Option "VertScrollDelta" "20"
         Option "HorizScrollDelta" "0"
         Option "UpDownScrolling" "1"
         Option "MinSpeed" "0.70"
         Option "MaxSpeed" "1.20"
         Option "AccelFactor" "0.080"
         Option "EdgeMotionMinSpeed" "200"
         Option "EdgeMotionMaxSpeed" "200"
         Option "SHMConfig" "on"
         Option "Emulate3Buttons" "on"
EndSection

The important bit is where I specify the Device as /dev/input/event3. Also note I set MaxTapTime to 0 to turn off taps acting as mouse clicks. You may have to muck around with the *Speed and AccelFactor settings to suit you: there is a GUI client called gsynaptics which can configure scroll speed and taps; and there is a more comprehensive command line application synclient which helps if you want to fiddle around with the settings in real time.

Suspend to RAM

I found this article which provided the necessary settings. After I'd made the changes suggested, everything worked brilliantly: even the wireless came back from suspend. (Note that I use the proprietary NVidia drivers, so the instructions are specific to that.)

First off, I edited /etc/X11/xorg.conf, adding a couple of settings to the NVidia driver setup:

Section "Device"
        Identifier      "nVidia Corporation G72M [GeForce Go 7400]"
        Driver          "nvidia"
        Busid           "PCI:1:0:0"
        Option          "UseEDIDDpi"            "False"
        Option          "AddARGBVisuals"        "True"
        Option          "AddARGBGLXVisuals"     "True"
        Option          "NoLogo"                "True"
        Option          "NvAGP"                 "1"
EndSection

Then I edited /etc/defaults/acpi-support (in all cases, the default setting was true):

SAVE_VBE_STATE=false
POST_VIDEO=false
USE_DPMS=false
LOCK_SCREEN=false

I find the command completion tools in a shell indispensable (i.e. when you press Tab to complete a path or show you the options for a command - this works for svn, for example). Plus I get quite annoyed when they don't work for a command I'm using. I found out today it is pretty easy to add your own completions using the complete command (at least, when using Bash under Linux).

So here, for example, is how to get Rake to auto-complete with the names of the tasks in your Rake file:

complete -W "$(rake -T | awk 'NR != 1 {print $2}')" rake

It's not flawless, but it's a damn sight better than having to do rake -T and scrolling through a wordy list. NB this only works if you run the complete command in a directory where you have a Rake file to start with. There's probably some switch to complete which dynamically generates the completions when you try to use them: I need to investigate.

You could also try something like this to put up a list of names of hosts in your /etc/hosts file when you're using SSH:

complete -W "$(cat /etc/hosts | awk '$1 != "#" {print $2}')" ssh

You can just add these to your Bash profile to get them activated when starting a new shell, I imagine.

I've been a VMware user for a couple of years now, and have found it invaluable for testing different operating systems and network configurations without the need for tons of hardware. (For example, I used VMware to simulate deployments of Rails applications from my laptop to a virtual Linux server, using Capistrano over SSH.) I have the luxury of a Workstation licence, which makes it simple to set up new virtual machines and network them.

However, having heard about Xen and the like, I thought I'd have a look over the open source offerings for virtualisation and see how they measured up. This week, I have been trying out VirtualBox, a nice system which offers much of the functionality of VMware (probably all if you're prepared to read the manual properly), but which you can use for nothing. The licensing model is to charge extra for some "enterprise" features, but for me the wholly-free community version (released under the GPL) is more than adequate. Also bear in mind that I installed it on Ubuntu Linux (Dapper) and used that as the host operating system (the one running the virtual machines), but that it is available for Windows too.

Installation

The first step is to download a package for your operating system. I used the Ubuntu Dapper .deb package provided on the downloads site, which I installed with no problems. The only slight issue is that you need a few dependencies first. I had most already, but I needed a couple more. For those of you who are interested, here's the output of dpkg-query -s virtualbox, which shows the dependencies:

Package: virtualbox
Status: install ok installed
Priority: optional
Section: misc
Installed-Size: 30936
Maintainer: innotek GmbH <info@virtualbox.org>
Architecture: i386
Version: 1.5.0-24069-1_Ubuntu_dapper
Depends: libasound2 (>> 1.0.10), libc6 (>= 2.3.4-1), libgcc1 (>= 1:4.0.2), libgl ib2.0-0 (>= 2.10.0), libice6, libidl0, libpng12-0 (>= 1.2.8rel), libqt3-mt (>= 3 :3.3.6), libsdl1.2debian (>> 1.2.7+1.2.8), libsm6, libssl0.9.8 (>= 0.9.8a-1), li bstdc++6 (>= 4.0.2-4), libx11-6, libxalan110, libxcursor1 (>> 1.1.2), libxerces2 7, libxext6, libxt6, zlib1g (>= 1:1.2.1), debconf (>= 0.5) | debconf-2.0, psmisc
Pre-Depends: debconf (>= 1.1) | debconf-2.0
Recommends: linux-headers, gcc, make, binutils, bridge-utils, uml-utilities, lib hal1 (>= 0.5)
Conffiles:
 /etc/init.d/vboxnet dba1d3dd5cf1dfa526df9f8be2cd17b8
 /etc/init.d/vboxdrv b3b219d047c0de1216db6c4b9481a233
 /etc/vbox/vbox.cfg 7fa8136a0f4330b1024b70f9d258c0a5 obsolete
 /etc/init.d/virtualbox 4946cb298821c69ebd17575c75c56fd1 obsolete
Description: innotek VirtualBox
 VirtualBox is a powerful PC virtualization solution allowing you to run a
 wide range of PC operating systems on your Linux system. This includes
 Windows, Linux, FreeBSD, DOS, OpenBSD and others. VirtualBox comes with a broad
 feature set and excellent performance, making it the premier virtualization
 software solution on the market.

I installed the VirtualBox .deb itself using:

sudo dpkg --install virtualbox_1.5.0-24069-1_Ubuntu_dapper_i386.deb

(Attempting to install highlighted my missing dependencies, which I then installed with apt-get.) Once installed, you can get it up and running from the command line with:

$ VirtualBox

The interface is plain but fairly easy to follow:

Installing Damn Small Linux as a guest operating system

My preferred distribution for testing virtualisation software is Damn Small Linux (DSL), a tiny but still useful Linux distribution. The download is only 50Mb, and it will run from a CD image; but you can also install it to a USB drive or proper hard drive, providing you give it at least 200Mb of space. I use it as the guest operating system on my Linux host, to quickly get a flavour of how the virtualisation software works.

I downloaded an .iso for DSL, created a new virtual machine, and pointed it at the .iso file as the CD image to boot from. This is what the VirtualBox configuration looks like; notice the options to mount multiple hard disks, mount a real CD drive, use USB, audio etc., as in VMware:

Once I clicked on the Start button, DSL booted off the CD image first time, blindingly fast: much faster than a VMware virtual machine booting DSL with the same amount of RAM. This was promising. Once up and running, everything worked as expected, and VirtualBox had set up the virtual machine's network connection so I could get out onto the internet using Firefox (1.0.6):

Installing Damn Small Linux to a virtual disk

However, trying to install to a hard disk was a bit more of a chore than I would have liked. To do this, I worked out this series of steps, with some trial and error. First, you have to attach a virtual disk to the machine, like so:

1. Close down the virtual machine so it is powered off.
2. Click on the Hard disks heading in the virtual machine configuration. This takes you to the Hard Disks tab in the configuration window.
3. Tick the Primary Master radio button, then on the icon next to the drop-down box. This will take you through to the Virtual Disks Manager.
4. Click on the New button to add a new virtual disk.
5. Click on the Next button. You're now offered a choice of creating a Fixed-size or Dynamically expanding virtual disk. I chose a fixed size one. Click on Next.
6. Set the size for the disk image. I went for 200Mb. You can leave the name for the disk at the default, and it will be stored in a .VirtualBox directory in your home directory.
7. Click through the rest of the wizard until your virtual disk is created.

Next, boot off the operating system off the CD again. Here's where I had a few problems. While the hard disk was recognised (as /dev/hda), it wasn't automatically mounted; in addition, it didn't have a valid partition table on it. Fine for technical users, but nowhere near as friendly as VMware: I seem to remember that when I installed DSL on VMware, I didn't need to create the filesystem myself.

In VirtualBox, when I tried to run the DSL "install to hard disk" option (right click on the desktop, and select Apps > Tools > Install to Hard Drive), it failed miserably. So I got a terminal up and created my own partition on the hard drive using fdisk. My session looked like this:

I then ran the installer, choosing to use an ext3 filesystem and using the Grub bootloader (Lilo didn't work). Here's what the DSL hard disk install script looks like while it's running:

I don't know if you'd need to do this for other Live CD to hard disk installers (e.g. Ubuntu's). In most cases, more sophisticated distributions provide a graphical tool for this, so it shouldn't be such a chore. However, notice that I think I needed to specify a bootable flag for the disk (with the a option in fdisk), which you might need to keep an eye out for when installing another distribution. This might not be necessary, but I did it so that I definitely wouldn't end up with an unbootable disk.

Once I'd done this, DSL happily booted off the virtual disk and I was done.

Pros and cons

So, what do I like about VirtualBox?

• It's open source :) and doesn't cost a penny for the "workstation" style edition. VMware's player is open source, but the workstation needs a paid-for licence key. It also means that installation is at least as easy (and will probably eventually be easier than) VMware, as it can become part of mainstream distribution repositories.
• It runs happily on Linux and was easy to install on Dapper (providing you remember the dependencies).
• It seems faster than VMware.
• One feature I really like is that you can attach any virtual disk to any machine, and that disks can be managed separately from virtual machines. This means you could mount a single virtual disk onto multiple machines; so if you upgrade or change the operating system, you could still easily access data from another virtual machine by mounting its disk. I'm not sure whether you can do this at all with VMware.
• Although I am not likely to use it, the API is fairly open; you can also run guest OSes headlessly, as I believe you can with VMware server. You can also programmatically create virtual machines, as they are configured through an XML file (I believe the virtual disk format is also compatible with the Xen virtualisation software; I don't think VMware's is). This open-ness should make it attractive to enterprises that don't want to get locked to a vendor.

What don't I like?

• Setting up things like bridged networking seems like a real faff, involving doing quite a bit of work on the host operating system. It looked so much of a pain that I decided not to bother. This makes it tricky to do networking between the guest and the host (e.g. so you can copy files from the host onto the guest). Similarly, sharing folders looks slightly trickier than on VMware.
• Cut and paste requires you to add the VirtualBox Guest Additions to a virtual machine, which requires compilation. This process is similar to VMware's toolbox, and a bit of a pain. I haven't bothered with this yet.
• It doesn't quite have the interface polish of VMware, though that doesn't bother me too much.
• It doesn't have quite the backing of tools like Xen. This could mean that it doesn't attract the same quality of engineers, and that it will have trouble innovating so fast.

Summary

This is a short, biased and probably inaccurate review of VirtualBox, but hopefully it will give you a flavour of why I think it is a useful tool, and help you get started with it yourself. I'll be using it in favour of VMware, at least in the short term, so that I don't have to pay a licence fee. It definitely suits my needs for the moment, and if you're interested in virtualisation, I'd recommend giving it a go.

I just bought a PDF, legally, from a publisher's website. However, in their wisdom, they decided it would be a good idea to password protect all legally-purchased PDFs. This means that each time you open the PDF using Acrobat Reader, you have to remember and type in the password to read it. (Evince, the built-in PDF viewer in Linux, allows you to permanently save the password, but I tend to use Acrobat as it copes better with some PDFs.)

So, if you know the password for a PDF and want to remove it, you can use the command line tools pdftops and ps2pdf to free your PDF from its chains.

• Install pdftops. On Ubuntu, you can do:
  apt-get install xpdf-utils
• Install ps2pdf. I think this is already included with a default Ubuntu.
• Convert the PDF to a postscript file, using the password:
  pdftops -upw <password> <file>.pdf
• Convert the resulting postscript (which is now sans password) back to a PDF:
  ps2pdf <file>.ps

The only thing you lose are any PDF-specific features which don't translate to postscript, e.g. hyperlinks.

Remember, this only works if you know the password for the PDF: it doesn't break the PDF password for you.

I use eMusic to satisfy my insatiable desire for music. It's great. However, you have to download each track individually: there's no way to get a whole album at once (at the moment). In Firefox, I found I could only download two tracks at a time; this meant it took quite a long time to download 10 tracks, say, as I could only have two downloads going at once.

I always thought this was down to eMusic, but it occurred to me today that it's probably a Firefox setting. Lo and behold, it is a Firefox setting. If you want to fix your browser to allow more than two simultaneous downloads over HTTP, do the following:

• Open up the URL about:config by typing it in the address bar and clicking the green arrow. This shows you all your Firefox settings.
• In the Filter text box, type connections to just show options relating to connections.
• I've got four settings, so I set all four to 12. Might seem excessive, but better to be safe than sorry. (To modify a value, right-click on it, select Modify and enter your new value). Here are the properties I changed:
  • network.http.max-connections
  • network.http.max-connections-per-server
  • network.http.max-persistent-connections-per-proxy
  • network.http.max-persistent-connections-per-server

I can now have 10 or so downloads running concurrently (the first typically finishes before I've got round to starting number 11!). Much better. This might even speed up standard web browsing, as you can download far more files from a single server at the same time. (Though I don't know whether setting it too high might make it look like you're trying to launch a denial-of-service attack or something.)

To further improve your download experience, I recommend the Download Statusbar for Firefox. Peachy.

Note: This is an update of my earlier set of instructions about installing Ubuntu Dapper as a guest operating system on VMware.

This time I'm using VMware Workstation 5.5.3 build-34685 and Ubuntu Dapper as the host operating system. I am installing Ubuntu Edgy as the guest operating system. I also worked out how to get the VMware Tools to work. Here's what I did:

1. Downloaded the Ubunutu .iso file from http://ubuntu.org/. (I tried to install off CD and failed miserably: it hung when it tried to create the Ubuntu Live CD user.)
2. Created a new virtual machine in VMware (I accepted all the defaults). I set the hard disk to 4Gb which should be big enough.
3. Edit the virtual machine and set its CD to use an ISO image; point it at the Ubuntu .iso file you downloaded.
4. Boot the virtual machine. Press enter when prompted to install Ubuntu. This will boot into the Live CD version of Ubuntu.
5. Once in, double-click the "Install" icon on the desktop to install the Live CD image onto the hard disk.
6. Follow through the installation procedure.
7. When you power off ready for reboot, you can point the CD for the virtual machine back at the physical drive (I used the "autodetect" setting). This prevents the virtual machine booting from the CD image in future (you can delete the image once you've completed the install, if you want).
8. Boot into Ubuntu. You are now using the hard disk installation rather than the Live CD image.
9. Upgrade all the packages on the system. You do this by clicking on the orange asterisk icon on the right-hand side of the top menu bar. You'll also need the networking to be up to do this. It takes quite a while, but it's worth doing.
10. Select the VM > Install VMware Tools option from the VMware workstation menu. You should get a CD icon on your Ubuntu desktop; if not, go to Places and choose the CD there.
11. Copy the .tar.gz file to your desktop. Right click on it and select "Extract here". This will give you a folder called vmware-tools-distrib.
12. Get a console up and cd to the vmware-tools-distrib directory.
13. Make yourself root (sudo su).
14. The installation requires a recent gcc and the Linux headers. Fortunately, these both appear to be installed by default on Edgy, so you don't need to do anything (unlike the previous set of instructions for Dapper).
15. It's worth creating a symlink to your kernel source to make the installer run more smoothly (it looks for /usr/src/linux and complains if it's not there):
    ln -s /usr/src/linux-headers-`uname -r` /usr/src/linux
16. While you're still in the vmware-tools-distrib directory, run the install script:
    ./vmware-install.pl
    I accepted all the default settings and said yes to everything.
17. This gets everything (pretty much) working except the mouse. You can fix this like so:
    1. sudo ln -s /usr/lib/vmware-tools/configurator/XOrg/7.0/vmmouse_drv.so /usr/lib/xorg/modules/input/
    2. Edit /etc/X11/xorg.conf and find the section headed Section "InputDevice", with the line Driver "mouse". Change the Driver "mouse" line so it reads Driver "vmmouse".
18. To get copy and paste working, you have to run vmware-toolbox inside the virtual machine; to get the "Autofit Guest" feature requires the vmware-user program to be running in the virtual machine. You can enable both of these applications to start when your Edgy virtual machine starts like so:
    1. First, you need to symlink a couple of VMware libraries so that Ubuntu knows where to find them (for running vmware-user). At a command prompt, run this command:
       ln -st /usr/lib /usr/lib/vmware-tools/lib32/lib*/lib*
    2. Go to the main Ubuntu menu (inside the virtual machine) and select System > Preferences > Sessions. Then select the Startup Programs tab.
    3. Click on the Add button and type vmware-user in the text box. Click on OK.
    4. Click on the Add button and type vmware-toolbox --minimize in the text box. Click on OK. Note that this window must always be open for cut and paste between the guest OS and the host one to work. If you want to push it into the system tray, you could use AllTray to do that. There is an --iconify flag for this command, which should presumably turn it into an icon, but it doesn't seem to work for me.
19. As I'm using a widescreen laptop, I also had to edit the settings for my screen resolution in the Section "Screen", e.g.:
    Modes "1280x768" "1024x768" "800x600" "640x480"
    This works fine on my IBM Z60t.
20. Logout and kill X with Ctrl-Alt-Backspace. This should make all your new settings come alive.

Hurrah! It works!

Credits: Thanks to Sean Flanigan for extra tips on enabling cut and paste and guest window resizing.

I bet you're wondering, why oh why would you want to compile Subversion? Can't you just install it with apt-get or Synaptic?

The answer is: yes, you can install it with apt. But my problem: if you set up a project with the latest EasyEclipse, the Subversion client version in EasyEclipse is different from the Ubuntu Subversion client version. This means any projects you create with EasyEclipse and link to a Subversion repository are unusable with the command line svn client in Ubuntu Dapper. You get this message if you try:

$ svn stat
svn: This client is too old to work with working copy '.'; please get a newer Subversion client

(By the way, I also get this error if I try to use script/plugin for a Rails application inside a project created with EasyEclipse. Presumably because the Ruby Subversion client library isn't up to date as a consequence of the C client library being out of date. This makes it difficult to use the plugin script, so I've been manually adding svn:externals to my plugins directory.)

First, download Subversion itself (I got the tarball) from the Subversion site. I'm using Subversion 1.4.3.

Next, install all the pre-requisites for compiling. I cribbed this list from http://packages.debian.org/stable/source/subversion:

sudo apt-get install m4 debhelper libneon24-dev libapr0-dev libdb4.3-dev libtool \
libexpat1-dev zlib1g-dev bison patch autotools-dev autoconf swig libsasl2-dev \
perl libperl-dev libkrb5-dev

Just for good measure (and because I don't really know what I'm doing :) I installed all the dependencies listed at http://packages.debian.org/stable/devel/subversion, plus all the -dev versions too:

sudo apt-get install db4.3-util libapr0 libapr0-dev libc6 libc6-dev libdb4.3 \
libexpat1 libexpat1-dev libldap2 libldap2-dev libneon24 libneon24-dev \
libssl0.9.7 libxml2 libxml2-dev patch zlib1g zlib1g-dev

Next, connect to the unpacked Subversion source directory and run these commands:

./configure --prefix=<path_to_install_directory>/subversion-1.4.3 --disable-nls
make
make install

Replacing <path_to_install_directory> with the path to the install directory. The --disable-nls turns off gettext support (for internationalisation) which otherwise causes problems.

Now try running:

<path_to_install_directory>/subversion-1.4.3/bin/svn --version
svn, version 1.4.3 (r23084)
   compiled Mar 12 2007, 23:33:51

It works for me. It could be there are other dependencies I've missed or not listed, but you can get what the Debian developers think the dependencies are from the links in the text above.

By the way, it took me about two hours of debugging wierd error messages to get this working, so there's a chance you might get the same, of course. Here are a few examples of what I ended up googling:

• NE_FEATURE_SSL undeclared
• PKCS12_unpack_authsafes@OPENSSL

I'm pretty sure these were caused by problems with OpenSSL and Neon, but I'm not 100% sure.

Jokosher is an audio production tool, designed for multi-track recording, which runs under Linux. The primary design goal was ease of use. Jono today called for Jokosher testers, to iron out bugs in Jokosher for version 0.9.

The application has come a long way, and is kind of usable in its current incarnation (not currently suitable for the kind of music I do [electronica], which needs lots of low-level wave editing, sampling and looping, but I'm just putting in bug reports and feature requests). It is impressive what's been achieved so far, and I'd like to see it become more general-purpose, so I can use it for my own audio production. I had got 0.2 working pretty easily, and submitted a few bug reports, then realised the team wouldn't be interested in that, and had probably fixed a load of the bugs. So I decided to bite the bullet and try it out properly from the latest version.

The biggest struggle is getting the thing installed. You're OK if you have a bleeding edge Ubuntu (Feisty) as it is packaged for that, but if you want to help with the testing, you'll need to get the CVS versions of Gstreamer and associated libraries, plus a checkout of Jokosher from Subversion. This turns out to be a bit of a pain, so here are some pointers if you're on an old Ubuntu (like me with Dapper). I won't replicate all the documentation here (there is a fine amount on the Jokosher userdocs website), but I'll try to summarise and provide pointers to the right places:

I got my instructions from http://userdocs.jokosher.org/InstallingCvsGstreamer and http://userdocs.jokosher.org/Installation. But here they are boiled down to the essentials (for Ubuntu Dapper):

# make a directory to put everything into
# (Gstreamer CVS, jokosher etc.)
mkdir ~/apps/jokosher
cd ~/apps/jokosher

# pull gstreamer (and associates) from CVS
mkdir gstreamer
mkdir gstreamer/head
cd gstreamer/head
cvs -d:pserver:anoncvs@anoncvs.freedesktop.org:/cvs/gstreamer co gstreamer \
gst-plugins-base gst-plugins-good gst-plugins-ugly gst-python gnonlin

# pull trunk jokosher from SVN
cd ~/apps/jokosher
svn checkout http://svn.jokosher.python-hosting.com/JonoEdit/trunk jokosher-trunk

# setup dependencies to build everything; NB there are a LOT of these: 
# this is just a list of the ones which have been identified, but there's a 
# chance there are others; you'll soon find out if the compilation step fails...
sudo apt-get install build-essential automake1.7 libtool libglib2.0-dev \
libxml2 liboil0.3-dev python-dev python-gtk2-dev bison flex libxml2-dev \
libgnomevfs2-dev libasound2-dev libspeex-dev libflac-dev libtag1-dev \
libhal-dev libogg-dev libvorbis-dev libid3tag0-dev libmad0-dev

# here's the tricky bit: getting a backport of liboil0.3.8, which 
# gst-plugins-base needs (Ubuntu Dapper has version 0.3.6 or something);
# note that I've done this rather recklessly, and hope it won't cause 
# horrendous clashes with other installed versions of liboil; 
# it doesn't complain, anyway
wget http://backports.org/debian/pool/main/libo/liboil/liboil0.3_0.3.8-0bpo1_...
wget http://backports.org/debian/pool/main/libo/liboil/liboil0.3-dev_0.3.8-0b...
sudo dpkg --install liboil0.3_0.3.8-0bpo1_i386.deb
sudo dpkg --install liboil0.3-dev_0.3.8-0bpo1_i386.deb

# setup a script which will run the CVS gstreamer alongside 
# any existing gstreamer you've installed
cp gstreamer/head/gstreamer/docs/faq/gst-uninstalled ./gst-head

# make it executable
chmod +x gst-head

# edit it: you need to change the line which starts MYGST so 
# it points to the directory ABOVE the head directory, where 
# you checked Gstreamer out; in my case:
#
# MYGST=$HOME/apps/jokosher/gstreamer

# create a script in ~/apps/jokosher/gstreamer/head called 
# build_all.sh to build all your gstreamer bobbins 
# from source, keeping it inside the ~/apps/jokosher directory; 
# it looks like this:

/* START SCRIPT */

#!/bin/bash
# build all the gstreamer stuff
for f in gstreamer liboil gst-plugins-base gst-plugins-good 
gst-plugins-ugly gst-python gnonlin; do
  cd $f
  echo "Building $f"
  make distclean
  rm -rf po
  cvs update -PAd
  ./autogen.sh
  make
  # don't do make install here, as this will trash existing libraries
  cd ..
done

/* END SCRIPT */

# make it executable
chmod +x ~/apps/jokosher/gstreamer/head/build_all.sh

# now we're ready to build
cd ~/apps/jokosher

# setup the build environment so everything points 
# at the right versions 
# of gstreamer etc.; 
# NB calling this drops you into ~/apps/jokosher/gstreamer/head
./gst-head

# and once there you can build everything with:
./build_all.sh

That does the building part. To run Jokosher once you've done all that:

cd ~/apps/jokosher
./gst-head
 ~/apps/jokosher/jokosher-trunk/Jokosher/Jokosher

This worked for me! I'm now ready to try out the delights of bleeding-edge Jokosher.

Below is a simple Spamassassin setup for use with Postfix; I also cover how to train Spamassassin on a Maildir mail box (I use Dovecot).

Some of the below is cribbed from http://www.debuntu.org/postfix-and-pamassassin-how-to-filter-spam. I've simplified it and left more of the configuration at the defaults. You don't have to do this on a locally-hosted Postfix server: the instructions are the same for any email server which can receive email (it just so happens mine isn't open to the outside world and is just passed email by fetchmail). I'm working on Ubuntu Breezy.

Disclaimer: Use these instructions at your own risk. I am not an expert on spam filtering: this set of instructions just gave me what I needed to do my own local spam filtering. If you set it up and it loses you the email clinching a one million dollar contract, I assume no responsibility.

By the way, you need to do all of the below as root.

Install Spamassassin

apt-get install spamassassin spamc

Configure Spamassassin user and group

We want it to run as non-root, so add a spamd user and group:

# groupadd spamd
# useradd -g spamd -s /bin/false -d /var/log/spamassassin spamd
# mkdir /var/log/spamassassin
# chown spamd:spamd /var/log/spamassassin

(This sets the spamd user's home directory as /var/log/spamassassin.)

Configure Spamassassin

Edit /etc/default/spamassassin so these options are set:

ENABLED=1

SAHOME="/var/log/spamassassin/"

OPTIONS="--create-prefs --max-children 2 --username spamd \
-H ${SAHOME} -s ${SAHOME}spamd.log"

--max-children spawns the specified number of child processes (you might need more on a busy server), --username specifies the username spamd runs under, -H sets the home directory, -s sets the log file.

(I left the rest of the file alone, and didn't touch /etc/spamassassin/local.cf.)

Start the Spamassassin daemon (spamd)

/etc/init.d/spamassassin start

(By default, spamassassin gets added to the startup scripts by Ubuntu, so it will start/stop with your system.)

Configure Postfix to use Spamassassin as a filter

This makes Postfix pipe email to Spamassassin once it's been received. Edit /etc/postfix/master.cf and add this line as the first line of the file:

smtp      inet  n       -       -       -       -       smtpd
        -o content_filter=spamassassin

Add this to the end of the same file:

spamassassin unix -     n       n       -       -       pipe
        user=spamd argv=/usr/bin/spamc -f -e  
        /usr/sbin/sendmail -oi -f ${sender} ${recipient}

I bet you're asking yourself "What the hell?!". I know I was. As far as I can tell, this sets up an after-queue content filter, and this config. does the following:

1. Sets up a content filter called spamassassin with type unix (i.e. listening on a Unix socket).
2. "- n n - -" means the filter is private, privileged (any pipe daemon needs privileges), not chrooted (pipe daemons can't be chrooted), has no wakeup set, and uses the default for maximum processes spawned, respectively.
3. pipe sets the filter up using the pipe daemon: stuff gets passed into it by Postfix, then it either returns what was passed in, possibly changed (e.g. with headers added or the body sanitised), or a code indicating what the filter did.
4. userd specifies the user to run the filter as.
5. argv sets the command for the pipe daemon to run over messages (i.e. this is the actual filter command). In this case, the command is:
   /usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
   This runs spamc (the Spamassassin client program), piping its output into the sendmail command; spamc passes the email being filtered to the Spamassassin daemon (spamd) which is running in the background; spamc gets the email back with a few spam headers to show the email has been checked, and possibly with its content rewritten to show the standard Spamassassin "Spam detection software, running on the system 'briar', has identified this incoming email as possible spam" if the email is spam. The ${sender} and ${recipient} markers are interpolated by Postfix, so that sendmail can take the output from spamc and deliver the message with the correct from and to addresses.

Reload Postfix for the configuration to take:

/etc/init.d/postfix reload

Train the Spamassassin filter

This was the bit where I had to do the most research. I'm using Maildir format for my email, under dovecot; email for localhost accounts goes in /home/user/Maildir.

To do my training, I created a new Junk folder for my localhost Postfix account: I did this by adding the folder via Thunderbird. Next I got a load of existing spam from my Trash folder, and moved it to the Junk folder.

Then, to train the filter on the Junk folder, I used sa-learn like this (you need to be root, which is why the sudo is there):

sudo sa-learn --spam -u spamd --dir /home/ell/Maildir/.Junk/* -D

You can also train it what the good stuff looks like, e.g. run it over a clean inbox (no spam):

sudo sa-learn --ham -u spamd --dir /home/ell/Maildir/.INBOX/* -D

There are probably spam corpuses (corpi?) you can use for this or some other smart method I missed, but this seems to work.

Testing

Get the content out of an existing spam, send it to an account on your protected server from a free email account (you could even set up a Hotmail account to get the most accurate spam scenario), and see if Spamassassin stops it. Check the logs (/var/log/spamassassin and /var/log/mail.log) to see what happened to the email. If Postfix is using Spamassassin properly, you should see something like this in the logs:

Jan 26 14:56:10 localhost postfix/pipe[12139]: 9CBD5DA4BF: \
to=<ell@localhost>, relay=spamassassin, delay=17, status=sent (localhost)

(Notice the mention of postfix/pipe.)

Spamassassin might not stop the email, as it will be coming from a legitimate email address. The important thing is that the logs report that Spamassassin was applied.

Results

I now get very little spam delivered to my inbox: most of it gets stopped by the Postfix server using Spamassassin, and Thunderbird catches the rest. I periodically train the filter to make it improve, using the stuff Thunderbird has decided is junk.

Why?

Originally, this was at the top of this post, but it's kind of irrelevant. I've included it here so you know why I put myself through this lunacy.

I have a couple of legacy email accounts which still get the occasional proper email, plus endless spam. I use the excellent fetchmail to download email from these moribund accounts and send it to my local Postfix server instead. I then read my local email account via the Dovecot imap server (setup to use Maildir layouts) in the Thunderbird email client. A fairly complex setup, but one which has worked well for me over the past three or four years.

However, a vast amount of spam used to hit my old email accounts, and I'd have to wait for Thunderbird to sort it out (which it does well) once fetchmail passed it to the local Postfix server. I got tired of having to teach the Thunderbird junk controls, so decided instead to setup my local Postfix server with Spamassassin: the stuff that fetchmail then pulls down and sends to my local Postfix server goes through Spamassassin before it hits Thunderbird. As you can tell Thunderbird to trust Spamassassin SPAM headers, it will automatically delete anything which Spamassassin has already decided is spam. Which means I don't have to teach Thunderbird, and can just teach Spamassassin instead (which is an automatic process), and let it learn by itself too as it finds more spam.