There were a number of interesting Talks. I especially liked the talk of our DPL, and those about piuparts and git-buildpackage. In contrast to the other LinuxTag talks, we had a livestream of our talks and recorded (most) of them. The kudos for setting this up goes to Alexander Wirt, who spent quite a few hours to get it up and running.

I have to apologize for being late in bringing my Notebook, which was intended to do the theora encoding of the livestream. This was a misunderstanding on my part, I should have known that this is not going to be setup in the night before show time… So to compensate the extra hours he had to put in for me, I offered to do the post processing of the videos.

Basic approach for post processing

The main goal of post processing the videos was (of course) to compress them to a usable size from the original 143 GB. I also wanted to have a title on each video, and show the sponsors at the end of the video.

My basic idea to implement that consisted of the following steps:

1. Create a title animation template.
2. Generate title animations from template for all talks.
3. Use a video editor to create a playlist of the parts title – talk – epilogue.
4. Run the video editor in batch mode to generate the combined video.
5. Encode the resulting video as ogg theora.

As always with technology, it turned out that the original plan needed a few modifications.

Title animations

Originally I wanted to use Blender for the title animation, but I knew it is quite a complicated bit of software. So I looked for something simpler, and stumbled across an article that pointed me towards Synfig Studio for 2D animation. This is also in Debian, so I gave it a try.

I was delighted that Synfig Studio has a command line renderer which is just called synfig and that the file format is XML, which would make it simple to batch-create the title animations. My title template can be found in this git repository.

Batch creation of title animations

I used a combination of make and a simple python script to replace the author name and the title of the talk into the synfig XML file. The data for all talks is another XML file talks.xml. Basically, I used a simple XPath expression to find the relevant text node and change the data using the ElementTree API of lxml python module.

The same could be done using XSLT of course (for a constant replacement, see this file) but I found it easier to combine two XML files in python.

Note that I create PNG files with synfig and use ffmpeg to generate a DV file from those. Originally, I had synfig create DV files directly but those turned out quite gray for some reason. I am now unable to reproduce this problem.

Combining the title animation with the talk

For joining the title animation with the talk, I originally went with OpenShot, which somebody of the video team had running at the conference. My idea was to mix a single video manually and just replace the underlying data files for each talk. I expected that this would be easy using the openshot-render command, which renders the output video from the input clips and the OpenShot project file. However, OpenShot stores the video lengths in the project file and will take those literally, so this did not work for talks of different play times…

I considered working with Kino or Kdenlive but they did not look more appropriate for this use case. I noticed that OpenShot and Kdenlive both use the Media Lovin’ Toolkit under the hood, and OpenShot actually serializes the MLT configuration to $HOME/.openshot/sequence.xml when rendering. I first tried to read that XML file from python (using the mlt python bindings from the python-mlt2 package) but did not find an API function to do that. So I just hard coded the video sequence in python.

I ran into a few gotchas on the way:

• it’s easy to segfault MLT from Python (something is wrong with the refcounting)
• generating ogg theora using MLT from Python (which uses libavformat) resulted in a bad audio/video synchronisation of the output (more than 1 second for 1 hour of video)
• generating dv from MLT and running ffmpeg2theora on the result is disk-bound on my quad core system – so use a pipe to ffmpeg2theora

Things to improve

While the results look quite okay for me now, there is a lot of room for improvement.

• What bothers me the most is the unequal audio volume between talks. We should apply something like replay gain for normalization. I did not find an easy way to add this to the processing pipeline, otherwise it would have been included already.
• Original recordings have different aspect ratios, so after combining them, there is a letterbox added. This is the case for Goswin’s talk on Multiarch for example. I’d rather have the title distorted or letter boxed, if I get around to it, I will fix it in the script.
• Most of the recordings concentrate on showing the slides. As we did not keep the raw camera data, but recorded the DVswitch output, we missed the opportunity to show the speaker more often in full screen. Looking at the same slide becomes quite boring after half a minute, so it would be better
  to show the speaker more often. That would require quite a bit of man power for post processing though, I have to admit that I did not watch all the hours of video.
• We could cut a bit off the talk at start and end, when the speaker discusses where he is available for questions. It is not of interest anymore after the conference is over.

Availability

• git repo (browsing): http://www.landschoff.net/git/?p=mdc-video.git;a=summary
• git repo (clone): git://landschoff.net/mdc-video
• Videos: http://meetings-archive.debian.net/pub/debian-meetings/2010/minidebconf-berlin/ (note: most of these are in german, apart from the DPL and the LXDE talk)

Thanks to this blog post, I have this fix in my bashrc:

The Debian bug tracker also knows about this problem, which is partly fixed for the eclipse packages. Bad luck that I am using a download from eclipse.org.

BTW, I changed it locally to check only for debian folder in parent directories and create debian/patches automatically if needed:

I’d rather end up with a patches dir inside the debian folder instead of having it at a random place in the package sources when adding the first quilt patch.

I wonder if we should ship this with the quilt package!?

Update: After discussion with James Vega on IRC, the quiltrc evolved into this version:

This works with any directory depth while the original would only support 5 levels from the package base directory.

Conventional wisdom is to use dpkg --get-selections and dpkg --set-selections to preserve the package selection. However, this does not keep the information about automatically installed packages.

Instead, I used

aptitude search '?installed?not(?automatic)' -F %p > packages.txt

to dump the packages I installed manually. Installing these was a matter of running:

aptitude install `cat packages.txt`

In fact this was taking so long that I loaded the packages in batches using

cat packages.txt|xargs -n 100 aptitude install --schedule-only; aptitude

While I was at it, it ran

aptitude remove '?obsolete'

on the original system, killing of a few hundred packages that accumulated since 2003.

Thanks to Holger Levsen for pointing me at the wiki entry which describes how a Debian developer can add himself to the Planet.

Specs

For the curious, here are the specs:

CPU

AMD Phenom II X4 965 (4×3.4 GHz)

Mainboard

MSI 790GX-G65

Graphics

ATI Radeon HD 3300 (on-board)

RAM

8 GB, DDR3

Hard drives

2x SATA 1.5 TB (by Western Digital)

Ubuntu installation

For testing and having a first look, I installed Ubuntu karmic koala (amd64). Installation went like a breeze, everything was detected and worked out of the box. Well, kind of – graphics performance was dreadful. Which was kind of expected.

Debian installation

I originally thought about using Debian only inside virtual machines or change roots. Still, Ubuntu does not really feel like “the real thing”. So I went to install Debian, aiming at a sid/unstable installation.

The first boot using the official squeeze snapshot netinst image (Binary-1 20091128-11:21) went fine. Up to the first prompt: No input was possible. Probably a problem with the USB keyboard I had connected (Logitech wireless). Out of curiosity I tried the graphical installer, which did not even get so far. It was stuck in an endless loop trying to start an X11 server.

So I dug for a PS/2 connected keyboard and had more luck. I got to the point where the installer searches for the CD-ROM drive. As I did not want to dismantle my old system yet, I used an external USB DVD/RW drive. This was not detected by the installer so I was unable to continue installation. I guess, the NIC driver was also on the disc so no ethernet either.

Today I installed the DVD burner into the tower and had more luck. It still amazes me how easy it is to create my default storage setup using d-i (LVM on RAID). However, the installer failed to reread the partitions after I created two on each drive, interestingly telling me about /dev/sda2 being busy. Perhaps a left-over swap partition from the Ubuntu install? Anyway, after running fdisk manually and writing the same partition table again, I was able to continue.

The remaining d-i steps went fine, with just a nuisance: I was asked for the console font setting – twice. And I had no idea what it was asking of me, AFAIK UTF-8 should be fine for all possible uses. I selected # Latin1 and Latin5 – western Europe and Turkic languages. Not that I will see the console unless Xorg fails to run…

After having the base system running, I rebooted without selecting any more software, partly because I knew that the resync on md1 will need restarting after booting into the new system.

The new system booted fine and I called aptitude to install build-essential, Xorg and both KDE and Gnome desktops. This pulled in MySQL via akonadi-server (ouch!) and I was asked for a MySQL root password. Seriously, I don’t care, this is a desktop system. I tend to forget the password anyway and the last time, root was able to reset it. So I just hit Enter, leaving no password set. This lead to the installation asking me two more times for the password, which really sucks given that the server is only used by Akonadi for whatever reason and the last time I looked, it creates its own MySQL configuration.

Albeit my old system is 8 years old, the new system still seemed slower. Which of course is due to disk latencies, given that /dev/md1 was being synchronized in the background.

Anyway, I was eager to test the desktop experience and started gdm. X came up fine, but again I had no mouse and no keyboard (USB mouse, keyboard connected as well as PS/2 keyboard). I also was unable to switch to a VT, so I logged in remotely and rebooted the new system.

After the reboot, hal obviously picked up the input devices and X11 worked fine now. At this point, I stopped as the sunday almost passed already.

Summary: Debian installation still needs some improvements I think. Maybe our distribution is just too stable, after all my last install is 3 years back – due to a disk crash…

Relevant Debian bugs

• #558679: My installation report with some logs included.
• #558681: Nagging about MySQL root password.
• #558686: Failure to reload partition table.
• #545933: No USB keyboard support
• #315553: Installation from optical drive on usb fails (but this is for kernel 2.4)
• #558691: No keyboard and mouse after fresh installation of xserver-xorg.

After a bit of searching, I stumbled across schroot, which turned out to be a really useful package. The blog post “Joys of Schroot” by Enrico had me going in no time (thanks Enrico!).

Since I built a number of packages multiple times, I did not want to download the required dependencies each time. I also did not want to do any tricks with approx as suggested by Enrico. Instead, I added another bind-mount to /etc/schroot/mount-defaults:

/var/cache/apt/archives	/var/cache/apt/archives		none	rw,bind		0	0

With this simple change, the chroots now share the apt cache with my main system.

These packages may also be of interest to other Debian users, therefore I make them available here:

• Roundcube packages
• jQuery package

Everything else you need is available at backports.org, most notably the database wrappers in php-mdb2.