Ben's technical blog

FOSDEM 10 Distribution Developer Room videos now available

2010-02-28T20:17:25Z

As previously announced elsewhere, video recordings from the Distribution Developer Rooms at FOSDEM 10 are now available. All but two talks were recorded and are available in Ogg Theora+Vorbis format, in low-bandwidth (~300 kbit/s) and high-bandwidth (~1.5 Mbit/s) versions.

These recordings should also be available later on the FOSDEM YouTube channel.

Video and audio were recorded by Dominique Dumont (H.1302) and the DebConf video team (H.1308) with the assistance of some additional volunteers. Thanks to everyone involved in recording, and thanks also to the speakers and the organisers of FOSDEM.

This season I have mostly been building kernels

2010-02-28T18:11:09Z

As some people noticed, a driver update to support a major hardware release in my day job and a long series of trivial patches motivated by Debian kernel work combined to make me joint most prolific change author for Linux 2.6.33. This was a fluke, and 2.6.34 is likely to be fairly quiet for me.

Team work

Meanwhile the kernel team is still working hard on fixing bugs in Linux 2.6.32. As previously announced, this kernel version will be used in both Debian 6.0 'squeeze' and Ubuntu 10.04 LTS. It is also known to be the kernel version for RHEL 6 though this has not been officially announced. If you find a serious bug that was fixed in 2.6.33 or later, please ask the author to send the fix to stable@kernel.org so that it will be included in all these distributions. Note that this happens automatically for commits with the line 'Cc: stable@kernel.org' in their description.

We are also continuing to backport new drivers and new hardware support in existing drivers into Debian 5.0 'lenny' (stable). Missing support for common hardware is considered an important bug and suitable for stable updates. However, if there were major changes to a driver before those that added new hardware support, we may be unable to produce a backport. We are not able to carry out comprehensive hardware testing and do not want to risk a regression existing hardware support.

libata transition - call for testing

Finally, a major change I have been working on is the transition from the old IDE drivers for PATA controllers to new drivers based on libata. The old IDE subsystem is no longer developed and some drivers do not properly support all the hardware that their libata-based counterparts do. However, while the IDE drivers generate device names beginning with 'hd', libata presents PATA devices as SCSI devices and generates device names beginning with 'sd'. In a system that already has other SCSI or SCSI-like disks, names may change somewhat unpredictably. Similar problems exist for PATA CD/DVD and tape devices.

So, while the transition doesn't involve any kernel hacking, it does require a complex upgrade process. Any configuration files that refer to IDE or SCSI device names may need to be changed. It is not enough to switch to only SCSI device names, since we cannot know what they are in advance and the configuration files should continue to work with kernel versions from before and after the transition. In the case of disks we can normally use the partition label or UUID: many tools understand the LABEL= and UUID= syntax, and for others we can refer to the symlinks under /dev/disk created by udev. In the case of CD/DVD devices, we can use the aliases /dev/cdrom etc. created by udev. In the case of tape devices, however, you're on your own.

In experimental, kernel image packages depend on a new package linux-base which implements this transition (from 2.6.33-1~experimental.2; the previous version was broken). The postinst script will prompt you to make changes automatically or manually. It can convert most bootloader configuration files, /etc/fstab, the udev CD aliases configuration and the initramfs-tools resume partition. It can also label partitions that don't have a label or UUID. Please do test it and verify that its changes are correct. All changed configuration files are backed up with a suffix of '.old' (or '^old' in one case).

Debian kernel team at Linux Plumbers Conference

2009-09-23T17:34:13Z

Most of the Debian kernel team members are attending the Linux Plumbers Conference in Portland over the next 3 days. We'll be discussing kernel packaging and integration issues among ourselves and with upstream and other distributors.~

You may have noticed that Linux 2.6.31 is out but not yet in sid. This kernel version contains a large number of known regressions which should be fixed in subsequent stable updates, so we will wait for those. In the mean time we've uploaded an update to 2.6.30 which should resolve the most serious bugs.

DebConf 9 videos - progress

2009-09-13T13:14:16Z

Low-bandwidth versions of the recorded talks are now available. There are currently two exceptions:

Some of the music for 'Welcome to DebConf 9' is licenced as CC BY-NC-ND, which unfortunately doesn't allow use in a video. I'm discussing this with Marga and hope to upload a new version soon.
'Debian por dentro' has a piece missing which should be restored from tape.

While the master versions of all these talks have been checked, not all the generated low-bandwidth versions have. Please let us know on debconf-video@lists.debconf.org if there is something wrong with them other than the following:

A few of the recordings were unfortunately incomplete; this is indicated by a fade-in or fade-out
Some of the recordings are in two pieces
There were a few drop-outs where you will see a black screen and no sound for up to a second
The text on some slides is unreadable; in this case, refer to the file provided by the speaker or wait for the high-bandwidth version

The high-bandwidth versions will be uploaded over the next week (slow connection is slow).

DebConf 9 videos

2009-09-10T03:24:33Z

Several people have been asking after the videos. My answer to them is "we release when it's ready".

Seriously, though, I'm sorry about the delay in publishing videos. There was some miscommunication between me and the sysadmin team at DebConf which meant I couldn't take files away with me then. Peter Palfrader reassembled the video RAID later, copied the necessary files onto a single hard disk, and posted that to me. This disk apparently got lost on the way, but eventually arrived here last week after a 3-week journey.

Over the weekend I dealt with most of the videos that required editing, and today I was able to resume the transcoding process. Tomorrow I intend to start uploading again.

Photos from DebConf 9

2009-08-11T11:43:37Z

My photos from DebConf 9. These are all taken with my new digital compact as my film SLR is in need of repair. I'm quite pleased with the results.

¡Olé!

2009-07-29T23:12:46Z

Geek Flamenco, by Nattie

Status of firmware in Debian

2009-07-29T16:23:35Z

A question from AJ reminded me that I haven't said much about the changes to packaging of firmware in Debian, and in particular the separation of non-free firmware from the Linux kernel.

Linux kernel packages

There is an ongoing process upstream to move firmware blobs from drivers into a firmware/ subdirectory of the source, which is now almost complete. Since most of this firmware is non-free, we remove it from the source tarballs for kernel packages but use it to update the firmware-nonfree "source" package.

We continue to patch some drivers to separate out firmware, and have been submitting our changes upstream. Most of these have been accepted though the DRI drivers matrox, r128 and radeon are notable exceptions.

A few months ago I attempted to make a new inventory of the remaining firmware blobs outside of the firmware/ subdirectory. I identified three that should still be addressed. The Linux-libre project, however, removes many other constant arrays from the kernel (and disables the affected drivers) where I judged the array to be a plausible preferred form of modification.

Firmware packages

Much of the non-free firmware removed from the kernel is now available in the firmware-linux package in the non-free section of the Debian archive. Starting with Linux 2.6.31, we will build the DFSG-free firmware shipped with Linux into a package called firmware-linux-free, which will be recommended by kernel image packages. The contents of firmware-linux will be moved to firmware-linux-nonfree and firmware-linux becomes a meta-package depending on the other two packages.

Many other firmware images never distributed with Linux are also packaged for the benefit of users that require them.

DebConf

2009-06-12T02:14:07Z

I'll be involved in the video team again.

Python relicencing

2009-04-01T01:51:06Z

One of the more interesting talks at last week's PyCon was on the new Python licence.

Busy at PyCon

2009-03-26T18:58:15Z

This week I'm at PyCon in Chicago, helping to train and supervise the A/V team in recording the tutorials and talks. We have a lot of inexperienced volunteers but they're learning quickly. Carl Karsten (in charge of A/V) and Ryan Verner (another outsider with long experience from LCA) are organising things. Ryan, Dave Noble and I provide technical assistance to the rest of the team.

I'm also working on my first Django app, which will be used for reviewing files. This is essentially a clone of the excellent work done by Gunnar Wolf and Damian Viano in Pentabarf. PyCon doesn't have a centralised database so this is a standalone application. Ryan is now applying styles to my horribly bare HTML, and I have a short list of bugs to fix before we ask the video team to start reviewing.

The tutorials started yesterday and talks wil start tomorrow in different rooms, so we have to set up all the equipment again and reconfigure for multiple cameras. Hopefully by Saturday things will be running smoothly and I can spend some time in the city. It would be a shame to come all this way and stay by the airport for the entire time!

Going to PyCon

2009-02-15T17:44:33Z

I'm going to

Carl Karsten of the PyCon organisers recruited three of us from the DebConf video team to lead and train the video team at PyCon.

Searching for invalid object code

2009-01-06T00:53:09Z

Jérémy Bobbio suggested that I should explain how I looked for packages affected by a compiler bug (Debian bug 506713; gcc bug 38287). I don't claim that this is a particularly good way to do it, but here it is:

First, I identified a pattern to search for. Unfortunately I don't really understand the cause or fix for the bug, but I did have the example which led to this bug report: Debian bug 490999. The bad code was a stack-pointer-relative load immediately after a stack allocation (SPARC save instruction) where the offset was not adjusted for the stack allocation:

save  %sp, -112, %sp
ld  [ %sp + 0x40 ], %i5

I generalised this to:

save  %sp, offset1, sp
...
ld  [ %sp + offset2 ], register

where offset1 + offset2 < 0. Of course, this may be valid if the intervening instructions include a restore, branch or store to the effective stack location that the last instruction loads from. I ended up allowing up to 10 intervening instructions and examining a disassembly to work out which cases were valid.

I looked up the instruction encoding for these two instructions. Thankfully SPARC is RISC so they are simple and regular:

save %sp, offset1, sp is encoded as 0x9de3a000 | offset1 & 0x1fff
ld [ %sp + offset2 ], register is encoded as 0xc003a000 | reg << 25 | offset2 & 0x1fff

I took the dumb but effective approach of scanning entire files for this pattern rather than only scanning their code sections. This seemed to work - I got no false hits for non-code - but might not work for other patterns that could match ASCII text.

I wrote the scanning program in Python, which is my default choice of language unless I know it's going to be too slow. I was hoping to be able to read the code files into arrays, but unfortunately the Python array type only supports the native byte-order (SPARC is big-endian and I was intending to use an x86 which is little-endian). I tried reading into a tuple using struct.unpack, which does support explicit byte-ordering, but this used so much memory for larger files that the program swapped to a crawl. So finally I resorted to reading the file into a string, doing a string search for '\x9d\xe3', rejecting matches that weren't appropriately aligned, then unpacking and comparing the code words from the point of the string match. (In Python 3.0 I would have to use the bytes type for this, as str is a Unicode string type.)

So that's how I scanned single files. The next step was to find, unpack and scan all the SPARC shared libraries in the archive. (This particular code generation bug is understood to affect only PIC code, and that is normally only used in shared libraries.) I wrote functions to search Contents-sparc for shared library files - assumed to match the pattern ([^\s]*/lib[^/\s]+\.so(?:\.[^/\s]*)?) - and to parse Packages to find the filenames for the packages containing those files. The latter uses the debian_bundle.deb822 module from python-debian. The last key function downloads and unpacks a package using wget and dpkg-deb. I could have used the httplib module for downloading but I correctly anticipated that I'd need to restart the script several times so I wanted to cache the packages which was easier to do using wget.

So, that's the explanation. If you really want to see it, here's the code.

FOSDEM 2009

2008-12-04T02:26:19Z

Are you?

My Debian packages in Git

2008-11-02T15:50:36Z

I finally got sufficiently fed up with Subversion yesterday to begin converting my package repositories to Git. This has many benefits, including the fact that I can now easily publish them. Browse or clone them as you prefer. Or just ignore them.

For those who care about firmware

2008-10-22T00:53:55Z

Some drivers in Linux include "firmware" - code that runs on the device the driver deals with, not the main processors in the system. Most of this is in a binary or equivalent form, which is probably not the form that it was originally written in. This makes it non-free by Debian's standards, an issue which has led to much argument around previous releases and has blown up once again.

The good news is that kernel developers are making a continuing effort to separate this from the driver code. I've collected together the patches that do this and written 2 of my own, which appear to cover all the remaining sourceless firmware. But I was only able to test one of these drivers (radeon) - the rest is up to you.

Affected drivers

Driver	Hardware which requires firmware	Bug report	Change made
cassini	Sun "Cassini" Ethernet controllers with "Neptune" PHY	498631	removed
dabusb	BayCom DAB-USB radio receivers	502663	moved to firmware-dabusb
e100	Intel PRO/100 Ethernet controllers with PCI revision 8, 9, 15 or 16 or built into ICH6 or ICH7 (LAN-on-motherboard)	494308	moved to firmware-e100
kaweth	Kawasaki LSI USB Ethernet controllers	502665	removed
mga	Matrox G200, G400, G450(?) and G550 graphics controllers, 3D acceleration only	502666	moved to firmware-matrox
qla1280	QLogic QLA1XXX SCSI controllers	502667	moved to firmware-qlogic
r128	ATI Rage 128 graphics controllers, 3D acceleration only	494007	moved to firmware-ati
starfire	Adaptec DuraLAN aka Starfire Ethernet controllers	501152	moved to firmware-adaptec
tehuti	Tehuti Ethernet controllers	501153	moved to firmware-tehuti
typhoon	3Com Typhoon (3CR990) Ethernet controllers	502669	moved to firmware-3com
whiteheat	ConnectTech WhiteHEAT USB serial adapters	502668	removed

How to test

Download and install the linux-source package from http://people.debian.org/~benh/firmware-removal/. There is a signed .changes file there that you can use to verify the package.
Build and install a kernel from that source (you should probably use make-kpkg).
Boot the kernel. Check that the driver fails cleanly: it should not crash, and there should be a useful error message in /var/log/messages
Download and install the relevant firmware package, if available. These are available from the same place as the linux-source package.
Reboot or reload the driver. Check that the driver is now working.
Add your results to the bug report as shown above.

Firmware that is removed

Do you depend on the drivers for which I have removed the firmware? Do you have contacts with the vendors who might be able to help relicence it? Please send mail to the bug report.

Update

I checked my email and found a response from ConnectTech; the WhiteHEAT firmware will probably be clear to package in firmware-nonfree. There is no need for anyone else to contact them.

chmod -x considered harmful

2008-09-06T22:24:56Z

I discovered an interesting "feature" of chmod(1), which caused a package build to fail. According to the GNU manual page, if no letters are used before a "-" or "+", "the effect is as if a were given, but bits that are set in the umask are not affected." The command will also fail with an error message when it does this!

The Single Unix Specification says this is correct, though there is some ambiguity over whether the exit status should be 0 or not.

Anyway, the result is that it is generally a bad idea to call chmod in this way in a script. Always specify who the permission changes should apply to.

DebConf videos on Planet Debian

2008-08-10T08:32:25Z

In addition to live streams, recordings of each event should be published on meetings-archive.debian.net. There is an RSS feed of these, which should work nicely in Miro: http://meetings-archive.debian.net/pub/debian-meetings/2008/debconf8/index.rss. That feed is also included on Planet Debian. This is the first time I've tried generating RSS so apologies in advance if I make a mess of the Planet.

Last of the DebConf 7 videos, no, really

2008-08-03T19:58:23Z

Sorry, I missed a few last time.

Dependency-based boot sequence	low	high
Data-mining Popcon	low	high
Debian-installer - an update	low	high
Wacky ideas II	low	high

Last of the DebConf 7 videos

2008-07-29T09:15:40Z

A few of the recordings from DebConf 7 had to be recovered in whole or in part from tape. For personal and technical reasons, a few of these were not done last year, and I have finally dealt with them now. The last few sessions are:

The secure Debian Desktop	low	high
Debian - The Universal Operating System?	low	high
Wacky ideas BoF	low	high
lintian BoF	low	high
Debian travels around the world	low	high
Leading a Free Software project	low	high
Method diffusion in large volunteer projects	low	high
Debtags is ready	low	high
OpenStreetMap	low	high
Maintaining packages with Git	low	high
netconf	low	high
Time for a better init system	low	high

Hopefully these are useful to someone. Sorry it took so long!

Becoming a kernel developer

2008-05-03T15:15:16Z

After a fair amount of clean-up, a lot of waiting for review, and some ruthless pruning, I and my colleagues at Solarflare finally got our net driver accepted for Linux 2.6.26. Meanwhile a larger version of the driver made it into Xen's Linux tree, along with dependent drivers to provide a secure fast path to the hardware from domU. Somewhere down the road we're going to have to resolve these two versions, but now I'm just pleased to have a working driver in-tree.

There are some other changes I've worked on over the past few months:

I maintain diagnostic scripts for our NICs and drivers. Among other things they include Vital Product Data (VPD). There's a standard way to read VPD in PCI (actually there are two, in different versions of the spec) but in Linux this is only accessible to root, and even then there are race conditions. We provide an MTD driver for flash and/or EEPROM attached to the NIC, and I know where the VPD appears in there, but most users won't be loading that driver. So I wrote this: Expose PCI VPD through sysfs
Along the way I found a bug in sysfs, which I fixed: Disallow truncation of files in sysfs
One day I accidentally bridged one of our network interfaces, which has Large Receive Offload (LRO) enabled by default, with one of the motherboard's network interfaces, and the results were not pretty. LRO isn't really compatible with packet forwarding, and currently causes the GSO code to crash or warn. I wanted to solve this by disabling forwarding of LRO skbs, but it turns out to be trickier than I thought.
We wanted to use topology information to allocate one IRQ per CPU package, not per core (or per thread, with HT enabled) for Receive-Side Scaling. Unfortunately this information is only available in some kernel configurations and not for all architectures. I had a go at providing reasonable defaults, but it didn't work too well!

However, this is still only part of my work. My main task at the moment is to write more unit tests for our next NIC.

My day job

2008-02-21T00:24:44Z

I'm employed at Solarflare Communications in the test group, supporting the work of testers and developers by developing test frameworks. (We actually have very few pure testers and a whole lot of automation.). In a major break from my previous jobs, I've been working mostly in Python on applications for use in-house.

More recently I've become involved in maintenance of our network driver for Linux, starting with updating it to use current kernel coding conventions, and submitting for inclusion in the kernel. This has become a bit frustrating because while there are a lot of people prepared to run checkpatch and pick nits, there are not so many who will do a thorough review of a driver that's 20,000-odd lines long. And without someone doing that, it won't go in.

Meanwhile some of my colleagues have done some great work to accelerate Linux networking on Xen by having drivers in dom0 and domU cooperate to expose hardware resources directly to domU. Our hardware architecture supports many virtual NICs with their own packet queues and set of buffers, so this does not compromise the security of the physical machine and dom0. The results are very impressive.

The original motivation for the virtual NIC architecture was as a basis for a user-level networking stack, which can provide very low latency and reduced CPU usage. This software was originally proprietary but has now been released under GPLv2 as OpenOnload. Google invited two of my senior colleagues to talk about this - here's a video of their talk about user-level networking.

Debian Miniconf 7 recordings

2008-02-07T22:44:50Z

Most of the Debian Miniconf 7 sessions at LCA 2008 in Melbourne were recorded by the LCA video team. The recordings are now mirrored on meetings-archive.debian.net.

Idea for a new PAM module

2008-01-19T03:43:17Z

To guard against late-night administrative mistakes: pam_breathalyser.

Separated at birth

2008-01-13T23:19:52Z

L: Bill Gates; R: Matthew Garrett

Introducing ParticleMan

2007-11-11T18:22:44Z

As ~~threatened~~ promised, I have produced a renamed version of Ion3 that should avoid upstream trademark claims and thus be DFSG-compliant. Some translations are out-of-date, but I'll release a first version now and try to get those updated later. The new name is ParticleMan. When I put the name to a vote some time back the winner was Iceparticle, but there's already an icewm and I couldn't resist the TMBG reference. (Whoever suggested this name to me: I had already thought of it but forgot to include it in the poll.)

The packages are in my repository for now.

Ion3 rc-20071109

2007-11-11T01:49:21Z

Since ries (ftp-master.debian.org) is still down, I've uploaded packages for the latest version of Ion3 to my own server. That's:

deb http://womble.decadent.org.uk/debian/ sid/
deb-src http://womble.decadent.org.uk/debian/ sid/

The repository is signed with my personal key.

Publication of DebConf 7 videos

2007-07-28T17:39:26Z

The DebConf 7 videos needed some post-processing, mostly to recover missing pieces from tape. I've done a large part of that, but it's not yet complete.

The video team to-do list has details of the recordings that still need work, and the sessions that we believe were not recorded. Please tell us if we did record some of these, and we will have another look through our tapes for them.

The conversion to "publication" formats (i.e. small enough to offer for download) was started at the conference and I have continued this at home over the past 2 weeks. Unfortunately this has been interrupted by some hardware failures. It looks like the CPUs in two of my PCs were running hot enough to be damaged (slowly) but not hot enough to trigger an alarm or automatic shutdown. The remaining conversions will be running mostly on my laptop, which is pretty fast and should be able to finish conversion of the recordings that are in a good state in about 2 days. However, some of the converted files may be broken and need some manual attention.

My upload bandwidth is only 448 kbit/s so the high-quality files will take about a week to upload.

As ever, the video files are being published on meetings-archive.debian.net.

Everything you Never Wanted to Know about PKI but were Forced to Find Out

2007-07-04T22:43:26Z

Not my title, but that of Peter Gutmann's presentation about PKI, specifically X.509 as used in SSL. It's informative, entertaining, and I suspect not as well known as it should be.

How to use a "system tray" in Ion3

2007-07-04T22:39:15Z

Ion3 supports docked applications and is compatible with the docking protocols used by WindowMaker and KDE. Normally you would use:

dopath("mod_dock")

to load mod_dock, which provides a standard corner dock, toggled using MOD1+D.

However, recent versions also support docking applications in the status bar, in "system tray" style:

-- In cfg_ion.lua:
defwinprop {
   class = "foo-window-class",
   statusbar = "foo"
}

-- In cfg_statusbar.lua:
mod_statusbar.create {
   ...
   template="... %systray_foo"
}

GNOME unfortunately uses a different dock protocol. However, the docker program can adapt from this to the WindowMaker protocol. So you can include a GNOME system tray in your status bar by including docker in your X session and this in your Ion3 configuration:

-- In cfg_ion.lua:
defwinprop {
   class = "Docker",
   statusbar = "dock"
}

-- In cfg_statusbar.lua:
mod_statusbar.create {
   ...
   template="... %systray_dock"
}

dvswitch in use

2007-06-19T01:09:14Z

It's day 2 of DebConf and dvswitch is being used by many members of the video team. So I have a pretty good idea of what it does well and what I need to fix in it.

The most serious problem is that there's no way to start and stop recording from the GUI. That's actually controlled by starting and stopping the file sink, which runs on a remote machine. Here I have arranged to start it by running a command with ssh in the background, which is stopped by quitting dvswitch or with pkill! The result is that we can record a lot of junk between events, which has to be checked and then discarded, or alternately stop recording and risk missing the next event.

Also, we like to number our cameras: 1 for the speaker, 2 for the audience, 3 for slides. But dvswitch assigns numbers to sources in order of connection. I fudged around this by again using ssh to start the sources in the proper order after dvswitch.

Despite these and a number of other flaws, it seems that several people have been impressed by the ability to cover talks from multiple angles. I hope this will attract more people to contribute to dvswitch so that it can serve DebConf and other conferences better.

DebConf video software

2007-06-19T01:07:41Z

The DebConf video team is using a large number of programs, some common and some of our own, spread across 20-30 computers. Some may be interested to know what this is.

FAI for package installation and configuration on DebConf machines.
dvgrab for video grabbing. We are using a variety of DV cameras, but dvgrab works well with almost all of them.
dvswitch for switching between multiple cameras and files.
ffmpeg2theora for transcoding to Ogg Theora/Vorbis, both for live streams and for the downloadable files slowly appearing on meetings-archive.
A small number of scripts for starting the above three programs and recording to files.
Icecast 2 for streaming. We are currently using the sarge package rebuilt for etch because the server in the etch package is unstable in our double-relayed configuration. (To limit bandwidth use between DebConf and the Internet, we have an internal server for internal use and an external relay that serves as master for all other relays.)
GeoDNS and a monitoring script to resolve streams.video.debconf.org to the nearest working relay server.
A newly-developed extension to the Pentabarf conference management system that presents recordings for review. Reviewers match up recordings to events, give the start and end time of the event in the recording so we can cut out extra material, and rate the quality of the recordings and transcoded files.
A script to import the details of new recordings into the database ready for review, run as a cron job.
A script to transcode recordings into multiple versions using the timing and naming information provided by reviewers. This spreads the work across multiple servers controlled using ssh (again!).

Most of the custom software and configurations should be included in the debconf-video project on Alioth.

MSN Video enforces "free software"

2007-05-09T23:36:17Z

Semi-annoying, semi-amusing:

I'm using ~~Firefox~~ Iceweasel 2.0.

Renaming of Ion3 in Debian

2007-05-07T12:55:09Z

Continuing a trend of upstream control-freakery, the author of Ion has claimed Ion and Ion3 as trademarks and set out a trademark policy that will require Debian to rename it. I've created a poll for selecting a new name.

Introducing dvswitch

2007-04-30T08:28:56Z

I just released version 0.5.0 of dvswitch, the software DV mixer intended for use at DebConf 7. I believe I've met all of the video team's essential requirements and most of the desirables.

Unfortunately real video mixing seems to take a faster machine than I have at the moment and I very much doubt I'll have time to improve that much. Although I have implemented picture-in-picture mixing, it won't be usable and may result in frame dropping, so I've started a stable branch intended for DebConf 7 that doesn't include it. Version 0.5.0 is released from this branch and can be considered the first alpha test version. There's now a little over a month to test, fix bugs and make minor improvements on this branch.

ACCU Conference

2007-04-14T17:16:08Z

I've been meaning to attend the ACCU Conference for some years and finally decided to pay my own way for a couple of days this year. It was an opportunity to hear world experts on C++ and other programming topics and to a smaller extent to meet and socialise with other professional programmers.

I attended the following sessions:

Keynote: Improving Collaboration in Open Source Projects, Mark Shuttleworth

This was a somewhat surprising choice of speaker and subject since I think most attendees work on proprietary software, but I think any sensible developer in the proprietary world will try to build on free libraries under non-copyleft licences, where appropriate. Mark talked about governance and leadership in "open source" projects (actually praising Debian for having a clear constitution) and about the need for good communication between projects, for example about release schedules. He seems to have accepted that Launchpad cannot be a single hub, and called for standard formats for, for example, exchanging information about related bugs between bug trackers. But it doesn't sound like he's ready to take the lead on that.

Choose your Poison: Exceptions or Error Codes, Andrei Alexandrescu

Andrei looked at the strengths and weaknesses of various error reporting mechanisms in C++ and introduced a technique for making error handling more flexible. Normally a function is designed either to indicate errors through its return value or to throw an exception on error. If a function returns an error indicator, it's easy for callers to ignore it. If it throws an exception on error, then a caller that provokes errors more often than the called function's author anticipated pays a high cost in processing time for those errors.

Andrei's solution involves a class template Likely<typename T> whose instances hold either a return value of type T or an exception. A function that can fail has a specialisation of this template as its return type. The caller can check for failure if it knows how to handle it, or it can ignore it - in which case the conversion to T or the destructor will throw the exception. This is clever but doesn't seem to be entirely satisfactory. The destructor which throws does first check that no exception is already being processed, avoiding collision of the two exceptions and consequent abrupt termination of the program, but this can result in errors being unexpectedly ignored.

Introduction to Component-Level Testing, John Lakos

A marathon 3-hour talk on designing and testing software, particularly C++ programs (the subject of his well-known book) as a set of well-defined components.

Lakos first defined these components as being "physical" i.e. sets of source files - normally a header file defining an interface, an implementation file, and a test driver file. They may also include metadata for the build system. He talked about the value of reusable components (as if this wasn't obvious!) and the risk of introducing circular dependencies - which essentially turn many components into one and reduce reusability - if we fail to keep track of dependencies. Software built out of components can be viewed as a hierarchy or stack defined by dependencies. In a large system, components with similar dependencies and dependents can be grouped together into packages or package groups to simplify a view of these dependencies.

He explained that the test driver's dependencies count just as much as the implementation's, because we cannot reasonably test a component until we have tested its dependencies. Test drivers should also be written to avoid depending on environmental settings and configuration, so far as possible.

Finally, and perhaps most importantly, he spent a long time talking about how to construct test cases and test data. I might try to summarise this in a later entry if anyone's interested.

The Appliance of Science: Things in Computer Science that every practitioner should know, Andrei Alexandrescu

Andrei began by complaining that OpenOffice Impress was even worse than PowerPoint, but being able to use Linux (specifically Ubuntu) more than made up for the pain.

He briefly presented four key ideas that he believes programmers should understand and be ready to make use of:

Dynamic programming. The Wikipedia article summarises this as "a method of solving problems exhibiting the properties of overlapping subproblems and optimal substructure (described below) that takes much less time than naive methods". As a simple example, Andrei presented a Fibonacci number generator. But since no-one really needs Fibonacci numbers, he also showed its application to calculation of the Levenshtein distance between words which is important in spell-checking and fuzzy string matching. A naive approach to either of these problems is so inefficient as to be useless.
Garbage collection and its connection to type-safety. Any object model can only provide 2 of the following 3 features without losing type-safety: mutable objects, aliasing (multiple pointers/references to the same objects), and explicit reuse of memory (via free, delete or similar). C and C++ have all three of these. Garbage collection eliminates the need for the third. However it does require more virtual memory and/or more processing time (there is a trade-off between these). (He quoted numbers from Hertz and Berger, Quantifying the Performance of Garbage Collection vs. Explicit Memory Management, 2005. However, these numbers show the cost of GC in Java, specifically Jikes, not as an addition to C or C++.)
Machine learning. Sometimes we don't know how to solve a problem, but we can have the program gather feedback that will allow it to learn how to do it better. This depends on the "smoothness assumption": small changes in inputs have a small effect on the result, rather than making sudden changes. Examples: image recognition, deciding whether changes to web page layout are helpful, disambiguating search terms.
Transactional memory is the most promising approach to achieving scalability across massively parallel systems. Most hardware performance improvements over the next 10 years will come from greater parallelism, not higher clock speeds.

Writing for ACCU (BoF)

I think I've been convinced to write an article about standardisation of multithreading in C++.

C++ Threads, Lawrence Crowl

Crowl gave an overview (similar to this) of the extensive work that's been done on standardisation of multithreading in the C++ language and library. This is one of the most important features in C++0x due to the increased parallelism mentioned above. The current semantics of multithreaded programs are not standardised, resulting in subtle but potentially serious differences between current C++ implementations that support multiple threads, and in the need for assembly-language code in programs that use lockless synchronisation. Much of this is also relevant to C and is being considered by the C standard committee.

C++0x Initialisation: Lists and a Uniform Syntax, Bjarne Stroustrup

Stroustrup recapped the 4 forms of syntax for initialising objects in C++, some inherited from C and some necessarily invented for C++, none of which are entirely generic. He noted the lack of a simple syntax to initialise the elements of containers, which violates the general principle that class types should have the same status as built-in types in C++. He then introduced a new initialiser syntax that he and others have defined for C++0x, which is a brace-enclosed comma-separated list of values, similar to that used for arrays and C-style structures (aggregates). Container types can define a constructor with a parameter of type const std::initializer_list<T> & and when constructed with this new form of initialiser they will receive this wrapper for a constant array of element values. For types that do not, the values in the list are treated as arguments to some other constructor, if one matches. Otherwise, for compatibility, this may be treated as aggregate initialisation.

C++ Standard Library report, Alisdair Meredith

There was far too much information here for me to even begin to summarise. Suffice to say there are lots of good things coming bu the library committee will have its work cut out to get them all properly specified in time for the Committe Draft coming late this year.

I also met old friends and new, put faces to names from newsgroups and books, and got too little sleep.

I can't draw cartoons

2007-03-24T15:19:01Z

Gecko and ATI graphics

2007-03-24T01:11:10Z

Just as Gecko exposes a bug in the nv driver, it can also provoke poor performance in the ati driver. This turns out to be avoidable by configuring X to use the newer driver interface.

Developing a DV mixer

2007-03-15T01:18:10Z

For this year's DebConf, we (the video team) want to be able to use multiple cameras like at DebConf 5, where we had one for the projection screen and one for the presenter fed to an analogue mixer and a digitiser. At DC6 we had no mixer and no budget for one. Same at DC7 - unless we can do it with software. Toresbe had a go at writing one but didn't finish. I want to give it a go. So:

Alioth project
Requirements (first try)
Camera
Action? Er, actually I need a Firewire card first.

New blog

2007-03-10T16:55:42Z

This is my new blog for technical/geeky/Debian stuff. I'm keeping my LiveJournal for social and private matters and syndicating this one to Planet Debian.

I hope this separation will make me less afraid of boring LiveJournal readers with technical minutiae and Debian politics or boring Planet Debian readers with local social stuff, so I actually write more. Don't get too hopeful though.