(This post is geekery of, if not the highest order, then fairly high order. It doesn’t contain any useful information about Ardour itself, but might be interesting for … people interested in such arcana)
The packaging issue that broke translations in our initial release of 8.11 for Linux and macOS was almost a cool bug. I thought I’d quickly describe it here for the geeks among us.
The problem came from a combination of two things: an actual error in our packaging scripts, and the subtle and generally not-considered behavior of the Unix find command.
Our wonderful translators work on files that end in “.po” and connect the original english strings in the source code with their translated versions. During the build process, tools from the GNU translation system are used to convert the .po files into .mo files (aka “message catalogs”), which contain the same information but in a binary format that can be more efficiently used by the program when it is running.
During packaging Ardour for distributions, we copy all the .mo files into a new location in preparation for “bundling” (e.g. as a DMG file for macOS or a .run file for Linux). The copy also requires a renaming, because the organization of the message catalogs for use by the program needs to be fairly different than the way they are organized in our source code.
So the first bug was that we used find(1) to locate all the .mo files and copy/rename them. We start in several locations within the source code, including the directory that holds the GUI source code (gtk2_ardour). The files we’re looking for are in the po folder, and we use find because we don’t want to hard-code the languages that have translations. However, it turns out that there are another set of message catalogs associated with the RedHat/Fedora “appdata” system, and these files not only also are somewhere under gtk2_ardour but also, because of the way the translation software works, they have the same name.
So, if find finds the “real” message catalogs first and copies/renames them during packaging, and then later finds the “appdata” message catalogs, the latter will overwrite the former in the package being built. This is a bug - the appdata message catalogs are placed in the packaging at a separate step of the process, and we should not have been using a command that was so generic. This was easy to fix (and has been).
But wait a minute … didn’t this work just fine for Ardour 8.10 and other releases? It did. How could that be? Well, recall that at the beginning of the previous paragraph I wrote “if find …”. It turns out that that the order in which find will find files, unless told otherwise, depends on the filesystem the files are located on. Consequently, if you use two different types of filesystem (e.g. on Linux the ext4 or xfs filesystems), find may very well return files in a different order on each.
However, it does deeper than this. Certain directory operations can also cause the filesystem to change its state in a way that will change the order in which find finds files. It turns out that this had happened within the build systems we use for macOS and Linux. At the time we released 8.10, find would locate the (unintended) appdata message catalogs first, copy/rename them into the package and then later repeat this for the real message catalaogs. Result? The package has the correct translation files and everything works. When we released 8.11, the ordering had changed, and the real message catalogs were found first, and then overwritten by the “appdata” ones. Result – translations do not work.
I thought this was an example of a fairly cool and unusual category of bug. There was an error in our packaging scripts - we used an unnecessarily generic command to find message catalogs that needed installing, which found files it should not have. But this mistake by itself did not matter on systems where the unintended files were found first. It only caused problems when the unintended files were found second.
This is not a perfect example of what programmers called “Abstraction Leakage”, but it’s not a bad one. We generally like to think of filesystems as things where the details of their internal organization do not really matter, and for the most part that is possible. But combine the fact that their internal organization does affect the order that a program like find will list files in, and the bug in our packaging script, and all of a sudden the internal details of how filesystems work becomes a thing we have to think about.
3 posts - 3 participants
Read full topic
