Why the KDE project switched to CMake -- and how (continued)

Please consider subscribing to LWN Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

Why the project revamped its build system for KDE4

KDE developer Alexander Neundorf explains the background for the move away from the traditional "autotools"

KDE is one of the largest Free and Open Source Software (FOSS) projects. It follows the typical "distributed development" model used by many other FOSS applications. More than 1.200 developers around the planet have accounts and credentials to access its central source code repository. This repository currently holds more than 4 million lines of program code, translations of approximately 100,000 user interface strings (and many more lines of application manuals) into more than 80 different languages. Every day there are about 300 or more "commits", adding new or modifying existing content.

Any software project of this size and scope can only prosper and go forward if it uses tools that are good enough to manage and build all its code, for all its contributors, on all supported OS and CPU platforms, all the time, without major problems. Oftentimes "good enough" here translates into: "the best one that is available for our purposes".

For its central source code management KDE last year migrated from the venerable "Concurrent Versioning System" (CVS) to the newer, more powerful Subversion (SVN) software. That change in itself was an enormous stress test for the capabilities of the still young SVN project: a year ago, several preparatory "dry runs" which simulated the pending move revealed quite a number of bugs and performance problems in advance. Close cooperation of some core KDE hackers with the Subversion developers lead to fixes and improvements to SVN itself before the real change-over for the huge KDE repository finally happened.

Now the next big change is happening: KDE is leaving the aging "autotool" build chain behind. Some developers, not only in KDE, like to nickname the autotools as "auto-hell" because of its difficult to comprehend architecture. So, KDE 4 will feature a completely different build system: CMake.

In typical KDE fashion the current move to CMake was not a "decision by committee". Instead, the old rule "who codes, decides" made itself felt once again. Let's look back at the history of this change.

The principal move away from autoconf, automake, configure, libtool & friends was decided at last year's annual KDE conference, akademy. KDE developers at the time discussed and evaluated several alternatives: back then, SCons (a Python-based build tool) was favored, spiced up with a KDE-wrapper called bksys to help with the actual work. SCons/bksys already worked well for a number of developers who used it in their sub-projects, and the tandem seemed to easily win the race. Up until January 2006, several people worked hard on replacing the existing autotools based build system of KDE3 with SCons/bksys for KDE4. Their first acid test was to make it compile kdelibs on various platforms.

However, various hurdles showed up unexpectedly. The KDE individuals who tried to bring SCons into a shape that made it fit for building such a huge project felt they didn't have any support from the upstream SCons developers. There were major problems building KDE on non-Linux platforms with SCons (e.g. on OS X); in general they felt it did not yet have a mature configuration system. The only option down that road was to create major SCons fixes and patches on their own. Since these changes would not likely be included in the upstream sources, it would require permanent maintenance of the fixes in a separate repository. In effect, this would have amounted to a fork of SCons. KDE developers would have had to maintain the new build system entirely on their own. So the rosy SCons/bksys image paled again....

It was decided that CMake would be the build system for KDE 4. Beginning now, CMake will be the tool that is used to base all of KDE4 development.

During the aKademy 2005 discussions, CMake had fallen behind, mainly due to the fact that no one who was present at the meeting was overly familiar with CMake's merits. SCons/bksys had some enthusiastic supporters and contributors doing work, and they had already made large portions of kdelibs4 build with it, at least on Linux. So most build system related work went into SCons during Autumn of last year.

However, in January, Jaison Lee drew this balance on the kde-buildsystem mailing list:

"scons/bksys has been in development for months and yet it seems to be months away at best from even being serviceable. Perhaps it's time to cut our losses and run."

David Faure, one of our core developers, responded:

"Honestly, that's my feeling too. As it is right now, we're stuck: no way to work on the configuration since it's not modular, and no way to compile kdelibs since kconfig_compiler can't be run."

I had previously used CMake for several small and medium-sized projects, none of them being KDE-related. So I already had a feeling of its merits (as well as some of its shortcomings) and I knew it would be a good fit for KDE's needs. I had had email contact with the CMake developers, and knew their stance on KDE's needs. Here is how I had proposed to evaluate CMake and what their developers had already pledged to change/improve for KDE's needs:

Pros:
  -they offered to implement the things which are missing to build KDE
  -they have a fully working configure-like framework, that is easy to use
  -I managed to build most of kdelibs (KDE3)
  -cmake has no other dependencies except for a C++ compiler
  -cmake supports basically every UNIX, MS Windows (MSVC, Borland, cygwin, mingw) and Mac OS X
  -cmake can generate Makefiles and *projects* for KDevelop3, MSVC 6,7, XCode
  -cmake has a simple syntax
  -it features a testing framework
  -it supports: compiling libs, apps, KDE kparts, KDE ioslaves, KDE loadable modules, -enable-final, la-file generation
  -there is an am2cmake ruby script which does around 90 percent of the work of converting Makefile.am's to cmake files
Cons:
  -the cmake commands for installing files are not very powerful, but the developers know this may fix it to support KDE
  -not sure: relinking during installation
  -no progress percentage: this can not be done in cmake, since cmake is no build system itself, it just generates input files for the native buildsystem

When David Faure responded:

me thinks someone should start having a look at compiling kdelibs trunk with cmake :)

I immediately started to work on implementing a CMake-based build system for KDE 4. And of course, my first targets were the KDE core libraries ("kdelibs") in the KDE SVN repository. For converting the Makefile.am's to CMake I wrote a script which does most of the job, but the configure checks still had to be ported manually. Without considering myself a buildsystem-expert, only two weeks later I was lucky enough to announce the first results:

"I think now I have reached a state were I can invite interested people to try compiling kdelibs with cmake.

I compiled kdelibs successfully on Slackware 9.1 (gcc 3.2.3) and FreeBSD 5.4 (gcc 3.4.2), Bill Hoffman was able to compile it on Debian (gcc 4.x)."

It didn't take long before more developers joined the effort. Among them were OS X developers Benjamin Reed and Tanner Lovelace. Just a few days later, Benjamin was able to announce the next major breakthrough:

"With my patch to knotify/daemon, I've now successfully compiled all of kdelibs on Mac OS X."

Then the Windows developers Christian Ehrlicher and Ralph Habacker joined followed by David Faure and Laurent Montel. Today, almost all of the KDE modules have been converted to CMake, mostly in a tremendous effort by Laurent Montel. The KDE 4 libraries (kdelibs) compile completely on all supported platforms, these are: Linux, *BSD, Mac OS X and MS Windows. The CMake build works both with GCC and with the MS compilers.

SCons support has been removed from the subversion checkouts some weeks ago, and just a few days ago the support for the autotools has been removed in some KDE modules. All this would not have been possible without the full and whole-hearted support by the CMake developers. Developers Bill Hoffman and Brad King even joined our kde-buildsystem mailing list. They were actively present every day, responded to all newbie questions (yes, KDE developers are CMake newbies!) and in general helped wherever they could. They even made several intermediate "KDE special" releases of CMake just to help us in our porting efforts. And in turn, CMake benefited a lot from adapting to KDE's needs and fixing bugs that showed up when compiling this large project.

As they put it:

"In the short time we have been working with the KDE developers in the migration to CMake, we have been impressed by the talents of the KDE developers and the quality of the source code. Building one of the world's largest open source projects has been an exciting challenge for CMake, and the result has benefited both projects. We feel that since CMake can build KDE on every supported platform, it can build anything !"

Everyone who will be building KDE4 (developers, power users and distribution packagers alike) now and in the future will have to make themselves familiar with a few things. So, what is CMake actually?

CMake is an open source build system published under a BSD-style license. CMake is developed by a commercial company: Kitware Inc. Kitware specializes in open source 3D visualization, medical imaging, and quality engineering tools. Besides CMake, Kitware supports and actively develops VTK, the Visualization ToolKit, ITK, the Insight Segmentation and Registration Toolkit and ParaView; all of these are also cross platform (Linux, Unix, Windows, Mac OS X) and Open Source software.

CMake's development was supported by the National Library of Medicine (NLM), the Advanced Computing Laboratory at Los Alamos National Lab, and National Alliance for Medical Image Computing (NAMIC).

But like the chain of autotools, CMake itself doesn't do the actual building itself. The actual building is still done by the "make" utility or the respective native build tool on each platform. CMake reads script files, named CMakeLists.txt; the output it produces serves as input ("Makefiles" or e.g. project files for XCode) for the native build tool of each platform .

A run of cmake replaces what the chain of "autotools" (autoconf, configure, automake, libtool, aclocal, amedit & co.) did in previous KDE versions (and still do for most FOSS projects). Developers need to provide or check CMakelists.txt files (handwritten or generated by the respective IDE) to make sure the build process works, instead of the Makefile.am files used to feed the autotools.

CMake can generate Makefiles for all supported platforms; additionally it can generate project files for KDevelop3 (KDE Integrated Development Environment), XCode (Mac OS X IDE) and several versions of MS Visual Studio (Windows IDE). To put it in other words: with the help of CMake....
   ...on Linux you can use KDevelop3;
   ...on OS X you can use XCode and
   ...on MS Windows you can use MS Visual Studio to work on KDE.

Did I say Windows? Yes. KDE4 will bring many of its best programs and technologies to the Windows platform, now that a GPL'd Qt4 is also available there. Right now the KDE4 libraries compile under Linux, FreeBSD, Mac OS X and MS Windows.

Scribus moves too! Scribus is one of the first projects to follow KDE's lead with CMake adoption. Peter Linnell, the Scribus webmaster, who is not a hard-core C++ coder himself has this to say: "It took me about all of 10 minutes after installing cmake to be able to grok and hack on cmake files in our source code. That rocks." He sees CMake's advantages in these points: "The CmakeLists.txt files are models of clarity and only require you to add the name of a file once, where makefile.am requires three different entries for a single file." "Speed - on my quite ordinary Linux system it takes 5-7 seconds to generate all the makefiles for Scribus, compared to 30-50 seconds for auto*" "Much more useful error messages when you make a mistake in editing files. Auto* error messages have been known to drive developers to a state of madness trying to fix the errors." "Smaller makefiles. 57kb for the main Scribus makefile from cmake vs 257kb from auto* In the future, compiling Scribus with cmake is as simple as: cmake ." Adding CMake support to Scribus was done by Craig Bradney, who is not a full time developer. Craig said: "It took a really short time to convert the Scribus build system. [...] It took me about a week from CMake newbie status to at least generally understanding how it works and having Scribus building. Running the very latest CVS was a major help as the KDE guys have switched and enhancements they required helped me out. Once CMake 2.4.2 flows into the distros out there, we will more than likely dump autotools completely."

Getting it to work under Windows was the hardest part. KDE pushes the limits on this platform. On Windows, you can compile KDE with MinGW, MSYS and MS Visual C++ >=7 (MSVC) -- and all three of them work differently! These were the main areas where CMake had to be changed to support KDE4's needs.

So, what does a CMakeLists.txt file look like?

Here's the most simple one. It will create an executable named "helloworld" (or helloworld.exe on Windows) from the source file main.cpp:

  add_executable(helloworld main.cpp)

Put this file into your source directory and run cmake on it:

  cmake <path_to_the_source_dir>

This will create a Makefile in the directory you ran cmake from (the "builddir"). That step works the same on each of the previously mentioned platforms, and you can compile the program with all of the before mentioned compilers available there. In general, both in-source as well as out-of-source builds are supported by CMake. However, building out-of-source is preferred and for building kdelibs it is currently even enforced (i.e. "builddir != sourcedir"); the minimum CMake version to be used for KDE4 is 2.4.2.

Running cmake as seen above is equivalent to running automake and autoconf and configure when using autotools.

Any developer who starts a project needs to make sure that their sources will build. Most developers find it painful to setup their build system when they use the current de facto standard, the "autotools". Once (and as long as) the autotool chain works, it is easy to handle. Most power users who occasionally build applications from sources see only a very small part of its components. But getting there is a big effort....

Running "./configure; make; make install" seems easy enough, right? But the magic behind making those commands tick the right way is extremely difficult to master. Any developer, even of a relatively simple project, who has to setup the build system on their own will likely confirm this. The developer has to use (and learn) syntaxes for the combo of automake + autoconf + libtool + make + sh + perl + m4. A measure of the autotool complication is indicated by the fact that source code tarballs of only a few hundred KBytes of size may require at least 0.5 MByte of space for the build scripts and autotool helper files.

In KDE3 development, we ended up with an autotools-based build system that most of our developers did not fully understand. I estimate that no more than 10 people were able to fix any major problem occurring with it, first and foremost our "buildsystem-guru" Stephan Kulow. Over the years, these guys must have been contacted by hundreds of co-developers with requests for help to fix build problems in specific applications and modules.

So, among KDE developers, and probably the developers of other autotools-based projects, there is the opinion that dealing with the build system is hard, almost higher magic. This is a myth. A myth created by autotools. CMake is able to destroy this myth. Yes, there is a new syntax for the CMakeLists.txt files, but it is easy to learn.

At first CMake syntax is a bit unusual for the average UNIX hacker with lots of CAPITALIZATION, CamelCasedWords and quite verbose "IF(SOME_SPECIFIC_CONDITION) ...<do_something>... ENDIF(SOME_SPECIFIC_CONDITION)" statements. But a good editor with auto-completion makes it a breeze to handle, and the repetition of the conditions in the ENDIF line makes the meaning very unambiguous.

It is my hope that in a few months time, we will see all experienced developers become familiar enough with CMake to enable them to maintain the build systems of their respective modules and applications on their own. The goal is to have 100 or more developers who can fix more difficult problems that may occur with CMake.

Changing to CMake gives us some additional benefits for free:

we can utilize the built-in support for continuous and/or nightly builds via Dart. Dart is a software quality assurance system also developed by Kitware Inc.

Kitware is currently running a so-called dashboard for the KDE libraries. This dashboard collects build results from all participating machines and platforms and displays them on a website.

To let your own machine feed its build results into the dashboard, just use "make Experimental" or "make Nightly" as your make command. The next hourly update of the dashboard will show your machine name with detailed build results in the table.

Dart and the Dashboard will become a very useful tools to ensure that during development, KDE4 always compiles on all supported platforms (or that any build problem will be fixed in no time), and that the unit tests keep run successfully. (Yes, there is an accelerating addition of built-in unit tests written by a rising number of KDE application developers; this makes us feel rather optimistic about KDE4's stability once it is ready for release). Just make sure that you let one compile machine of yours build "Experimental" and/or "Nightly" targets (especially if it sports one of the more uncommon platforms supported by KDE).

To find out more about CMake, your first source of information is http://www.cmake.org/. Go through the introduction, the tutorial, and don't miss out the CMake wiki; they all contain a lot of useful information. There is also a special "entry point" to cmake for KDE developers: KDECMakeIntro.

If there are any other project developers who are interested in switching their build system , they surely will get advice, help and support from us, as well as from the CMake developers. Just ask.

P.S.: Let me say one last word about the current state of KDE4: it compiles, and it runs -- but it crashes very often right now. It does not yet show off many features beyond what KDE3 offers. The compiled kdelibs4 and kdebase4 at this point in time look very much like KDE 3.5 (but made completely unstable). The main difference is that under its hood now ticks Qt4 instead of Qt3. This is the required foundation that now lets us go forward with the "real" development. Right now, KDE4's code base gets added in code modifications, code refactorizations and new code. But none of the cool new core technologies: (solid, plasma, phonon, oxygen, akonadi, decibel and more) are in a usable state. KDE4 at this moment is completely uninteresting to end-users, they should wait at least for the first technology preview tentatively planned for late autumn of this year. If you are a developer, things are different: come and help us write the code. The fun starts now! :-)

Index entries for this article
GuestArticles	Neundorf, Alexander

---

(Log in to post comments)