Building

The building again involves builder.rb. This time more parameters are needed:

    ruby builder.rb -u -n -s 1,2,bootconf \
        -p config/pkglist_neutral_rescue.txt \
        -c config/general_neutral_rescue.xml \
        -b config/branding_neutral_rescue.xml \
        -k config/kernels_rescue_unstable.xml > ../../llbuild/stage03.log 2>&1

The parameters are as follows:

• -u: build unstable, must match stage02
• -s: skip stages 1 and 2 and skip the generation of boot configuration files
• -p: use this package list
• -c: use this configuration file (specifies paths like the overlay)
• -b: use this branding
• -k: use the kernels noted here

You might also use --full to build an ISO containing all files from the chroot environment – this is self containing and can be used for building. Or use --skip-files to specify a filename containing a list of files to omit. After 30 to 40 minutes you’ll find the new ISO in /mnt/archiv/LessLinux/llbuild/stage03.

My next post will tell how to customize kernels and branding.

• LessLinux now supports tracking of dependencies after building packages. This allows to define the order of packages to build.

• As a consequence of the dependency tracking it is now also possible to parallelize the build. On a three core Athlon this brought down build times from around three days to less than 20 hours. With a faster machine you should even get better results.

Prerequesites

I assume that you already built stage01 on the same machine. If not, just create the directories as we did in the last article and mount the hard disk or partition containing sources and the stage01 chroot. The unprivileged user is not required this time.

Building

I prefer redirecting the console messages to a log file (and “tailing” it in another terminal). The build and install of the respective pacakges are separately logged to an package specific log file. I will later show how to analyze and them. So change to the build directory and run:

        ruby -I. builder.rb -s 1,3 -n -u -l -t 7 > /tmp/stage02.log 2>&1

This command uses six threads maximum. It really depends on your machine how many is ideal. As a rule of thumb, take twice the number of processor cores plus one. Less than three do not make sense, and above ten too many seeks may appear. Anyway: If you have a pretty fast box with a six core processor and you are maybe building on a SSD: try 15 or 20 threads and tell me how it compares to just ten.

Trouble

You will occasionally run into a problem where no modified files were found. In this case the builder exits. To find out which package is affected just grep for “no modified” in the log file:

        grep -A2 -i 'no modified' /tmp/stage02.log

This is most likely due to some race condition: The build or install script is not yet closed, but the builder already tries to run it. Take a look at the files /mnt/archiv/LessLinux/llbuild/stage02/build/packagename-version.build.log and packagename-version.install.log. If it reads bad interpreter … text file busy this is the problem. Sometimes the maximum number of open files is the problem, so you should compare the current number of open files with the maximum number during the threaded build.

To re-build the affected package delete the package and the file /tmp/LessLinux_Emergency_Exit and run the builder again:

        rm -rf /mnt/archiv/LessLinux/llbuild/stage02/build/packagename-version* /tmp/LessLinux_Emergency_Exit
        ruby -I. builder.rb -s 1,3 -n -u -l -t 7 >> /tmp/stage02.log 2>&1

When restarting the build all packages already built are ignored and the builder picks up where it left.

The next post will show how to assemble a working live system from the freshly created stage02.

• We switched from MySQL to SQLite3 as database during build

• LessLinux is self containing now, this means you can use LessLinux to build LessLinux to build LessLinux…

The stage01 build of LessLinux very closely resembles the chapter Chapter 5 – Constructing a Temporary System from Linux from Scratch. You’ll even notice similar environment variables. There will always be slight differences in package versions – this is intended. So you might be able to build LessLinux on any machine equipped with the right programs to build Linux from Scratch. But in my opinion do one of the two following things:

• Setup a clean Ubuntu 10.04 LTS and install all necessary packages by running
  apt-get install ` wget -O - http://download.lesslinux.org/builder/packagelists/20110202_ubuntu_1004_packages.txt `

• Download the latest FULL ISO, a build of LessLinux containing compilers, headers and stuff. FULL ISO images are currently around 1.5GB and they allow you to build LessLinux without having to permanently setup a build environment. As of this post being written the current FULL ISO is lesslinux-search-and-rescue-uluru-20110202-155012-FULL-en.iso

I strongly recommend to take a dedicated build box – with the FULL ISO and an USB hard drive the conversion will be temporary. This can be some fairly recent PC (2GHz, 1GB RAM) and a USB hard drive (30GB upwards – a 32GB thumb drive might also do the job). In the mid term we will concentrate on the self containing builds, so it’s best to get used to building with FULL ISO. The FULL ISO might also be a good starting point for anyone diving into Linux from Scratch since the versions of GCC and Glibc are usually close to those in LFS.

Directories and users

You have great freedom on directory layout and the unprivileged user that is used for building some packages, but for start you should stick with the defaults in LessLinux.

• I use /mnt/archiv/LessLinux as base for building. This path is used in the default config, so I suggest to keep it by mounting a dedicated build partition to /mnt/archiv/LessLinux:

  mkdir -p /mnt/archiv/LessLinux
  mkfs.ext3 /dev/sdx1
  mount /dev/sdx1 /mnt/archiv/LessLinux
  mkdir /mnt/archiv/LessLinux/llbuild
  mkdir /mnt/archiv/LessLinux/llbuilder
  mkdir /mnt/archiv/LessLinux/src

• Add an unprivileged user for building. He must have /bin/bash as a shell, otherwise the build will fail. The default scripts use mattias as unprivileged user:

  adduser -s /bin/bash mattias

Downloading and Building

You’ll need some buildscripts from http://download.lesslinux.org/src/. Take either the most recent ones or some that match a certain version you want to build. As I write this, lesslinux-search-and-rescue-uluru-20110202-155012-buildscripts.tbz is recommended.

• Go to the directory /mnt/archiv/LessLinux/llbuilder where you unpack the tarball:

  tar xvjf *-buildscripts.tbz

• Edit the file lesslinux-builder/config/general.xml and change the node unpriv to contain the username of the unprivileged user created above

• Change to the newly created directory lesslinux-builder, from there run:

  ruby -I. builder.rb -s 2,3 -n -u

This tells the builder to do an unstable (-u) build (glibc 2.12 and kernel 2.6.37 instead of glibc 2.11 and kernel 2.6.36). In a view days, I’ll move it to stable and take 2.6.38-rc and glibc 2.13 as unstable. The -n tells to omit tests. When making changes to stage01, you would probably omit this to run tests to discover problems. Finally -s makes the builder skipping stages 2 and 3.

Cleaning up

The build will now take somewhere between two and eight hours, strongly depending on how fast your machine is. After the build, the freshly created chroot environment and the database file containing will be backed up so that you can start any stage02 build without having to re-build stage01. Thus you should delete the build directory for stage01:

rm -rf /mnt/archiv/LessLinux/llbuild/stage01/build

And the next step…

Tomorrow I’ll describe how to build stage02 – which means building all packages in a chroot environment – and finally assembling an ISO image from it.

mmmm is built around the XML output of lshw and uses just Ruby and Ruby/Gtk. No DBUS, HAL or Policy Kit required which makes it attractive for builds that try to be as light as possible.

The interface of mmmm is clearly inspired by GtkMount of the Pmagic live distribution, but using the XML output of lshw is a bit more precise than just parsing “fdisk -l”.

TODO: mmmm is very usable right now, but for the future a button to reread devices would be nice.

Please test: Please test the latest build (lesslinux-20090604-1326) and tell me if mmmm correctly finds all your volumes. You might also send me your hardware information, so I can debug possible problems more easily.

Update: A small bug in mmmm prevents extended partitions from being correctly detected. A fixed build lesslinux-20090608-1708 is available.

Still there are some drawbacks when using Glibc, especially the dependence on Bash as shell and the relative big size. Now there seems to be a solution for this problem. Eglibc is a fork of Glibc that aims to be source and binary compatible with Glibc but removes the dependency to Bash as sole shell and will allow for lighter configuration. In the medium term this seems like a perfect solution: LessLinux does not need NIS and for some applications even localization might be skipped.

In the next weeks I will not have the time to play around with Eglibc, but as soon as the release of Glibc 2.10 is near, I will give Eglibc a try.

• Aurélienn Jarno zum Import der EGLIBC in das Debian-Paket-Repository
• OS News “Debian Switching to EGLIBC”
• Golem: “Debian wechselt zur Eglibc”