Building

The building again involves builder.rb. This time more parameters are needed:

    ruby builder.rb -u -n -s 1,2,bootconf \
        -p config/pkglist_neutral_rescue.txt \
        -c config/general_neutral_rescue.xml \
        -b config/branding_neutral_rescue.xml \
        -k config/kernels_rescue_unstable.xml > ../../llbuild/stage03.log 2>&1

The parameters are as follows:

• -u: build unstable, must match stage02
• -s: skip stages 1 and 2 and skip the generation of boot configuration files
• -p: use this package list
• -c: use this configuration file (specifies paths like the overlay)
• -b: use this branding
• -k: use the kernels noted here

You might also use --full to build an ISO containing all files from the chroot environment – this is self containing and can be used for building. Or use --skip-files to specify a filename containing a list of files to omit. After 30 to 40 minutes you’ll find the new ISO in /mnt/archiv/LessLinux/llbuild/stage03.

My next post will tell how to customize kernels and branding.

• LessLinux now supports tracking of dependencies after building packages. This allows to define the order of packages to build.

• As a consequence of the dependency tracking it is now also possible to parallelize the build. On a three core Athlon this brought down build times from around three days to less than 20 hours. With a faster machine you should even get better results.

Prerequesites

I assume that you already built stage01 on the same machine. If not, just create the directories as we did in the last article and mount the hard disk or partition containing sources and the stage01 chroot. The unprivileged user is not required this time.

Building

I prefer redirecting the console messages to a log file (and “tailing” it in another terminal). The build and install of the respective pacakges are separately logged to an package specific log file. I will later show how to analyze and them. So change to the build directory and run:

        ruby -I. builder.rb -s 1,3 -n -u -l -t 7 > /tmp/stage02.log 2>&1

This command uses six threads maximum. It really depends on your machine how many is ideal. As a rule of thumb, take twice the number of processor cores plus one. Less than three do not make sense, and above ten too many seeks may appear. Anyway: If you have a pretty fast box with a six core processor and you are maybe building on a SSD: try 15 or 20 threads and tell me how it compares to just ten.

Trouble

You will occasionally run into a problem where no modified files were found. In this case the builder exits. To find out which package is affected just grep for “no modified” in the log file:

        grep -A2 -i 'no modified' /tmp/stage02.log

This is most likely due to some race condition: The build or install script is not yet closed, but the builder already tries to run it. Take a look at the files /mnt/archiv/LessLinux/llbuild/stage02/build/packagename-version.build.log and packagename-version.install.log. If it reads bad interpreter … text file busy this is the problem. Sometimes the maximum number of open files is the problem, so you should compare the current number of open files with the maximum number during the threaded build.

To re-build the affected package delete the package and the file /tmp/LessLinux_Emergency_Exit and run the builder again:

        rm -rf /mnt/archiv/LessLinux/llbuild/stage02/build/packagename-version* /tmp/LessLinux_Emergency_Exit
        ruby -I. builder.rb -s 1,3 -n -u -l -t 7 >> /tmp/stage02.log 2>&1

When restarting the build all packages already built are ignored and the builder picks up where it left.

The next post will show how to assemble a working live system from the freshly created stage02.

• We switched from MySQL to SQLite3 as database during build

• LessLinux is self containing now, this means you can use LessLinux to build LessLinux to build LessLinux…

The stage01 build of LessLinux very closely resembles the chapter Chapter 5 – Constructing a Temporary System from Linux from Scratch. You’ll even notice similar environment variables. There will always be slight differences in package versions – this is intended. So you might be able to build LessLinux on any machine equipped with the right programs to build Linux from Scratch. But in my opinion do one of the two following things:

• Setup a clean Ubuntu 10.04 LTS and install all necessary packages by running
  apt-get install ` wget -O - http://download.lesslinux.org/builder/packagelists/20110202_ubuntu_1004_packages.txt `

• Download the latest FULL ISO, a build of LessLinux containing compilers, headers and stuff. FULL ISO images are currently around 1.5GB and they allow you to build LessLinux without having to permanently setup a build environment. As of this post being written the current FULL ISO is lesslinux-search-and-rescue-uluru-20110202-155012-FULL-en.iso

I strongly recommend to take a dedicated build box – with the FULL ISO and an USB hard drive the conversion will be temporary. This can be some fairly recent PC (2GHz, 1GB RAM) and a USB hard drive (30GB upwards – a 32GB thumb drive might also do the job). In the mid term we will concentrate on the self containing builds, so it’s best to get used to building with FULL ISO. The FULL ISO might also be a good starting point for anyone diving into Linux from Scratch since the versions of GCC and Glibc are usually close to those in LFS.

Directories and users

You have great freedom on directory layout and the unprivileged user that is used for building some packages, but for start you should stick with the defaults in LessLinux.

• I use /mnt/archiv/LessLinux as base for building. This path is used in the default config, so I suggest to keep it by mounting a dedicated build partition to /mnt/archiv/LessLinux:

  mkdir -p /mnt/archiv/LessLinux
  mkfs.ext3 /dev/sdx1
  mount /dev/sdx1 /mnt/archiv/LessLinux
  mkdir /mnt/archiv/LessLinux/llbuild
  mkdir /mnt/archiv/LessLinux/llbuilder
  mkdir /mnt/archiv/LessLinux/src

• Add an unprivileged user for building. He must have /bin/bash as a shell, otherwise the build will fail. The default scripts use mattias as unprivileged user:

  adduser -s /bin/bash mattias

Downloading and Building

You’ll need some buildscripts from http://download.lesslinux.org/src/. Take either the most recent ones or some that match a certain version you want to build. As I write this, lesslinux-search-and-rescue-uluru-20110202-155012-buildscripts.tbz is recommended.

• Go to the directory /mnt/archiv/LessLinux/llbuilder where you unpack the tarball:

  tar xvjf *-buildscripts.tbz

• Edit the file lesslinux-builder/config/general.xml and change the node unpriv to contain the username of the unprivileged user created above

• Change to the newly created directory lesslinux-builder, from there run:

  ruby -I. builder.rb -s 2,3 -n -u

This tells the builder to do an unstable (-u) build (glibc 2.12 and kernel 2.6.37 instead of glibc 2.11 and kernel 2.6.36). In a view days, I’ll move it to stable and take 2.6.38-rc and glibc 2.13 as unstable. The -n tells to omit tests. When making changes to stage01, you would probably omit this to run tests to discover problems. Finally -s makes the builder skipping stages 2 and 3.

Cleaning up

The build will now take somewhere between two and eight hours, strongly depending on how fast your machine is. After the build, the freshly created chroot environment and the database file containing will be backed up so that you can start any stage02 build without having to re-build stage01. Thus you should delete the build directory for stage01:

rm -rf /mnt/archiv/LessLinux/llbuild/stage01/build

And the next step…

Tomorrow I’ll describe how to build stage02 – which means building all packages in a chroot environment – and finally assembling an ISO image from it.

Well it can be easier. Just configure your DSL router to forward inbound traffic on port 5500 to your desktop machine and get a nice DynDNS hostname. Then remaster any LessLinux Search and Rescue CD to include one boot entry that does not disable earlynet and add the cheatcode

        xvnc=|reverse|1280x800|24|my.dyndns.name|

In case of emergency tell your grandma to boot the CD with the respective entry. The earlynet script requests an IP address on all wired interfaces and instead of starting a local Xserver Xvnc is started and a reverse connection to your host is made where a listening VNC viewer must be running.

There is one drawback: the data is transmitted unencrypted, which might be an issue with sensitive data. You might however use this first VNC connection to build up a SSH tunnel between two hosts.

The background of this feature: I got some customers quite far away from here. With this cheatcode I can help them no matter where they are – no special configuration of their router required. This feature also will work over WLAN soon. I just introduced some cheatcodes to attach to a wireless network while startup. Those are not tested yet, so please be patient for a few more days.

Should work in all builds higher than 20100701-000000.

        nfs=192.168.1.1:/dir/containing/iso

In this case, the NFS share is mounted read only (and no locking takes place), then all ISO files down to depth 2 are searched, loop back mounted and scanned for the proper version.txt. If the system is found, the settings for toram=value take place. In reasonable fast networks you should keep this threshold low, then the start of a program via network is quick, on networks with bad latencies you might set a low threshold to copy the entire system to RAM during startup.

You might download the ISO and the sources here:

• lesslinux-search-and-rescue-uluru-20100726-105414.iso
• lesslinux-search-and-rescue-uluru-20100726-105414-buildscripts.tar.bz2

To build yourself follow the instructions in those two articles: Build stage 01 and 02 and Build stage03.

• lesslinux-search-and-rescue-uluru-20100528-114807.iso

Currently for netbooting a DHCP and PXE server are required, a HTTP daemon should be present, although it is possible to load the ISO directly from download.lesslinux.org. For now the ISO is completely stored in volatile memory — future versions will probably also allow NFS access. To load an ISO via HTTP just one additional parameter is necessary:

     wgetiso=http://host/path/lesslinux.iso

Of course you can use the download link above for your first tests, but please remember to change it to a local HTTP server as soon as your setup works to keep traffic at an acceptable level. Now you may combine the information from the last post with a PXE server to build a netbooting, remote accessible version of “LessLinux Search and Rescue”. Take a look at the documentation of PXELINUX and you determine when which system boots via network.

1. Setup a PXE boot environment

German speaking readers might head here, where I describe the setup of a DHCP-PXE-TFTP-boot-chain. Build the setup and test with a simple OS like memtest to confirm that it works flawlessly.

2. Copy the boot files from the LessLinux CD

Our example expects the contents of /boot/isolinux in /llsar within the TFTP root directory, so just use rsync or cp to copy all files contained in this directory.

3. Setup the boot loader

Use isolinux.cfg as a template for your entry in pxelinux.cfg/default. Make sure that earlynet is not contained in the services to skip (upcoming releses will skip this service since it waits for about 15 seconds for DHCP configuration). Then decide which options you want to add. I ended up with three entries. One for a local desktop, two to encrypted and unencrypted VNC access:

LABEL llsar
MENU LABEL LessLinux Search and Rescue LOCAL
KERNEL /llsar/l2634vn
APPEND initrd=/llsar/devs.img,/llsar/initram.img,/llsar/i2634vn.img ramdisk_size=100000 vga=788
    ultraquiet=1 security=none skipcheck=1 quiet lang=de ejectonumass=1
    skipservices=|installer|xconfgui|firewall|mountdrives| hwid=unknown laxsudo=1
    wgetiso=http://192.168.1.56/ll.iso
TEXT HELP
  Start LessLinux Search and Rescue with local desktop.
ENDTEXT

LABEL llsarvnc
MENU LABEL LessLinux Search and Rescue VNC
KERNEL /llsar/l2634vn
APPEND initrd=/llsar/devs.img,/llsar/initram.img,/llsar/i2634vn.img ramdisk_size=100000 vga=788
    ultraquiet=1 security=none skipcheck=1 quiet lang=de ejectonumass=1
    skipservices=|installer|xconfgui|firewall|mountdrives|runtimeconf| hwid=unknown laxsudo=1
    wgetiso=http://192.168.1.56/ll.iso xvnc=|remote|1280x800|24|
TEXT HELP
  Start LessLinux Search and Rescue with insecure VNC desktop.
ENDTEXT

LABEL llsarvnc
MENU LABEL LessLinux Search and Rescue SSH
KERNEL /llsar/l2634vn
APPEND initrd=/llsar/devs.img,/llsar/initram.img,/llsar/i2634vn.img ramdisk_size=100000 vga=788
    ultraquiet=1 security=none skipcheck=1 quiet lang=de ejectonumass=1
    skipservices=|installer|xconfgui|firewall|mountdrives|runtimeconf| hwid=unknown laxsudo=1
    wgetiso=http://192.168.1.56/ll.iso xvnc=|local|1280x800|24|
    rootpwhash=$1$ILXt/Dc3$DmYp.51WCDNjGuTL90eju/
TEXT HELP
  Start LessLinux Search and Rescue with VNC-over-SSH desktop.
  (SSH password is "test", please change!)
ENDTEXT

Please note that the indented lines after APPEND are part of the append parameter line. I had to break the lines to prevent the formatting from exploding. With this setup you can try the full netboot of “LessLinux Search and Rescue”.

• Booting the kernel and initramfs with PXE and loading the system ISO with wget by http or ftp
• Starting the desktop with Xvnc instead a local X server

Currently both functions just work with a wired network interface. This post covers VNC access, I will write on PXE booting later.

To test VNC access, download the latest build:

• lesslinux-search-and-rescue-uluru-20100527-082411.iso

…or build it yourself according to the last two articles withe those buildscripts and overlays (in this case the extra switch -u for unstable is necessary):

• lesslinux-search-and-rescue-uluru-20100527-082411-buildscripts.tar.bz2
• lesslinux-search-and-rescue-uluru-20100527-082411-overlays-neutral.tar.bz2

Doing it the easy way

In trusted networks, you may start Xvnc on the external interface, this means VNC access is granted without password. Just press the Tab key in the boot menu and add

        xvnc=|remote|1280x800|24|

The first value tells Xvnc to bind to the external interface, the second is the screen resolution to use (any value is accepted here), and the third parameter is the color depth. On congested networks you might reduce the depth to 16. It also makes sense to add runtimeconf to the list skipservices. When finished booting, you can connect on port 5900.

Doing it the secure way

Specifying

        xvnc=|local|1280x800|24|

tells Xvnc to bind to 127.0.0.1. So no external connections are allowed. To access, you must tunnel port 5900 over SSH. To make this possible, first remove ssh and roothash from the list skipservices, then add a roothash by specifying:

        rootpwhash=$1$ILXt/Dc3$DmYp.51WCDNjGuTL90eju/

This is the MD5 hash for the simple password test. I created this Hash with the command openssl passwd -1, Apaches htpasswd might also work. Then after the system is fully booted, you can first connect with SSH and use -L to forward :

        ssh -L 5900:localhost:5900 root@192.168.1.23

Now VNC is available on port 5900 on you local machine, simply connect this way:

        vncviewer localhost:0

The current build also is the first one to use kernel 2.6.34 and it introduces libraries from Gnome 2.30. So please test and give me feedback!

Start with the overlay that has to be unpacked directly in /mnt/archiv/LessLinux:

    cd /mnt/archiv/LessLinux
    wget http://download.lesslinux.org/overlays/lesslinux-search-and-rescue-uluru-20100507-000000-overlays-neutral.tar.bz2
    tar xvjf lesslinux-search-and-rescue-uluru-20100507-000000-overlays-neutral.tar.bz2

Now change directory to /mnt/archiv/LessLinux/llbuilder/lesslinux-builder and run builder.rb:

    cd /mnt/archiv/LessLinux/llbuilder/lesslinux-builder
    ruby builder.rb -n -s 1,2,bootconf \
        -p config/pkglist_neutral_rescue.txt \
        -c config/general_neutral_rescue.xml \
        -b config/branding_neutral_rescue.xml \
        -k config/kernels_rescue.xml

The parameters are as follows:

• -p use a certain pkg_list, this is a simple text file containing one package per line
• -c use a certain config file, this contains paths and assumptions on the layout of the final system (for example how kernel modules are loaded)
• -b use a certain branding, this would be the first file to customize when building your own version of LessLinux
• -k use a certain kernel configuration, you would modify this file when using different kernels

After the build is done, take a look at the stage03 directory, you will find the finished ISO image here:

    # ls -lah /mnt/archiv/LessLinux/llbuild/stage03
    total 391M
    drwxr-xr-x   7 root root 4.0K 2010-05-07 09:33 .
    drwxr-xr-x   5 root root 4.0K 2010-05-07 08:45 ..
    drwxr-xr-x 300 root root  12K 2010-05-07 09:13 build
    drwxr-xr-x   6 root root 4.0K 2010-04-13 12:50 cdmaster
    drwxr-xr-x   4 root root 4.0K 2010-05-07 09:32 cpio-2633vn
    -rw-r--r--   1 root root  18M 2010-05-07 09:32 cpio-2633vn.gz
    drwxr-xr-x  14 root root 4.0K 2009-10-13 17:25 initramfs
    -rw-r--r--   1 root root 7.6M 2010-05-07 09:32 initramfs.gz
    -rw-r--r--   1 root root 365M 2010-05-07 09:33 lesslinux-search-and-rescue-uluru-20100507-084527.iso
    drwxr-xr-x   9 root root 4.0K 2010-05-07 09:32 squash

You might now dd the ISO to an USB stick to create a bootable device without having to burn:

    dd if=lesslinux-search-and-rescue-uluru-timestamp.iso of=/dev/sdx

That’s it for today, in the next days we will add some more software and change the branding of your own build.

Currently the only supported build environment is VMware, this means you must have either VMware Workstation, VMware Player (Win, Linux) or VMware Fusion (Mac OS X) installed. Player is sufficient, since the build environments are considered as some kind of “disposable environment”. Note that the VMware image expands to up to 80GB, to prevent fragmentation and reach best performance you might want to put it on a separate partition.

You might download the VMware image from: http://download.lesslinux.org/builder/vmware-images/. This howto uses the file LessLinux_Builder_20100504_Ubuntu_910.zip. Unpack it and run it by calling ubuntu.vmx. As soon as the virtualized Ubuntu is fully started up, you can log in as root with an empty password. Set the password with the command passwd (note that at this point an english keyboard is active – for german readers this means “z” and “y” are swapped!). Log out and log in again using ssh.

Building stage01

The build of stage01 closely ressembles this chapter of the Linux From Scratch handbook. We basically build a set of tools under /tools that are later used to build the system itself in the second stage. Since just glibc, the toolchain and some of the basic Linux utilities are build, this step takes usually one and a half to four hours.

When logged in, change your directory to /mnt/archiv/LessLinux, there create three new subdirectories:

    cd /mnt/archiv/LessLinux
    mkdir llbuild llbuilder src

Change to llbuilder and download and unpack the source/script tarball of the release you intend to build:

    cd /mnt/archiv/LessLinux/llbuilder
    wget http://download.lesslinux.org/src/lesslinux-search-and-rescue-uluru-20100507-000000-buildscripts.tar.bz2
    tar xvjf lesslinux-search-and-rescue-uluru-20100507-000000-buildscripts.tar.bz2

This results in a new directory lesslinux-builder, change to this directory and start building of the first stage:

    cd /mnt/archiv/LessLinux/llbuilder/lesslinux-builder
    ruby builder.rb -n -s 2,3

This command tells the script builder.rb to run no tests and to skip the second stage (build of software within the chroot) as well as the third stage (assembly of the live system). When finished a backup of the resulting chroot environment as well as of the database at this point will be made — we will refer to those later.

Building stage02

In stage02 all the software defined in the XML build definitions in /mnt/archiv/LessLinux/llbuilder/lesslinux-builder/scripts/stage02 will be built. Since two kernels, Firefox, Thunderbird and lots of “expensive” C++-stuff are built you should calculate between one and four days for this build:

    cd /mnt/archiv/LessLinux/llbuilder/lesslinux-builder
    ruby builder.rb -n -s 1,3

It is also possible to specify breakpoints. This means to interrupt the build procedure after a certain package is being built. This feature is usually intended to build package by package after large changes to the package versions or build parameters. For example to stop right after xz (before building the kernels):

    ruby builder.rb -n -s 1,3 -d xz

You might want to take a look at the XML files located in scripts/stage02 to determine where to take breaks.

That’s it. As soon as you (and my stage02) build is complete, I will tell you how to assemble an actual working live system in stage03!

Notes

• If a FTP download hangs, you might interrupt it by pressing Ctrl+C, then the download should try again via HTTP
• If the build cannot start due to missing files you might either remove the stage02 build definition that uses the missing file or you manually download the file
• VMware will suspend if you work on a nearly full file system, to prevent a dropped SSH connection from interrupting a build, use screen
• If you started with lesslinux-search-and-rescue-uluru-20100413-192604-buildscripts.tar.bz2 the build fails at gnutls, in this case unpack lesslinux-search-and-rescue-uluru-20100507-000000-buildscripts.tar.bz2 in /mnt/archiv/LessLinux/llbuilder (this will replace scripts/stage02/5282_gnutls.xml and scripts/pkg_content/gnutls-2.8.6.xml), then run again builder.rb, the build will now continue and finish.

I would be very happy if you sent me as many hardware protocols as possible. Please note: hardware protocols might contain data that allows to uniquely identify a single machine. In environments where this is a problem you might open the archive in hwinfo and just send me the files you wish.

To edit the boot line, just press Tab:

You might also edit the boot command line to enable faster boot. To skip loading all containers to RAM you might set the value toram=999999999. This defines an insanely high minimum RAM. You might also want to skip the hardware protocol in some cases. Do this by adding hwproto to skipservices.

Please send hardware protocols to hardware@lesslinux.org