Section 5.1 Introduction
Contains a short introduction about creating the temporary system with needed tools installed under $LFS/tools/. This tool chain will be used to build the final version of the basic LFS system. There is no steps or instructions needed for this section.Section 5.2 Toolchain Technical Notes
This section contains useful information about the temporary toolchain that is about to be built. The technical details has some notes about cross-compiling and gives a brief overview of the upcoming steps. There is no steps or instructions needed for this section, feel free to read the LFS book for additional info.Section 5.3 General Compilation Instructions
This section contains generic information about build steps, such as:- Ensure $LFS is set
- Sources directory is available under the $LFS location and is NOT under the tools directory
- When building a package:
- Change to the sources directory.
- Untar the given package.
- Change to the unpacked directory.
- Use the book instructions for compiling and installing the given package.
- When done change back to the sources directory, and remove the unpacked package, unless otherwise instructed.
Section 5.4 Binutils-2.33 Pass 1
Package Information:
The binutils package contains a linker, an assembler, and other tools for handling object files.Approximate build time: 1 SBU
Required disk space: 659 MB
Build Prep and Timing
Recommends wrapping all the commands in a time { ... } to get a feel for how long '1 SBU' takes. For details on what all the options in the following commands mean, I recommend just reading the LFS book. My prep output is as follows:gentoo@livecd ~ $ echo $LFS /rootPool/root_fsgentoo@livecd~ $env TERM=xterm MAKEFLAGS=-j44 LC_ALL=POSIX LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: LFS=/rootPool/root_fs PATH=/tools/bin:/sbin:/bin:/usr/bin PWD=/home/gentoo LFS_TGT=x86_64-lfs-linux-gnu PS1=\[\033]0;\u@\h:\w\007\]\[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\] SHLVL=1 HOME=/home/gentoo _=/bin/envgentoo@livecd~ $cd $LFS/sources/binutilsgentoo@livecd/rootPool/root_fs/sources/binutils $ cd binutils-gdb/gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $ git status HEAD detached at binutils-2_33_1 nothing to commit, working directory cleangentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $/sbin/zfs snapshot rootPool/root_fs/sources/binutils@lfs_prep_section_5.4_pass1gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $
Configure/Build/Install
So I could easily time and recreate the commands need to build this pass, I created the following script:gentoo@livecd /rootPool/root_fs/sources/binutils $ cat build_binutils_temp_pass1.sh #!/bin/bash set -x # Change into the repository cd binutils-gdb # Create a build location, and change to it. mkdir -v build cd build # Run the configuration ../configure --prefix=/tools \ --with-sysroot=$LFS \ --with-lib-path=/tools/lib \ --target=$LFS_TGT \ --disable-nls \ --disable-werror # Build bin-utils make # create reequired install locations for x64 as needed case $(uname -m) in x86_64) mkdir -v /tools/lib && ln -sv lib /tools/lib64 ;; esac # Install the bin-utils make installgentoo@livecd /rootPool/root_fs/sources/binutils $
When running the script I received the following error(s):
mkdir: cannot create directory '/tools/lib': No such file or directory
+ make install
make[1]: Entering directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build'
/bin/sh ../mkinstalldirs /tools /tools
make[1]: Nothing to be done for 'install-target'.
mkdir -p -- /tools /tools
mkdir: cannot create directory '/tools': Permission denied
mkdir: cannot create directory '/tools': Permission denied
make[1]: *** [Makefile:2505: installdirs] Error 1
make[1]: Leaving directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build'
make: *** [Makefile:2258: install] Error 2
What I forgot about is that in section 4,
livecd gentoo # ln -sv $LFS/tools /
'/tools' -> '/rootPool/root_fs//tools'
gentoo@livecd ~ $ ls -l /
total 1
drwxr-xr-x 2 root root 1660 Jul 3 2016 bin
drwxr-xr-x 2 root root 28 Jul 3 2016 boot
drwxr-xr-x 18 root root 5700 Jan 13 01:13 dev
drwxr-xr-x 127 root root 420 Jan 13 01:19 etc
drwxr-xr-x 4 gentoo users 5 May 28 2019 gentooScratch
drwxr-xr-x 4 root root 60 Jan 13 01:13 home
lrwxrwxrwx 1 root root 5 Jun 23 2016 lib -> lib64
drwxr-xr-x 2 root root 1070 Jun 23 2016 lib32
drwxr-xr-x 18 root root 60 Jan 13 01:13 lib64
drwxr-xr-x 2 root root 28 Oct 28 2015 media
drwxr-xr-x 6 root root 120 Jan 13 01:12 mnt
drwxr-xr-x 2 root root 40 Jan 13 01:12 newroot
drwxr-xr-x 5 root root 64 Oct 28 2015 opt
dr-xr-xr-x 626 root root 0 Jan 13 01:12 proc
drwx------ 3 root root 60 Jan 13 01:13 root
drwxr-xr-x 3 root root 3 Jun 3 2019 rootPool
drwxr-xr-x 19 root root 660 Jan 13 01:47 run
drwxr-xr-x 2 root root 5023 Jul 3 2016 sbin
dr-xr-xr-x 13 root root 0 Jan 13 01:12 sys
drwxrwxrwt 6 root root 220 Jan 13 01:47 tmp
lrwxrwxrwx 1 root root 23 Jan 13 01:19 tools -> /rootPool/root_fs/tools
drwxr-xr-x 21 root root 180 Jan 13 01:17 usr
drwxr-xr-x 13 root root 100 Jan 13 01:13 var
Now I can rollback to the rootPool/root_fs/sources/binutils@lfs_prep_section_5.4_pass1 snapshot I made. This time I will redploy my script and make a new snapshot with my build script, and run again.
gentoo@livecd ~ $ sudo zfs rollback -r rootPool/root_fs/sources/binutils@lfs_prep_section_5.4_pass1gentoo@livecd~ $mv build_binutils_temp_pass1.sh /rootPool/root_fs/sources/binutilsgentoo@livecd~ $/sbin/zfs snapshot rootPool/root_fs/sources/binutils@lfs_prep_section_5.4_pass1_scriptedgentoo@livecd~ $/sbin/zfs snapshot rootPool/root_fs/tools@lfs_prep_section_5.4_pass1_scriptedgentoo@livecd~ $mv lfs.bash_profile .bash_profilegentoo@livecd~ $mv lfs.bashrc .bashrcgentoo@livecd~ $source .bash_profilegentoo@livecd~ $env TERM=xterm MAKEFLAGS=-j44 LC_ALL=POSIX LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: LFS=/rootPool/root_fs PATH=/tools/bin:/sbin:/bin:/usr/bin PWD=/home/gentoo LFS_TGT=x86_64-lfs-linux-gnu PS1=\[\033]0;\u@\h:\w\007\]\[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\] SHLVL=1 HOME=/home/gentoo _=/bin/env gentoo@livecd ~ $ ls -ld /tools lrwxrwxrwx 1 root root 23 Jan 13 01:19 /tools -> /rootPool/root_fs/toolsgentoo@livecd~ $cd $LFS/sources/binutilsgentoo@livecd/rootPool/root_fs/sources/binutils $ ls binutils-gdb build_binutils_temp_pass1.shgentoo@livecd/rootPool/root_fs/sources/binutils $ time ./build_binutils_temp_pass1.sh... [WHOLE BUNCH OF CONFIGURE, MAKE, COMPILER OUTPUT REMOVED FOR BREVITY] ... make[3]: Leaving directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build/gdb' make[2]: Leaving directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build/gdb' make[1]: Leaving directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build' real 0m58.416s user 10m43.418s sys 1m8.464s gentoo@livecd /rootPool/root_fs/sources/binutils $
SBU Timings
Because I am using ZFS snapshots, I can perform the rollbacks as many times as I want to re-perform the build steps. This will actually allow me to benchmark the approximate amount of time per SBU with different thread counts. Also if there are build errors I can effectively reset, determine what went wrong, fix it, and re-perform the build steps, with a clean state as if I had not attempted the build at all:gentoo@livecd ~ $ sudo zfs rollback rootPool/root_fs/sources/binutils@lfs_prep_section_5.4_pass1_scripted gentoo@livecd~ $sudo zfs rollback rootPool/root_fs/tools@lfs_prep_section_5.4_pass1_scriptedgentoo@livecd~ $export MAKEFLAGS='-j1'gentoo@livecd~ $cd /rootPool/root_fs/sources/binutils/gentoo@livecd/rootPool/root_fs/sources/binutils $ time ./build_binutils_temp_pass1.sh + cd binutils-gdb + mkdir -v build mkdir: created directory 'build' + cd build + ../configure --prefix=/tools --with-sysroot=/rootPool/root_fs --with-lib-path=/tools/lib --target=x86_64-lfs-linux-gnu --disable-nls --disable-werror checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-pc-linux-gnu checking target system type... x86_64-lfs-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c... [WHOLE BUNCH OF CONFIGURE, MAKE, COMPILER OUTPUT REMOVED FOR BREVITY] ...make[1]: Nothing to be done for 'install-target'. make[1]: Leaving directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build' real 6m52.944s user 5m50.666s sys 0m35.333s
A quick explination about the time values. The 'real' time is the wall clock time that has passed between entering the command and the command is complete. The user time is the accounting time of how much cpu time was executing userland code. The sys time is the accounting time of how much cpu time was executing kernel/syscall functions, including IO waits. As an simplified example, in the first scenerio where we had the real/user/sys times of 0:58/10:43/1:08, with code running on all 32 threads, each thread on average spent about 2 seconds waiting on IO or waiting on the OS to perform a task for its behalf. Also each thread spent on average about 20 seconds actually compiling. Because of the the multiple cores/threads available in the system it was able to schedule all the work in parallel in about a minute. In the second example, since we only ran with 1 thread on 1 core, nothing was done in parallel therefore the total/real time had to wait for that one thread to perform all the user and system tasks in order, therefore taking as much or more wall clock as the actual work...
The next question you may ask is, why did the total user and sys times take less with just 1 thread. There are several things that could cause this difference in time.
- Thread-Resource Contention.
This is where one thread can not execute because another thread is currently exclusively using a resource it needs. This resource could be an execution unit on the CPU itself, a memory operation to main-memory, a file operation on the disk, or some-other resource both compiling threads need access to. Depending on the type of resource the thread is waiting on will determine which accounting time is incremented while the thread is waiting. - Cached Resources.
Since the '1 thread' version was run after the '44 thread version', some items may have already been loaded into memory, so the make process did not actually have to go back to disk durring the 2nd compile. This can include both the tool chains (make, shell, compiler, assembler, linker, libraries) which have to load from the slow gentoo usb flash OR which portions of the source code are already in the ZFS ARC, based on usage and available memory. Typically when benchmarking one will throw out the 'first' run, because that run will warm up (assuming enough memory to hold the data set in memory) the tools and data into main memory and/or other system caches. - Compile Dependencies.
Not all of the items can be run in parallel. Certain steps require other portions of the build to have already been completed. With multiple threads, there can be time spent waiting for a previous step to have completed, before running the current step on the current thread. For the single thread version, all steps are done sequentially so there is no time spent waiting for a previous step, since it won't be queued for work untill the prior dependencies have all completed. - Other Computer Tasks.
Other programs running on the computer during compile. This includes both background, scheduled, and user UI tasks that may have been going on while I was waiting for the compile to complete.
| Threads | 1 | 2 | 4 | 8 | 12 | 16 | 17 | 24 | 32 | 32* | 33 | 40 | 44 | 64 |
| Real | 6:52.127 | 3:39.783 | 1:59.355 | 1:24.866 | 1:04.054 | 1:00.332 | 0:59.636 | 0:58.843 | 0:58.156 | 0:57.727 | 0:58.141 | 0:58.279 | 0:58.658 | 0:58.945 |
| User | 5:49.268 | 5:54.943 | 6:03.927 | 6:28.973 | 6:49.164 | 7:23.114 | 7:37.813 | 9:26.583 | 10:48:781 | 10:45.696 | 10:55.746 | 10:56.867 | 11:03.175 | 11:01.951 |
| Sys | 0:35.268 | 0:35.745 | 0:37.309 | 0:41.800 | 0:44.180 | 0:48.328 | 0:50.493 | 1:00.726 | 1:08.731 | 1:06.232 | 1:08.551 | 1:08.171 | 1:10.394 | 1:08.338 |
It looks like binutils build had diminishing returns after 16 threads, and seems to compile fastest at 32 threads. Using more threads (such as my current default value of 44) does not significantly slow it back down past 32, so I will probably leave that value set. Note that we now have a base SBU value of about 1 minute per SBU. When compiling with multiple threads there will be more variance per SBU based on the package layout, design, make configuration, and internal dependencies. Because of this some packages may do significantly better or worse then 1 minute per SBU.
Final Cleanup.
Once a package has successfully built we are left with some clean up steps. The LFS book talks about deleting the unpacked package. Instead we are going to do a different set of steps. We will snapshot the build and install, we will perform a clean, we will perform a snapshot, and then if needed manually clean any remaining changes and perform a final snapshot as follows:
In this case, git did not show any modified or untracked files, so the source repository is clean, and this section is now complete.gentoo@livecd /rootPool/root_fs/sources/binutils $ zfs snapshot rootPool/root_fs/sources/binutils@lfs_built_section_5.4_pass1gentoo@livecd/rootPool/root_fs/sources/binutils $zfs snapshot rootPool/root_fs/tools@lfs_install_section_5.4_pass1gentoo@livecd/rootPool/root_fs/sources/binutils $cd binutils-gdb/build/gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb/build $ make clean && make distclean rm -f *.a TEMP errs core *.o *~ \#* TAGS *.E *.log make[1]: Entering directory '/rootPool/root_fs/sources/binutils/binutils-gdb/build' make[1]: Nothing to be done for 'clean-target'. Doing clean in binutils [...ADDITIONAL MAKE OUTPUT FOR CLEAN OMITTED FOR BREVITY...] find . -name config.cache -exec rm -f {} \; \; 2>/dev/null make: [Makefile:2104: local-distclean] Error 1 (ignored)gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb/build $cd ..gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $ rm -rf buildgentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $zfs snapshot rootPool/root_fs/sources/binutils@lfs_cleaned_section_5.4_pass1gentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $git status HEAD detached at binutils-2_33_1 nothing to commit, working directory cleangentoo@livecd/rootPool/root_fs/sources/binutils/binutils-gdb $cd ../..gentoo@livecd/rootPool/root_fs/sources $
Next time I will start in on section 5.5 to cross-compile gcc for the temporary system.
