LFS sections 2.5 through 3.1

 Section 2 Preparing the Host System

We have already completed sections 2.1-2.4 in the previous blog.

 Section 2.5 'Creating a File System on the Partition'

Section 2.5 drones on about the 'ext' family of file systems, which I am not using. In addition I need to create a fat32 system on the EFI esp partition, and create some swap.  Since I will be using ZFS, I will create a ZPOOL and the subsuquent file systems.

My deviations from the book include running the following commands:

gentoo@livecd ~ $ sudo mkfs.fat -F 32 /dev/nvme0n1p1
mkfs.fat 4.0 (2016-05-06)
gentoo@livecd ~ $ sudo mkswap /dev/nvme0n1p2
Setting up swapspace version 1, size = 32 GiB (34359734272 bytes)
no label, UUID=f17abe47-d409-489e-920b-4b8341f3f1a8
gentoo@livecd ~ $ sudo swapon /dev/nvme0n1p2
gentoo@livecd ~ $ free
              total        used        free      shared  buff/cache   available
Mem:       65871904     1453280    61927076       88796     2491548    63707984
Swap:      33554428           0    33554428
 gentoo@livecd ~ $ sudo zpool create -o ashift=12 -o cachefile=/mnt/cdrom/scratch/lfs_root.cache -o comment="LFS Root" rootPool /dev/nvme0n1p6
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs/home
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs/usr
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs/opt
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs/usr/src
 gentoo@livecd ~ $ sudo zfs create rootPool/root_fs/tmp
 gentoo@livecd ~ $ sudo mkdir /rootPool/root_fs/esp
 gentoo@livecd ~ $ sudo mount /dev/nvme0n1p1 /rootPool/root_fs/esp
 gentoo@livecd ~ $ sudo zfs snapshot -r rootPool@init

 gentoo@livecd ~ $ sudo zpool list
NAME            SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
gentooScratch  3.72G   565M  3.17G         -    16%    14%  1.00x  ONLINE  -
rootPool        119G   206K   119G         -     0%     0%  1.00x  ONLINE  -
 gentoo@livecd ~ $ sudo zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
gentooScratch                                 564M  3.05G  79.1M  /gentooScratch
gentooScratch/persistentHomeGentoo            418M  3.05G   335M  /home/gentoo
gentooScratch/persistentPortageDistfiles      714K  3.05G   660K  /usr/portage/distfiles
gentooScratch/persistentPortageTmp             28K  3.05G    19K  /tmp/portage
gentooScratch/scripts                         101K  3.05G  23.5K  /gentooScratch/scripts
gentooScratch/scripts/data                   40.5K  3.05G  22.5K  /gentooScratch/scripts/data
gentooScratch/sources                        65.3M  3.05G    19K  /gentooScratch/sources
gentooScratch/sources/gentoo                  205K  3.05G    19K  /gentooScratch/sources/gentoo
gentooScratch/sources/gentoo/cpuid2cpuflags   159K  3.05G   123K  /gentooScratch/sources/gentoo/cpuid2cpuflags
gentooScratch/sources/lfs_book               65.0M  3.05G  62.0M  /gentooScratch/sources/lfs_book
rootPool                                      196K   115G    19K  /rootPool
rootPool/root_fs                              115K   115G    20K  /rootPool/root_fs
rootPool/root_fs/home                          19K   115G    19K  /rootPool/root_fs/home
rootPool/root_fs/opt                           19K   115G    19K  /rootPool/root_fs/opt
rootPool/root_fs/tmp                           19K   115G    19K  /rootPool/root_fs/tmp
rootPool/root_fs/usr                           38K   115G    19K  /rootPool/root_fs/usr
rootPool/root_fs/usr/src                       19K   115G    19K  /rootPool/root_fs/usr/src

Eight steps down... lots to go.

Section 2.6 Setting the $LFS Variable

We need to setup the LFS variable, since we want this to be persistent and set everytime we reboot, we will update our persitence scripts.

The /mnt/cdrom/scratch/mountfs.sh contents will be updated to also import our new LFS pool:

#!/bin/bash -x
# Script removes the existing directories on the Gentoo LiveUSB
# Then the script imports the persistent zfs storage, including                                           
# mounting/replacing the removed directories. Finally the script                                                      
# creates a new snapshot marking the begining of the live session.                   
                             
rm -rf /usr/portage/distfiles
rm -rf /tmp/portage
rm -rf /home/gentoo
                                                         
zpool import -a -c /mnt/cdrom/scratch/zfs_bootstrap.cache                 
zfs snapshot -r gentooScratch@`date +Persistence_Remount_%Y%h%d_%H%M%S.%N`
                                                                        
# Now the script will import our LFS root pool for the target system,   
# create a new snapshot marking the begining of this LFS modification   
# session, and then restore any gentoo/lfs values needed                
                                                                        
zpool import -a -c /mnt/cdrom/scratch/lfs_root.cache                    
mount /dev/nvme0n1p1 /rootPool/root_fs/esp
zfs snapshot -r rootPool@`date +Persistence_Remount_%Y%h%d_%H%M%S.%N`   
                                                                          
# Restore the portage settings, and the LFS bootstrap packages/settings.
/gentooScratch/scripts/restoreGentooEnv.sh      

The updated /gentooScratch/scripts/restoreGentooEnv.sh becomes:

#!/bin/bash -x

# Restore make/portage settings for this box.
cp /gentooScratch/scripts/data/make.conf /etc/portage/make.conf
cp /gentooScratch/scripts/data/00cpuflags /etc/portage/package.use/

# Restore needed gentoo packages
emerge app-text/tidy-html5

# Setup our LFS environment
export LFS=/rootPool/root_fs/
cp /gentooScratch/scripts/data/lfs_var.sh /etc/profile.d

And the new /gentooScratch/scripts/data/lfs_var.sh file is:

export LFS=/rootPool/root_fs/

Of course we create a new snapshot to mark this improvement.

sudo zfs snapshot -r gentooScratch/scripts@lfs_var

Now we will reboot to ensure that LFS is properly set.  Upon logging back in:

gentoo@livecd ~ $ echo $LFS
/rootPool/root_fs/
gentoo@livecd ~ $ df -h
Filesystem                                   Size  Used Avail Use% Mounted on
udev                                          10M  4.0K   10M   1% /dev
/dev/sda1                                     29G  5.9G   23G  21% /mnt/cdrom
tmpfs                                         32G   33M   32G   1% /.unions/memory
aufs                                          32G   33M   32G   1% /
/dev/loop0                                   2.0G  2.0G     0 100% /mnt/livecd
none                                          32G   20K   32G   1% /mnt/aufs-rw-branch
tmpfs                                        6.3G  1.7M  6.3G   1% /run
shm                                           32G     0   32G   0% /dev/shm
cgroup_root                                   10M     0   10M   0% /sys/fs/cgroup
vartmp                                        32G   36K   32G   1% /var/tmp
tmp                                           32G  4.0K   32G   1% /tmp
gentooScratch                                3.2G   79M  3.1G   3% /gentooScratch
gentooScratch/scripts                        3.1G     0  3.1G   0% /gentooScratch/scripts
gentooScratch/scripts/data                   3.1G     0  3.1G   0% /gentooScratch/scripts/data
gentooScratch/sources                        3.1G     0  3.1G   0% /gentooScratch/sources
gentooScratch/sources/gentoo                 3.1G     0  3.1G   0% /gentooScratch/sources/gentoo
gentooScratch/sources/gentoo/cpuid2cpuflags  3.1G  128K  3.1G   1% /gentooScratch/sources/gentoo/cpuid2cpuflags
gentooScratch/sources/lfs_book               3.1G   62M  3.1G   2% /gentooScratch/sources/lfs_book
gentooScratch/persistentHomeGentoo           3.4G  338M  3.1G  10% /home/gentoo
gentooScratch/persistentPortageTmp           3.1G     0  3.1G   0% /tmp/portage
gentooScratch/persistentPortageDistfiles     3.1G  640K  3.1G   1% /usr/portage/distfiles
rootPool                                     116G     0  116G   0% /rootPool
rootPool/root_fs                             116G     0  116G   0% /rootPool/root_fs
rootPool/root_fs/home                        116G     0  116G   0% /rootPool/root_fs/home
rootPool/root_fs/opt                         116G     0  116G   0% /rootPool/root_fs/opt
rootPool/root_fs/tmp                         116G     0  116G   0% /rootPool/root_fs/tmp
rootPool/root_fs/usr                         116G     0  116G   0% /rootPool/root_fs/usr
rootPool/root_fs/usr/src                     116G     0  116G   0% /rootPool/root_fs/usr/src
/dev/nvme0n1p1                               1.2G  4.0K  1.2G   1% /rootPool/root_fs/esp
none                                          32G  8.0K   32G   1% /run/user/1000

Nine steps down... lots to go.

Section 2.7 Mounting the new Partition

We will be basically skipping this step. ZPool management automounts the filesystems as seen in the previous output, furthermore this mounting and LFS are persistent by re-running our mountfs.sh script.

Ten steps down... lots to go.

Section 3 Packages and Patches

Section 3.1 Introduction

Rather then create a new directory for the sources, we just create a descendant ZFS file system, and the the permissions as specified:

sudo zfs create rootPool/root_fs/sources
sudo zfs create rootPool/root_fs/sources/lfs_tarballs
sudo chmod -v a+wt $LFS/sources
sudo chmod -v a+wt $LFS/sources/lfs_tarballs
sudo zfs snapshot -r rootPool/root_fs/sources@init

We will start with the 'easy way' using the wget-list file provided by the book and downloading all the packages and patches.

wget --input-file=/gentooScratch/sources/lfs_book/book_output/wget-list --continue --directory-prefix=$LFS/sources/lfs_tarballs/ 2&> $LFS/sources/tarball_download.log

pushd $LFS/sources/lfs_tarballs
md5sum -c /gentooScratch/sources/lfs_book/book_output/md5sums
popd  
sudo zfs snapshot rootPool/root_fs/sources/lfs_tarballs@initial_download

This lists 7 files as not being downloaded, all of which appear to be related to SSL connections to the sourceforge server.  My initial guess is an expired certificate, so I will attempt to re-download them using a non ssl connection, relying on the MD5 sums to validate the source. [I know MD5 is no longer considered secured, it is possible, though not probable, that a MITM attack could replace the source tar with a file that appears to be valid and has a valid md5sum, but this is so little of a chance... and later on where possible I plan on cloning source repositories...]
  
The list of packages that were not downloaded was:

• e2fsprogs-1.45.1.tar.gz
• expat-2.2.6.tar.bz2
• expect5.45.4.tar.gz
• procps-ng-3.3.15.tar.xz
• psmisc-23.2.tar.xz
• tcl8.6.9-src.tar.gz
• xz-5.2.4.tar.xz

By manually replacing the https to http, or going to the sourceforge project page I was able to download the packages. Running MD5 sums again, and every file checks out as ok, so yet another snapshot.

So now quite a bit of time has been spent getting the packages, so for now we will call that a session.
 

LFS book steps through 2.4

So I have started reading through the book I produced at the end of my last session.

Step 1.1 'How to build an LFS System' says one will need an existing distribution or live system for the tools to prep the LFS system.  WaHOO!!! my last 3 blog posts on just getting a persistent gentoo live usb system should qualify for this step.  One step down... lots to go.

Step 1.2 is just the list of new features.  Since I am working from the SVN repo of the book, the most recent changes are about 2 weeks old.  I did note that one of the updated packages in this revision is E2fsprogs version.  Debating if I even need/want support for the ext family of filesystems.  I plan on going whole-hog with ZFS, which will require deviating from the instructions to support any way.  Two steps down... lots to go

Step 1.3 is the change log, I did review the change log and most of the track tickets.  Most appear to be updating to new minor revisions of the software from the last 'stable' version of the book.  Some are fixes to minor build issues.  Three steps down... lots to go.

Step 1.4 is resources, good to know, nothing to do.  Four steps down... lots to go.

Step 1.5 asking for help, also good to know, nothing to do.  Five steps down... lots to go.

Step 2.2 version checks of host system:

• Bash 3.2 (/bin/sh should be a symbolic or hard link to bash)
• bash --version 

GNU bash, version 4.3.46(1)-release (x86_64-pc-linux-gnu)                                                             
Copyright (C) 2013 Free Software Foundation, Inc.                                                                     
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>                                         
                                                                                                                      
This is free software; you are free to change and redistribute it.                                                    
There is NO WARRANTY, to the extent permitted by law.     

• ls -l /bin/sh

lrwxrwxrwx 1 root root 4 Jun 23  2016 /bin/sh -> bash 

• Binutils-2.25 (Versions greater than 2.32 are not recommended as they have not been tested)
• emerge --search binutils

*  sys-devel/binutils                                                                                                 
      Latest version available: 2.25.1-r1                                                                             
      Latest version installed: 2.25.1-r1                                                                             
      Size of files: 23616 KiB                                                                                        
      Homepage:      https://sourceware.org/binutils/                                                                 
      Description:   Tools necessary to build programs                                                                
      License:       || ( GPL-3 LGPL-3 )                    

•  Bison-2.7 (/usr/bin/yacc should be a link to bison or small script that executes bison) 
• yacc --version

bison (GNU Bison) 3.0.4
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• ls ls -l `which yacc`

lrwxrwxrwx 1 root root 10 Oct 28  2015 /usr/bin/yacc -> yacc.bison

• Bzip2-1.0.4
• bzip2 --version

bzip2, a block-sorting file compressor.  Version 1.0.6, 6-Sept-2010.
   
   Copyright (C) 1996-2010 by Julian Seward.
   
   This program is free software; you can redistribute it and/or modify
   it under the terms set out in the LICENSE file, which is included
   in the bzip2-1.0.6 source distribution.
   
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   LICENSE file for more details.
   
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.

•  Coreutils-6.9 
• emerge --search coreutils
  

*  sys-apps/coreutils
      Latest version available: 8.25
      Latest version installed: 8.25
      Size of files: 5641 KiB
      Homepage:      https://www.gnu.org/software/coreutils/
      Description:   Standard GNU file utilities (chmod, cp, dd, dir, ls...), text utilities (sort, tr, head, wc..), and shell utilities (whoami, who,...)
      License:       GPL-3

•  Diffutils-2.8.1
• emerge --search diffutils 

*  sys-apps/diffutils
      Latest version available: 3.3
      Latest version installed: 3.3
      Size of files: 1170 KiB
      Homepage:      https://www.gnu.org/software/diffutils/
      Description:   Tools to make diffs and compare files
      License:       GPL-2

•  Findutils-4.2.31
•  emerge --search findutils

*  sys-apps/findutils
      Latest version available: 4.6.0-r1
      Latest version installed: 4.6.0-r1
      Size of files: 3692 KiB
      Homepage:      https://www.gnu.org/software/findutils/
      Description:   GNU utilities for finding files
      License:       GPL-3+

•  Gawk-4.0.1 (/usr/bin/awk should be a link to gawk) 
• awk --version

GNU Awk 4.1.3, API: 1.1
Copyright (C) 1989, 1991-2015 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

• ls -l `which awk`

lrwxrwxrwx 1 root root 4 Oct 28  2015 /usr/bin/awk -> gawk

•  GCC-5.2 including the C++ compiler, g++ (Versions greater than 9.1.0 are not recommended as they have not been tested) 
• gcc --version

gcc (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• g++ --version

g++ (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

•  Glibc-2.11 (Versions greater than 2.29 are not recommended as they have not been tested) 
• emerge --search glibc

*  sys-libs/glibc
      Latest version available: 2.23-r2
      Latest version installed: 2.23-r2
      Size of files: 21241 KiB
      Homepage:      https://www.gnu.org/software/libc/libc.html
      Description:   GNU libc6 (also called glibc2) C library
      License:       LGPL-2.1+ BSD HPND ISC inner-net rc PCRE

•  Grep-2.5.1a
• grep --version

grep (GNU grep) 2.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

•  Gzip-1.3.12
• gzip --version
  

gzip 1.8
Copyright (C) 2016 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.

•  Linux Kernel-3.2
• uname -a 

Linux livecd 4.5.2-aufs-r1 #1 SMP Sun Jul 3 17:17:11 UTC 2016 x86_64 AMD Ryzen Threadripper 2950X 16-Core Processor AuthenticAMD GNU/Linux

•  M4-1.4.10 
• m4 --version 

m4 (GNU M4) 1.4.17
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Rene' Seindal.

•  Make-4.0
• make --version 

GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

•  Patch-2.5.4
•  patch --version

GNU patch 2.7.5
Copyright (C) 2003, 2009-2012 Free Software Foundation, Inc.
Copyright (C) 1988 Larry Wall

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Larry Wall and Paul Eggert

•  Perl-5.8.8 
•  perl --version
  

This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-linux
(with 28 registered patches, see perl -V for more detail)

Copyright 1987-2016, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

•  Python-3.4
•  python --version

Python 3.4.4

•  Sed-4.1.5 
•  sed --version

sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.

•  Tar-1.22
•  tar --version

tar (GNU tar) 1.29
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

•  Texinfo-4.7
•  info --version

info (GNU texinfo) 6.1

Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

•  Xz-5.0.0
•  xz --version

xz (XZ Utils) 5.2.2
liblzma 5.2.2

After I get to the bottom of the list, manually verifying as I go, I note that the instructions include a script to verify that the host system has all the proper tools and versions.  The script is as follows:

cat > version-check.sh << "EOF"
#!/bin/bash
# Simple script to list version numbers of critical development tools
export LC_ALL=C
bash --version | head -n1 | cut -d" " -f2-4
MYSH=$(readlink -f /bin/sh)
echo "/bin/sh -> $MYSH"
echo $MYSH | grep -q bash || echo "ERROR: /bin/sh does not point to bash"
unset MYSH

echo -n "Binutils: "; ld --version | head -n1 | cut -d" " -f3-
bison --version | head -n1

if [ -h /usr/bin/yacc ]; then
  echo "/usr/bin/yacc -> `readlink -f /usr/bin/yacc`";
elif [ -x /usr/bin/yacc ]; then
  echo yacc is `/usr/bin/yacc --version | head -n1`
else
  echo "yacc not found" 
fi

bzip2 --version 2>&1 < /dev/null | head -n1 | cut -d" " -f1,6-
echo -n "Coreutils: "; chown --version | head -n1 | cut -d")" -f2
diff --version | head -n1
find --version | head -n1
gawk --version | head -n1

if [ -h /usr/bin/awk ]; then
  echo "/usr/bin/awk -> `readlink -f /usr/bin/awk`";
elif [ -x /usr/bin/awk ]; then
  echo awk is `/usr/bin/awk --version | head -n1`
else 
  echo "awk not found" 
fi

gcc --version | head -n1
g++ --version | head -n1
ldd --version | head -n1 | cut -d" " -f2-  # glibc version
grep --version | head -n1
gzip --version | head -n1
cat /proc/version
m4 --version | head -n1
make --version | head -n1
patch --version | head -n1
echo Perl `perl -V:version`
python3 --version
sed --version | head -n1
tar --version | head -n1
makeinfo --version | head -n1  # texinfo version
xz --version | head -n1

echo 'int main(){}' > dummy.c && g++ -o dummy dummy.c
if [ -x dummy ]
  then echo "g++ compilation OK";
  else echo "g++ compilation failed"; fi
rm -f dummy.c dummy
EOF

bash version-check.sh

So after reading through the script for hidden dangers (you should always audit scripts and source...) I put the contents of the 'version-check.sh' into /gentooScratch/scripts/host-version-check.sh, mark it is executable and obtained the following output:

gentoo@livecd /gentooScratch/scripts $ vi ./host-version-check.sh 
gentoo@livecd /gentooScratch/scripts $ ./host-version-check.sh 
bash, version 4.3.46(1)-release
/bin/sh -> /bin/bash
Binutils: (Gentoo 2.25.1 p1.1) 2.25.1
bison (GNU Bison) 3.0.4
/usr/bin/yacc -> /usr/bin/yacc.bison
bzip2,  Version 1.0.6, 6-Sept-2010.
Coreutils:  8.25
diff (GNU diffutils) 3.3
find (GNU findutils) 4.6.0
GNU Awk 4.1.3, API: 1.1
/usr/bin/awk -> /usr/bin/gawk
gcc (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0
g++ (Gentoo 5.4.0 p1.0, pie-0.6.5) 5.4.0
(Gentoo 2.23-r2 p4) 2.23
grep (GNU grep) 2.25
gzip 1.8
Linux version 4.5.2-aufs-r1 (root@jasmin) (gcc version 5.4.0 (Gentoo 5.4.0 p1.0, pie-0.6.5) ) #1 SMP Sun Jul 3 17:17:11 UTC 2016
m4 (GNU M4) 1.4.17
GNU Make 4.2.1
GNU patch 2.7.5
Perl version='5.24.0';
Python 3.4.4
sed (GNU sed) 4.2.2
tar (GNU tar) 1.29
texi2any (GNU texinfo) 6.1
xz (XZ Utils) 5.2.2
g++ compilation OK
gentoo@livecd /gentooScratch/scripts $ sudo zfs snapshot -r gentooScratch/scripts@host_versions_checked

So now my host system has passed all the prerequisites for building LFS. Six steps down... lots to go.

Section 2.3 discusses the state of the system, needing to be careful when restarting/rebooting.  Part of the reason I spent so much effort on setting up my Gentoo live USB persistent system was to make it easier to reconfigure any system environments needed for a perticular set of steps.  I will be making use of my /gentooScratch/scripts directory to record import environment info to allow me to reboot and get back to the proper state.  I will probably need to refer back to this section as I work.  Seven steps down... lots to go.

Section 2.4 'Creating a New Partition'.  I will be heavily deviating in this section.  For one I will be building a ZFS root pool, this will require an initramfs (advanced topic in the Note, typically not recommended for first time LFS users...).  In addition the instructions appear to be tailored to a BIOS based boot.  My new modern system is all EFI.  The old-tried-and-true amount of swap=2x available sounds a bit ridicules on my 64GB server, however since I do plan on running multiple VMs, some which may at time not be very active, and I have plenty of space on my nVME system drive I will over-provision the swap space at 128GB. Initially I will only activate 1/4 of the swap, untill I find additional need for swap latter. I will provide an overly large EFI partition for future kernels and other EFI boot utilities of 1200MB.  The section also lists convenience partitions:

• /boot
• Will not be creating this partition, my EFI partition will be mounted /esp (EFI System Partition)
• /home
• Will create a temporary zfs system on the root pool, will be migrated to my redundant zpool later, and will be the migrated data from my existing file server
• /usr
• Will create a zfs file system on the root pool.
• /opt
• Will crate a temporary zfs system on the root pool, will be migrated to my redundant zpool later.
• /tmp
• Will create a zfs mount point, but probably overlay with a tmpfs using ram.
• /usr/src
• Will create a temporary zfs system on the root pool, will be migrated to my redundant zpool later.

In addition I will be using the tool gdisk, not fdisk or cfdisk with the command:

sudo gdisk /dev/nvme0n1

After I layout my partitions, this is what the primary nVME boot device looks like

gentoo@livecd ~ $ sudo gdisk -l /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 3750748848 sectors, 1.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): C55485FA-4146-4D15-9E74-30036CAFF5C6
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 3750748814
Partitions will be aligned on 2048-sector boundaries
Total free space is 3228197485 sectors (1.5 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         2459647   1.2 GiB     EF00  EFI System
   2         2459648        69568511   32.0 GiB    8200  Linux swap
   3        69568512       136677375   32.0 GiB    8200  Linux swap
   4       136677376       203786239   32.0 GiB    8200  Linux swap
   5       203786240       270895103   32.0 GiB    8200  Linux swap
   6       270895104       522553343   120.0 GiB   BF00  Solaris root

Seven steps down... lots to go. For now this will be a good stopping point.

Additional Persistence for LFS bootstrap

In my previous post I determined I needed my Gentoo live system to include the tidy utility to even build the LFS systemd installation book from source.  Since my live system is not persistent I can not currently make system changes on the Live USB permanent across reboots.  However I can add some additional persistence and create some utility scripts to get my Live system back to a known state for future LFS work.

The last command I used prior to shutdown was:

sudo /sbin/zpool export gentooScratch

Now that  I have rebooted, I can remount all the ZFS filesystems by issuing the following command:  

sudo /sbin/zpool import -a -c /mnt/cdrom/scratch/zfs_bootstrap.cache  
/sbin/zfs list

Producing the following output:
gentoo@livecd ~ $ sudo /sbin/zpool import -a -c /mnt/cdrom/scratch/zfs_bootstrap.cache
gentoo@livecd ~ $ /sbin/zfs list
NAME                             USED  AVAIL  REFER  MOUNTPOINT
gentooScratch                   60.0M  3.54G    19K  /gentooScratch
gentooScratch/sources           59.9M  3.54G    19K  /gentooScratch/sources
gentooScratch/sources/lfs_book  59.8M  3.54G  59.6M  /gentooScratch/sources/lfs_book
gentoo@livecd ~ $

Previously I forgot to mention, that I typically take snapshots for different steps, so I can roll-back if something goes wrong.  The current state of the system is as follows:
gentoo@livecd ~ $ /sbin/zfs list -t snapshot
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
gentooScratch/sources/lfs_book@initial                   9K      -    19K  -
gentooScratch/sources/lfs_book@SVN_CO_rev_11610        201K      -  59.6M  -
gentooScratch/sources/lfs_book@make_failed_need_tidy      0      -  59.6M  -
gentoo@livecd ~ $

So my current persistent storage is working, but I now need additional storage for portage package downloads and would also like my desktop environment to persist. To that end, I need to create a few additional filesystems, and backup the gentoo user's home directory so I can apply the values on the over-layed/masked home directory after the zpool comes up.  Note these commands (particularly those altering the gentoo user's home directory) need to be run from a standard console as root so that the /home/gentoo directory does not have open files in use from using the graphical login.

rm -rf /usr/portage/distfiles  
zfs create -o mountpoint=/usr/portage/distfiles gentooScratch/persistentPortageDistfiles
rm -rf /tmp/portage
zfs create -o mountpoint=/tmp/portage gentooScratch/persistentPortageTmp
tar -cf /gentooScratch/home_gentoo.tar /home/gentoo
rm -rf /home/gentoo
zfs create -o mountpoint=/home/gentoo gentooScratch/persistentHomeGentoo
tar -C / -xf /gentooScratch/home_gentoo.tar
df -h
zfs snapshot gentooScratch/persistentPortageDistfiles@init
zfs snapshot gentooScratch/persistentPortageTmp@init
zfs snapshot gentooScratch/persistentHomeGentoo@init

Producing the following output on the console (CTRL-ALT-F1):
livecd ~ # rm -rf /usr/portage/distfiles
livecd ~ # zfs create -o mountpoint=/usr/portage/distfiles gentooScratch/persistentPortageDistfiles
livecd ~ # rm -rf /tmp/portage
livecd ~ # zfs create -o mountpoint=/tmp/portage gentooScratch/persistentPortageTmp
livecd ~ # tar -cf /gentooScratch/home_gentoo.tar /home/gentoo
tar: Removing leading `/' from member name
livecd ~ # rm -rf /home/gentoo
livecd ~ # zfs create -o mountpoint=/home/gentoo gentooScratch/persistentHomeGentoo
livecd ~ # tar -C / -xf /gentooScratch/home_gentoo.tar  
livecd ~ # df -h
Filesystem                                Size  Used Avail Use% Mounted on
udev                                       10M  4.0K   10M   1% /dev
/dev/sda1                                  29G  5.9G   23G  21% /mnt/cdrom
tmpfs                                     3.9G   29M  3.9G   1% /.unions/memory
aufs                                      3.9G   29M  3.9G   1% /
/dev/loop0                                2.0G  2.0G     0 100% /mnt/livecd
none                                      3.9G   20K  3.9G   1% /mnt/aufs-rw-branch
tmpfs                                     786M  1.6M  784M   1% /run
shm                                       3.9G     0  3.9G   0% /dev/shm
cgroup_root                                10M     0   10M   0% /sys/fs/cgroup
vartmp                                    3.9G   36K  3.9G   1% /var/tmp
tmp                                       3.9G  4.0K  3.9G   1% /tmp
gentooScratch                             3.3G   79M  3.3G   3% /gentooScratch
gentooScratch/sources                     3.3G     0  3.3G   0% /gentooScratch/sources
gentooScratch/sources/lfs_book            3.3G   60M  3.3G   2% /gentooScratch/sources/lfs_book
gentooScratch/persistentPortageDistfiles  3.3G     0  3.3G   0% /usr/portage/distfiles
gentooScratch/persistentPortageTmp        3.3G     0  3.3G   0% /tmp/portage
gentooScratch/persistentHomeGentoo        3.5G  244M  3.3G   7% /home/gentoo
none                                      3.9G  8.0K  3.9G   1% /run/user/1000
livecd ~ # zfs snapshot gentooScratch/persistentPortageDistfiles@init
livecd ~ # zfs snapshot gentooScratch/persistentPortageTmp@init
livecd ~ # zfs snapshot gentooScratch/persistentHomeGentoo@init

The next step is I need a persistent script to prep and mount the zfs filesystems after a restart.  This script will of course need to also be run as root from the console, prior to logging into gentoo system to remove the filesystems that exist in the Live squash filesystem.

Created a /mnt/cdrom/scratch/mountfs.sh with the following contents:

#!/bin/bash -x
# Script removes the existing directories on the Gentoo LiveUSB
# Then the script imports the persistent zfs storage, including mounting/replacing the removed directories
# Finally the script creates a new snapshot marking the begining of the live session.

rm -rf /usr/portage/distfiles
rm -rf /tmp/portage
rm -rf /home/gentoo

zpool import -a -c /mnt/cdrom/scratch/zfs_bootstrap.cache
zfs snapshot -r gentooScratch@`date +Persistence_Remount_%Y%h%d_%H%M%S.%N`

*tangent* Found the following code formatter for the above script, and now I am debating changing previous entries with the above format. *end tangent*

After logging out, exporting the zpool, rebooting,  running the mountfs.sh from the root console, and finally logging back in as gentoo user... The system state is:

gentoo@livecd ~ $ /sbin/zfs list -t snapshot
NAME                                                                                      USED  AVAIL  REFER  MOUNTPOINT
gentooScratch@Persistence_Remount_2019May28_003919.036868361                                 0      -  79.1M  -
gentooScratch/persistentHomeGentoo@init                                                  12.2M      -  84.4M  -
gentooScratch/persistentHomeGentoo@Persistence_Remount_2019May28_003919.036868361        21.5M      -   255M  -
gentooScratch/persistentPortageDistfiles@init                                                0      -    19K  -
gentooScratch/persistentPortageDistfiles@Persistence_Remount_2019May28_003919.036868361      0      -    19K  -
gentooScratch/persistentPortageTmp@init                                                      0      -    19K  -
gentooScratch/persistentPortageTmp@Persistence_Remount_2019May28_003919.036868361            0      -    19K  -
gentooScratch/sources@Persistence_Remount_2019May28_003919.036868361                         0      -    19K  -
gentooScratch/sources/lfs_book@initial                                                      9K      -    19K  -
gentooScratch/sources/lfs_book@SVN_CO_rev_11610                                           201K      -  59.6M  -
gentooScratch/sources/lfs_book@make_failed_need_tidy                                         0      -  59.6M  -
gentooScratch/sources/lfs_book@Persistence_Remount_2019May28_003919.036868361                0      -  59.6M  -
gentoo@livecd ~ $ df -h
Filesystem                                Size  Used Avail Use% Mounted on
udev                                       10M  4.0K   10M   1% /dev
/dev/sda1                                  29G  5.9G   23G  21% /mnt/cdrom
tmpfs                                     3.9G   29M  3.9G   1% /.unions/memory
aufs                                      3.9G   29M  3.9G   1% /
/dev/loop0                                2.0G  2.0G     0 100% /mnt/livecd
none                                      3.9G   20K  3.9G   1% /mnt/aufs-rw-branch
tmpfs                                     786M  1.6M  784M   1% /run
shm                                       3.9G     0  3.9G   0% /dev/shm
cgroup_root                                10M     0   10M   0% /sys/fs/cgroup
vartmp                                    3.9G   36K  3.9G   1% /var/tmp
tmp                                       3.9G  4.0K  3.9G   1% /tmp
gentooScratch                             3.3G   80M  3.2G   3% /gentooScratch
gentooScratch/sources                     3.2G     0  3.2G   0% /gentooScratch/sources
gentooScratch/sources/lfs_book            3.3G   60M  3.2G   2% /gentooScratch/sources/lfs_book
gentooScratch/persistentHomeGentoo        3.5G  248M  3.2G   8% /home/gentoo
gentooScratch/persistentPortageTmp        3.2G     0  3.2G   0% /tmp/portage
gentooScratch/persistentPortageDistfiles  3.2G     0  3.2G   0% /usr/portage/distfiles
none                                      3.9G  8.0K  3.9G   1% /run/user/1000
gentoo@livecd ~ $ 

Now that the portage system will be mostly persistent (for downloads/builds), we can build tidy for our Gentoo Live system using the following command 

sudo emerge app-text/tidy-html5
sudo zfs snapshot gentooScratch/persistentPortageTmp@tidy_html_cached 
sudo zfs snapshot gentooScratch/persistentPortageDistfiles@tidy_html_cached

The next step is to provide a persistent location that can be used for utility scripts to track changes and reconfigure the environment and settings needed for the LFS boot strapping project.

sudo zfs create gentooScratch/scriptssudo zfs create gentooScratch/scripts/datasudo zfs create gentooScratch/scripts/data
sudo chown -R gentoo:users  /gentooScratch/scripts
sudo zfs snapshot -r gentooScratch/scripts@init
cp /etc/portage/make.conf /gentooScratch/scripts/data/

Now I can edit the /gentooScratch/scripts/data/make.conf file, to customize the gentoo build options for any tools that we need on the way to initially bootstrapping the LFS system. The first thing we want to update is the CPU_FLAGS_X86 property to optimize the tools we have to build to setup our LFS system.  A good way to generate this is to use the app-portage/cpuinfo2cpuflags package.  So the next step is:

sudo emerge app-portage/cpuinfo2cpuflags

Unfortunately the emerge fails, unable to fetch the required package.  After a bit of digging I came across a similar piece of software, cpuid2cpuflags, available on github here.  So now we need to download and compile that program.

sudo zfs create gentooScratch/sources/gentoo
sudo zfs create gentooScratch/sources/gentoo/cpuid2cpuflags
sudo chown -R gentoo:users /gentooScratch/sources/gentoo
sudo zfs snapshot -r gentooScratch/sources/gentoo@init
cd /gentooScratch/sources/gentoo/cpuid2cpuflags
git clone https://github.com/mgorny/cpuid2cpuflags
sudo zfs snapshot gentooScratch/sources/gentoo/cpuid2cpuflags@init_git_clone
cd cpuid2cpuflags/

Looking around at the source there appears to be no Makefile, and only some of the files needed for input to automake are present.  The README file does not help with instructions on how to compile it either.  It looks like I will have to manually figure out how to compile and build from this source.  After a bit of experimentation I came up with the following command line:

cd src; g++ -DPACKAGE_STRING='"v5"' -DHAVE_CPUID_H=1 -DHAVE_STDINT_H=1 -o ../cpuid2cpuflags  main.c x86.c ; cd ..
 
Running the program I get the following output, which looks about right for a modern 64-bit CPU:

CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3

I  can now take these values and modify my persistence system to incorporate these values into gentoo's emerge system. So first I will edit the /gentooScratch/scripts/data/make.conf file and add these flags.  Line 16 will go from 'CPU_FLAGS_X86="mmx sse sse2"' to 'CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3"'.  I will also change the MAKEOPTS on line 25 from =-j2 to -j48, which is 1.5 * the number of total threads I have on this AMD Threadripper system.  I will also create a 00cpuflags file for portage to use with the following contents:
*/* CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3

The final changes will be to create a 2nd executable restore script /gentooScratch/scripts/restoreGentooEnv.sh with the following contents:

#!/bin/bash -x                                                    

# Restore make/portage settings for this box.         
cp /gentooScratch/scripts/data/make.conf /etc/portage/make.conf
cp /gentooScratch/scripts/data/00cpuflags /etc/portage/package.use/

# Restore needed packages
emerge app-text/tidy-html5

Now I can snapshot the scripts directory and change the /mnt/cdrom/scratch/mountfs.sh script to run the restore script from the persistent storage after it mounts the storage.  Test everything with another reboot and now we should be able to finally build our LFS instruction book from scratch.

cd /gentooScratch/sources/lfs_book
mkdir book_output
cd BOOK
svn update
make REV=systemd BASEDIR=/gentooScratch/sources/lfs_book/book_output
sudo zfs snapshot gentooScratch/sources/lfs_book@book_11611_built

I now check loading the book and get the following:

So now I can FINALLY start in on the LFS install, using these instructions.  My next blog post will hopefully make progress on getting LFS actually installed, now that my live USB has persistence to get me back to this point after any reboot.