new hint version : trip package manager

Pierre Hébert pierrox at pierrox.net
Sat Aug 18 11:16:52 PDT 2007


Hello,

I would like to submit a new version of my hint.
Trip is a trivial package manager focusing on LFS. It makes use of 
unionfs to monitor altered files during an install, and to package them 
automatically and safely.
This new version should be easier to read and describes the new 
options available in trip-0.3.
It can also be read in html version at http://www.pierrox.net/trip, and 
binary/source packages can be found at this address.

Best regards,

Pierre Hebert.
-------------- next part --------------
AUTHOR: Pierre Hebert <pierrox at pierrox dot net>

DATE: 2007-07-31

LICENSE: GNU Free Documentation License Version 1.2

SYNOPSIS: TRIP, a TRIvial Packager for LFS (and possibly other linux systems)

DESCRIPTION:
Trip is a trivial package manager. Its aim is to provide a secure way to build
and install packages from sources, without hassle and with as few dependencies
as possible.

Trip will record all created, modified or deleted files during the "make
install" step, while preventing the root filesystem to be altered. This fine and
safe monitoring has been made possible thanks to unionfs. Trip make use of
unionfs when constructing a package. Basically it puts a temporary filesystem on
top of the root filesystem. This temporary filesystem will receive a copy of all
created/modified files, while the underlying root filesystem remains untouched.
In no case you will overwrite a file by mistake, or create a conflict without
being alerted.
 
Trip package format is almost as simple as possible : it consists in a tarball,
meaning that you can always verify and extract its content with common tools.
Trip is a bash shell script and does not rely on other tools than those already
present in the /tools directory when building LFS... and unionfs, which is at
time of writing still not in mainline Linux kernel. Fortunately this module is
widely available though.
Trip has been designed with LFS in mind, but can also be used on other systems.

PREREQUISITES:
- some knowledge about the linux/unix problematic of package management
(suggested reading among other documents :
http://www.linuxfromscratch.org/hints/downloads/files/fakeroot.txt from Tushar
Teredesai)
- a linux 2.6 system with unionfs >= 1.1.3
- optional : you should have already built a LFS system yourself, in case you
plan to use trip to build and package a new LFS system. Some explanations about
this process are given later in this hint and may be enough, but the LFS book is
far better written and more instructive.


HINT:

--------------------------------------------------------------------------------
What trip is, and what trip is not
--------------------------------------------------------------------------------
Trip is a simple package manager, mainly designed for LFS users. It is a quite
low level tool, probably with bugs and rough edges. It will safely monitor file
alterations during compilation and installation from a source package. It will
manage packages with tarball and text files. It will let the user do what it
wants to do, no matter if that is a mistake or not.

Trip is not RPM nor DPKG (to name only two). It will not fetch packages
automatically for you from internet. It will not check dependencies. It will not
split packages into various pieces automatically (executables, devel, libs,
etc.). It will not allow you to upgrade your whole system automagically.

All that said, trip won't be in you way when you decide to do something, and
trip will not require tons of dependencies to run.



--------------------------------------------------------------------------------
Trip usage in short
--------------------------------------------------------------------------------
Trip -h should give something like that :
+ Trip, a Trivial Package manager, using Unionfs. You are using version 0.3
+ usage : trip -i,   --install <binary package file> [--no-conflict]
+                    (install from a pre-built binary package)
+              -u,   --uninstall <package name>
+                    (uninstall a package installed with trip)
+              -b,   --build <source package directory>
+                    (create a binary package from sources)
+              -r,   --rebuild <package name>
+                    (rebuild a binary package from an installed package)
+              -w,   --wizard
+                    (ask some questions, create a source package, build it and
install it)
+              -l,   --list [<package name>|<binary package file>]
+                    (list installed packages, files from an installed package
or files from a binary package)
+              -f,   --find-file <file>
+                    (find the package(s) containing "file")
+              -bbi, --batch-build-install <package list>
+                    (build and install a set of packages from "package list")
+              -bu,  --batch-uninstall <package list>
+                    (uninstall a set of packages from "package list")
+                    --upgrade-bd-from-0.1-to-0.2
+                    --upgrade-bd-from-0.2-to-0.3
+                    (migrate data from a previous package database format)
+              -c,   --config-dir <dir>
+                    (specify an alternate configuration directory)
+              -t,   --trip-path <path>
+                    (specify an alternate path to this trip shell script)
+              -v,   --verbose-level <0|1|2|3>
+                    (verbosity where 0=quiet, 1=errors, 2=errors/warnings,
3=errors/warnings/infos)
+              -h,   --help
+                    (print this short help, see also
http://www.pierrox.net/trip/ for more)

The essential switches are -b, -i, -u : build, install and uninstall a package.
* trip -b : will build a binary package from a source package, while making use
of unionfs to monitor created/modified/deleted files during the "make install"
step. The argument is the path (absolute or relative) to the source package. The
result (unless compilation or installation failure) is a trip binary package
containing created files, and eventually modified files. The binary package is
created into the current directory. (see "Trip package and database format"
below).
* trip -i : will install all files from a binary package (previously built using
trip -b) into the running system. Trip  will refuse to overwrite a file if it
already exists, unless you specify --no-conflict. After a sucessfull install an
entry in the trip package database is added.
* trip -u : remove files previously installed by trip, and also remove the
database entry. Directories belonging to a package are removed only if non
empty.

Additionally trip provides some other usefull switches :
* trip -r rebuild a binary package from the description found in the package
database and files present on the system. Such a reconstructed package may not
be exactly like the one used originally (in case some files have been modified
since the install), but it is a handy way to backup a package before
uninstallation. It will be possible to reinstall it later with trip as a normal
package if needed.
* trip -w will help the user to create a source package, build it and finally
install it. This is a step by step process where questions like package name,
version, source file location, etc. are asked interactively to the user.
* trip -l with no argument gives the list of already installed packages, sorted
by ascending installation date. Trip -l <package name> will give the list of
files belonging to an already installed package (package name can be either
"foo" or "foo-1.2.3"). Trip -l <binary package file> will do the same for trip
binary package. In these both cases the output will be affected by the verbosity
level. At level 3 the output will be the one of "tar --list --verbose". At any
other verbosity level trip will give the list of files, a file by line.
* trip -f will tell the user which package(s) contains the named file.
* trip -bbi build and install a set of packages. The <list> argument is the name
of a file containing the list of packages to build and install. Each line of the
file is a path to a source package, the same way it should be used for "trip
-b". The switch "--no-conflict" can be used in order to ignore potential
conflicts during install. It is not recommended however to use this switch
unless you know that these conflicts can be safely discarded.

Finally the behaviour of trip can be impacted by the use of these switches :
* -c allows the user to specify an alternate configuration directory. This can
be usefull for test purpose or for managing different set of installation.
* -v will set the verbosity level. Default is 0 (quiet). Currently all messages
from trip are thrown on stdout. The output of build and install steps during a
package build is not concerned by -v.
* -t is used to specify the path of the trip schell script. The path of trip is
needed when entering chroot. It should be specified in the trip configuration
file but may also be overriden by this command line option.



--------------------------------------------------------------------------------
Trip internals
--------------------------------------------------------------------------------
The idea of trip is to use two filesystems in a union filesystem. See
http://www.unionfs.org/.
One filesystem is the target filesystem on which the user wants to do "make
install", and is used read-only, the second is an empty filesystem merged with
the first, and used read-write.

Let's take an example fstab :
/dev/hda1      /            ext3   defaults                           1    1
/dev/hdc1      /mnt/pkg     ext3   defaults                           0    0
unionfs        /mnt/union   unionfs dirs=/mnt/pkg:/=ro,noauto         0    0

The result of this is :
- if we read a file in /mnt/union, it will be read from /, if it doesn't
exists in /mnt/pkg
- if we create a file in /mnt/union it will be really created in /mnt/pkg
- if we modify a file in /mnt/union belonging to / it will be copied and
modified in /mnt/pkg
- if we delete a file in /mnt/union belonging to /, a special file will
be created in /mnt/pkg (whiteout mode)
- the overall behaviour of the program executing in /mnt/union is as if there
were only one normal filesystem.

Hence a chroot in /mnt/union gives these benefits : 
- in no case the / filesystem will be altered
- created, deleted and modified files are all stored in /mnt/pkg
- all libraries and tools are available in their real prefix, running
./configure --prefix=/usr is safe, a standard make install is enough.

This principle being expressed, the work of trip is to manage the execution of
build/install steps of a source package, and put all created/modified files in a
tarball, so that they can be copied later into the real filesystem, if
everything goes well.



--------------------------------------------------------------------------------
Trip installation
--------------------------------------------------------------------------------
Installing trip involves two steps.
1) configure the unionfs mount point in /etc/fstab
 - first create mount points : mkdir -p /mnt/pkg /mnt/union
 - then edit your fstab in order to add this two lines :
        /dev/hdc1      /mnt/pkg     ext3   defaults                           0 
  0
        unionfs        /mnt/union   unionfs dirs=/mnt/pkg:/=ro,noauto         0 
  0
Modify /dev/hdc1 according to your system. The option "dirs=/mnt/pkg:/=ro" means
that /mnt/pkg and / are layered, and as / is specified as read-only (ro), all
file modifications will take place into /mnt/pkg.
Finally test your setup :
    mount /mnt/union && ls -l /mnt/union
(you should see the content of / and /mnt/pkg merged)

WARNING : before to build a binary package, trip removes all files in /mnt/pkg.
WARNING : Trip wont work as expected if your system is based on several
filesystems, for example / and /usr. In this case /usr will not be visible
inside /mnt/union and so a lot of things will probably not work. Work is in
progress to drop this limitation.


2) download and install trip (using trip itself of course !) :
    TRIP=trip-0.3
    wget -O - http://www.pierrox.net/trip/$TRIP.tar.gz | tar xvz
    ./$TRIP/src/trip -v 3 -c $PWD/$TRIP/src/conf -t $PWD/$TRIP/src/trip -b $TRIP
    ./$TRIP/src/trip -v 3 -c $PWD/$TRIP/src/conf -i $TRIP-1.noarch.tar.gz
This will dowload trip, construct an installable trip package from the trip
source package, and finally install the package. You can have a look at the list
of installed files using "trip -l trip". You may have to edit
$TRIP/src/conf/conf if you named the mount points other than /mnt/pkg or
/mnt/union.



--------------------------------------------------------------------------------
Using trip to package LFS from the beginning
--------------------------------------------------------------------------------
Trip as been designed from the ground to build a packaged LFS, starting at the
creation of directories till the last package Vim.
LFS-6.2 is used as an example, but the process may be applied to other versions
of LFS, even CLFS (tested on ARM).

* Prerequisites :
Follow the LFS book till the chapter "6.1 Introduction". At this stage the
$LFS/tools directory is ready.

* Trip installation and configuration
When used to build a LFS system, trip requires a special configuration and
directory layout. As trip must be reachable from $LFS the same way it would be
from /, it will be put in $LFS/trip and a symlink will be created into / (the
same way it has previously be done for $LFS/tools). Once the new LFS system is
up and ready, trip can be installed and used as explain in the previous
paragraph "Trip installation". For now use these commands to install trip in a
temporary  way :
    TRIP=trip-0.3
    wget -O - http://www.pierrox.net/trip/$TRIP.tar.gz | tar xvz -C $LFS
    mv $LFS/$TRIP $LFS/trip
    ln -s $LFS/trip /trip
    mkdir /trip/tmp /trip/pkg

This will download trip and copy files into a temporary location into $LFS. The
trip shell script is now located at /trip/src/trip, and the configuration
directory is now located at /trip/src/conf.

Finally edit /trip/src/conf/conf and use these values, where path are adjusted
to your $LFS :
---------- cut after this line -------------
# TRIP configuration file
# TRIP is a TRIvial Packager
# copyright 2006-2007 Pierre Hebert <pierrox at pierrox.net>
# http://www.pierrox.net/trip/


# the real root filesystem, in which resides the operating system
TRIP_FS_ROOT=/mnt/lfs

# the filesystem on which are temporarily created the files during the install
phase
TRIP_FS_PKG=/mnt/pkg

# the union filesystem mergin $TRIP_FS_ROOT(ro) and $TRIP_FS_PKG(rw)
TRIP_FS_UNION=/mnt/union

# where to install binary packages
TRIP_INSTALL_DIR=/mnt/lfs

# installed package db location
TRIP_DB=/mnt/lfs/var/lib/trip

# the mode tell if we use /tools or / as build environment (hosted = build from
/tools, normal=/)
TRIP_MODE=hosted

# the location of the trip shell script
TRIP_PATH=/trip/src/trip

# root location of temporary files
TRIP_TMPDIR=/trip/tmp
---------- cut before this line -------------


* Download trip source packages for LFS
    cd /trip/pkg
    wget -O - http://www.pierrox.net/trip/source-packages-lfs-6.2.tar.gz | tar
xvz

* Prepare virtual filesystem and some essential link to the shell
    mkdir -p $LFS/{dev,proc,sys,bin,tmp}
    mknod -m 600 $LFS/dev/console c 5 1
    mknod -m 666 $LFS/dev/null c 1 3
    ln -s /tools/bin/bash $LFS/bin/bash
    ln -s bash $LFS/bin/sh

* Use trip to build and install packages

  + When using trip during the LFS system building, trip can be executed this
way :
    /trip/src/trip -c /trip/src/conf -b <package>
Alternatively some environments variables can be adjusted to simplify the
command line :
    PATH=$PATH:/trip/src
    export TRIP_CONFIG_DIR=/trip/src/conf
    trip -b <package>
(We will use this second form.)

  + Now that your host system is ready to build the new LFS system using trip,
it is time to start the easy job.
Issue the following command to construct the very first package :
    cd /trip/pkg
    trip -b lfs-base-6.2
It will create a package named "lfs-base-6.2-1.noarch.tar.gz" in the current
directory, containing the standard directory structure. Install it with :
    trip -i lfs-base-6.2-1.noarch.tar.gz

  + Build and install linux-libc-headers-2.6.12.0 and man-pages-2.34 :
    trip -b linux-libc-headers-2.6.12.0 && trip -i
linux-libc-headers-2.6.10-1.i386.tar.gz
    trip -b man-pages-2.34 && trip -i man-pages-2.34-1.noarch.tar.gz
(if you don't build for i386 architecture edit the linux-libc-headers-2.6.12.0
package)

  + Create some essential symlinks
    ln -s /tools/bin/{cat,grep,pwd,stty} $LFS/bin
    ln -s /tools/bin/perl $LFS/usr/bin
    ln -s /tools/lib/libgcc_s.so{,.1} $LFS/lib

  + Build and install glibc-2.3.6
    trip -b glibc-2.3.6 && trip -i glibc-2.3.6-1.i686.tar.gz
(replace i686 with your architecture)

  + Re-adjusting the toolchain
    chroot "$LFS" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
    PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h
    mv /tools/bin/{ld,ld-old}
    mv /tools/$(gcc -dumpmachine)/bin/{ld,ld-old}
    mv /tools/bin/{ld-new,ld}
    ln -s /tools/bin/ld /tools/$(gcc -dumpmachine)/bin/ld
    gcc -dumpspecs | \
    perl -p -e 's@/tools/lib/ld-linux.so.2@/lib/ld-linux.so.2 at g;' \
         -e 's@\*startfile_prefix_spec:\n@$_/usr/lib/ @g;' > \
          `dirname $(gcc --print-libgcc-file-name)`/specs
    exit

  + Build and install all other packages : we will use the
"--batch-build-install" option to automate the process :
    cat << EOF > pkg_list
    binutils-2.16.1
    gcc-4.0.3
    db-4.4.20
    coreutils-5.96
    iana-etc-2.10
    m4-1.4.4
    bison-2.2
    ncurses-5.5
    procps-3.2.6
    sed-4.1.5
    libtool-1.5.22
    perl-5.8.8
    readline-5.1
    zlib-shared-1.2.3
    zlib-static-1.2.3
    autoconf-2.59
    automake-1.9.6
    bash-3.1
    bzip2-1.0.3
    diffutils-2.8.1
    e2fsprogs-1.39
    file-4.17
    findutils-4.2.27
    flex-2.5.33
    grub-0.97
    gawk-3.1.5
    gettext-0.14.5
    grep-2.5.1a
    groff-1.18.1.1
    gzip-1.3.5
    inetutils-1.4.2
    iproute2-2.6.16-060323
    kbd-1.12
    less-394
    make-3.80
    man-db-2.4.3
    mktemp-1.5
    module-init-tools-3.2.2
    patch-2.5.4
    psmisc-22.2
    shadow-4.0.15
    sysklogd-1.4.1
    sysvinit-2.86
    tar-1.15.1
    texinfo-4.8
    udev-096
    util-linux-2.12r
    vim-7.0
    lfs-bootscripts-6.2
    lfs-configuration-6.2
    EOF
    
    trip --no-conflict --batch-build-install pkg_list

Note that we use the "--no-conflict" option because the installation of some
packages overwrite manually created symlinks, like cat,grep,pwd,stty.

  + Final steps
The "shadow" package requires some manual configuration :
    chroot "$LFS" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
    PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h
    pwconv
    grpconv
    rm /etc/{passwd-,group-,shadow-,gshadow-}
    passwd root

You also need to build your kernel. It is possible to package linux, but as you
will probably want to choose your own configuration depending on your hardware,
it is probably easier to keep it under a well known location like
/usr/src/linux.. Follow the LFS instructions about it.

The last two packages lfs-bootscripts-6.2 and lfs-configuration-6.2 contains
some default configurations that you will certainly want to change. So have a 
look at the content of these packages in order to adapt them to your needs. At
this point come back to the LFS book at "7.5. Configuring the setclock Script"
for the last steps. You will in particular need to :
- edit /boot/grum/menu.lst
- edit /etc/fstab
- run grub



--------------------------------------------------------------------------------
Trip package and database format
--------------------------------------------------------------------------------
* Source package
The source package describes how to build and install a software. Its name
should be composed on the <name>-<version> schema, for example "glibc-2.3.6".
Inside the package directory are located two subdirectories : "src" and
"support"
 - src : contains required source files, eventually some patches or default
configuration files. More generally it contains every input materials needed to
build the package.
 - support : contains package meta data, instructions on how to build and
install the source package, and eventually some additionnal scripts :
    - identification : this is a list of variables describing the package 
        TRIP_NAME : for example glibc
        TRIP_VERSION : for example 2.3.6
        TRIP_RELEASE : used to distinguish between two different packages of the
same version
        TRIP_URL : where to find the package
        TRIP_DESCRIPTION : short comment about the package
      As the file "identification" is meant to be sourced by bash, you should
take care of syntax, especially quotes and escape some characters.
    - build.sh : this optionnal shell script should be executable. It contains
instructions on how to build the package (something like tar xzf
$TRIP_NAME-$TRIP_VERSION.tar.gz && cd $TRIP_NAME-$TRIP_VERSION && ./configure &&
make)
    - install.sh : this file is mandatory and should be executable. It contains
instructions on how to install the package (something like make install)
    - {pre,post}_{un,}install.sh : these four files are optionnal and should be
executable. If present, they will be incorporated into the binary package, and
executed during installation of the binary package. They can be used to start a
daemon after installation, or stop it before removal of the package, for
example.

The build.sh and install.sh scripts are executed in a chrooted environment
inside the /mnt/union filesystem. If these two scripts complete sucessfully,
trip will build a binary archive with all created/modified files, the
identification file and {pre,post}_{un,}install.sh scripts if available (see
Binary package below).
When building a binary package from a source package, trip sets some environment
variables that can be used in build.sh and install.sh :
- TMP_BUILD_DIR : temporary location, mainly used to extract and compile a
package. Files created under this directory won't be stored in the binary
package.
- SRC_DIR : points to the src subdirectory of the currently builded source
package.
- TRIP_* : meta-data found in the identification file.

* Binary package
A binary package is a gzipped tar archive containinig the files to install,
meta-data and eventually {pre,post}_{un,}install.sh scripts.
Files are stored in an uncompressed archive named "files.tar". Other files are
stored in a directory named "support". Running "tar tvf
lfs-base-6.2-1.noarch.tar.gz" will show :
drwxr-xr-x root/root         0 2007-07-30 15:18:24 lfs-base-6.2-1.noarch/
drwxr-xr-x root/root         0 2007-07-30 15:18:24
lfs-base-6.2-1.noarch/support/
-rw-r--r-- root/root       162 2007-07-30 15:18:24
lfs-base-6.2-1.noarch/support/identification
-rw-r--r-- root/root     51200 2007-07-30 15:18:24
lfs-base-6.2-1.noarch/files.tar


* Database of installed packages
The location of this database is specified by the TRIP_DB variable in the
configuration file. Its default value is /var/lib/trip.
The list of installed packages is stored in $TRIP_DB/list (one package per
line), sorted by installation date ascending.
Each installed package has a subdirectory of its name in $TRIP_DB. The
subdirectory contains the following files :
- files : contains the list of installed files, a line per file
- files-detail : contains the output of tar --list --verbose. This file may be
used later to compare date/permissions of actually installed files with their
expected state
- identification : contains meta data about the package
- {pre,post}_{un,}install.sh : eventually scripts run before/after install and
uninstall.



--------------------------------------------------------------------------------
Known limitations
--------------------------------------------------------------------------------
 * As we merge two filesystem there is a problem if your linux installation is
based on several filesystem (for example / and /usr), because /usr won't be
visible from /mnt/union. A workaround is scheduled for trip-0.4.

 * trip works only on GNU/Linux, with bash. It has few prerequisites though,
such as common tools like tar, coreutils, grep, etc. and unionfs.



--------------------------------------------------------------------------------
Where to find trip and packages
--------------------------------------------------------------------------------
Trip can be found at http://www.pierrox.net/trip/.
Source packages can be fetched with svn at svn://svn.pierrox.net/trip/lfs-6.2
(please use svn ls before, avoid to download everything since there are actually
more than 500 packages in the repository).



ACKNOWLEDGEMENTS:
  * http://www.linuxfromscratch.org/ (and BLFS too)
  * http://www.unionfs.org/
  * http://www.linuxfromscratch.org/hints/

CHANGELOG:
[2007-07-31]
  * Third revision, major rewrite and new trip version 0.3.

[2006-11-19]
  * Second revision, corrected some minor things.

[2006-05-13]
  * Initial hint.


More information about the hints mailing list