r1065 - trunk

tushar at linuxfromscratch.org tushar at linuxfromscratch.org
Sat Nov 11 15:15:46 PST 2006

Author: tushar
Date: 2006-11-11 16:15:45 -0700 (Sat, 11 Nov 2006)
New Revision: 1065

Added: crablfs

Added: trunk/crablfs.txt
--- trunk/crablfs.txt	                        (rev 0)
+++ trunk/crablfs.txt	2006-11-11 23:15:45 UTC (rev 1065)
@@ -0,0 +1,491 @@
+AUTHOR: ????(Chowroc) <chowroc dot z at gmail dot com>
+DATE: 2006-10-25
+LICENSE: GNU Free Documentation License Version 1.2
+SYNOPSIS: crablfs -- User Based Package Management System
+	Every package and its files belong to a system normal user, and
+a batch script to rebuild or migrate your personalized LFS/BLFS
+system by install all your packages all at once.
+	http://cheeseshop.python.org/packages/source/c/crablfs/
+	python-2.4
+	pexpect-2.1
+(1) Introduction:
+crablfs: User Based Package Management System which is
+written in python. Every package and its files belong to a system
+normal user, and a batch script to rebuild or migrate your
+personalized LFS/BLFS system by install all your packages
+automatically all at once. The goal of "crablfs" is to exert more
+control on packages that compiled from source, and reduce the costs
+of system migration and backup.
+The basic theory is: every package in the system has a corresponding
+user it belongs to, the installation actions(copy, mkdir, etc) will
+run by this user, thus all the files a package generated have the
+mode of this user. At the same time, all those users belongs to a
+'install' group, and the key directories in the system such as
+/usr/local, /usr/local/bin, etc have g+s,o+t mode, and owned to the
+'install' group, to make different packages create their own files
+and directories in those directories.
+There are many benefits to use this method. which can be looked up
+from the "LFS hints":
+More Control and Package Management using Package Users (v1.2)
+Based on this theory, the package manager "userpack" is a command
+line interface like rpm and paco, for finishing some necessary
+operations. It can install, remove, query a package, or list all
+packages of current system, or find the package owner of a specified
+file. the package name($pkgname-$version) will add to the list file
+(/usr/src/packages.list for default).
+Every package user has its own HOME dir, which is default to
+/usr/src/$pkgname. It will stores compressed source archives, all
+patches and supplementary files for installation, and installation
+profiles which contains the information of the package, such as
+user/group, source archive, patches, create time and those commands
+to build this package.
+By default, "userpack" use interactive mode, which will start a
+shell like command line, in which you type "cmd $command" to execute
+the corresponding command, and those commands will been recorded to
+"installation profile"(described before). This profile can be used
+to execute installation commands automatically in noninteractive
+The script "crablfs" is a batch script write in python, it is
+another ALFS instance in other words, which make use of the
+automated ability of "userpack". It can use the list
+file(/usr/src/packages.list) mentioned before, to operate the
+compressed source archives and installation profiles you appointed
+automatically. And you can also specified a directory that has the
+same structure as $homepre(/usr/src) to build a ALFS system
+automatically. This means that you can migrate from an old system to
+new ones more easily -- and rebuild your personalized system more
+The essential differences among Linux distributions is the
+differences of the package management ways. As a distribution, LFS
+actually does not has a mature package manager itself, this of
+course roots to the flexibility of LFS, but for daily use, a mature
+package management system is necessary. It is recommended to use
+User Based Management in the BLFS document, but the LFS hints only
+give the basic principles which is very creative, but the shell
+scripts is not enough to form a self-consistency system. So I write
+one regardless of unveil my superficialness, a hope that my crude
+remarks may draw forth by abler people.
+Of course crablfs is not only used for LFS, it aims to those
+packages that build from source code, so it can used on other
+distributions, too.
+(2) Install crablfs
+To make sure that crablfs also under the control of package
+management, to install by this way:
+# export PYTHONPATH=/opt/lib/python2.4/site-packages
+// sys.path will not be influenced by PYTHONPATH
+# tar xfz pexpect-2.1.tar.gz
+# cd pexpect-2.1
+# python setup.py install --prefix=/opt
+# cd ..
+# tar xfz crablfs-0.1.tar.gz
+# cd crablfs-0.1
+# vi userpack.dirs
+// define all key directory
+// these dirs will belong to install group??and have g+s,o+t mode
+// Adjust it for you own needs
+# python setup.py install --prefix=/opt
+# cd ..
+# /opt/bin/userpack init
+# /opt/bin/userpack install -f /tmp/crablfs-0.1.tar.gz crablfs-0.1
+crablfs> cmd tar xfz crablfs-0.1.tar.gz
+crablfs> cmd cd crablfs-0.1
+crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
+crablfs> cmd cd ..
+crablfs> cmd rm -rf crablfs-0.1
+crablfs> commit
+# userpack install -f /tmp/pexpect-2.1.tar.gz pexpect-2.1
+crablfs> cmd tar xfz pexpect-2.1.tar.gz
+crablfs> cmd cd pexpect-2.1
+crablfs> cmd python setup.py install
+crablfs> cmd cd ..
+crablfs> cmd rm -rf pexpect-2.1
+crablfs> commit
+(3) Package Management:
+manager: userpack
+a. install a package:
+# userpack install $pkgname-$version
+for example:
+# userpack install rxvt-2.7.10
+you can classify:
+# userpack install meida.mplayer-1.0pre8
+because the system support '.' for user name.
+* It is not true for LFS system,the shadow package that LFS use
+only support [-a-z0-9_] chars, so I wrote a shadow module to access
+/etc/passwd, /etc/group, /etc/shadow, /etc/gshadow directly.
+*. The classification function have not been tested strictly.
+Under the conditions above, the default action will taken, that is:
+the compressed source archive and patches are considered to have
+been in the dir of HOME(/usr/src/$pkgname), other will you should
+specify it with argument:
+# userpack install -f /mnt/file/packages/rxvt-2.7.10.tar.gz rxvt-2.7.10
+to make those archive and patches to be copied to HOME.
+This command will start an interactive command line, it is simple
+and only several command. In fact it is like a shell, you can type
+any 'shell' command to install the package, only need add 'cmd'
+before the command to make the 'userpack' records the commands.
+These commands will be stored into HOME/.config(installation
+crablfs> cmd tar xfz rxvt-2.7.10.tar.gz
+crablfs> cmd cd rxvt-2.7.10
+crablfs> cmd ./configure
+crablfs> cmd make
+crablfs> cmd make install
+crablfs> cmd cd ..
+crablfs> cmd rm -rf rxvt-2.7.10
+crablfs> commit
+You should known that the current user is the package user, not
+root; and current working directory is the user's HOME, so there may
+be ownership problem(What is the purpose of User Based Package
+Management System).
+When you type commands, you can use 'list' to see what you have typed:
+crablfs> list
+0, tar xfz rxvt-2.7.10.tar.gz
+1, cd rxvt-2.7.10
+2, ./configure
+3, make
+4, make install
+Use 'rollback' to clean up these, and 'del N' to delete specified
+You must 'commit/cmt' at last, thus all the commands will be
+recorded to the .config, the $pkgname-$version couple will been add
+to list file(/usr/src/packages.list for default), to indicate that
+the package has been installed. If there is a command error makes
+the installation failed, and the fault can't been clean up by
+yourself, just type 'quit' and the installation terminated, but the
+archive and patches will still be copied to HOME.
+If there is any other supplementary files:
+# userpack install -f /mnt/file/packages/MPlayer-1.0per8.tar.bz2 \
+    -p /mnt/file/packages/all-20060611.tar.bz2 \
+    -p /mnt/file/packages/Blue-1.6.tar.bz2 mplayer-1.0pre8
+// all is mplayer's codecs??Blue is GUI Skin.
+If you have the installation profile before, you can use -a|--auto
+to install automatically:
+# userpack install -a rxvt-2.7.10
+This means the HOME/.config exists, otherwise use -c|--profile:
+# userpack install -ac /mnt/packages/confiles/rxvt-2.7.10 rxvt-2.7.10
+If you have burned /usr/src to DCD-R's sources dir, you can also
+execute the automatically installation below:
+# userpack install -as /mnt/dvdrom/sources rxvt-2.7.10
+Moreover, the sources could also on the net such as ftp or ssh:
+the 2 above is same,
+for example:
+# userpack install \
+    -f mlterm-2.9.3
+If the package name(mlterm-2.9.3) is not given, it can be parsed
+from the archive name(mlterm-2.9.3.tar.gz) automatically by
+If a archive has problem and makes installation failed, the archive
+still has been copied to HOME, and the next time 'userpack' will
+check whether the archive exists in HOME, if true, copy will not
+occur. To change this, use -C|--copy-force:
+# userpack install -Caf libesmtp-1.0.3r1.tar.bz2
+About the user/group ID, 'userpack' can define its own range,
+default from 1000 to 20000, U can define in /etc/login.defs:
+PACK_UID_MIN    10000
+PACK_UID_MAX    25000
+PACK_GID_MIN    10000
+PACK_GID_MAX    25000
+b. remove:
+# userpack remove rxvt-2.7.10
+c. list all packages under the control:
+# userpack packs
+d. query a file's package owner:
+# userpack owner /usr/local/bin/mlterm
+e. list all files a package contains:
+# userpack files pexpect-2.1
+(4) About the pkgname & version rule:
+This is the argument to
+'userpack', the form is '$pkgname-$version', the python re pattern
+That is '$pkgname' part is 'part1[-part2][-part3][...]', every part
+could contains '[-a-zA-Z0-9_]', no rules for start char, but it is
+recommended to use alphabet; when meet '-[0-9][a-zA-Z0-9_]\.'
+pattern('-' follows a string start with digit, and follows a '.'),
+this will be considered as start part of $version, $version can also
+contains other '-', and when meet a string not start with digit,
+$version ends. chars follows are ignored.
+(5) ALFS
+It is mentioned before that all commands will be recorded to
+HOME/.config, and 'userpack' can also use this commands to run
+automatically, so there could be a batch script to use packages.list
+to install those packages all at once. Based on this, we can get an
+instance of ALFS.
+The target of the instance is extended to tranditional nALFS and
+jhalfs, for more personalized environment, for example: The first
+time, I build LFS/BLFS step by step, and the packages installed,
+related with their own's information & operations are recorded to
+the profiles, the next time, I can use those profiles to build the
+software envrionment I need with a 'turn key' way, exactly as those
+before. This means I can migrate many times, and others can also use
+my profiles to build their own personalized system. The end target
+also contains a platform independent operation.
+This batch script is 'crablfs'.
+So far, it is only tested under the condition that the LFS base
+system has been built, to use 'crablfs' to build a bunch of BLFS
+packages, so it is the future to make the LFS base system under the
+control of package management system.
+At first, adjust the tool chain:
+# cd /blfs-sources
+# tar xfj python-2.4.1.tar.bz2
+# cd python-2.4.1
+# patch -Np1 -i ../python-2.4.1-gdbm-1.patch
+# ./configure --prefix=/opt --enable-shared
+# make && make install
+# cd ..
+# rm -rf python-2.4.1
+// U'd better delete it, otherwise may be problems
+# export PYTHONPATH=/opt/lib/python2.4/site-packages
+# tar xfz pexpect-2.1.tar.gz
+# cd pexpect-2.1
+# /opt/bin/python setup.py install --prefix=/opt
+# cd ..
+# rm -rf pexpect-2.1
+# tar xfz crablfs-0.1.tar.gz
+# cd crablfs-0.1
+# cp userpack.dirs.blfs userpack.dirs
+// Adjust for you own need and system
+// Whenever U meet pemission problem, just adjust /etc/userpack.dirs
+//  and run 'userpack init'
+# /opt/bin/python setup.py install --prefix=/opt
+# /opt/bin/userpack init
+# /opt/bin/userpack install -f python-2.4.1.tar.bz2 -p
+# python-2.4.1-gdbm-1.patch python-2.4.1
+crablfs> cmd tar xfj python-2.4.1.tar.bz2
+crablfs> cmd cd python-2.4.1
+crablfs> cmd ./configure --prefix=/usr --enable-shared
+crablfs> cmd make
+crablfs> cmd make install
+crablfs> cmd cd ..
+crablfs> cmd rm -rf python-2.4.1
+crablfs> commit
+# /opt/bin/userpack install -f crablfs-0.1.tar.gz crablfs-0.1
+crablfs> cmd tar xfz crablfs-0.1.tar.gz
+crablfs> cmd cd crablfs-0.1
+crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
+crablfs> cmd cd ..
+crablfs> cmd rm -rf crablfs-0.1
+crablfs> commit
+# userpack install -f pexpect-2.1.tar.gz pexpect-2.1
+crablfs> cmd tar xfz pexpect-2.1.tar.gz
+crablfs> cmd cd pexpect-2.1
+crablfs> cmd python setup.py install
+crablfs> cmd cd ..
+crablfs> cmd rm -rf pexpect-2.1
+crablfs> commit
+# userpack packs
+And use the same way, I installed some packages:
+# userpack packs
+# cat /usr/src/packages.list
+# mv /usr/src/packages.list /blfs-sources/
+# config.copy /usr/src
+// This will generate a profiles dir in current dir,
+// which contains all the profiles from $homepre(/usr/src)
+// This shell script will not be installed,
+// you can find it in the archive
+# mv profiles /blfs-sources
+# crablfs -t alfs \
+    -C /blfs-sources/profiles/ \
+    -F /blfs-sources/ \
+    /blfs-sources/packages.list
+If you have installed such a BLFS system and burned /usr/src all to
+the DVD-R, you can do this:
+# crablfs -t crablfs[/default] \
+    -s /mnt/dvdrom/sources/ \
+    /mnt/dvdrom/sources/packages.list
+crablfs will record current successfully installed package name to
+/var/log/crablfs/.mark. Because of the packages dependencies
+problems, if one package failed, the 'crablfs' process terminated.
+After you clean up the problem, you can continue from the
+So if .mark point to the last item of packages.list, 'crablfs' will
+do nothing. Adjust it for you need.
+Moreover, 'crablfs' will check packages.list, illegal package name
+will terminate 'crablfs' and do nothing; if the name in .mark not in
+packages.list, you will get a "x not in list" error report, you can
+adjust corresponding, or just delete the .mark file.
+So far, there are few permission problems, and all can avoid by
+changing the installtion arguments. The wrappers for chmod, chgrp,
+chown, install may be needed, but so far I have not see the
+neccessary. To dicide this for further using and testing.
+(6) The Next Plans
+* BLFS crablfs tool chain adjusting shell script
+* Add upgrade part
+* Add gettext internationlized support
+* Whole infomations show of a package
+* PyUnit
+* Enganced cmdline completion function
+* Add code for cmdline to read .config history commands
+* A command editor?
+* File lock of packages.list
+* Change .config to the form of '$pkgname-$version'
+* Enganced platform independent support:
+    Add var and env var to cmdline for different $ARCH & $LOCALE
+* Support multi installation sources
+* Enganced classification mechanism
+* Intergrated checking
+* Enganced configuration getting & setting: Plain Tree & XML
+* Consider package dependencies problems
+* Better documents: internal, design doc and manual
+	Matthias S. Benkmann built the original principles
+	crablfs-0.1 has been distributed at 2006-10-20, so I write this

More information about the hints mailing list