| PyRPM Tool descriptions |
|
In order to understand what each of the tools really does this section describes the behaviour of the tools in more detail for each tool.
There are currently 4 basic tests in pyrpmcheck. The first checks for all packages if any two dirs differ in user/group/mode. The second check looks for duplicate provides, meaning if 2 rpms provide the same thing. The third test looks for dangling symlinks, that is, symlinks that point to nowhere. The last check simply checks if any package names appear in the various given dirs. This check is only useful for FC and FC-Extras, as there should be no overlap there.
Check some common problems with packages and interdependencies for Fedora Core Devel:
pyrpmcheck /mirror/fedora/development/i386/os/Packages |
In order to test problems with installs or removals this tool can be used to test a whole repository against missing prereqs, failing scripts and other common problems. It does so by iterating over all packages of the given repos and installing and removing them just like a yum install <package>/yum remove <package> would have done. Doing this in an empty buildroot allows one to find missing dependencies and other script failures easily.
Check a complete FC4 repository:
pyrpmcheckinstall -y -v -c fc4-i386.conf -r /mnt/build/test &> /tmp/installtest |
To only get the errors and have a minimal log make it quiet:
pyrpmcheckinstall -y -q -c fc4-i386.conf -r /mnt/build/test &> /tmp/installtest |
To work on a directory use this:
DEST=/mnt/build/test for d in dev dev/shm dev/pts proc sys ; do mkdir -p $DEST/$d mount --bind /$d $DEST/$d done mkdir -p $DEST/etc touch $DEST/etc/fstab pyrpmcheckinstall -y -c /dev/null -r $DEST /mirror/fedora/development/i386/os/Packages &> /tmp/installtest for d in sys proc dev/pts dev/shm dev ; do umount $DEST/$d done # grep for "ERROR" in the logs |
If you have a multiple CPUs and a lot of ram you can run the whole test in a tmpfs and store the restults in logfiles like this:
TOP=/mnt/build/tmpfs
NAME=fc-extras-devel-i386
DEST=$TOP/$NAME
NUMCPU=8
MEM=7G
mkdir -p $TOP
mount -t tmpfs -o size=$MEM none $TOP/
for i in `seq 0 $NUMCPU`; do
D=$DEST.$i
for d in dev dev/shm dev/pts proc sys ; do
mkdir -p $D/$d
mount --bind /$d $D/$d
done
mkdir -p $D/etc
touch $D/etc/fstab
done
pyrpmcheckinstall --ignoresize -y -v -c /root/$NAME.conf -r $DEST numcpu=$NUMCPU logfile=/var/tmp/$NAME.log
for i in `seq 0 $NUMCPU`; do
killall pyrpmcheckinstall
sleep 1
done
for i in `seq 0 $NUMCPU`; do
D=$DEST.$i
for d in sys proc dev/pts dev/shm dev ; do
umount $D/$d
done
done
umount $TOP
# grep for "Error" in the logs
|
This tool should actually be renamed (and will so soon) to pyrpmcheckinstall as that's what it does. What the tool does it to simulate an installation (just like you could do with pyrpmyum --test) and then simulates an update to the second set of rpms or repository. A typical test would be to use a FC-3 or FC-4 yum.conf repo as a install base and then use a FC-devel repo or rpm tree as the update tree. The nice thing about the simulation is that it doesn't actually perform the installation and so doesn't need any disk space. Additionally you can use the (undocumented ;)) option --arch where you can specify any arch you like. The host arch doesn't matter for the simulation as the whole code "thinks" it runs under arch. So simulated installs and updates for all kinds of archs can be performed this way on one single machine.
Check if an update from Fedora Core 4 to Fedora Core Devel works or what problems come up:
pyrpmcheckrepo -r /mnt/foo install /mirror/fedora/4/i386/os/Fedora/RPMS update /mirror/fedora/development/i386/os/Packages |
Using the original createrepo as a basis we integrated our pyrpm code into it instead of using rpmlib.
Create a repo for FC-4:
pyrpmcreaterepo /mirror/fedora/4/i386/os/ |
Converts one rpm database to another. The 2 currently supported databases are rpmdb (DB4 based) and a SQLite rpm database. It can be used to either convert a rpm database from one format to another or to recreate a rpm database.
Convert your system rpmdb into a sqlite database:
pyrpmconvertdb /var/lib/rpm sqlitedb://var/lib/rpm.sqlite |
Rebuild your database:
pyrpmconvertdb /var/lib/rpm /var/lib/rpm.rebuild |
This tool actually really reads all DB4 files in /var/lib/rpm (or, if you use the --dbpath option, in that directory) and tries to cross check the entries with the rpm header that is found in Packages. The pyrpm-devel documentation contains a lot more detail about the structure of rpmdb. Simply put we check if for every entry in every file there is a corresponding entry in Packages and vice versa. There are some exceptions and special cases, but all known ones are handled correctly. The one check that is very unreliable is the SHA1 header checksum tests as this test has to rely on a very specific ordering of the rpm binary header and about the tags that appear in the original rpm. If those assumptions don't hold this checksum can't be computed correctly. Additional binary packages or a yum config file with yum repositories can be specified to test the rpmdb headers against those packages for additional verification.
Link to a page with more information on rpmdb repair information is available at http://www.rpm.org/hintskinks/repairdb/.
Check the rpmdb of the current system with internal cross checks
pyrpmdbverify |
Same, but this time we use external rpms to verify against, too
pyrpmdbverify /mirror/fedora/development/i386/os/Packages |
Verify our rpm database in our FC-4 buildroot using external rpms as well
pyrpmdbverify --dbpath /mnt/build/fc4-i386 /mirror/fedora/4/i386/os/Fedora/RPMS |
Verify our rpm database in our FC-4 buildroot using external rpms using a yum repository:
pyrpmdbverify -c fc4-i386.conf --dbpath /mnt/build/fc4-i386 |
Simply put this tool allows you to install, update and erase rpms just like you would with rpm itself. It supports most of the common options from rpm and is mostly using the same syntax, too. As it's "only" rpm though there is no depresolver in this tool, so if you have any missing dependencies you'll have to fiddle them out yourself, just as you had to do with rpm.
Usage: pyrpmkickstart <options> <kickstartfile> [[<disk name>:]<disk image>|<disk device>]*
This tool can install Red Hat Enterprise Linux (>=3) or Fedora Core distributions on hard disks, partitions or disk images. pyrpmkickstart uses a kickstart file for setup and installation of the system. Here is a very simple kickstart file:
lang en_US langsupport en_US.UTF-8 keyboard us mouse timezone EST rootpw test install nfs --server=server --dir=/dist/fedora/core/4/i386/os #url --url=file:///dist/fedora/core/4/i386/os bootloader --location=mbr #zerombr yes #clearpart --all --initlabel partition swap --size=100 partition /boot --fstype=ext3 --size=100 partition / --fstype=ext3 --size=1 --grow auth --useshadow --enablemd5 network --bootproto=dhcp --device=eth0 firewall --enabled --ssh selinux --enforcing %packages --resolvedeps @base @x |
You can create kickstart files with "redhat-config-kickstart" or "system-config-kickstart" according to your needs.
In the kickstart example, pyrpmkickstart will create new partitions in an IDE hard disk or disk image. If you want to use existing partitions, then append "—usepart=/dev/hdaX" to the partition entry. If a partition should not be formatted, then add "—noformat".
The source has to be a installable tree. It is not possible to install from iso images. There are currently two supported installation methods: NFS install and url-file install. Contrary to the kickstart documentation, it is possible to install from a local installation tree with "url —url=file://path", where path is a full path name with a leading slash.
The destination is either a hard disk, a partition on a hard disk or a complete disk image (usable e.g. with XEN, QEMU, VMware). If you are using a hard disk, please be careful with "zerombr" and "clearpart" in the kickstart file. These will remove some or all partitions on this hard drive according to the usage. Do not use them, if you want to install on a single partition on your hard drive. You would have to install the bootloader into the root partition of your new install as well: "bootloader —location=partition".
A disk image is similar to a real hard disk for the installer. It assumes, that it is a IDE hard disk and maps it to hda. Qemu emulates a disk image as a IDE hard drive internally, so you can use it as is. If you want to use sda instead, you have to map the disk image with "sda:disk.img". This is essential for installation in vmware disk images, because vmware defaults to SCSI disks. Please keep in mind that you have to preallocate the hard disk in vmware to be usable with pyrpmkickstart. Here is the command to easily generate a 2G disk image:
dd if=/dev/zero of=disk2G.img bs=1M count=2048
Disk image installations are usable by XEN, QEMU and VMware. Please remember to increase loop device amount for large disk image installs in /etc/modprobe.conf:
options loop max_loop=64
Here is the command to install into the disk image "disk2G.img" with the kickstart file "ks.cfg":
pyrpmkickstart ks.cfg disk2G.img
Here is the command to use a disk image with QEMU.
qemu -m 256 disk2G.img
The default virtual memory size in qemu is 128M, therefore it is advisable to increase it.
Say you have just completed a normal kickstart installation with qemu. Create a second IDE disk and check if your kickstart file can be used to install onto the second IDE disk with:
pyrpmkickstart --no-stage2 ks.cfg hda:/dev/hdb
pyrpmkickstart first downloads all rpm packages before the installation begins. This can be changed with the option "—no-cache".
pyrpmkickstart now has SELinux support. You can install a SELinux guest even if your host system has no SELinux support. This is done via a autrelabeling on first boot of the guest.
Fedora Core 7 and later:
If you need ata disk drive support, you have to add a device entry for the driver to the kickstart file:
device scsi ata_piix
This is our main stress test tool. With it you can run literally thousands of update and erase operations in a buildroot and after several days scan the output for any problems. It works simply by alternating between trying to randomly install one rpm package (including needed dependent rpms) and randomly removing one package. There are several special options which help circumvent some strange problems. The first is --servicehack which simply changes /sbin/service to "exit 0" as especially some of the postuninstall scripts really misbehave otherwise and you don't need any services started or stopped in a buildroot anyway for testing. The second is the --autoerase option which I'll describe a little more in detail for pyrpmyum.
Run 10000 install and erase operations in the /mnt/build/test buildroot and save the resulting output in /tmp/stresstest
pyrpmrandomizer --servicehack -y -v -c fc3-updates-i386.conf -r /mnt/build/test 10000 &>/tmp/stresstest |
Ever had to use rpm --rebuilddb? Well, here is our version of it. It tries to do the same thing as rpm where it takes /var/lib/rpm/Packages and rebuilds the database with that information. The default path for the new db is ./rpmdb so that your original one won't be overwritten. Currently /var/lib/rpm/Pubkeys is not yet rebuild.
As we often have to work with specfiles, too, we've written a small tool with which we can extract the different sections from a specfile in order to use that info via pipe in other tools.
A tool to verify installed rpm packages. Command-line arguments specify packages to verify. When no arguments are specified, all installed packages are verified.
With the —diff option pyrpmverify attempts to download the original packages and outputs diffs of the changed files. Changes will be shown for all nonempty regular files which are marked as %config.
Some packages have content verification explicitly disabled for some files (e. g. /etc/nsswitch.conf in FC5 glibc-2.4-4), you can use the —verifyallconfig option to force verification (and diff output) for such files.
Multilib installations are currently untested and SELinux context testing depends on https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188106.
Our main workhorse and real puppy. This tool, as the name already suggests is basically a yum replacement of sorts. It uses the same command line options as yum and uses the same config files, too. It contains a complete depresolver and allows complex installs, updates and erases from systems. We have used it to install FC1 and then step by step update to FC1-updates, FC2, FC2-updates, FC3, FC3-updates and finally FC4. Some of the updates need the --autoerase option though because some dependencies just can't be met and packages need to be removed in order to fulfill them. The autoerase option tries exactly that by semi-intelligently selecting the package that needs to be removed in order for the update to work properly without user intervention. For the described update chain only about 6-7 packages were deleted up to FC3, and all of them would have to be removed manually anyway. But a word of warning here: The autoerase option might run amok if things go really wrong or you try to update to a completely broken tree and would then try to remove almost all of your packages. This has never happened for real trees for us, but just imagine a broken repository and things could go really wrong. That's why this option isn't directly listed in the help and should only be used for a --test run or without the -y option where you get listed separately from the "normal" transaction the packages that pyrpmyum would autoerase for you.
Install a full Fedora Core Development tree in /mnt/build/fc-devel-i386 buildroot:
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 install "*" |
Update kdebase in /mnt/build/fc-devel-i386 buildroot to latest version, resolving all dependencies:
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase |
Same, only if we get some unresolvable problems try to remove packages that don't fit automatically:
pyrpmyum -y -v --autoerase -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase |
Make a dry-run of the above command to be sure that not everything gets automatically deleted :)
pyrpmyum -y -v --test --autoerase -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase |
Remove glibc and all depending packages from our buildroot :)
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 remove glibc |
This script does not depend on the rest of the pyrpm module, but is all copied into one file. It is not as complete as the new pyrpm code, but can still be used for many use cases. oldpyrpm.py only depends on python-2.2 or newer and you can use parts of it even if one of the addon python modules are not available. Also feel free to check the script for more hidden usage cases.
You can list the changes between two source packages. If you specify --explode it tries to also show the changes within *.tar.gz files if they have changed.
oldpyrpm.py [--explode] --diff 1.src.rpm 2.src.rpm |
You can extract source or normal binary packages. This always overwrites existing files and does not start any scripts in binary packages, so it is often only used for source packages:
oldpyrpm.py [--buildroot=/chroot] --extract *.rpm |
You can check source rpm packages and mark the ones which are excluded from building on certain architectures.
oldpyrpm.py --checkarch /mirror/fedora/development/source/SRPMS |
You can check the rpm database rpmdb. The complete information that rpm needs is included in /var/lib/rpm/Packages and the opengpg data in /var/lib/rpm/Pubkeys. All other db4 files in the rpmdb directory contain duplicate data from Packages to speed up information lookup.
This is how the rpmdb is being checked:
Verify all data from /var/lib/rpm/Packages is the same as in all other db4 files (two-way check).
Most of the rpm header data does have a sha1 checksum. For most newer rpm packages in rpmdb we can try to verify that sha1 to be correct.
If scripts fail often duplicate rpm packages stay installed. Apart from kernel packages and a few other special packages, you get a warning about all duplicated packages.
Check all dependency information for Requires:, Provides:, Conflicts: and Obsoletes: is correctly matching up and if the installed rpm packages do have any fileconflicts.
If you specify a yum configuration file, all installed files are checked against the yum repositories. Which rpm packages do not match, which installed packages are not available in one of the repos. You can specify more than on yum config file by repeating the -c option.
If installed rpms are compared against a repository, then a check is done if pyrpm would also write the same rpmdb entry to compare if our functions work the same as original /bin/rpm ones.
Only rpmdb files in /var/lib/rpm are accessed read-only, no actual installed files from the rpm packages are verified. If checks against repositories are done, those are also read-only and all tests can be run as non-root user.
The check can be run on any architecture for any other architecture with automatic detection of the right endian setting of the db files as well as automatic arch setting by looking at the installed kernels in the rpmdb. So you can copy the rpmdb files onto a central server to check them all.
Check if only packages with the correct architecture are installed. Make sure no arch-dependent rpm and a noarch package are installed at the same time. Warn if we have more than one package installed for any given name/arch pair (with the exception of "installonly" packages like the kernel). Also warn if we have more than one package with the same name that has different epoch/version/release. This is checking multilib (also called compatarch) installations for some additional sanity.
-q and --quiet output less progress information. -v or --verbose add more output information. --rpmdbpath /var/lib/rpm/ can specify the directory which contains the database files. —nofileconflicts disables the checks for fileconflicts.
For special setups you might want to specify --releasever 4, --arch i686 or --buildroot /mnt/build (buildroot is actually only used to read the yum config files.
Example usages to check your rpm database rpmdb:
oldpyrpm.py [--verbose|-v|--quiet|-q] [--rpmdbpath=/var/lib/rpm/] --checkrpmdb oldpyrpm.py --checkrpmdb [--enablerepos] [--fileconflicts] [-v] |
You can generate repo metadata like createrepo does. You need to copy the comps.xml file yourself into the repodata directory if you need one. Also xml pretty printing is hardcoded (-p option for upstream createrepo). Createrepo files are compatible with the output from createrepo-0.4.3.
cp comps.xml /mirror/fedora/development/i386/repodata/comps.xml oldpyrpm.py --createrepo /mirror/fedora/development/i386/ |
Per default oldpyrpm.py is only reading in rpm packages and doing some sanity checks on them. It can read in rpm packages from Red Hat Linux 5.2 and newer. The pyrpmyum code only works on RHEL3 or newer due to how missing epoch are treated in dependency information. Here some options that can be useful:
If you specify a directory, automatically all files in that dir are read.
--nodigest skips verifying the md5 checksum of the header and payload data and the sha1 checksum of the header data.
--nopayload skips reading in the cpio data of all files (called payload).
Add -c /etc/yum.conf to read in yum repositories.
--strict is adding checks which are only useful for the Fedora Core development tree.
--verbose or -v increases output logging information. This is nearly unused in the current code, so doesn't really change anything.
--quiet or -q sets the verbose level to 0.
Specify a certain architecture via --arch i686. Default value is detected via uname(2).
To override the release version information add --releasever 4. The default is the version number of the installed redhat-release or fedora-release package (used rpm packages can be specified via --distroverpkg="redhat-release fedora-release").
-y to never prompt for yes/no questions. This is unused until now.
--noverify is skipping the special debug tests and then mainly all rpm packages are only read in. If also --small is specified, not all rpm tags are read in, only the ones in importanttags.
Specify a cache directory to hold data from yum repositories via --cachedir ~/.pyrpm/cache/.
Dependency and fileconflict checking is done if either --strict or --checkdeps is added. --nofileconflicts is disabling fileconflict checking.
--runorderer is ordering the set of rpms according to their dependencies in a way they should get installed.
Example usages:
find /mirror/ -name "*.rpm" -type f -print0 2>/dev/null | xargs -0 oldpyrpm.py --nodigest --nopayload
locate '*.rpm' | xargs oldpyrpm.py --nodigest --nopayload
oldpyrpm.py --strict --nodigest --nopayload --fileconflicts --completerepo /mirror/fedora/development/{i386,source}
|
Only read in the needed rpm tags and do dependency checking without further debug checks. This shows the full speed of the current implementation and finishes in under 14 seconds for FC-development on some laptop:
time oldpyrpm.py --checkdeps --small --nodigest --nopayload --noverify /mirror/fedora/development/i386/os/Packages time oldpyrpm.py --checkdeps --small --nodigest --nopayload --noverify /mirror/fedora/4/i386/os/Fedora/RPMS /mirror/fedora/updates/4/i386 /mirror/fedora-extras/4/i386 |
Last updated 2008-02-18 08:02:44 CEST