DCC/Debian cluster FAQ

To add a user to the cluster use the dcc_useradd(8) script, for example:

# dcc_useradd -m bob

To delete a user from the cluster use the cpu(8) command, for example:

# cpu userdel -r bob

To change the user's password, use the passwd(1) command:

# passwd bob
bob@cluster$ passwd

To change the default shell for the user, use the chsh(1) command:

# chsh bob
bob@cluster$ chsh

To change the Geckos field for the user, use the chfn(1) command:

# chfn bob
bob@cluster$ chfn

First enter the work-node image:

# dcc_editimage node-image

CHROOT# apt-get install libpvm3

CHROOT# ./configure && make && make install

# cpushimage node_image

	If upgrading/installing a daemon
	If you are installing/upgrading a package that provides a server daemon (which is started from /etc/init.d/...), take care that you restart the relevant service on the actual node after the image is pushed to it. It is done like this: # cexec /etc/init.d/service-name stop # cexec /etc/init.d/service-name start Of course, change 'service-name' to the desired service you are to restart.

Install the kernel from the package:

CHROOT# apt-get install kernel-image-2.4-686-smp

CHROOT/usr/src/linux-2.4.28# make menuconfig
CHROOT/usr/src/linux-2.4.28# make dep
CHROOT/usr/src/linux-2.4.28# make bzImage modules
CHROOT/usr/src/linux-2.4.28# make modules_install
CHROOT/usr/src/linux-2.4.28# cp arch/i386/boot/bzImage /boot/vmlinuz-2.4.28
CHROOT/usr/src/linux-2.4.28# cp System.map /boot/System.map-2.4.28

# mksidisk -A --file /etc/dcc/disktable --name image_name

# cpushimage node_image

In the default installation both MPICH and LAM are installed. MPICH is used by default. To temporarily switch to LAM use mpirun.lam, mpicc.lam etc. To permanently switch to LAM use:

# update-alternatives --config mpi
# update-alternatives --config mpirun

To submit a job to the queueing system use qsub(1). If you want an interactive session, use it like this:

$ qsub -I

$ qsub /home/user/testjob.sh

$ qsub -l nodes=2:ppn=2 /home/user/testjob.sh

As you have successfully submitted your job to the queueing system, you are able to monitor it's status. It is done either with qstat(1), pbstop or pestat commands.

Additionally, you can monitor overall cluster performance at your Ganglia cluster monitor web interface. Your local Ganglia URL is http://your.cluster.fqdn/ganglia

For information on fine-tuning your queueing-system configuration please refer to the TORQUE Admin Manual

If you want to change disk partition configuration on the work-nodes, first change the settings in /etc/dcc/disktable so that it reflects your desired partitioning scheme. After that, you have to issue two commands for the changes to take effect:

mksidisk -A --name node --file /etc/dcc/disktable
mkautoinstallscript --image node --force --ip-assignment dhcp --post-install reboot

To change the front-node's external hostname after installation, you have to update following configuration files on the front-node: /etc/hosts , /etc/hostname , /etc/network/interfaces , /etc/torque/server_name , /etc/c3.conf and /etc/gmond.conf. You will also want to reconfigure your MTA, which can be done like this (if you're using exim4):

# dpkg-reconfigure exim4-config

If the jobs spanning multiple work-nodes won't start try running the following two commands on the front-node:

# cpushimage image_name
# /etc/init.d/torque-server restart-quick
# cexec /etc/init.d/torque-mom restart-quick

1. dcc_discovernode creates node1

2. node1 gets installed

3. node1 reboots

4. dcc_discovernode creates node2

5. node2 gets installed

6. node2 reboots

As you start working on DCC clusters you may notice that root password is not set on the work-nodes, root's SSH key doesn't exist on the work-nodes etc. We decided on this because it is not safe to put sensitive information into the images. Images are served anonymously via rsync(1) so any cluster user can retrieve sensitive data from the image:

bob@cluster$ rsync -v rsync://node0/image_name/etc/shadow .
shadow

sent 91 bytes  received 784 bytes  1750.00 bytes/sec
total size is 679  speedup is 0.78

debconf-dcc package is installed first. It will ask you (through debconf) to enter the names of front-end's internal and external interface. It will deduce other information from these answers (using ifconfig(8) and friends). As a result, /etc/dcc/config and /etc/dcc/debconf are created. /etc/dcc/config is the main DCC configuration file, and /etc/dcc/debconf contains a debconf database with preloaded answers for packages DCC depends on. This way when dcc-front and its dependencies (like slapd, ssh etc.) are installed they won't ask you for any questions. Hence you have a silent install. Moreover, dcc-front depends on a series of packages whose only purpose is to configure existing Debian packages. slapd-dcc will, for example, configure LDAP server to be used as an user database for the cluster. For details look at the postinst scripts of those packages.

DCC installation in the work-node image is similar. After the debootstrap builds the base system, /etc/dcc/config is copied into the image. When debconf-dcc is installed in the image it will use this file instead of asking questions. Just like dcc-front on the front-node, dcc-node and its dependencies will be silently installed because of the preloaded answers to debconf questions (/etc/dcc/debconf). See dcc_buildimage(8) for details of the image build process.

Although all this makes the installation simple, it also violates the strict Debian policy guidelines. slapd-dcc, for example, modifies /etc/ldap/slapd.conf which is owned by another package. Moreover, there is no sense in installing e.g. slapd-dcc if debconf-dcc isn't installed first. This applies to the other DCC packages too. Also there might be other issues not mentioned here.

Several manual steps are required. More info in the HOWTO provided by Gordon Grubert.