Programs of the StarBED Project

Table of Contents
Terminology Experiment Overview Facility-Side Servers User-Side Programs Kuroyuri SheepDog xant SpringOS Installation References Appendix

node

The unit of experiment resource. Most of them, the term means PC.

RM, ERM -- (experiment) resource manager

Resource manager serves information of resources and stores ownership for exclusive using. The program dispatches resources according to the condition of requests.

ENCD -- experiment node configuration driver

The program drives nodes of experiments.

FNCP -- facility node configuration pilot

The program navigates nodes to their driver(ENCD) for configuration.

DMAN -- directory manipulator

Dman makes and removes symlinks on the remote filesystem. It is useful to switch the node behavior on TFTP and FTP servers' directories

NA -- node agent

NA is the program on the node.

NI -- node initiator

An implemenation of NA. The program does a data transfer between local disk and remote file server. So, it is upload/download of diskimage.

wolagent, wol_agent -- Wake on LAN (WoL) agent

The program issues Magick Packet to boot node, but it is not relay.

An experiment consists of a driver host and runnner hosts. The driver sends to the runners a sequence of experiment steps to be performed on the runner. Each runner executes the steps in the sequence. The driver and the runner exchange messages. These messages are used as parameters and commands on the experiment.

To save cost, network experiment facilities are generally designed to support running many experiments at the same time. However, when many experiments run on the same facility, it requires an arbitration of resources. In addition, data storage is required. To this purpose we employ dedicated facility side management servers to control the experiment hosts (driver and runner).

Programs included in SpringOS and related to it are divided into 2 sides and 3 categories.

In order to manage multiple simultaneous experiments running on a facility with a wide range of equipments, we designed the resource manager, erm. According to the database of resource manager, a node monitoring tool, sheepdog, probes the state of resources by using protocols such as ICMP, TCP and IPMI.

Initially nodes are not aware of their role in an experiment, and have no configuration. To avoid setting them up manually, we employ PXE, DHCP and TFTP related technologies in boot phase, and employ FNCP (Facility Node Configuration Pilot) and ENCD (Experiment Node Configuration Driver) negotiation in the experiment phase.

Here we only describe FNCP and ENCD negotiation. Detailed information about PXE, DHCP and TFTP can be found in REFERENCES. The agent program on each node aims to retrieve node configuration from the coresponding storage. However, the agent doesn't know in the beginning which host stores this data. We decided to use a single contact point for node agents. On the contact host, the program named FNCP redirects the agent to ENCD. Through the redirect of FNCP, the agent knows which is the configuration folder, ENCD.

ENCD has to register with FNCP in order to benefit of its service. The following figure depicts the flow of the node configuration retrieval by node agent, NI, via FNCP.

SpringOS expects WoL-capable PCs for node booting, since WoL is the de-facto standard for PC booting. The program wolagent is the WoL packet sender. On request from other programs, the program sends WoL packets in the data link network. Therefore one should build the program on each data link.

As an optional feature, we implemented a directory manipulator, dman. Using dman, one can access the directory of TFTP daemon. Thus one can change the boot loader, the disk images, etc. In StarBED we employ this program to switch the operating systems of experiment nodes.

Several programs can be used on the user side, and they will be briefly described in what follows.

The node initiator, NI, is a node agent software. The program controls disk partition and network configuration. According to configuration specified for each node, the program downloads the partition contents from FTP server programs. As it was mentioned above, the program retrieves the configuration from ENCD (usually via FNCP). This program runs before the experiment, in the node setup phase. Most users do not need to modify its behaviour.

StarBED already supports multi-OS bootable nodes. The boot loader is a key compornent for this purpose. The original boot loader was supplied by NEC. However, it is not open source. Therefore we developed a custom-made boot loader, coil. This program runs before the experiment, in the node setup phase. Most users do not need to modify its behaviour.

Kuroyuri is package of ENCDs and related programs.

• master and slave are evaluators of the experiment description language. By interpreting the experiment description, the master manages resources and executes the global scenario. The slave evaluates each node's scenario, and controls how each node performs its role. The experiment is driven through the colaboration of these evaluators.
• pickup and wipeout are managers for disk image saving and restoring. When saving, pickup reads a disk image by controlling ni and writes that to the FTP server. On the other hand, wipeout orders ni to retrieve a disk image from the FTP server, and write that image to the disk. The execution duration of wipeout depends on the size of disk image (and compression ratio), as well as the number of target hosts. For example, the installation of a 2 GB Windows-XP operating system on 114 hosts takes about 370 minutes.
• sbpsh is a shell for StarBED. It can be used to power on/off and reboot any hosts by command-line operation. One can also change the OS of hosts using this shell interface.
• Since disk operation takes a long time, users may want to skip node setup sometime, and use the same conditions with the last experiment. The program skipnsetup or sns makes this possible. This functionality is also useful during demonstrations.
• During network experiments the low-level approach is to check text messages and LEDs of equipments. However, non-specialists and executives prefer graphical demonstrations. We made a demo program ks.pl for such situations. Please note that this program is not suited for large experiments with complicated topologies. The image below depicts a scenario with 2 hosts (server and client) connected via 4 networks paths (ethnet-0,-1,-2, and -3).

Several programs are used in connection with Kuroyuri. ifsetup is a package managing network interfaces on nodes. ifscan scans all network interfaces on the node. slave (an evaluator of Kuroyuri) expects these programs to run in order to setup interfaces. ifscan should be installed on target nodes. In many cases, the user installs that on the disk image of the reference machine. For automatic reboot and shutdown SNMP is used in StarBED. snmpmine is an implementation used for this purpose.

SheepDog is a program that probes host status on the facility, and allows users to display it. The program is an http server (httpd), therefore users can check status of hosts using WWW clients. Futhermore, this program supports host power control. Users can boot any hosts using the provided interface.

By clicking on the image, a larger version can be displayed.

Xant is a program that shows the direction and volume of traffic in StarBED. The program scans interfaces on specified switches, and retrieves statistical values. These values are graphically displayed over a specified topology figure.

By clicking on the image, a larger version can be displayed.

In large-scale testbed, it is difficult for operators and users to have a global knowledge about the topology and the traffic. Xant was designed to help them with these issues. The table below indicates the configurable elements of Xant.

For SpringOS installation, 4 categories of programs are required:

• Category A: facility-side servers
  • dhcpd
  • tftpd
  • erm
  • fncp
  • dman
  • wolagent
  • ftpd
• Category B: node setup disk image
  • A raw disk image for diskless booting. (At the moment, ni expects FreeBSD 4.x. The use of other platforms was not tested so far.)
  • ni
  • A mechanism to wake up ni (e.g., /etc/rc.local)
• Category C: experiment disk image
  • Raw disk images for experiment
  • slave
  • ifscan
  • snmpmine
  • A mechanism to wake up ni (e.g., /etc/rc.local)
  • Programs that you want to run on the experiment
• Category D: experiment driving
  • master
  • experiment description file(s)

Other recommended programs may be installed as well:

• xant on category A
• sheepdog on category A or D
• sbpsh on category D

[IPMI]

Intel, Hewlett-Packard, NEC, Dell: Intelligent Platform Management Interface Specification Second Generation, Feb, 2004.
See IPMI v2.0 specification Document Revision 1.0 in http://developer.intel.com/design/servers/ipmi/

[PXE]

Intel Corporation: Preboot Execution Environment (PXE) Specifiction Version 2.1, Sep, 1990.

[DHCP] RFC2131

R. Droms: Dynamic Host Configuration Protocol, RFC2131, March, 1997.

[TFTPD] RFC1350

K. Sollins: The TFTP Protocol (Revision 2), RFC1350, July, 1992.

[FTPD] RFC959

J. Postel and J.K. Reynolds: File Transfer Protocol, RFC959, October, 1985.