Package Building Procedures

Pick a unique hostname. It does not have to be a publicly resolvable hostname (it can be a name on your internal network).

By default, package building requires the following TCP ports to be accessible: 22 (ssh), 414 (infoseek), and 8649 (ganglia). If these are not accessible, pick others and ensure that an ssh tunnel is set up (see below).

(Note: if you have more than one machine at your site, you will need an individual TCP port for each service on each machine, and thus ssh tunnels will be necessary. As such, you will probably need to configure port forwarding on your firewall.)

Decide if you will be booting natively or via pxeboot. You will find that it is easier to keep up with changes to -current with the latter, especially if you have multiple machines at your site.

Pick a directory to hold ports configuration and chroot subdirectories. It may be best to put it this on its own partition. (Example: /usr2/.)

Create a directory to contain the latest -current source tree and check it out. (Since your machine will likely be asked to build packages for -current, the kernel it runs should be reasonably up-to-date with the bindist that will be exported by our scripts.)

If you are using pxeboot: create a directory to contain the install bits. You will probably want to use a subdirectory of /pxeroot, e.g., /pxeroot/${arch}-${branch}. Export that as DESTDIR.

If you are cross-building, export TARGET_ARCH=${arch}.

Generate a kernel config file. Include GENERIC (or, if on i386, and you are using more than 3.5G, PAE).

Required options:

options         NULLFS
options         TMPFS

Suggested options:

options         GEOM_CONCAT
options         GEOM_STRIPE
options         SHMMAXPGS=65536
options         SEMMNI=40
options         SEMMNS=240
options         SEMUME=40
options         SEMMNU=120

If you are interested in debugging general problems, you may wish to use the following. However, for unattended operations, it is best to leave it out:

options         ALT_BREAK_TO_DEBUGGER

For PAE, it is not currently possible to load modules. Therefore, if you are running an architecture that supports Linux emulation, you will need to add:

options         COMPAT_LINUX
options         LINPROCFS

Also for PAE, as of 20110912 you need the following. This needs to be investigated:

nooption        NFSD                    # New Network Filesystem Server
options         NFSCLIENT               # Network Filesystem Client
options         NFSSERVER               # Network Filesystem Server

As root, do the usual build steps, e.g.:

# make -j4 buildworld
# make buildkernel KERNCONF=${kernconf}
# make installkernel KERNCONF=${kernconf}
# make installworld

The install steps use DESTDIR.

Customize files in etc/. Whether you do this on the client itself, or another machine, will depend on whether you are using pxeboot.

If you are using pxeboot: create a subdirectory of ${DESTDIR} called conf/. Create one subdirectory default/etc/, and (if your site will host multiple nodes), subdirectories ${ip-address}/etc/ to contain override files for individual hosts. (You may find it handy to symlink each of those directories to a hostname.) Copy the entire contents of ${DESTDIR}/etc/ to default/etc/; that is where you will edit your files. The by-ip-address etc/ directories will probably only need customized rc.conf files.

In either case, apply the following steps:

• Create a portbuild user and group. It can have the '*' password.

  Create /home/portbuild/.ssh/ and populate authorized_keys.

• Also add the following users:

  squid:*:100:100::0:0:User &:/usr/local/squid:/bin/sh
  ganglia:*:102:102::0:0:User &:/usr/local/ganglia:/bin/sh
  

  Add them to etc/group as well.

• Create the appropriate files in etc/.ssh/.

• In etc/crontab: add

  *       *       *       *       *       root    /var/portbuild/scripts/client-metrics
  

• Create the appropriate etc/fstab. (If you have multiple, different, machines, you will need to put those in the override directories.)

• In etc/inetd.conf: add

  infoseek        stream  tcp     nowait  nobody  /var/portbuild/scripts/reportload
  

• You should run the cluster on UTC. If you have not set the clock to UTC:

  # cp -p /usr/share/zoneinfo/Etc/UTC etc/localtime
  

• Create the appropriate etc/rc.conf. (If you are using pxeboot, and have multiple, different, machines, you will need to put those in the override directories.)

  Recommended entries for physical nodes:

  hostname="${hostname}"
  inetd_enable="YES"
  linux_enable="YES"
  nfs_client_enable="YES"
  ntpd_enable="YES"
  sendmail_enable="NONE"
  sshd_enable="YES"
  sshd_program="/usr/local/sbin/sshd"
  
  gmond_enable="YES"
  squid_enable="YES"
  squid_chdir="/usr2/squid/logs"
  squid_pidfile="/usr2/squid/logs/squid.pid"
  

  Required entries for VMWare-based nodes:

  vmware_guest_vmmemctl_enable="YES"
  vmware_guest_guestd_enable="YES"
  

  Recommended entries for VMWare-based nodes:

  hostname=""
  ifconfig_em0="DHCP"
  fsck_y_enable="YES"
  
  inetd_enable="YES"
  linux_enable="YES"
  nfs_client_enable="YES"
  sendmail_enable="NONE"
  sshd_enable="YES"
  sshd_program="/usr/local/sbin/sshd"
  
  gmond_enable="YES"
  squid_enable="YES"
  squid_chdir="/usr2/squid/logs"
  squid_pidfile="/usr2/squid/logs/squid.pid"
  

  ntpd(8) should not be enabled for VMWare instances.

  Also, it may be possible to leave squid disabled by default so as to not have /usr2 persistent (which should save instantiation time.) Work is still ongoing.

• Create etc/resolv.conf, if necessary.

• Modify etc/sysctl.conf:

  9a10,30
  > kern.corefile=/usr2/%N.core
  > kern.sugid_coredump=1
  > #debug.witness_ddb=0
  > #debug.witness_watch=0
  >
  > # squid needs a lot of fds (leak?)
  > kern.maxfiles=40000
  > kern.maxfilesperproc=30000
  >
  > # Since the NFS root is static we do not need to check frequently for file changes
  > # This saves >75% of NFS traffic
  > vfs.nfs.access_cache_timeout=300
  > debug.debugger_on_panic=1
  >
  > # For jailing
  > security.jail.sysvipc_allowed=1
  > security.jail.allow_raw_sockets=1
  > security.jail.chflags_allowed=1
  > security.jail.enforce_statfs=1
  >
  > vfs.lookup_shared=1
  

• If desired, modify etc/syslog.conf to change the logging destinations to @pointyhat.freebsd.org.

Install the following ports:

net/rsync
security/openssh-portable (with HPN on)
security/sudo
sysutils/ganglia-monitor-core (with GMETAD off)
www/squid (with SQUID_AUFS on)

There is a WIP to create a meta-port, but it is not yet complete.

Customize files in usr/local/etc/. Whether you do this on the client itself, or another machine, will depend on whether you are using pxeboot.

Apply the following steps:

• Modify usr/local/etc/gmond.conf:

  21,22c21,22
  <   name = "unspecified"
  <   owner = "unspecified"
  ---
  >   name = "${arch} package build cluster"
  >   owner = "portmgr@FreeBSD.org"
  24c24
  <   url = "unspecified"
  ---
  >   url = "http://pointyhat.freebsd.org"
  

  If there are machines from more than one cluster in the same multicast domain (basically = LAN) then change the multicast groups to different values (.71, .72, etc).

• Create usr/local/etc/rc.d/portbuild.sh, using the appropriate value for scratchdir:

  #!/bin/sh
  #
  # Configure a package build system post-boot
  
  scratchdir=/usr2
  
  ln -sf ${scratchdir}/portbuild /var/
  
  # Identify builds ready for use
  cd /var/portbuild/arch
  for i in */builds/*; do
      if [ -f ${i}/.ready ]; then
          mkdir /tmp/.setup-${i##*/}
      fi
  done
  
  # Flag that we are ready to accept jobs
  touch /tmp/.boot_finished
  

• Modify usr/local/etc/squid/squid.conf:

  288,290c288,290
  < #auth_param basic children 5
  < #auth_param basic realm Squid proxy-caching web server
  < #auth_param basic credentialsttl 2 hours
  ---
  > auth_param basic children 5
  > auth_param basic realm Squid proxy-caching web server
  > auth_param basic credentialsttl 2 hours
  611a612
  > acl localnet src 127.0.0.0/255.0.0.0
  655a657
  > http_access allow localnet
  2007a2011
  > maximum_object_size 400 MB
  2828a2838
  > negative_ttl 0 minutes
  

  Also, change usr/local to usr2 in cache_dir, access_log, cache_log, cache_store_log, pid_filename, netdb_filename, coredump_dir.

  Finally, change the cache_dir storage scheme from ufs to aufs (offers better performance).

• Configure ssh: copy etc/ssh to usr/local/etc/ssh and add NoneEnabled yes to sshd_config.

• Note: This step is under review.

  Create usr/local/etc/sudoers/sudoers.d/portbuild:

  # local changes for package building
  portbuild     ALL=(ALL) NOPASSWD: ALL
  

Change into the port/package directory you picked above, e.g., cd /usr2.

As root:

# mkdir portbuild
# chown portbuild:portbuild portbuild
# mkdir pkgbuild
# chown portbuild:portbuild pkgbuild
# mkdir squid
# mkdir squid/cache
# mkdir squid/logs
# chown -R squid:squid squid

If clients preserve /var/portbuild between boots then they must either preserve their /tmp, or revalidate their available builds at boot time (see the script on the amd64 machines). They must also clean up stale jails from previous builds before creating /tmp/.boot_finished.

Boot the client.

As root, initialize the squid directories:

squid -z

If any of the default TCP ports is not available (see above), you will need to create an ssh tunnel for them and include its invocation command in portbuild's crontab.

Unless you can use the defaults, add an entry to /home/portbuild/.ssh/config to specify the public IP address, TCP port for ssh, username, and any other necessary information.

Create /a/portbuild/${arch}/clients/bindist-${hostname}.tar.

• Copy one of the existing ones as a template and unpack it in a temporary directory.

• Customize etc/resolv.conf for the local site.

• Customize etc/make.conf for FTP fetches for the local site. Note: the nulling-out of MASTER_SITE_BACKUP must be common to all nodes, but the first entry in MASTER_SITE_OVERRIDE should be the nearest local FTP mirror. Example:

  .if defined(FETCH_ORIGINAL)
  MASTER_SITE_BACKUP=
  .else
  MASTER_SITE_OVERRIDE= \
  	ftp://friendly-local-ftp-mirror/pub/FreeBSD/ports/distfiles/${DIST_SUBDIR}/ \
  	ftp://${BACKUP_FTP_SITE}/pub/FreeBSD/distfiles/${DIST_SUBDIR}/
  .endif
  

• tar it up and move it to the right location.

Hint: you will need one of these for each machine; however, if you have multiple machines at one site, you should create a site-specific one (e.g., in /a/portbuild/conf/clients/) and symlink to it.

Create /a/portbuild/${arch}/portbuild-${hostname} using one of the existing ones as a guide. This file contains overrides to /a/portbuild/${arch}/portbuild.conf.

Suggested values:

disconnected=1
http_proxy="http://localhost:3128/"
squid_dir=/usr2/squid
scratchdir=/usr2/pkgbuild
client_user=portbuild
sudo_cmd="sudo -H"
rsync_gzip=-z

infoseek_host=localhost
infoseek_port=${tunelled-tcp-port}

Possible other values:

use_md_swap=1
md_size=9g
use_zfs=1
scp_cmd="/usr/local/bin/scp"
ssh_cmd="/usr/local/bin/ssh"

Add the public IP address to /etc/hosts.allow. (Remember, multiple machines can be on the same IP address.)

Add an appropriate data_source entry to /usr/local/etc/gmetad.conf:

data_source "arch/location Package Build Cluster" 30 hostname

You will need to restart gmetad.

Ensure that ssh to the client is working by executing ssh hostname uname -a. The actual command is not important; what is important is to confirm the setup, and also add an entry into known_hosts, once you have confirmed the node's identity.

Populate the client's copy of /var/portbuild/scripts/ by something like /a/portbuild/scripts/dosetupnode arch major latest hostname. Verify that you now have files in that directory.

Test the other TCP ports by executing telnet hostname portnumber. 414 (or its tunnel) should give you a few lines of status information including arch and osversion; 8649 should give you an XML response from ganglia.

Tell qmanager about the node. Example:

python path/qmanager/qclient add name=uniquename arch=arch osversion=osversion numcpus=number haszfs=0 online=1 domain=domain primarypool=package pools="package all" maxjobs=1 acl="ports-arch,deny_all"

Once you are sure that the client is working, tell pollmachine about it by adding it to /a/portbuild/${arch}/mlist.

Install FreeBSD.

Create a user to own the portbuild repository, such as portbuild. It should have the '*' password.

Similarly, create a user to own the administration functions and manage the svn repositories, such as srcbuild. It should have the '*' password.

Add the following to /boot/loader.conf:

console="vidconsole,comconsole"

You should run the cluster on UTC. If you have not set the clock to UTC:

# cp -p /usr/share/zoneinfo/Etc/UTC /etc/localtime

Create the appropriate /etc/rc.conf.

Required entries:

hostname="${hostname}"
sshd_enable="YES"
zfs_enable="YES"

Recommended entries:

background_fsck="NO"
clear_tmp_enable="YES"
dumpdev="AUTO"
fsck_y_enable="YES"

apache22_enable="YES"
apache_flags=""
apache_pidfile="/var/run/httpd.pid"
gmetad_enable="YES"
gmond_enable="YES"
inetd_enable="YES"
inetd_flags="-l -w"
mountd_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 12"
nfs_remote_port_only="YES"
ntpd_enable="YES"
rpcbind_enable="YES"
rpc_lockd_enable="NO"
rpc_statd_enable="YES"
sendmail_enable="NONE"
smartd_enable="YES"

Create /etc/resolv.conf, if necessary.

Create the appropriate files in /etc/ssh/.

Add the following to /etc/sysctl.conf:

kern.maxfiles=40000
kern.maxfilesperproc=38000
sysctl vfs.usermount=1
sysctl vfs.zfs.super_owner=1

Make sure the following change is made to /etc/ttys:

ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure

The following ports (or their latest successors) are required:

databases/py-sqlite3
databases/py-sqlalchemy (only SQLITE is needed)
devel/git (WITH_SVN)
devel/py-configobj
devel/py-setuptools
devel/subversion
net/nc
net/rsync
sysutils/ganglia-monitor-core (with GMETAD off)
sysutils/ganglia-webfrontend (compile with -DWITHOUT_X11)
www/apache22 (with EXT_FILTER)

Expect those to bring in, among others:

databases/sqlite3
lang/perl-5.14 (or successor)
lang/python27 (or sucessor)

The following ports (or their latest successors) are strongly suggested:

devel/ccache
mail/postfix
net/isc-dhcp41-server
ports-mgmt/pkg
ports-mgmt/portaudit
ports-mgmt/portmaster
shells/bash
shells/zsh
sysutils/screen

The following ports (or their latest successors) are handy:

benchmarks/bonnie++
ports-mgmt/pkg_tree
sysutils/dmidecode
sysutils/smartmontools
sysutils/zfs-stats

Export the value of PORTBUILD_USER:

# export PORTBUILD_USER=portbuild

Export the value of SRCBUILD_USER:

# export SRCBUILD_USER=srcbuild

Pick a zfs volume name and export it. We have used a so far to date.

# export ZFS_VOLUME=a

Pick a mountpoint and export it. We have used /a so far to date.

# export ZFS_MOUNTPOINT=/a

Create the zfs volume and mount it.

# zpool create ${ZFS_VOLUME} mirror da1 da2 mirror da3 da4 mirror da5 da6 mirror da7 da8

Select an svn repository and export it. See the FreeBSD Handbook for the currently supported list.

# export VCS_REPOSITORY=svn://svn0.us-east.FreeBSD.org

Obtain a copy of the kickstart script into a temporary directory. (You will not need to keep this directory later.)

# mkdir -p /home/portbuild/tmp
# svn checkout ${VCS_REPOSITORY}/base/projects/portbuild/admin/tools /home/portbuild/tmp

Run the kickstart script:

# sh /home/portbuild/tmp/mkportbuild

This will accomplish all the following steps:

Configure the server by making the following changes to /a/portbuild/admin/conf/admin.conf:

• Set SUPPORTED_ARCHS to the list of architectures you wish to build packages for.

• For each source branch you will be building for, set SRC_BRANCHES and SRC_BRANCH_branch_SUBDIR as detailed in Section 14.1. You should not need to change SRC_BRANCHES_PATTERN.

• Set ZFS_VOLUME and ZFS_MOUNTPOINT to whatever you chose above.

• Set VCS_REPOSITORY to whatever you chose above.

• Set MASTER_URL to the http URL of your server. This will be stamped into the package build logs and the indices thereof.

Most of the other default values should be fine.

Configure how build slaves will talk to your server by making the following changes to /a/portbuild/conf/client.conf:

• Set CLIENT_NFS_MASTER to wherever your build slaves will PXE boot from. (Possibly, the hostname of your server.)

• Set CLIENT_BACKUP_FTP_SITE to a backup site for FTP fetches; again, possibly the hostname of your server.

• Set CLIENT_UPLOAD_HOST to where completed packages will be uploaded.

Most of the other default values should be fine.

Most of the default values in /a/portbuild/conf/common.conf should be fine. This file holds definitions used by both the server and all its clients.

Configure the server by making the following changes to /a/portbuild/conf/server.conf:

• Set UPLOAD_DIRECTORY, UPLOAD_TARGET, and UPLOAD_USER as appropriate for your site.

Most of the other default values should be fine.

For each architecture, follow the steps in Section 17.1.

As root, copy the following files from /a/portbuild/admin/etc/rc.d/ to /usr/local/etc/rc.d/:

pollmachine
qmanager

As root, start each one of them. You may find it handy to start each under screen for debugging purposes.

Initialize the qmanager database's acl list:

# python /a/portbuild/qmanager/qclient add_acl name=deny_all uidlist= gidlist= sense=0

As the srcbuild user, run the following commands manually to create the src and ports repositories, respectively:

% /a/portbuild/admin/scripts/updatesnap.ports
% /a/portbuild/admin/scripts/updatesnap

These will be periodically run from the srcbuild crontab, which you will install below.

Configure /usr/local/etc/apache22/httpd.conf as appropriate for your site.

Copy /a/portbuild/admin/conf/apache.conf to the appropriate Includes/ subdirectory, e.g., /usr/local/etc/apache22/Includes/portbuild.conf. Configure it as appropriate for your site.

Install /a/portbuild/admin/crontabs/portbuild as the portbuild crontab via crontab -u portbuild -e. If you do not support all the archs listed there, make sure to comment out the appropriate dologs entries.

Install /a/portbuild/admin/crontabs/srcbuild as the srcbuild crontab via crontab -u srcbuild -e.

If your build slaves will be pxebooted, make sure to enable the tftp entries in /etc/inetd.conf.

Configure mail by doing the following:

newaliases.

For each architecture, follow the steps in Section 17.2.

You will probably find it handy to append the following to the PATH definition for the portbuild user:

/a/portbuild/scripts:/a/portbuild/tools

You will also probably find it handy to append the following to the PATH definition for the srcbuild user:

/a/portbuild/admin/scripts:/a/portbuild/admin/tools