typedef int (*funcptr)();

An engineers technical notebook

Building custom ports with Poudriere and Portshaker

Guest post by Scott Sturdivant.

Maintaining custom ports and integrating them into your build process doesn't need to be difficult. The documentation surrounding this process however is either non-existent, or lacking in its clarity. At the end of the day, it really is as simple as maintaining a repository whose structure matches the ports tree layout, then managing that repository and the standard ports tree with portshaker, and finally handing the end result off to poudriere.

Your Custom Repository

For this example, we'll assume a git repo is used and that you're already familiar with how to build FreeBSD ports. We'll also assume that we have but a single port that we're maintaining and that it is called myport. The hierarchy of your repo should simply be category/myport. We'll refer to this repo simply as myrepo.


Portshaker is the tool responsible for taking multiple ports sources and then merging them down into a single target. In our case, we have two sources: our git repo (myrepo) containing myport, and the standard FreeBSD ports tree. We aim to merge this down into a single ports tree that poudriere will then use for its builds.

To configure portshaker, add the following to the /usr/local/etc/portshaker.conf file:

# vim:set syntax=sh:
# $Id: portshaker.conf.sample 116 2008-09-30 16:15:02Z romain.tartiere $

#---[ Base directory for mirrored Ports Trees ]---

#---[ Directories where to merge ports ]---

default_merge_from="freebsd myrepo"

Some key points here are that the two items listed in for the default_merge_from argument need to have scripts present in the /usr/local/etc/portshaker.d directory. Further more, the combination of the poudriere_ports_mountpoint and default_poudriere_tree needs to be a ports tree that is then registered with poudriere.

Next, we need to tell portshaker how to go off and fetch our two types of ports trees, freebsd and myrepo. For the freebsd ports tree, create /usr/local/etc/portshaker.d/freebsd with the following contents and make it executable:

. /usr/local/share/portshaker/portshaker.subr
run_portshaker_command $*

Next, create a similar script to handle our repository containing our custom port. /usr/local/etc/portshaker.d/myrepo should contain the following and similarly be executable:

. /usr/local/share/portshaker/portshaker.subr
run_portshaker_command $*

Obviously replace the git_clone_uri and git_branch variables to reflect your actual configuration. For more information about the values and what they can contain, consult man portshaker.d

Now, portshaker should be all set. Execute portshaker -U to update your merge_from ports trees (freebsd and myrepo). You'll see the standard portsnap fetch and extract process as well as a git clone. After a good bit of time, these will both be present in the /var/cache/portshaker directory. Go ahead and merge them together by executing portshaker -M.

Hooray! You now have /usr/local/poudriere/ports/default/ports that is a combination of the normal ports tree and your custom one.

We're effectively complete with configuring portshaker. Whenever your port is updated, just re-run portshaker -U and portshaker -M to grab the latest changes and perform the merge.


Poudriere is a good tool for building ports. We will use it to handle our merged directory. Begin by configuring poudriere (/usr/local/etc/poudriere.conf):


Really there's nothing here that is specific to the problem at hand, so feel free to consult the provided configuration file to tune it to your needs.

Now, the step that is specific is to set poudriere up with a ports tree that it does not manage, specifically our resultant merged directory. If you consult man poudriere, it specifies that for the ports subcommand, there is a -m method switch which controls the methodology used to create the ports tree. By default, it is portsnap. This is confusing as in our case, we do not want poudriere to actually do anything. We want it to just use an existing path. Fortunately, there is a way!

The poudriere wiki has an entry for using the system ports tree, so we adopt it for our needs by executing:

poudriere ports -c -F -f none -M /usr/local/poudriere/ports/default \
-p default

If you've consulted the poudriere manpage, you'll see that the -F and -f switches both reference ZFS in their help. As we're not using ZFS, it's not clear how they will behave. However, in conjunction with the custom mountpoint (-M /usr/local/poudriere/ports/default), we ultimately wind up with what we want, a ports tree that poudriere can use, but does not manage:

# poudriere ports -l
default              -          /usr/local/poudriere/ports/default

Note that this resulting PATH is the combination of the poudriere_ports_mountpoint and default_poudriere_tree variables present in our /usr/local/etc/portshaker.conf configuration file.

Building software from your custom ports tree

Go ahead and create your jail(s) like you normally would (i.e. poudriere -c -j 92amd64 -V 9.2-RELEASE -a amd64) and any other configuration you would like, and then go ahead and build myport with poudriere bulk -j 92amd64 -p default category/myport. Success!

IPv6 -- getaddrinfo() and bind() ordering with V6ONLY

Recently I ran into an issue that took me a while to sort out, and it is regarding inconsistent behaviour on various OS's with regards to IPv6 sockets (AF_INET61) and calling bind(2) after getting the results back from getaddrinfo(3).

A call to getaddrinfo() with the hints set to AF_UNSPEC in ai_family and AI_PASSIVE in ai_flags will return to us 1 or more results that we can bind() to. Sample code for that looks like this:

struct addrinfo hints, *addrlist;

memset(&hints, 0, sizeof(hints));

// Ask for TCP
hints.ai_socktype = SOCK_STREAM;

// Any family works for us ...
hints.ai_family = AF_UNSPEC;

// Set some hints
hints.ai_flags = 
            AI_PASSIVE    | // We want to use this with bind
            AI_ADDRCONFIG;  // Only return IPv4 or IPv6 if they are configured

int rv;

if ((rv = getaddrinfo(0, "7020", &hints, &addrlist)) != 0) {
    fprintf(stderr, "getaddrinfo: %s", gai_strerror(rv));
    return 1;

// Use the list in *addrlist
for (addr = addrlist; addr != 0; addr = addr->ai_next) {
    // use *addr as appropriate

// Clean up the memory from getaddrinfo()

On Linux there are two entries returned when the host it is run on has both IPv4 and IPv6 enabled. An AF_INET which was followed by an AF_INET6. Now, it is not said that you are required to use all of the results that are returned, but if you want to listen on all address families it is off course suggested.

Following the steps below for each of the returned results should result in having 1 or more different sockets that are bound to a single port.

  1. Create the socket()
  2. Set any socket options you want (SO_REUSEADDR for example)
  3. Then bind() the socket
  4. After that call listen() (followed off course by accept() on the socket)

Only for some unknown reason (and errno is no help) bind() fails when you get to the AF_INET6, which was returned second. Searching online as to why the bind would fail doesn't give you any good results and the thing that is even worse is that if you run the same code on another platform such as FreeBSD, OpenIndiana or Mac OS X no such failure exists. However I started suspecting something was up when I started looking at the output from netstat -lan | grep 7020 on Mac OS X. Where 7020 is the port I passed into getaddrinfo().

tcp46      0      0  *.7020                 *.*  LISTEN     
tcp4       0      0  *.7020                 *.*  LISTEN

Wait a minute ... one of the sockets is on both IPv4 and on IPv6. Some more time spent searching the internet I came across RFC 3493 section 5.3, which is titled "IPV6_V6ONLY option for AF_INET6 Sockets".

As stated in section <3.7 Compatibility with IPv4 Nodes>, AF_INET6 sockets may be used for both IPv4 and IPv6 communications. Some applications may want to restrict their use of an AF_INET6 socket to IPv6 communications only.

This was going down the right route, so I changed my code so that in the steps listed above in number 2 I added the following code if the socket type is AF_INET6:

if (setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, &yes, sizeof(int)) == -1) {
    fprintf(stderr, "setsockopt: %s IPV6_V6ONLY\n", strerror(errno));

The RFC 3493 section 5.3 also states that this option should be turned off by default, which means that all IPv6 sockets can also communicate over IPv4. Thus technically setting the option manually in code the best way to fix the issue. FreeBSD has had this feature turned on (as in IPv6 sockets can only communicate with IPv6 and NOT IPv4) since 5.x.

The biggest issue is that the remaining operating systems (OS X and OpenIndiana) don't have the same behaviour as Linux which makes troubleshooting this issue more difficult than it should be. The issue is that the RFC doesn't specify what exactly the operating should do when it encounters a request to bind to the same port on IPv4 and IPv6. The only place where I have found this documented is in "IPv6 Network Programming" under "Tips in IPv6 Programming" chapter 4, section 4, appropriately titled "bind(2) Ordering and Conflicts".

If you get a bind() error when attempting to bind to an AF_INET6 socket please make sure that you set the socket option IPV6_V6ONLY on the AF_INET6 socket. The default as required by RFC 3493 is to have that option be off. The default is wrong, and the RFC should have been more specific regarding what the right behaviour is when attempting to bind on an AF_INET6 socket when already bound on an AF_INET while IPV6_V6ONLY is set to false.

The full code that I used for testing, along with a little bit more information is available as a gist on github.

  1. The old BSD style socket() called for defines starting with PF_ such as PF_INET and PF_INET6 with the PF standing for protocol family. POSIX starts them with AF_, and calls them an address family. On almost every operating system PF_INET is the same as AF_INET. If the define doesn't exist you can always create it. 

FreeBSD 8.2 to 9.0 update with ZFS on root (mirror)

Upgrading from one version of FreeBSD to another has become much simpler with freebsd-update than before, a full rebuild of the FreeBSD kernel/world source used to be required (other upgrade paths existed but were not widely touted). freebsd-update works extremely well when using UFS as your file system, there are however some gotcha's that one needs to look out for when using FreeBSD Root on ZFS. Two machines needed upgrading, both are running with a ZFS root mirror on FreeBSD 8.2-RELEASE. I had followed the FreeBSD Root on ZFS (mirror) wiki article and everything has been running smoothly ever since.

  pool: zroot
 state: ONLINE
 scrub: scrub completed after 0h35m with 0 errors on Mon Mar  5 10:58:19 2012

    zroot          ONLINE       0     0     0
      mirror       ONLINE       0     0     0
        gpt/disk0  ONLINE       0     0     0
        gpt/disk1  ONLINE       0     0     0

errors: No known data errors

Wanting to take advantage of some of the advancements made in FreeBSD 9.0-RELEASE such as the updated ZFS, a bug that was fixed that stopped jails from owning a ZFS file system, improved dtrace, and better IPv6 support required that the plunge be taken and an upgrade was in order. There were a couple reports of people having done source upgrades and then having issues booting their system, but I found one mailling list post that suggested that everything went well using freebsd-update, so that is the route I went with.

I read through the entirety of the FreeBSD 9.0-ERLEASE Release Notes before moving forward to make sure I didn't miss any important changes that would cause my system to not restart correctly or would require changing system configuration files before attempting the upgrade. I would advise you to do the same.

I ran freebsd-update -r 9.0-RELEASE upgrade and after accepting the list of components that were installed, I got an error saying:

# freebsd-update -r 9.0-RELEASE upgrade
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 8.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic src/base src/bin src/cddl src/contrib src/crypto src/etc
src/games src/gnu src/include src/krb5 src/lib src/libexec src/release
src/rescue src/sbin src/secure src/share src/sys src/tools src/ubin
src/usbin world/base world/catpages world/dict world/doc world/games
world/info world/lib32 world/manpages

The following components of FreeBSD do not seem to be installed:

Does this look reasonable (y/n)? y

Fetching metadata signature for 9.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.

The update metadata is correctly signed, but
failed an integrity check.
Cowardly refusing to proceed any further.

Specifically the line stating "Cowardly refusing to proceed further", a quick Google later I found the 9.0-RC1 mailling list post that suggested the following one line fix:

sed -i '' -e 's/=_/=%@_/' /usr/sbin/freebsd-update

After that quick fix, a re-run of freebsd-update functioned without issues and finished, showing me what it would change, asking me if it seemed acceptable and telling me what it was going to remove and what was going to be installed.

Next we run freebsd-update for the first time in this three step process:

freebsd-update install

Don't restart yet (although it tells you to)! We need to go back and make sure we update the boot records first.

Going back to the mailling list post made by Magnus Strahlert, I made note of the following:

Had to upgrade the gpt bootcode as instructed when upgrading the zpool for the system to boot.

This is the only comment that worried me, and nowhere in the email thread does the poster describe what commands he ran. In any case I figure he meant re-running the boot code commands from the FreeBSD Root on ZFS wiki article:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

After running these commands, it told me that it had updated the boot code and then I ran:

shutdown -r now

After the machine came back online (it is in those moments when you have a small heartattack when the machine doesn't come up as fast as you had expected ...) you log back in and run:

freebsd-update install

This will install all of the updated binaries that came with FreeBSD 9.0, at this point FreeBSD update will tell you to upgrade all installed software and ports, and then run freebsd-update one more time to remove all unnecessary shared libraries. If you want to have certain software continue functioning, yet you want freebsd-update to remove any other remaining old 8.x files, you can install misc/compat8x from the ports tree. This is the suggested route to go if you want to keep the 8.x libraries around for compatibility reasons.

cd /usr/ports/misc/compat8x
make install clean
# or
# portmaster misc/compat8x

I use portmaster and the following command will simply rebuild every port installed on the system:

portmaster -aPf # -P means look for package and if available use it, otherwise build from port

Run freebsd-update one last time:

freebsd-update install

Since we are running ZFS we will most likely also want to upgrade our pool to the latest version of ZFS to take advantage of any new goodies (such as deduplication!), upgrading a zpool is easy:

zpool upgrade zroot

That upgrades to the latest pool version, now we will want to upgrade our existing file systems to the latest version of ZFS, do note that this will make them incompatible with older file system verisons (so if you use zfs send for backups, your other endpoint will need to be of the same ZFS version)

zfs upgrade # List all ZFS file systems that need to be upgraded
zfs upgrade -a # Upgrade all file systems

After this one last reboot to make sure that all the processes come back online using the newer libraries and that you didn't accidentally miss something.

shutdown -r now

At this point the system has been upgraded from FreeBSD 8.0-RELEASE to FreeBSD \9.0-RELEASE and has all of the latest updates and enhancements.

Converting KVM virtual machines to VirtualBox

Recently the requirement came up to take a KVM based virtual machine and move it over to a VirtualBox image. Which turned out to be a fairly simple endeavour, and was fairly painless. The longest part was transferring over the 40 GB image from one machine to the other where the conversion could take place. The machine the image was coming from was only on a 100 Mbit/sec connection so that took a good hour.

Converting from KVM to VirtualBox for a FreeBSD image was pretty simple, the VBoxManage command has a convertdd command that allows you to convert from raw disk .img format to .vdi format.

VBoxManage convertdd KVM-image.img VB-image.vdi

After this, unfortunately, there is no way to to automatically convert over the settings that the virtual machine had, such as the network cards, the memory allocations and hard drive settings. You will have to go to VirtualBox and create a new virtual machine and replicate all of the settings. Once that is done make sure to select the same type of disk controller (SATA or IDE) so that the drive will hopefully be assigned the same name in the device tree so that you don't need to alter your /etc/fstab.

Hopefully everything boots without any issues. If not try creating a new virtual machine, attach the converted image as a secondary drive and see if you can mount the converted image within your new install. If so maybe transferring the data using rsync or dump/restore would be an option.

NAT with PF on an interface with multiple IP addresses

If you want to do NAT for your currently running jail instances on FreeBSD so that they can have outgoing connections you could try the following pf.conf to set up NAT on the interface that the jails have their IP addresses on.

ext_if="em0"    # The network card your default gateway is on
jail_if="lo1"   # The interface that your jails have IP addresses on

# Set some options
set optimization aggressive
set block-policy drop
set skip on lo

# NAT on the external interface when coming from the jail interface
nat on $ext_if from $jail_if:network:0 to any -> ($ext_if)

# We just pass everything
pass quick all

What I have done is create IP addresses within the private network range (10/8, 172.16/12, 192.168/16) on the lo1 interface. These can't be in the 127/8 range because those addresses can't be NAT'ed (not sure if this is a limitation in pf or if this is a FreeBSD limitation), which is a shame because using would be pretty awesome in my opinion.

What I did find though is that the above will not work correctly if your main network card (the one your default gateway is on, in the example em0) contains multiple IP addresses. At that point the syntax ($ext_if) does not function correctly and will cause packet loss/drop. 1

So instead of using the syntax above we simply replace ($ext_if) with the actual IP address of the interface. This takes care of the issue and will let your jails have proper internet access without issues.


set optimization aggressive
set block-policy drop
set skip on lo

nat on $ext_if from $jail_if:network:0 to any -> # Your IP!

pass quick all

The other way you can solve this problem off course is to provide your jails with IP addresses within the same range that go to your default gateway thereby solving the problem of needing to NAT in the first place.

  1. Do note that I am using FreeBSD 8.2, so this may be fixed in the new FreeBSD 9 that is due to be released soon.