funcptr - FreeBSD

Building custom ports with Poudriere and Portshaker

2013-12-11T00:10:11+00:00

Guest post by Scott Sturdivant</a>.

Maintaining custom ports and integrating them into your build process doesn't need to be difficult. The documentation surrounding this process however is either non-existent, or lacking in its clarity. At the end of the day, it really is as simple as maintaining a repository whose structure matches the ports tree layout, then managing that repository and the standard ports tree with portshaker</a>, and finally handing the end result off to poudriere</a>.

Your Custom Repository</h2>
For this example, we'll assume a git</a> repo is used and that you're already familiar with how to build FreeBSD</a> ports</a>. We'll also assume that we have but a single port that we're maintaining and that it is called myport</code>. The hierarchy of your repo should simply be category/myport</code>. We'll refer to this repo simply as myrepo</code>.
Portshaker</h2> Portshaker is the tool responsible for taking multiple ports sources and then merging them down into a single target. In our case, we have two sources: our git repo (myrepo</code>) containing myport</code>, and the standard FreeBSD ports tree. We aim to merge this down into a single ports tree that poudriere will then use for its builds. To configure portshaker, add the following to the /usr/local/etc/portshaker.conf</code> file:
# vim:set syntax=sh: # $Id: portshaker.conf.sample 116 2008-09-30 16:15:02Z romain.tartiere $ #---[ Base directory for mirrored Ports Trees ]--- mirror_base_dir="/var/cache/portshaker" #---[ Directories where to merge ports ]--- ports_trees="default" use_zfs="no" poudriere_ports_mountpoint="/usr/local/poudriere/ports" default_poudriere_tree="default" default_merge_from="freebsd myrepo"</code></pre> Some key points here are that the two items listed in for the default_merge_from</code> argument need to have scripts present in the /usr/local/etc/portshaker.d</code> directory. Further more, the combination of the poudriere_ports_mountpoint</code> and default_poudriere_tree</code> needs to be a ports tree that is then registered with poudriere. Next, we need to tell portshaker how to go off and fetch our two types of ports trees, freebsd</code> and myrepo</code>. For the freebsd</code> ports tree, create /usr/local/etc/portshaker.d/freebsd</code> with the following contents and make it executable:
#!/bin/sh . /usr/local/share/portshaker/portshaker.subr method="portsnap" run_portshaker_command $*</code></pre> Next, create a similar script to handle our repository containing our custom port. /usr/local/etc/portshaker.d/myrepo</code> should contain the following and similarly be executable:
#!/bin/sh . /usr/local/share/portshaker/portshaker.subr method="git" git_clone_uri="http://github.com/scottsturdivant/packaging.git" git_branch="master" run_portshaker_command $*</code></pre> Obviously replace the git_clone_uri</code> and git_branch</code> variables to reflect your actual configuration. For more information about the values and what they can contain, consult man portshaker.d</code>
Now, portshaker should be all set. Execute portshaker -U</code> to update your merge_from</code> ports trees (freebsd</code> and myrepo</code>). You'll see the standardportsnap</a> fetch and extract process as well as a git clone. After a good bit of time, these will both be present in the /var/cache/portshaker</code> directory. Go ahead and merge them together by executing portshaker -M</code>.
Hooray! You now have /usr/local/poudriere/ports/default/ports</code> that is a combination of the normal ports tree and your custom one.
We're effectively complete with configuring portshaker. Whenever your port is updated, just re-run portshaker -U</code> and portshaker -M</code> to grab the latest changes and perform the merge.
Poudriere</h2> Poudriere is a good tool for building ports. We will use it to handle our merged directory. Begin by configuring poudriere (/usr/local/etc/poudriere.conf</code>):
NO_ZFS=yes FREEBSD_HOST=ftp://ftp.freebsd.org RESOLV_CONF=/etc/resolv.conf BASEFS=/usr/local/poudriere USE_PORTLINT=no USE_TMPFS=yes DISTFILES_CACHE=/usr/ports/distfiles CHECK_CHANGED_OPTIONS=yes</code></pre> Really there's nothing here that is specific to the problem at hand, so feel free to consult the provided configuration file to tune it to your needs. Now, the step that is specific is to set poudriere up with a ports tree that it does not manage, specifically our resultant merged directory. If you consult man poudriere</code>, it specifies that for the ports</code> subcommand, there is a -m method</code> switch which controls the methodology used to create the ports tree. By default, it is portsnap. This is confusing as in our case, we do not want poudriere to actually do anything. We want it to just use an existing path. Fortunately, there is a way!
The poudriere wiki has an entry for using the system ports tree</a>, so we adopt it for our needs by executing:
poudriere ports -c -F -f none -M /usr/local/poudriere/ports/default \ -p default</code></pre> If you've consulted the poudriere manpage, you'll see that the -F</code> and -f</code> switches both reference ZFS in their help. As we're not using ZFS, it's not clear how they will behave. However, in conjunction with the custom mountpoint (-M /usr/local/poudriere/ports/default</code>), we ultimately wind up with what we want, a ports tree that poudriere can use, but does not manage: # poudriere ports -l PORTSTREE METHOD PATH default - /usr/local/poudriere/ports/default</code></pre> Note that this resulting PATH is the combination of the poudriere_ports_mountpoint</code> and default_poudriere_tree</code> variables present in our /usr/local/etc/portshaker.conf</code> configuration file. Building software from your custom ports tree</h2> Go ahead and create your jail(s) like you normally would (i.e. poudriere -c -j 92amd64 -V 9.2-RELEASE -a amd64</code>) and any other configuration you would like, and then go ahead and build myport</code> with poudriere bulk -j 92amd64 -p default category/myport</code>. Success!

IPv6 -- getaddrinfo() and bind() ordering with V6ONLY 2012-08-07T22:39:56+00:00 Recently I ran into an issue that took me a while to sort out, and it is regarding inconsistent behaviour on various OS's with regards to IPv6 sockets (AF_INET6</code></a>1</a>) and calling bind(2)</code></a> after getting the results back from getaddrinfo(3)</code></a>. A call to getaddrinfo()</code> with the hints set to AF_UNSPEC</code></a> in ai_family</code> and AI_PASSIVE</code></a> in ai_flags</code> will return to us 1 or more results that we can bind()</code> to. Sample code for that looks like this: struct addrinfo hints, *addrlist; memset(&hints, 0, sizeof(hints)); // Ask for TCP hints.ai_socktype = SOCK_STREAM; // Any family works for us ... hints.ai_family = AF_UNSPEC; // Set some hints hints.ai_flags = AI_PASSIVE | // We want to use this with bind AI_ADDRCONFIG; // Only return IPv4 or IPv6 if they are configured int rv; if ((rv = getaddrinfo(0, "7020", &hints, &addrlist)) != 0) { fprintf(stderr, "getaddrinfo: %s", gai_strerror(rv)); return 1; } // Use the list in *addrlist for (addr = addrlist; addr != 0; addr = addr->ai_next) { // use *addr as appropriate } // Clean up the memory from getaddrinfo() freeaddrinfo(addrlist);</code></pre> On Linux there are two entries returned when the host it is run on has both IPv4 and IPv6 enabled. An AF_INET</code> which was followed by an AF_INET6</code>. Now, it is not said that you are required to use all of the results that are returned, but if you want to listen on all address families it is off course suggested. Following the steps below for each of the returned results should result in having 1 or more different sockets that are bound to a single port. Create the socket()</li> Set any socket options you want (SO_REUSEADDR</code> for example)</li> Then bind() the socket</li> After that call listen() (followed off course by accept() on the socket)</li> </ol> Only for some unknown reason (and errno</code> is no help) bind()</code> fails when you get to the AF_INET6</code>, which was returned second. Searching online as to why the bind would fail doesn't give you any good results and the thing that is even worse is that if you run the same code on another platform such as FreeBSD</a>, OpenIndiana</a> or Mac OS X</a> no such failure exists. However I started suspecting something was up when I started looking at the output from netstat -lan | grep 7020</code> on Mac OS X. Where 7020 is the port I passed into getaddrinfo()</code>. tcp46 0 0 *.7020 *.* LISTEN tcp4 0 0 *.7020 *.* LISTEN </code></pre> Wait a minute ... one of the sockets is on both IPv4 and on IPv6. Some more time spent searching the internet I came across RFC 3493 section 5.3</a>, which is titled "IPV6_V6ONLY</code> option for AF_INET6</code> Sockets". As stated in section <3.7 Compatibility with IPv4 Nodes>, AF_INET6</code> sockets may be used for both IPv4 and IPv6 communications. Some applications may want to restrict their use of an AF_INET6</code> socket to IPv6 communications only. </blockquote> This was going down the right route, so I changed my code so that in the steps listed above in number 2 I added the following code if the socket type is AF_INET6</code>: if (setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, &yes, sizeof(int)) == -1) { close(sockfd); fprintf(stderr, "setsockopt: %s IPV6_V6ONLY\n", strerror(errno)); continue; }</code></pre> The RFC 3493 section 5.3 also states that this option should be turned off by default, which means that all IPv6 sockets can also communicate over IPv4. Thus technically setting the option manually in code the best way to fix the issue. FreeBSD has had this feature turned on (as in IPv6 sockets can only communicate with IPv6 and NOT IPv4) since 5.x. The biggest issue is that the remaining operating systems (OS X and OpenIndiana) don't have the same behaviour as Linux which makes troubleshooting this issue more difficult than it should be. The issue is that the RFC doesn't specify what exactly the operating should do when it encounters a request to bind to the same port on IPv4 and IPv6. The only place where I have found this documented is in "IPv6 Network Programming"</a> under "Tips in IPv6 Programming" chapter 4, section 4, appropriately titled "bind(2) Ordering and Conflicts". If you get a bind()</code> error when attempting to bind to an AF_INET6</code> socket please make sure that you set the socket option IPV6_V6ONLY</code> on the AF_INET6</code> socket. The default as required by RFC 3493 is to have that option be off. The default is wrong, and the RFC should have been more specific regarding what the right behaviour is when attempting to bind on an AF_INET6</code> socket when already bound on an AF_INET</code> while IPV6_V6ONLY</code> is set to false. The full code that I used for testing, along with a little bit more information is available as a gist on github</a>. The old BSD style socket()</code></a> called for defines starting with PF_</code> such as PF_INET</code> and PF_INET6</code> with the PF</code> standing for protocol family. POSIX starts them with AF_</code>, and calls them an address family. On almost every operating system PF_INET</code> is the same as AF_INET</code>. If the define doesn't exist you can always create it. ↩</a> </li> </ol> </section> FreeBSD 8.2 to 9.0 update with ZFS on root (mirror) 2012-04-24T18:36:20+00:00 Upgrading from one version of FreeBSD to another has become much simpler with freebsd-update</code></a> than before, a full rebuild of the FreeBSD kernel/world source used to be required (other upgrade paths existed but were not widely touted). freebsd-update</code> works extremely well when using UFS as your file system, there are however some gotcha's that one needs to look out for when using FreeBSD Root on ZFS. Two machines needed upgrading, both are running with a ZFS root mirror on FreeBSD 8.2-RELEASE. I had followed the FreeBSD Root on ZFS (mirror)</a> wiki article and everything has been running smoothly ever since. pool: zroot state: ONLINE scrub: scrub completed after 0h35m with 0 errors on Mon Mar 5 10:58:19 2012 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors</code></pre> Wanting to take advantage of some of the advancements made in FreeBSD 9.0-RELEASE</a> such as the updated ZFS, a bug that was fixed that stopped jails from owning a ZFS file system, improved dtrace, and better IPv6 support required that the plunge be taken and an upgrade was in order. There were a couple reports of people having done source upgrades and then having issues booting their system, but I found one mailling list post</a> that suggested that everything went well using freebsd-update</code>, so that is the route I went with. I read through the entirety of the FreeBSD 9.0-ERLEASE Release Notes</a> before moving forward to make sure I didn't miss any important changes that would cause my system to not restart correctly or would require changing system configuration files before attempting the upgrade. I would advise you to do the same. I ran freebsd-update -r 9.0-RELEASE upgrade</code> and after accepting the list of components that were installed, I got an error saying: # freebsd-update -r 9.0-RELEASE upgrade Looking up update.FreeBSD.org mirrors... 4 mirrors found. Fetching metadata signature for 8.2-RELEASE from update2.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. The following components of FreeBSD seem to be installed: kernel/generic src/base src/bin src/cddl src/contrib src/crypto src/etc src/games src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin world/base world/catpages world/dict world/doc world/games world/info world/lib32 world/manpages The following components of FreeBSD do not seem to be installed: world/proflibs Does this look reasonable (y/n)? y Fetching metadata signature for 9.0-RELEASE from update2.freebsd.org... done. Fetching metadata index... done. The update metadata is correctly signed, but failed an integrity check. Cowardly refusing to proceed any further.</code></pre> Specifically the line stating "Cowardly refusing to proceed further", a quick Google</a> later I found the 9.0-RC1 mailling list post</a> that suggested the following one line fix: sed -i '' -e 's/=_/=%@_/' /usr/sbin/freebsd-update</code></pre> After that quick fix, a re-run of freebsd-update</code> functioned without issues and finished, showing me what it would change, asking me if it seemed acceptable and telling me what it was going to remove and what was going to be installed. Next we run freebsd-update</code> for the first time in this three step process: freebsd-update install</code></pre> Don't restart yet (although it tells you to)! We need to go back and make sure we update the boot records first. Going back to the mailling list post made by Magnus Strahlert</a>, I made note of the following: Had to upgrade the gpt bootcode as instructed when upgrading the zpool for the system to boot. </blockquote> This is the only comment that worried me, and nowhere in the email thread does the poster describe what commands he ran. In any case I figure he meant re-running the boot code commands from the FreeBSD Root on ZFS wiki article: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1</code></pre> After running these commands, it told me that it had updated the boot code and then I ran: shutdown -r now</code></pre> After the machine came back online (it is in those moments when you have a small heartattack when the machine doesn't come up as fast as you had expected ...) you log back in and run: freebsd-update install</code></pre> This will install all of the updated binaries that came with FreeBSD 9.0, at this point FreeBSD update will tell you to upgrade all installed software and ports, and then run freebsd-update</code> one more time to remove all unnecessary shared libraries. If you want to have certain software continue functioning, yet you want freebsd-update</code> to remove any other remaining old 8.x files, you can install misc/compat8x</code></a> from the ports tree. This is the suggested route to go if you want to keep the 8.x libraries around for compatibility reasons. cd /usr/ports/misc/compat8x make install clean # or # portmaster misc/compat8x</code></pre> I use portmaster</code> and the following command will simply rebuild every port installed on the system: portmaster -aPf</code></pre> The -P</code> to portmaster</code> means look for package and if available use it, otherwise build from port. Run freebsd-update</code> one last time: freebsd-update install</code></pre> Since we are running ZFS we will most likely also want to upgrade our pool to the latest version of ZFS to take advantage of any new goodies (such as deduplication!), upgrading a zpool is easy: zpool upgrade zroot</code></pre> That upgrades to the latest pool version, now we will want to upgrade our existing file systems to the latest version of ZFS, do note that this will make them incompatible with older file system verisons (so if you use zfs send for backups, your other endpoint will need to be of the same ZFS version) zfs upgrade # List all ZFS file systems that need to be upgraded zfs upgrade -a # Upgrade all file systems</code></pre> After this one last reboot to make sure that all the processes come back online using the newer libraries and that you didn't accidentally miss something. shutdown -r now</code></pre> At this point the system has been upgraded from FreeBSD 8.0-RELEASE to FreeBSD 9.0-RELEASE and has all of the latest updates and enhancements. NAT with PF on an interface with multiple IP addresses 2011-09-08T02:40:00+00:00 If you want to do NAT for your currently running jail instances on FreeBSD so that they can have outgoing connections you could try the following pf.conf</code> to set up NAT on the interface that the jails have their IP addresses on. ext_if="em0" # The network card your default gateway is on jail_if="lo1" # The interface that your jails have IP addresses on # Set some options set optimization aggressive set block-policy drop set skip on lo # NAT on the external interface when coming from the jail interface nat on $ext_if from $jail_if:network:0 to any -> ($ext_if) # We just pass everything pass quick all</code></pre> What I have done is create IP addresses within the private network range (10/8</code>, 172.16/12</code>, 192.168/16</code>) on the lo1</code> interface. These can't be in the 127/8</code> range because those addresses can't be NAT'ed (not sure if this is a limitation in pf or if this is a FreeBSD limitation), which is a shame because using 127.1.0.1/24</code> would be pretty awesome in my opinion. What I did find though is that the above will not work correctly if your main network card (the one your default gateway is on, in the example em0</code>) contains multiple IP addresses. At that point the syntax ($ext_if)</code> does not function correctly and will cause packet loss/drop. 1</a> So instead of using the syntax above we simply replace ($ext_if)</code> with the actual IP address of the interface. This takes care of the issue and will let your jails have proper internet access without issues. ext_if="em0" jail_if="lo1" set optimization aggressive set block-policy drop set skip on lo nat on $ext_if from $jail_if:network:0 to any -> 192.168.1.2 # Your IP! pass quick all</code></pre> The other way you can solve this problem off course is to provide your jails with IP addresses within the same range that go to your default gateway thereby solving the problem of needing to NAT in the first place. Do note that I am using FreeBSD 8.2, so this may be fixed in the new FreeBSD 9 that is due to be released soon. ↩</a> </li> </ol> </section> FreeBSD running within KVM: Serial Console 2011-08-21T18:53:52+00:00 Instead of using virtualisation technology such as VMWare of VirtualBox I decided to give the new KVM</a> technology that is included with Linux. It is fast, and has been extremely reliable. However I wanted to be able to access the console on the guest FreeBSD instances. I used virt-manager, a graphical tool, to set up everything, including creating the virtual machine, I didn't want to have to deal with XML configuration files and whatnot. After installing FreeBSD there are a few changes I made to get a virtual console working, this allows virsh console <domain></code> to function as expected. First we add a new file to the root file system (which should be the first slice on the drive): echo "-Dh" > /boot.config</code></pre> See man 5 boot.config</a> for more information. Basically -D says that FreeBSD should boot with a dual console configuration (built-in over VGA) and serial console, and -h says to force the serial console to on. If you were to reboot now and use the virsh command: virsh console FreeBSD</code></pre> you would see the system go through the boot process and show output starting from when the boot block gains control, at this point however you still have no way to login to this new console since we never specified that we wanted any ttys on the serial console, for that we need to edit /etc/ttys. We are looking for the ttyu0 entry, we want to make sure to change dialup to vt100, and off to on, so that it should look like this: ttyu0 "/usr/libexec/getty std.9600" vt100 on secure</code></pre> Now when we reboot once more we will get the usual login display that we have come to expect of console on FreeBSD, and we can login over the serial console. This way we don't have to open up the virtual machine manager to get a console, and can continue to do almost everything from a terminal window. PostgreSQL setup for external connections 2011-08-16T17:48:54+00:00 PostgreSQL is a fantastic open-source RDBMS which is extremely powerful, once installed on FreeBSD getting it set up is pretty simple. We do want to change some of the defaults though. By default the system will be set up to trust any user using the local socket to connect as any user to any database. By setting up the initdb</code></a> flags we can change how PostgreSQL generates the default configuration files. Adding -A md5</code> tells it to use the md5 authentication mechanism</a>, which requires that passwords sent over the wire are md5 hashed. Add the following to /etc/rc.conf</code></a> for the initdb flags, and enable PostgreSQL as a service: postgresql_initdb_flags="-D /usr/local/pgsql/data -W -A md5" postgresql_enable="YES"</code></pre> And then run: /usr/local/etc/rc.d/postgresql initdb</code></pre> This will initialise and create the configuration as well as setting up some of the default tables. You will want to navigate to the folder specified in the initdb_flags</code> above: cd /usr/local/pgsql/data</code></pre> this is where you will find the files that we are going to modify next to get PostgreSQL set up to listen to connections coming in on a private network (192.168.1.1/24</code>) and where we allow connections coming in from servers on that network. This is strictly for setting up PostgreSQL in a situation where you have multiple servers accessing it as a client on a private network. After this we need to modify a few files to get it to listen to TCP/IPv4 connections, first we start with postgresql.conf</code></a> and add: listen_addresses = '127.0.0.1,192.168.1.1'</code></pre> Then in pg_hba.conf</code></a> we add a new line: host all all 192.168.1.1/24 md5</code></pre> This allows anyone in the 192.168.1.1/24 range to connect to our PostgreSQL instance and to use the databases. This is exactly what we want to allow, now we start PostgreSQL using the rc script: /usr/local/etc/rc.d/postgresql start</code></pre> Setting up jails with multiple IPs and providing it with internet access 2011-08-15T15:40:04+00:00 I have multiple VMs up and running and generally I have an internal to the VMs only network set up, in the current project I am working on I have three completely different networks set up. DHCP assigned network that routes to the outside network on the host (192.168.0.1/24)</li> An internal to the VM's network interface (10.99.12.1/24)</li> A network that is tied to the host's wireless network (setup to be an AP) (10.99.11.1/24)</li> </ol> So within my FreeBSD VM I have the following: em0</code>, em1</code>, em2</code> as "network cards". In my FreeBSD VM, I am creating two jails that are going to contain different pieces of server software. The jails need IP's in the following ranges, 10.99.12.1/24 (to connect to another VM), 10.99.11.1/24 (Wifi network) and then it needs to have an IP address that is considered "local" and we can NAT on, so in addition to the above networks we also needed to add another network. The easiest way to create new interface with IP addresses is using a bridge. They can be assigned IP addresses, which is all we need. At this point inside of a jail it kinda looks like this: em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:6b:6a:49 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:2f:a5:55 inet 10.99.12.4 netmask 0xffffffff broadcast 10.99.12.4 media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 52:54:00:bb:b6:43 inet 10.99.11.14 netmask 0xffffffff broadcast 10.99.11.14 media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3<RXCSUM,TXCSUM> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether da:de:63:c9:55:6c inet 10.0.1.4 netmask 0xffffffff broadcast 10.0.1.4 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0</code></pre> em0</code> doesn't have an address since that is our DHCP address, em1</code>, em2</code> and bridge0</code> all have IP addresses, in the jail configuration the bridge0</code> IP address is listed first, and thus when something binds or sends it sends from that IP address. The thing is that at this point the jail does not have internet access. Jails inherit the routing table from their host environment (yes, you can compile in more routing tables ... but I don't want to modify stock), any traffic that goes to the outside is going to go out over the default route, thing is you'd expect that traffic to show up on bridge0</code> where the IP address exists, instead using tcpdump</code> you see it show up on em0</code>, which is where the default route is. I would have expected having to do NAT on the bridge, not just the external interface. Here is the pf</code> configuration. ext_if="em0" jail_if="bridge0" set optimization aggressive set block-policy drop set skip on lo nat on $ext_if from $jail_if:network:0 to any -> ($ext_if) pass quick all</code></pre> And we've got internet within our jail. Moving from mod_fastcgi to mod_fcgid 2009-08-25T23:22:03+00:00 Update: I've since moved to a PHP-FPM with mod_fastcgi</code> based setup which works much better than mod_fcgid</code> could ever dream of, taking process management away from Apache and putting it where it actually makes sense has improved server load and improved website speed. See my new PHP-FPM with mod_fastcgi post</a> for more information. -- Errors, all they cause is trouble. The dreaded 500 error showed up when visiting my favourite tech website, the one I am the administrator for. This time I had enough, there was going to be no more playing around with PHP settings, attempting to figure out why PHP was suddenly dying and or why mod_fastcgi</code> refused to retry a select() when it failed due to a system signal. The server in question runs Apache in MPM worker mode since a threaded Apache is going to be faster than a pre-fork, besides this server does not have as much memory so it seemed to be better to have multiple threads rather than multiple processes, each of which would have their own memory segment. There however is the issue that this server also needs to run PHP, the accepted method to do so is to use mod_php</code> along with Apache; however PHP is not thread safe. The alternative is FastCGI, basically it spawns PHP processes as a separate stand-alone process, with their own memory space, much like Apache pre-fork, however now when one process grows to big, or when a process is no longer needed it can be cleaned up, keeping memory to a minimum. Also, FastCGI is perfectly thread safe, this means that with Apache running in MPM worker mode we could now still run our PHP scripts even when they were not thread safe. Setting up mod_fastcgi</code> is not that hard, it takes some httpd.conf configuration values, and off course the loadmodule is assumed here: # Set up mod_fastcgi <IfModule `mod_fastcgi`.c> FastCgiIpcDir /var/tmp/fcgi-ipc/ FastCgiConfig -autoUpdate -singleThreshold 100 -killInterval 300 -idle-timeout 240 -pass-header HTTP_AUTHORIZATION AddHandler fastcgi-script .fcgi .fcg .fpl Action application/x-httpd-php5 /fastcgi-bin/php5.fcgi AddType application/x-httpd-php5 .php .php5 </IfModule> # Set up the script alias, basically anything in this directory gets executed as a fastcgi script ScriptAlias /fastcgi-bin/ "/usr/local/www/fastcgi-bin/" <Location /fastcgi-bin/> Options ExecCGI SetHandler fastcgi-script Order allow,deny Allow from all </Location></code></pre> Because of errors that mod_fastcgi</code> was throwing out at me, I figured mod_fcgid</code> is worth a try. However all of the configurations I found required me to add an FCGIWrapper line into each of my VirtualHost blocks, which was an immediate no-no since there is well over 100 of those on the server, and I'd rather spend my time doing other things. With some trial and error, and some Googling I put the following together for mod_fcgid</code>, tested it out, and once it worked perfectly set Apache off running with the following configuration: # Set up mod_fcgid <IfModule `mod_fcgid`.c> AddHandler fcgid-script .fcgi .fcg .fpl IPCCommTimeout 60 SocketPath /var/tmp/fcgi-ipc/ Action application/x-httpd-php5 /fastcgi-bin/php5.fcgi AddType application/x-httpd-php5 .php .php5 </IfModule> # Set up the script alias, basically anything in this directory gets executed as a fastcgi script ScriptAlias /fastcgi-bin/ "/usr/local/www/fastcgi-bin/" <Location /fastcgi-bin/> Options ExecCGI SetHandler fcgid-script Order allow,deny Allow from all </Location></code></pre> Notice how similar they both are, that was the whole goal. mod_fcgid</code> was supposed to be a drop in replacement, and thankfully it was. PHP was served as it once was, with a twist. The php5.fcgi script is as follows for those of you trying to set this up as well: #!/bin/sh # To use your own php.ini, comment the next line and uncomment the following one #PHPRC="/usr/local/etc" #export PHPRC PHP_FCGI_CHILDREN=4 export PHP_FCGI_CHILDREN exec /usr/local/bin/php-cgi</code></pre> Some things that that intrigued me is that my php processes seemed to be capped now. Where mod_fastcgi</code> would spawn processes but never remove them or kill them when they were no longer needed, mod_fcgid</code> keeps the running php processes to a sane limit instantly starting new processes when required to handle the requests coming in. This means that the server now has more free memory for MySQL, or file caches which has helped speed up other types of transfers as well. Even with the relatively short time that mod_fcgid</code> has been in place it has been faster, more reliable and more sane than mod_fastcgi</code>. Whichever one you pick, make sure to benchmark your server and check your error logs to solve common issues, the issue may not be what FastCGI module you picked but rather PHP itself that is causing the errors! FreeBSD GEOM striping and concatenation 2006-12-13T06:44:20+00:00 Some of the hard drives in my FreeBSD box. Just a little back-story: /dev/ad0 - 20 GB /dev/ad1 - 40 GB /dev/ad2 - 100 GB /dev/ad3 - 60 GB /dev/da0 - 8 GB /dev/da1 - 8 GB</code></pre> I had already made a gstripe(8)</code> out of the two SCSI drives, da[1|2]</code>, however I wanted to see if I could do the same with ad[1|2|3]</code>. As I did not want to resort to gvinum, which is rather heavy, and I was unable to figure out how it worked. The entire point behind gstripe is that since it is striped, the writes/reads are faster since it writes/reads to all the drives at the same time. So first I tried: gstripe label -v st1 /dev/ad1 /dev/ad2 /dev/ad3</code></pre> That however complained that it would only use 40 GB's on all of the drives, since the smallest drive in the collection was 40 GB. I then read up on gconcat(8)</code>, which concatenates all the drives together to look like one big drive, without the speed improvement of striping. So I figured, since FreeBSD's gstripe is GEOM based, it should not care about what the underlying data sink is, just that it is a device, so just maybe this could work: gconcat label -v gb100 /dev/ad1 /dev/ad3</code></pre> So now I had created a 100 GB "hd" at /dev/concat/gb100</code>. I now did what I was planning to do: gstripe label -v st1 /dev/ad2 /dev/concat/gb100</code></pre> No problem. I now had 180.48 GB, of which only about 166 GB was usable. Which is still excellent. As it still came with a speed improvement. Followed up with the standard advice from the handbook</a> I did the following: bsdlabel -wB /dev/stripe/st1 newfs -U /dev/stripe/st1a mount /dev/stripe/st1a /mnt</code></pre> Which got me the following: /dev/stripe/st1a 180G 5.5G 161G 3% /mnt</code></pre> Full: Filesystem Size Used Avail Capacity Mounted on /dev/ad0s1a 496M 63M 393M 14% / devfs 1.0K 1.0K 0B 100% /dev /dev/ad0s1e 496M 230K 456M 0% /tmp /dev/ad0s1f 15G 2.2G 11G 17% /usr /dev/ad0s1d 1.4G 115M 1.2G 9% /var /dev/stripe/st0a 16G 4.0K 15G 0% /usr/scsi /dev/stripe/st1a 180G 5.5G 161G 3% /mnt</code></pre> Next up I might actually add gdbe(8)</code> or geli(8)</code> even in front of the stripe, so that I will have an encrypted stripe. Talk about stacking. Might give gvinum a try sometime. All working: keyhole# gconcat status Name Status Components concat/gb100 UP ad1 ad3 keyhole# gstripe status Name Status Components stripe/st0 UP da0 da1 stripe/st1 UP ad2 concat/gb100</code></pre> FastCGI quick and dirty 2006-09-13T00:19:31+00:00 In this quick and dirty guide for FreeBSD I will assume you have already compiled and installed PHP5 with fcgi mode enabled, which can be checked by running: /usr/local/bin/php-cgi --version</code></pre> I will also assume you have installed mod_fastcgi</code> from www/mod_fastcgi</code> in the ports. Create the fast cgi ipc temp dir, make sure you place this somewhere it will keep existing: cd /var/tmp mkdir -p /var/tmp/fcgi-ipc/dynamic chmod -R 0777 /var/tmp/fcgi-ipc</code></pre> Let's create the directory where we are going to place our PHP wrapper, this config is for server wide using the same FastCGI PHP config. This is on purpose, since it will allow server wide PHP usage. If you want you could then split it up into virtualhosts if one required a standalone php running for that. mkdir /usr/local/www/fastcgi-bin/ chmod 755 /usr/local/www/fastcgi-bin/</code></pre> Next up we create the very simple wrapper: cat << EOF > /usr/local/www/fastcgi-bin/php5.fcgi #!/bin/sh # To use your own php.ini, comment the next line and uncomment the following one #PHPRC="/usr/local/etc" export PHPRC PHP_FCGI_CHILDREN=8 export PHP_FCGI_CHILDREN exec /usr/local/bin/php-cgi EOF</code></pre> Set PHP_FCGI_CHILDREN</code> to whatever value you want, that is how many processes it will use. I suggest the standard which is 8, if you make it more, you use more ram, you make it less Apache will have to wait if all of the PHP processes are in use. Next up, make it executable: chmod +x /usr/local/www/fastcgi-bin/php5.fcgi</code></pre> Last but not least, edit httpd.conf</code> and make sure to add a comment in front of the mod_php5</code> line, and uncomment the mod_fastcgi</code> line. Then add this to the end of your Apache config: <IfModule mod_fastcgi.c> FastCgiIpcDir /var/tmp/fcgi-ipc/ FastCgiConfig -pass-header HTTP_AUTHORIZATION AddHandler fastcgi-script .fcgi .fcg .fpl Action application/x-httpd-php5 /fastcgi-bin/php5.fcgi AddType application/x-httpd-php5 .php .php5 </IfModule> ScriptAlias /fastcgi-bin/ "/usr/local/www/fastcgi-bin/" <Location /fastcgi-bin/> Options ExecCGI SetHandler fastcgi-script Order allow,deny Allow from all </Location></code></pre> That's it. Restart apache and watch as the first time you access a PHP script FastCGI will spawn several of them, and they start serving requests. Load usage may seem higher, that is because unlike Apache the standalone running PHP's handle the jobs as they come in with whatever one is first, Apache will try to put as much work in one of its preforked modules. As the PHP processes come and go through FastCGI the memory usage will go up and down. Mine has an average of about 300 MB, which is very good compared to the 800 MB Apache with mod_php</code> would rack up. Backup Script 2006-09-09T00:33:02+00:00 I was working on some backup scripts and I decided I might as well generalise almost all of it (I am sorry about the hard coded paths) so almost anyone can come grab it and use it on their servers. The shell script is pretty self explanatory and I am not very good at documenting code, so the script source itself is your best bet as to why I did something some way. This script is in production use on several servers I administrate, I hope someone can use it as well. Documentation on how to use it: A backup script which will mount /backup</code>, then use rsync to copy files from /usr/home</code> to /backup/yyyy-mm-dd</code>. After it is done, it will remove the oldest backup, 14 days ago. All paths are hardcoded. If a file .exclude</code> is in a user's homedir, it is looked at, and is used to exclude certain directories or files from the rsync. For example: In /usr/home/xistence/.exclude</code>: public_html/iso</code></pre> No trailing slash, this will cause the folder /usr/home/xistence/public_html/iso</code> to not be backed up. This might be for various reasons, but for this example it is because backing up GB sized files should not be something that is done, only because it takes ages and wastes valuable space. Thing to keep in mind is, if you want to backup a 40 GB HD, with 22 GB used, to keep even a 3 day backup of that would require 3 * 22 GB, or 66 GB of available space on the backup drive. It is therefore in one's best interest to exclude very large files, and back them up in a different method since they are most likely not going to change much across revisions. This script should be set to cron every 30 minutes, as rsync will do one full backup when the day changes, and then from there on do incremental backups. Maybe removing the --delete</code> option might help with customers that delete files and expect them to still be in the backup for that day, this way however if a file is removed it will still exist in the backup until it is deleted after 14 days. #!/usr/local/bin/bash # This file REQUIRES bash, and can not be run under /bin/sh! ### # Copyright 2006 Delta Regeer. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # The views and conclusions contained in the software and documentation are # those of the authors and should not be interpreted as representing official # policies, either expressed or implied, of bsdPanel project. # ### for HOMEDIR in `/usr/bin/find /usr/home -type d -maxdepth 1 | grep -v '^.$'`; do if [ -f $HOMEDIR/.exclude ]; then echo "Exclude found for $HOMEDIR" while read line; do if [ "${line:0:1}" != "/" ]; then # They are missing a full path, we will be nice and add it TMPLINE="$HOMEDIR/$line" line=$TMPLINE fi echo " Excluding $line" echo $line >> /tmp/exclude.$$ done < $HOMEDIR/.exclude fi done mount | grep /backup if [ $? -ne 0 ]; then /sbin/mount /backup if [ $? -ne 0 ]; then logger -s -p user.err -t backup "Backup drive could not be mounted! Bailing!" /sbin/umount -f /backup exit 1 fi fi BACKUPDIR="/backup/`date "+%Y-%m-%d"`" OLDDIR="/backup/`date -v-14d "+%Y-%m-%d"`" if [ ! -d $BACKUPDIR ]; then mkdir $BACKUPDIR fi if [ $? -ne 0 ]; then logger -s -p user.err -t backup "Could not create backup folder $BACKUPDIR. Bailing!" /sbin/umount -f /backup exit 1 fi # We got this far, now we have to run rsync. Rsync is very verbose on purpose /usr/local/bin/rsync -av --relative --delete /usr/home/ $BACKUPDIR --exclude-from=/tmp/exclude.$$ if [ $? -ne 0 ]; then logger -s -p user.err -t backup "rsync failed. Do not trust backup ($BACKUPDIR)" /sbin/umount /backup exit 1 fi echo "Backup complete! Deleting oldest backup (14 days ago)" if [ -d $OLDDIR ]; then rm -rf $OLDDIR fi /sbin/umount /backup /bin/rm -f /tmp/exclude.$$ logger -s -p user.notice -t backup "Backup complete at $BACKUPDIR."</code></pre>