Tuning ipfilter on SmartOS

By Alasdair Lumsden on 23 Jul 2013

Recently we were having some connectivity issues between EveryCity and the Joyent Public Cloud in Amsterdam, and as part of the troubleshooting I built iperf3 and fired it up to perform some tests.

Rather quickly, I hit some strange errors:

[root ~]# iperf3 -c 95.131.y.y  -u -p 5779 -b 1m -t 10 -i 1 -f m ; done
Connecting to host 95.131.y.y, port 5779
[  4] local 37.153.x.x port 35482 connected to 95.131.y.y port 5779
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec  96.0 KBytes  0.78 Mbits/sec
[  4]   1.01-2.03   sec   112 KBytes  0.90 Mbits/sec
[  4]   2.03-3.00   sec   112 KBytes  0.94 Mbits/sec
[  4]   3.00-4.00   sec   104 KBytes  0.85 Mbits/sec
[  4]   4.00-5.03   sec  88.0 KBytes  0.70 Mbits/sec
[  4]   5.03-6.00   sec   112 KBytes  0.94 Mbits/sec
[  4]   6.00-7.00   sec   104 KBytes  0.85 Mbits/sec
[  4]   7.00-8.00   sec   104 KBytes  0.85 Mbits/sec
[  4]   8.00-9.05   sec   104 KBytes  0.81 Mbits/sec
iperf3: error - unable to write to stream socket: Network is unreachable
Connecting to host 95.131.y.y, port 5779
[  4] local 37.153.x.x port 55454 connected to 95.131.252.130 port 5779
iperf3: error - unable to write to stream socket: Network is unreachable

Very strange – iperf is unable to send packets. Why might that be? Well I had a hunch, and this box was running ipfilter – I disabled ipfilter, and things started working again (note that the iperf server won’t time out quickly, so if the test fails in the middle, you need to restart the iperf server).

With a stateful firewall, outbound communication must be tracked to allow the return packets in, and this uses up state in a table in the kernel. If it becomes full, you can’t track more connections until the old state times out. You can check to see if you’re losing packets due to the state table overflowing by looking at the lost counters via ipfstat:

# ipfstat  | grep ^packet
packet state(in):       kept 2898599    lost 1654
packet state(out):      kept 384694     lost 325

You can get traffic flowing again by flushing the ipfilter state table with “ipf -F S”, but this will up quickly again as the default size is very conservative. I found the rather excellent article over at letsgetdugg.com on how to increase the ipf state table size, and this fixed the problem. iperf was also using UDP, which is a connectionless protocol and harder to track and time out. The default timeout is 240 seconds, so I lowered this to 30.

The short version of how to do this is:

# svcadm disable -s ipfilter
# ipf -T fr_statemax=113279,fr_statesize=151007,fr_udptimeout=30
# svcadm enable ipfilter
# ipf -F S

The problem was, I wanted to make this change persistent across Zone reboots. On the Joyent Public Cloud I had no access to change the IPF kernel driver settings as suggested on letsgetdugg.com. So to work around this, I wrote a transient SMF service to apply the settings post-boot:

cat <<EOF > /var/svc/method/ipf-tuning
#!/sbin/sh

#
# This file and its contents are supplied under the terms of the
# Common Development and Distribution License ("CDDL)". You may
# only use this file in accordance with the terms of the CDDL.
#
# A full copy of the text of the CDDL should have accompanied this
# source. A copy of the CDDL is also available via the Internet at
# http://www.illumos.org/license/CDDL.
#

#
# Copyright 2013 EveryCity Ltd. All rights reserved.
#

. /lib/svc/share/ipf_include.sh

ipf_settings=`svcprop -p ipfilter/tunables $SMF_FMRI`

if [[ "x$ipf_settings" == "x" ]] ; then
  echo "Error: No tunables set in $SMF_FMRI under ipfilter/tunables"
  exit $SMF_EXIT_ERR_FATAL
fi

tune_ipf() {
  echo "Applying $ipf_settings"
  ipf -T $ipf_settings
  if [[ $? -ne 0 ]] ; then
    echo "Error: Unable to apply ipf settings"
    exit $SMF_EXIT_ERR_FATAL
  fi
}

# Check to ensure ipf is running
state=`svcs -H -o state $IPFILTER_FMRI`
if [[ "x${state}" == "xonline" ]] ; then
  svcadm disable -s $IPFILTER_FMRI && 
  tune_ipf && 
  svcadm enable -s $IPFILTER_FMRI
  if [[ $? -ne 0 ]] ; then
    echo "Error: Problem when disabling, tuning and then starting ipfilter"
    exit $SMF_EXIT_ERR_FATAL
  fi
elif [[ "x${state}" == "xdisabled" ]] ; then
  tune_ipf
else
  echo "Error: Unable to operate on ipfilter when in state $state"
  exit $SMF_EXIT_ERR_FATAL
fi

exit $SMF_EXIT_OK
EOF
chmod 755 /var/svc/method/ipf-tuning

cat <<EOF > /tmp/ipf-tune.tmp
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
//
// This file and its contents are supplied under the terms of the
// Common Development and Distribution License ("CDDL)". You may
// only use this file in accordance with the terms of the CDDL.
//
// A full copy of the text of the CDDL should have accompanied this
// source. A copy of the CDDL is also available via the Internet at
// http://www.illumos.org/license/CDDL.
//
//
// Copyright 2011 EveryCity Ltd. All rights reserved.
//
-->
<service_bundle type='manifest' name='ipfilter-tuning'>
  <service name='network/ipfilter/tuning' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <dependency name='milestone' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/ipfilter:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='/var/svc/method/ipf-tuning' timeout_seconds='60'/>
    <exec_method name='stop' type='method' exec=':true' timeout_seconds='60'/>
    <property_group name='startd' type='framework'>
      <propval name='duration' type='astring' value='transient' />
    </property_group>
    <property_group name='ipfilter' type='application'>
      <stability value='Evolving' />
      <propval name='tunables' type='astring' value='fr_statemax=113279,fr_statesize=151007' />
    </property_group>
  </service>
</service_bundle>
EOF
svccfg import /tmp/ipf-tune.tmp

This will then run at boot time after ipfilter has come online. To adjust the tunables you can do:

# svccfg -s ipfilter/tuning setprop ipfilter/tunables = ""fr_statemax=ABC,fr_statesize=XYZ""

Enjoy!