ZFS runs *really* slowly when free disk usage goes above 80%

0 Flares 0 Flares ×

You’re sat at your desk, sipping a nice beverage, all is well in the world. Your busy database is sat happily in the background as always. Suddenly, out of the blue, performance drops significantly. You’re perplexed – nothing has changed. The DBAs go wild – fingers get pointed. They claim the DB is fine – the hardware/OS is at fault. You investigate, you see a lot of IO, but you can’t pin it down. What’s going on?

Well, you could very well just have hit 80% disk space usage, and now your disk performance has gone through the toilet.

You can fix the issue by running this:

echo "metaslab_df_free_pct/W 4" | mdb -kw

And you can make it permanent by doing:

echo "set zfs:metaslab_df_free_pct=4" >> /etc/system

What does this do? Well, ZFS normally uses “first fit” block allocation policy. When you hit 80% disk space usage, it switches to best fit. To quote the source code:

     50  * The minimum free space, in percent, which must be available
     51  * in a space map to continue allocations in a first-fit fashion.
     52  * Once the space_map's free space drops below this level we dynamically
     53  * switch to using best-fit allocations.

All current Solaris 10 releases, and versions of OpenSolaris prior to 22-Nov-2009 use a default of 30 for this value. How does a value of 30 equate to 80% disk space usage? I have no idea – I’ve never figured that one out. All I know is, run the above commands, and the problem goes away. Perhaps someone with more knowledge of how metaslabs work can enlighten me :)

0 Flares Twitter 0 Facebook 0 Google+ 0 Reddit 0 LinkedIn 0 StumbleUpon 0 Buffer 0 Email -- 0 Flares ×

Tags: , , , , , , , , , , , , ,

3 Responses to “ZFS runs *really* slowly when free disk usage goes above 80%”

  1. OpenSolaris has changed this in onnv revision 11146 but along with other changes.

    See the following OpenSolaris Bug IDs:

  2. [...] I was aware that the more a zpool got filled up the more effort it was for the system to find free blocks and that this leads to slower performance. What I did not know is that zfs actually switches to a different algorithm to find free blocks at a certain level and that this level is configurable with metaslab_df_free_pct. Older releases switch at about 80% full and try to find the “best fit” for new data which slows things down even more. Read more about it here. [...]

  3. Alasdair says:

    You can also get poor performance with imbalanced zpools.

    Let’s say you’ve got 4 mirrored pairs of disks (so 8 disks in total), and these fill to 80%. Lets say you now add another 4 mirrored pairs (so you now have 16 disks). You’ve suddenly got lots of free space, but you can still hit the aforementioned issue as ZFS tries to squeeze data into the full disks instead of quickly using the empty ones.

    George Wilson at Delphix fixed this in Illumos just over a year ago:


    This fix is therefore available in recent releases of OpenIndiana, SmartOS and OmniOS. I have no idea if Oracle have also addressed this separately in Solaris 10 or Solaris 11.

Leave a Reply

Back to top