Using mbuffer to speed up slow zfs send | zfs receive

By Alasdair Lumsden on 18 Jul 2010

Using MBuffer to Speed

So, you find yourself doing a zfs send | receive, perhaps a large incremental send. You find the transfer process going really slowly – trickling along at less than 1MB/sec. Yet, you know the sender and the receiver is capable of far far more than this. What’s the deal?

Well, basically, zfs receive is bursty – it can spend ages computing something, doing no receiving, then blat the data out to disk. The issue with this is that it stalls the sender, resulting in a bursty and slow transfer process.

The solution is to deploy mbuffer into the mix. MBuffer will buffer the stream, which you can do at both ends. While ZFS receive can’t receive, mbuffer buffers, when zfs receive can receive, mbuffer sends it data as fast as it can. Let’s see an example:

# Start the receiver first. This listens on port 9090, has a 1GB buffer,
    and uses 128kb chunks (same as zfs):

mbuffer -s 128k -m 1G -I 9090 | zfs receive data/filesystem

# Now we send the data, also sending it through mbuffer:

zfs send -i data/filesystem@1 data/filesystem@2 | mbuffer -s
    128k -m 1G -O

You’ll get a lovely output such as:

in @ 15.8 MB/s, out @ 8923 kB/s, 1.0 GB total, buffer   6% full

And hopefully, you’ll find your zfs send|receive suddenly go a lot lot quicker.

Here is the link to download the useful mbuffer program.

I personally found this approach decreased my send time for 40GB from over 4 hours down to around 30 minutes. Hurrah! :)