Performance comparison with Ubuntu 20.04
-
TrueNAS was a little slower than Ubuntu 20.04 on 1MiB random writes—which doesn’t matter, unless you’ve got terabit(!) Ethernet. Writes over gigabit Ethernet bottleneck at around 110MiB/sec, about an eighth of TrueNAS’s storage bottleneck here.
Jim Salter -
TrueNAS loses the race again on 4KiB sync write throughput. While this is well underneath typical network bottlenecks, it’s still unlikely to be a problem for most NAS use-cases.
Jim Salter -
Latency, not throughput, is really the killer metric on 4KiB writes. What we’re seeing here are the average and worst-case times to sync a single block to disk.
Jim Salter
TrueNAS was noticeably slower than Ubuntu 20.04 in our limited fio testing, using a set of eight Ironwolf 12TB disks in a single RAIDz2 vdev. With that said, we don’t think the performance difference is likely to actually matter for most NAS users.
We tested two simple metrics here—1MiB async write throughput and 4KiB sync write latency. In each case, we used eight parallel processes with an I/O queue depth of 8, and we tuned recordsize to match the operation block size. Sticking to eight processes and iodepth=8 gives the operating system and filesystem implementation a little extra breathing room in which to attempt to efficiently order writes. This helps expose differences in the software stacks, since the actual hardware remains unchanged.
For the 1MiB tests, all we really need to do is outrun the network. Gigabit Ethernet bottles at roughly 110MiB/sec, so although Ubuntu “outran” TrueNAS here, it really doesn’t matter—TrueNAS is already beating the network by a solid factor of eight.
For the 4KiB test, we included throughput numbers—but mostly just for comfort. The interesting statistic here isn’t throughput, it’s latency. What we’re really doing here is finding out how long it takes to sync an individual 4KiB block to disk. In addition to the average latency—which is just the inverse of throughput—we can check peak latencies here.
TrueNAS is noticeably slower than Ubuntu here, with thirty-three percent higher average latency, and more than double the peak latency. This still should not typically be considered a deal-breaker for TrueNAS. Most NAS workloads won’t rely heavily on 4KiB operations—the kind of photos, movies, and other large files most users primarily store on NAS appliances won’t be affected by poor 4KiB sync performance.
Advertisement
If you want to get deeper into TrueNAS’s full featureset than we did and run VMs or potentially database-backed services directly on the system, those 4KiB numbers will start becoming more important. But if you’re doing that, you really shouldn’t be relying on a wide RAIDz2 vdev the way we did here, either. In order to accelerate that workload, you’ll instead want mirrors, narrower stripes, and possibly a LOG vdev as well.
Conclusions
We’ve been following TrueNAS Core’s development off and on since March, and we’re pretty pleased with it. Unifying the codebase between FreeNAS and TrueNAS allows iXsystems to devote more focus to developing a single, coherent, effective interface—and to squashing bugs as they pop up.
FreeNAS, and now TrueNAS Core, have come a long way in the past several years. TrueNAS Core is an easy way for a home admin or hobbyist who’s a little nervous about the command line to maintain a truly robust, feature-rich ZFS storage server. It’s also good for potential TrueNAS Enterprise customers to get their feet wet with a free edition that looks just like what they’ll be working with if they pull the trigger on a commercial license.
Although we didn’t see the best absolute performance out of TrueNAS, we don’t think the differences in performance between it and Ubuntu will be as relevant for most users as the expansive interface and easy setup are. There are also plenty of ways to drastically increase performance above the simple setup we explored here; for instance, a fast LOG vdev would have accelerated those 4.8MiB/sec 4KiB writes to 100MiB/sec or more.
TrueNAS Core is still in beta, and unlike some open source projects that live in fully production-ready “beta” status for years, it means it. We aren’t ready to recommend TrueNAS Core for prime-time use by users who aren’t prepared to take bugs in stride and actively participate in finding them, reporting them, and living with them in the meantime.
But this is an exciting project with an unusually deep featureset, and we look forward to looking again after the bugs are ironed out and it reaches release status later this year.
