Slow transfer rates to RAID5 array in FreeNAS file server

T

TC10284

Limp Gawd
Joined
Jan 8, 2006
Messages
213
I am considering tearing down an old dual P3 1GHz FreeNAS file server with 4x WD 1.5TB Green hard drives in RAID5. It has been consistently giving me 6MB/sec transfer rates. It does only have a Intel 100mbps PCI NIC, but it should top out at around 12.5MB/sec then.

However, if I rebuild it in another machine how am I to know whether it could be the PCI SATA RAID controller that is just not working well with FreeNAS, or a fully saturated PCI bus causing the slowness? A PCI 2.3 bus should top out at 133MB/sec for 33MHz. When transferring to this fileserver, the CPUs hit 85% or so usage.

Here is the spec sheet of the Promise SATA controller: http://promise.com/media_bank/Download Bank/Datasheet/FT_TX2300_DS091406.pdf
I realize it says the card is RAID0/1 capable, but I am only using the ports for non-RAID mode and I'm running a software RAID5.

It has 3GB of PC133, dual P3 1GHz, and the latest version of FreeNAS. I cannot recall the motherboard model but it is a Gigabyte dual socket 370.

I have another FreeNAS file server with 4x 1TB SATA Seagate HDs on a Phenom 9600, 8GB of DDR2, onboard gigabit NIC, and all onboard SATA ports. I see transfer rates of 20-35MB/sec to that RAID5 array. Yes, I realize 8GB of RAM is probably overkill for FreeNAS. =)
 
Do some local tests first to rule out disk I/O performance:

On freenas execute something like:

# write /raid5/zerofile.000 file
dd if=/dev/zero of=/raid5/zerofile.000 bs=1m count=10000

# read /raid5/zerofile.000 file
dd if=/raid5/zerofile.000 of=/dev/null bs=1m

Warning: don't make mistakes with this command! Replace /raid5/zerofile.000 to the location of your RAID5 array, check with df -h to see where your RAID5 is mounted.

All this you should do while in SSH session. If local disk performance is good, look at Samba tuning instead.
 
sub.mesa said:
# write /raid5/zerofile.000 file
dd if=/dev/zero of=/raid5/zerofile.000 bs=1m count=10000
Click to expand...

I changed the count to 1000 since it was taking forever.
250.00 sec, 4180894 bytes/sec

# read /raid5/zerofile.000 file
dd if=/raid5/zerofile.000 of=/dev/null bs=1m
Click to expand...
9.91 sec, 105768329 bytes/sec

Wow - a little under 4MB/sec write and 100MB/sec read. =(
 
Interesting! Could you get me the output of:

sysctl -a | grep kmem
sysctl -a | grep raid5
 
sub.mesa said:
Interesting! Could you get me the output of:

sysctl -a | grep kmem
sysctl -a | grep raid5
Click to expand...

sysctl -a | grep -i kmem
vm.kmem_size_scale: 3
vm.kmem_size_max: 335544320
vm.kmem_size_min: 0
vm.kmem_size: 335544320




sysctl -a | grep -i RAID5
kern.geom.raid5.wqf: 0
kern.geom.raid5.wqp: 104
kern.geom.raid5.blked2: 2229
kern.geom.raid5.blked1: 53
kern.geom.raid5.dsk_ok: 50
kern.geom.raid5.wreq2_cnt: 16455
kern.geom.raid5.wreq1_cnt: 34594
kern.geom.raid5.wreq_cnt: 126051
kern.geom.raid5.rreq_cnt: 732504
kern.geom.raid5.mhm: 81923
kern.geom.raid5.mhh: 169138
kern.geom.raid5.veri_w: 0
kern.geom.raid5.veri: 0
kern.geom.raid5.veri_nice: 100
kern.geom.raid5.veri_fac: 25
kern.geom.raid5.maxmem: 8000000
kern.geom.raid5.maxwql: 50
kern.geom.raid5.wdt: 3
kern.geom.raid5.tooc: 5
kern.geom.raid5.debug: 0
<118>graid5:
<118>Cannot destroy device RAID5 (error=16).
GEOM_RAID5: RAID5: worker thread exiting.
GEOM_RAID5: RAID5: ad4(0): disk removed.
GEOM_RAID5: RAID5: ad6(1): disk removed.
GEOM_RAID5: RAID5: ad8(2): disk removed.
GEOM_RAID5: RAID5: ad10(3): disk removed.
GEOM_RAID5: RAID5: device removed.
GEOM_RAID5: RAID5: device destroyed.
GEOM_RAID5: registered shutdown event handler.
GEOM_RAID5: RAID5: device created (stripesize=131072).
GEOM_RAID5: RAID5: ad10(3): disk attached.
GEOM_RAID5: RAID5: ad8(2): disk attached.
GEOM_RAID5: RAID5: ad6(1): disk attached.
GEOM_RAID5: RAID5: ad4(0): disk attached.
GEOM_RAID5: RAID5: activated.
GEOM_LABEL: Label for provider raid5/RAID5p1 is ufsid/4af206eba5422e57.
GEOM_LABEL: Label for provider raid5/RAID5p1 is ufs/RAID5.
GEOM_LABEL: Label ufs/RAID5 removed.
<118>/dev/raid5/RAID5p1: FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>/dev/raid5/RAID5p1: clean, 1643817626 free (7514 frags, 205476264 blocks, 0.0% fragmentation)
GEOM_LABEL: Label for provider raid5/RAID5p1 is ufsid/4af206eba5422e57.
GEOM_LABEL: Label for provider raid5/RAID5p1 is ufs/RAID5.
GEOM_LABEL: Label ufs/RAID5 removed.
 
When transferring to this fileserver, the CPUs hit 85% or so usage.
Click to expand...

You guys think that a 1GHz chip running RAID5 with 5400 rpm hard drives should give over 6MB/sec transfer rates?
 
Isn't your array rebuilding or something?

graid5 status

If it is, you can speed up the rebuild with:
sysctl kern.geom.raid5.veri_nice=1
sysctl kern.geom.raid5.veri_fac=1000
 
$ graid5 status
Name Status Components
raid5/RAID5 COMPLETE CALM ad10
ad8
ad6
ad4
 
The problem is this:

kern.geom.raid5.wreq2_cnt: 16455
kern.geom.raid5.wreq1_cnt: 34594
kern.geom.raid5.wreq_cnt: 126051

The wreq1 and 2 should be wreq total i believe.

Could you try these tunings? (warning: incorrect tuning may lead to panics!)

sysctl kern.geom.raid5.maxwql=500

You can keep the wdt on 3 seconds. Try benchmarking now; locally with DD just like u did before.
If still low speeds, run this benchmark on background and show me gstat -a output (screenshot would be best).

What is the stripesize of your RAID5? (check with graid5 list)
 
are you using the WD15EARS drives?

If so these are 4k drives and you need to put a jumper on pins 7-8 (closest to sata connector) to get normal speeds. The drive may be doing up to 8 read/write operations to translate the 512b blocks to 4k if you haven't set this jumper. You should also reset the intelliseek park time if you're using these drives.
 
The drives are all this model:
WDC WD15EADS-00P8B0/01.00A01

After looking up the model number, it looks like it is 512 bytes/block.
 
Last edited:
EADS = 512 byte sector

Also, Apollo686, i highly recommend NOT touching the jumper on the EARS. It is a crude hack for Windows XP's misaligned partitions. It changes the usual 63 sectors per track to 64 sectors per track. This would 'fix' the misaligned XP partitions, but only causes disaster for ZFS. Don't use it!

Also, if you look at his graid5 status output, you can see raw devices nodes like ad4, ad6. These have no partitions on them and thus can't be misaligned; UNLESS you use that jumper which upsets things horribly. It's a very bad bugfix for XP. EARS is also no true 4K sector device in that it lies about its sector size; it still identifies itself as having 512 byte sectors. That's where all the problems come from; because this product had to still work with 10-year-old Windows XP. If they dropped that requirement, no performance issue with EARS would exist.
 
sub.mesa said:
The problem is this:

kern.geom.raid5.wreq2_cnt: 16455
kern.geom.raid5.wreq1_cnt: 34594
kern.geom.raid5.wreq_cnt: 126051

The wreq1 and 2 should be wreq total i believe.

Could you try these tunings? (warning: incorrect tuning may lead to panics!)

sysctl kern.geom.raid5.maxwql=500

You can keep the wdt on 3 seconds. Try benchmarking now; locally with DD just like u did before.
If still low speeds, run this benchmark on background and show me gstat -a output (screenshot would be best).

What is the stripesize of your RAID5? (check with graid5 list)
Click to expand...

I haven't tried your other suggestions, but here is the other info you requested.

$ graid5 list
Geom name: RAID5
State: COMPLETE CALM
Status: Total=4, Online=4
Type: AUTOMATIC
Pending: (wqp 0 // 0)
Stripesize: 131072
MemUse: 2128896 (msl 157)
Newest: -1
ID: 2672069847
Providers:
1. Name: raid5/RAID5
Mediasize: 4500905656320 (4.1T)
Sectorsize: 512
Mode: r1w1e2
Consumers:
1. Name: ad10
Mediasize: 1500301910016 (1.4T)
Sectorsize: 512
Mode: r2w2e3
DiskNo: 3
Error: No
2. Name: ad8
Mediasize: 1500301910016 (1.4T)
Sectorsize: 512
Mode: r2w2e3
DiskNo: 2
Error: No
3. Name: ad6
Mediasize: 1500301910016 (1.4T)
Sectorsize: 512
Mode: r2w2e3
DiskNo: 1
Error: No
4. Name: ad4
Mediasize: 1500301910016 (1.4T)
Sectorsize: 512
Mode: r2w2e3
DiskNo: 0
Error: No
 
sub.mesa said:
EADS = 512 byte sector

Also, Apollo686, i highly recommend NOT touching the jumper on the EARS. It is a crude hack for Windows XP's misaligned partitions. It changes the usual 63 sectors per track to 64 sectors per track. This would 'fix' the misaligned XP partitions, but only causes disaster for ZFS. Don't use it!

Also, if you look at his graid5 status output, you can see raw devices nodes like ad4, ad6. These have no partitions on them and thus can't be misaligned; UNLESS you use that jumper which upsets things horribly. It's a very bad bugfix for XP. EARS is also no true 4K sector device in that it lies about its sector size; it still identifies itself as having 512 byte sectors. That's where all the problems come from; because this product had to still work with 10-year-old Windows XP. If they dropped that requirement, no performance issue with EARS would exist.
Click to expand...

Is there a better solution for me using the WD20EARS drives? I am using them in FreeNAS and without going to an unstable build I don't believe there is 4k block support. I used the jumpers to fix the terrible speed problems people have without the jumper. I'm just using software raid5, not ZFS.

Do you have a recommendation for me or should I just use the jumper until a new stable build of FreeNAS comes out?
 
Are you using partitions? Test with graid5 status output. If it says ad4s1 for example, then you are using partitions, which likely are misaligned. FreeBSD only creates aligned partitions by default if you use GPT partition scheme instead.

I have no idea whether removing the jumper on an existing configuration causes dataloss of any kind; thread carefully here!

Either way, if you just use raw devices, then you wouldn't need to worry about misalignment. ZFS RAID-Z is another story, though.
 
I think this is part of my problem.

http://community.wdc.com/t5/Desktop...-I-just-crazy/m-p/49249/highlight/false#M2439

I ended up RMA'ing the drives.

Here's a copy/paste of what I posted there:
Was there any solution to this problem? I have 5x WD15EADS-00P8B0. The past week I have been fighting with my file server with these drives on why the read/write speeds are so slow. First I attributed it to my Dual P3 1GHz with 1GB RAM hardware. It ran FreeNAS. Then I stuck the drives in a P4 2.4GHz with 2GB of DDR on a Highpoint RocketRAID 2210 controller with FreeNAS. Then I still got slow transfers. So I installed Windows Server 2003 with SP2 just to see if it was a driver issue or something with FreeBSD. Still no go. The transfer rates fluctuate wildly from 30MB/sec down to 1MB/sec.



I am running them in a RAID10. There is no background initilization going on and nothing else is reading/writing to the array. I don't know what else to do but RMA the drives. Why do I have to go to this hassle WD?
Click to expand...
 
An update:

After getting the RMA drives back, I put them back in the P4 2.4GHz system. I tried them both in RAID10 and RAID5. I saw little performance difference between the two modes. I get about 45MB/sec disk to array, and anywhere from 25 to 35MB/sec over gigabit. I'm sure some will say that still sucks, but it certainly beats 1MB/sec to 6MB/sec.

I am running a hardware RAID5 on a RocketRaid 2210 with FreeNAS 0.7.1.
 
You must log in or register to reply here.
Back
Top