iSCSI Target Server Choice Woes

I manage a small a set of Citrix Xenserver hosts for various infrastructure functions, for storage, I’ve been running openfiler for about 3 years now, since the last reboot, my uptime is 1614 days! It’s pretty solid, but the interface seems buggy, there’s a lot of things in there I don’t use. When I do need to go change something, it’s so long in between uses, that I have to re-read documentation to figure out what the heck it’s doing. I’ve got a new Xenserver cluster coming online soon, and have been researching, thinking, dreaming, of what I’m going to use for VM storage this time.

Openfiler, really has been mostly great. My server load runs about 1.13 always, which somewhat bugs me, mostly due to conary (its package manager) running. Openfiler is almost never updated which isn’t a bad thing, since the machine is inside our firewall, without internet access unless I set a specific nat rule for it. I’m running it on an old Dell 310 server with two 2TB drives running RAID1, it’s got 4GB ram and boots to the same drives as openfiler runs its magic on (this server was originally implemented as a quick fix, to get us off local Xen storage, so we could do rolling restarts). It’s not a problem, but now, 3 years later, I notice, the latest version, IS THE SAME version I have installed and have been running for the last 1614 days… So maybe it’s time to find something new.

So I build out a nice Dell 530 server, dual 16gb flash cards, dual 120gig write intensive SSDs, a bunch of 2TB SATA drives, dual six core procs, and 32gig ram, dual power supplies, nice RAID card. The system arrived, and I had a lot of good feedback for NAS4Free, both online (googling, lots of reddit threads), and even in person recommendations. I was pretty excited about it honestly, I’m a little unfamiliar with FreeBSD, but have used it on and off in my now 20 year Linux career. I went ahead and installed the thing to the 16gb flash, as recommended. I disabled RAID on the server, and setup all the drives as SATA. Booted to the system and got rolling. It was really simple, seems easy to use, does WAY more than I could even actually want, in a storage device. I setup a big lun, with ZFS and iSCSI, added the write intensive SSDs as cache, installed all the recent updates, and was ready.. Then I read documentation a bit.

  • iSCSI can’t make use of SSD write cache.. Well, I guess I get an all SSD lun.
    • “A dedicated log device will have no effect on CIFS, AFP, or iSCSI as these protocols rarely use synchronous writes.”
  • Don’t use more than 50% of your storage space with ZFS and iSCSI.. WHAT?
    • “At 90% capacity, ZFS switches from performance- to space-based optimization, which has massive performance implications. For maximum write performance and to prevent problems with drive replacement, add more capacity before a pool reaches 80%. If you are using iSCSI, it is recommended to not let the pool go over 50% capacity to prevent fragmentation issues.”

So, this was some sad news, no write caching, cant use more than 50% of my disk space, but, I decided to press on. I went home for the night. The next morning I got a friendly email from my new server that it had some critical updates, cool, I though, so I installed the updates, now it wants to reboot. So, I let NAS4free reboot, two days later, more critical updates and a reboot required.. This is a bad thing for me. I run servers that really need to be up 24/7/365, yes, we run everything clustered, and redundant, and can reboot a server without anyone noticing, but not the entire storage device, that kills the point of having my VMs all stay up. This is still okay, because we have a second VM cluster, which has “the sister machines” to all our cluster nodes going into it. I just dont want to have to fully shutdown a VM cluster so the storage host can reboot once or twice a week. Kudos to the NAS4Free guys though, it’s a really good thing they are so active, it’s just not going to be the device for me.

So, I ripped it apart. Created 2xRAID1 SSD, a RAID10 set out of the 2TB drives, and installed my best friend Debian. Debian is rock solid, I only need to reboot for kernel updates, and that’s very few. Installed iscsitarget, setup my block devices using lvm, and bam! Within 30 minutes I had an iSCSI target setup and connected to Xen.

Reliability? I see a lot of ZFS fanboys touting that hardware RAID sucks, ZFS is awesome, good luck recovering your data, etc. I really haven’t had problems with RAID in the 15+ years I’ve been using it. We buy vendor supported hardware, if something dies, Dell sends me a new one. I backup onsite and offsite. I haven’t had to restore from a backup (other than testing restores), in years. I think this will all be okay.

Next article, I’ll write about setting up my iSCSI target, since there wasn’t many decent articles out there, I’ll write about it. It’s really pretty simple. Even have multipath IO working.

No country for old men

Retiring a bunch of old servers over the next few months. I actually feel bad letting these guys go, they’ve done such a good job. This guy was about 10 years old, last reboot was in 2011, and he’s still running like a champ, we replaced him about a year ago and left him running just in case, but it’s time to retire the old bugger. Thanks for lasting 2299 days without a reboot Centos 5!Server uptime


It’s been a year, MS150 Again.

Tim took a selfie with me
Tim Took a selfie with me.

Wow, it seems like every year around this time, I’m blogging, my beard is completely out of control, and I’m saying that I haven’t blogged in a year, and I should start doing it again. I can’t believe how old this blog is… I started it TWELVE YEARS ago.. In March 2003!! Now that’s a blog with some history.

Maybe I will blog more. Or not. We’ll see.

In the mean time, I’m riding my bike from Houston, TX to Austin, TX (about 160 miles), to help find a cure for Multiple Sclerosis, I’m asking for donations.

Here’s a link to donate!


2014 Houston BPMS150 Ride

Again it’s that time of the year, only four more weeks and I’ll be riding my bicycle from Houston to Austin, TX. The ride is to raise funding for multiple sclerosis. I’d love for everyone to donate to my cause! You can donate as little as $10 or as much as you’d like.

My MS150 Page with donate link:

Multiple sclerosis is a progressive neurological disease that affects people in many different ways. It could be paralysis one day, loss of vision the next or impaired memory the day after that. Living with MS means living with uncertainty.

MS150 Finish Line Michael
MS150 Finish Line



Drivers Hating Cyclist on Twitter Gratitude

Drivers Hating Cyclist on Twitter Gratitude.

“Drivers: You sit behind the wheel of the most advanced personal transport device in the history of the world. All you need to do is apply slight pressure to a pedal with your foot and you are instantly propelled forward at an incredible rate of speed. Some slight side-to-side motions with your hands, or even one hand if you like, steers you almost anywhere your heart desires. At your disposal are turn-by-turn directions to any location on earth. You sit eating snacks in a protected, climate-controlled chamber on top of a seat more comfortable than most kings’ thrones. If something should slow you down temporarily you can almost instantly make that speed up by applying a little extra pressure to that pedal. Simply amazing how far we’ve come.


So how do you show your thanks for such good fortune, such freedom?” … More: Drivers Hating Cyclist on Twitter Gratitude.

If money wasn’t an issue in life…

I was thinking today, if I could do anything with my life and money wasn’t an issue, these are the things i’d do:

  1. Own land
    1. Ride Dirt Bikes
    2. Own a jeep again
    3. Build my own house from complete scratch
    4. Fish
    5. I think i’d need at least 200 acres!
  2. Photography
    1. Mostly travel or nature, or maybe public events
    2. figure out how to get into doing major sporting events on the field like baseball and football games..
    3. build my own studio warehouse, where i could photograph large things, like cars and motorcycles in house with full lighting setups.
    4. work on a movie set as a photographer
    5. have enough good photos to hold an exhibit somewhere
  3. Bicycles
    1. I’d like to ride a lot, maybe from my house to my photography studio every day :)
    2. Build – i want to learn how to build bicycles and completely build my own stable of different bikes from the frame up.
    3. i’d love to setup some type of charity where we build bikes for underprivileged kids.
  4. Fun Stuff
    1. I’d buy a good quality saxophone and play it again. I dreamed I was playing in a jazz club the other night, made me miss it.
    2. Learn to be a chef
    3. Take vacations with the kids.
    4. Stay in shape!
    5. Get back to volunteering regularly. It’s been so long.
    6. (With my Jeep) – Go on a Jeep Jamboree someday.
    7. Mountain Bike in amazing places like Colorado, Utah and some of the US’s famous parks.
    8. Get dual citizenship with the UK, so i can spend a few months in England with all my family whenever I want.

Cycling! A whole new hobby

So I’ve recently got into cycling. It’s been an awesome way to exercise and super fun.

I picked up a bunch of awesome gear from Amazon:
Giros Savant Road Bike Helmet
Shimano 105 PD-5700 SPD SL Pedal
Shimano SH-R087 Road Bike Shoes – Men’s
Pearl Izumi Attack Short
Pearl Izumi Men’s Quest Jersey

Pretty much everything is working out great! I’m normally a size 9.5US in Nike shoes and went with the 44 Shimano road shoes, I think maybe a 43.5 would have been better, but I’ve used them a lot so it’s too late to return them. Clipping in to the 105 pedals is pretty easy, although I did fall off at a stop light on the first ride! Shorts are great and highly recommended!

LeatherCraft 1539 18 Multi-Compartment Tool Carrier

I’m constantly struggling with keeping my network tool bag organized. It’s great after I organize it, but one job needs to be done and it’s completely disorganized again.

I have a ton of tools, carry everything I need to install network cabling, turn up T1 lines, troubleshoot analog lines, I can even do basic electrical work with what’s in my bag. It’s exploding with stuff, since I carry everything I could possibly need.

So I’ve been looking into a new bag.

The one I’m probably going to settle on is: Custom LeatherCraft 1539 18 Multi-Compartment Tool Carrier. It looks like it’ll do the trick. My current bag is a “bucketmouth” type bag, so there’s not really much organization, it’s so full of stuff I can’t zip it, so maybe this one will be great!