Infiniband SAN

[James]

I've got a dream, a dream started by browsing Supermicro's website.

Namely, I'm interested in creating a "Blade storage cluster". 1 7U Blade enclosure with the server nodes in it. This enclosure has a Quad DDR Infiniband switch in it (20Gbps). I want to use the Supermicro CSE-846E1's to create storage nodes, 24 drives in 4U each. From what I understand, I'm basically setting up a server, putting an Infiniband HCA (or TCA) into it, and connecting it to the infiniband switch.

Do I need a head end? Or only if I'm planning on sharing storage between server nodes (blades)? Does anyone actually make a TCA? That is, does someone make a TCA that takes 4x DDR infiniband, and includes a SAS RAID controller?

It seems like a lot of wasted power (and money) to build up a system that has the sole purpose to pass traffic from the Infiniband adapter to the RAID controller.

Any thoughts or suggestions? Has anyone had any experience with Infiniband networks, SAN's, etc? I'd love to hear some first hand experience.

[Nater]

I bought a big book on SANs last week, probably going to buy another next paycheck. Honestly, I think that using Infiniband for storage networking is fairly new, as the book I bought didn't really cover it and only mentioned it in passing (the usual 'it's used in HPC...'). Just searching for 'Infiniband' on Amazon yields next to nothing.

Are you doing this relatively soon or is this just an idea? Because CNAs are starting to creep onto the market, I don't know if FCoE requires a special switch compared to 10GbE or not. Obviously not as fast, but it's going to be hard to use 20Gb/s for just storage.

[James]

Originally Posted by Nater

I bought a big book on SANs last week, probably going to buy another next paycheck. Honestly, I think that using Infiniband for storage networking is fairly new, as the book I bought didn't really cover it and only mentioned it in passing (the usual 'it's used in HPC...'). Just searching for 'Infiniband' on Amazon yields next to nothing.

Are you doing this relatively soon or is this just an idea? Because CNAs are starting to creep onto the market, I don't know if FCoE requires a special switch compared to 10GbE or not. Obviously not as fast, but it's going to be hard to use 20Gb/s for just storage.

Agreed. It is more of a "headroom" thing.

And no, this isn't near future. I've just been researching SAN technology. I've become disillusioned with both iSCSI and FC as of late, because neither offers the scalability that I'm looking for. As for 10GbE, the adapters and switches for that are actually more expensive than the Infiniband adapters. So yeah, I'd go Infiniband before 10GbE. (Though that would be sweet on the networking side.) I just find it amusing that a lot of the sites I look at for infiniband talk about all of these awesome adapters that allow you high speed connections to your storage unit, but none of them actually offer or describe the storage units.

[proxops-pete]

My high school friend works for the company as CTO on FCoE.
I will ask him to drop in here and comment on your stuff ...

[fcoe]

Hi,
You do need a special switch for FCoE. Today the switch for FCoE must have what is called an FCF which is provides the encapsulation/decapsulation of FCoE frames and the FCoE name server functionality.
For more information on FCoE, see my recent blog post http://blogstu.wordpress.com/2009/09...nt-state-fcoe/
Let me know if I can be any further help.
Note that while InfiniBand is a nice solution for low cost, low latency cluster environments (popular in supercomputers), there are few storage solutions and the whole market has much more momentum for moving to Ethernet based options (including FCoE which will help the existing FC customers move to Ethernet).

[ua549]

fcoe, thanks for the post and link to your blog.

We need to resuscitate this forum with a few emerging technology topics.

James, is this research for your lab or for a client? It sounds expensive for a lab.
Are you thinking of a SuperBlade® server as well as the storage chassis?

[Nater]

I heard fibre channel over infiniband mentioned over at The Reg, but I've never been able to find any information on it.

How exactly do they make FC work over ethernet anyway? I've always understood ethernet to be lossy, which would be a big problem for a storage network. What are they doing to the ethernet packets to deal with this?

[proxops-pete]

As fcoe (aka. Stu) pointed out, here's some Youtube video of the explanation

EDIT: And that's fcoe, Stu, my high school friend now CFO of EMC. My only claim to even the slightest fame! LOL

http://www.youtube.com/watch?v=EZWaOda8mVY

[James]

Originally Posted by Nater

I heard fibre channel over infiniband mentioned over at The Reg, but I've never been able to find any information on it.

How exactly do they make FC work over ethernet anyway? I've always understood ethernet to be lossy, which would be a big problem for a storage network. What are they doing to the ethernet packets to deal with this?

From what I understand, they use ethernet as the physical hardware only. The actual transfer protocol (which as fcoe mentioned requires special switches) is a bit more robust than your standard 10/100/1000 type protocol.


ua549, Yes, the blade enclosure was the origination of this little thought/planning experiment. Because it has an integrated multi-port Infiniband switch, I wanted to see what it would take to get a full blown Blade enclosure + infiniband based storage network connected and working. It's not for anything my company will actually do at the moment, it's for perhaps the future when it becomes a more viable option.


The downside I'm finding lately is that there doesn't seem to be a company that offers a single board solution for an Infiniband connection to a SAS backplane/enclosure. It seems most of the solutions are quite literally a full blown system with a nice storage backplane (like the Supermicro 846 series), RAID card, and an Infiniband adapter. This means it's running an OS, it's got all of the normal issues that a regular system does, etc. What I really wanted to find was a board that had the infiniband adapter built in, an SAS RAID card built in, and bam, that would be it.

[bustedtyre]

The closest i could find was Mellanox MTD2000, but that's really for test/development, not a production model.

I guess if you are technical enough you can build your own, Mellanox have the drivers and IB stack which I think are Open-Source.

Our task is very similar to what you're describing: a telco site with a cluster of 10 modern virtualised (VMW ESX4) servers needs an extremely fast SAN for shared storage.

Putting QDR IB cards in the servers makes for 400 gbps throughput, most of which can be dedicated for storage. If you think about it you only need 150 Intel X25-E or similar SLC-based disks. That is just 6 2U enclosures holding 25 disks each, such as HP Storageworks 70. It is very tempting, isn't it?

Please let me know if you're closer. If you tried the Mellanox unit and found it suitable and scalable enough and decided to build a larger unit we might be able to help with components.