Feature #7659
Jun 25, 2020 defaultvolumetype = ixsystems-iscsi enabledbackends = ixsystems-iscsi, lvm Add these parameters and the appropriate values: ixsystems-iscsi iscsihelper = volumeddblocksize = volumedriver = iXsystems cinder driver. The ConnectX-4 adapters use a different driver than the early series. Both are included in the nightlies, but AFAIK everyone testing only has the older series cards (eg ConnectX, ConnectX-2, ConnectX-3). Some testing of the ConnectX-4 driver would help. . Talks to the host to discover device(s) and presents the device(s) to FreeBSD's pci driver via PCI configuration space. Allocates resources for the device(s) and initialize the related BARs, when the device driver's attach method is invoked. Talks to the host to create MSI/MSI-X interrupt remapping between the guest and the host. See full list on github.com.
Description
Hi,
Any way to include Mellanox CoonectX-4 40Gb support in FreeNAS 9.3/10?
History
#1 Updated by Josh Paetzel about 6 years ago
- Status changed from Unscreened to Screened
The needed drivers are FreeBSD 10 only right now. So for the moment this is a FreeNAS 10 thing.
#2 Updated by Jordan Hubbard about 6 years ago
- Category set to 76
- Assignee set to Xin Li
- Target version changed from 49 to 134
Let's put this in M3 then. Just in case we want to do it.
#3 Updated by Josh Paetzel about 6 years ago
- Target version changed from 134 to 49
I think you get it for free by switching to FreeBSD 10.x
#4 Updated by Josh Paetzel about 6 years ago
- Status changed from Screened to Unscreened
#5 Updated by Jordan Hubbard about 6 years ago
- Target version changed from 49 to 10.0-M1
Then by that logic, this will be present in M1. Let's not overuse FUTURE - it becomes a dumping ground that we're going to have to spend 8 hours (at least) going through sometime, just to figure out when and where the future is 'sometime more recent'.
#6 Updated by Xin Li about 6 years ago
- Status changed from Unscreened to Screened
I've contacted a FreeBSD developer regarding to this. Looking at the website there is a full open source driver at: http://www.mellanox.com/page/products_dyn?product_family=193&mtag=freebsd_driver but we would like to confirm with the vendor and see if we can avoid some diverge with upstream by adopting the in tree driver.
#7 Updated by Xin Li about 6 years ago
- Status changed from Screened to Investigation
#8 Updated by Jordan Hubbard almost 6 years ago
- Target version changed from 10.0-M1 to 134
#9 Updated by Anonymous over 5 years ago
- Project changed from FreeNAS to 9
- Category changed from 76 to 184
- Target version changed from 134 to 234
#10 Updated by Jordan Hubbard almost 5 years ago
- Status changed from Investigation to Closed: Not To Be Fixed
#11 Updated by Chris Glazebrook almost 5 years ago
Jordan Hubbard wrote:
Not going with InfiniBand.
I noticed you closed this because 'not going with Infiniband' but these cards do 10gb/40GB Ethernet as well..there are versions that run in Ethernet only as well. Will we see Ethernet support at least?
#12 Updated by aurf alien almost 5 years ago
Thanks Chris. I was hoping some one would notice.
This was an ethernet centric request.
#13 Updated by Justin Clift almost 5 years ago
As a possibly useful data point, I'm putting time into getting Mellanox cards (InfiniHost, ConnectX, ConnectX-2, ConnectX-3) working with FreeNAS.
My focus is in IB mode at present (my switch doesn't do Ethernet :<), but I can put up an experiment ISO with things set up for Ethernet mode if anyone has time to test it and give feedback? :)
#14 Updated by Justin Clift almost 5 years ago
As a thought, I've added an .iso download to that page with native ethernet mode support in it too:
Seems to work fine (so far), but it only has drivers for ConnectX, ConnectX-2, and ConnectX-3 in it so far. The ConnectX-4 drivers are only in FreeBSD 11.0 (development) as far as I know, and that's a bit too much of a stretch for me at the moment.
However, if it's useful for multiple people, I can probably put some time into trying to get it happening.
#15 Updated by aurf alien almost 5 years ago
Well, I'm stoked for the ConnectX-3 fromage.
Thank you.
I will most certainly download your ISO of 9.10 alpha v10. Thank you very much for this.
#16 Updated by aurf alien almost 5 years ago
Curious if this was merged with the latest 9.10 as of 4/26?
#17 Updated by aurf alien almost 5 years ago
Sorry to post again. You mind merging this into nightlies?
#18 Updated by Justin Clift over 4 years ago
aurf alien wrote:
Sorry to post again. You mind merging this into nightlies?
Sorry for the complete lack of reply. I must have had my account here misconfigured, so didn't receive any kind of email notification and didn't think to manually check. :(
Anyway, the drivers for Mellanox Ethernet support (not Infiniband nor IPoIB support though) have just been merged into the official FreeNAS 9.10 and FreeNAS 10 source trees. The FreeNAS 9.10 nightly builds will include it from here on.
Aurf, would you be ok to test the nightlies (any time from tomorrow onwards) and report back?
It needs testing before it'll be promoted into 9.10-STABLE for wider release. :)
#19 Updated by aurf alien over 4 years ago
Of course and many many thanks.
I'll update tomorrow night and report back by end of week how it goes.
#20 Updated by Justin Clift over 4 years ago
aurf alien wrote:
Of course and many many thanks.
I'll update tomorrow night and report back by end of week how it goes.
Awesome. :)
Download link for the 9.10 nightlies if that helps:
#21 Updated by Justin Clift over 4 years ago
aurf alien wrote:
Of course and many many thanks.
I'll update tomorrow night and report back by end of week how it goes.
Reminder ping? :)
#22 Updated by Justin Clift over 4 years ago
As a general note, it'd be really useful for someone with ConnectX-4 adapters to test the 9.10 nightlies.
The ConnectX-4 adapters use a different driver than the early series. Both are included in the nightlies, but AFAIK everyone testing only has the older series cards (eg ConnectX, ConnectX-2, ConnectX-3).
Some testing of the ConnectX-4 driver would help. :)
#23 Updated by aurf alien over 4 years ago
My CoonectX3 Pro works BTW. I'm a tad late on getting back to you :)
However I do not have a CoonectX4, I was wrong as mine is simply a 3 Pro. My switch is set to 56Gb so I will test 56 this week as its at 40Gbs now.
#24 Updated by Justin Clift over 4 years ago
aurf alien wrote:
My CoonectX3 Pro works BTW. I'm a tad late on getting back to you :)
However I do not have a CoonectX4, I was wrong as mine is simply a 3 Pro. My switch is set to 56Gb so I will test 56 this week as its at 40Gbs now.
Cool, not a problem. It'll be an interesting data point for the 56Gb/s, so please do. :D
#25 Updated by Jordan Hubbard over 4 years ago
- Project changed from 9 to FreeNAS
- Category changed from 184 to 129
- Status changed from Closed: Not To Be Fixed to Ready For Release
- Target version changed from 234 to 261
#26 Updated by aurf alien over 4 years ago
Man, workin like a champ.
I didn't even have to manually set line speed as it auto negotiated on its own once I set the switch port to 56Gb.
My Linux clients must be set manually.
All in all, I wished Linux wasn't as hacked together. BSD is simply evolved. Oh well it iz what it iz.
#27 Updated by Justin Clift over 4 years ago
Cool, that's a good result.
And yeah, agree about Linux. :/
#28 Updated by Justin Clift over 4 years ago
Out of curiosity, do you have storage behind it which can fill the pipes?
I don't have anything in my personal home network setup that can go beyond a few hundred MB/s, so no real ability to give a good 'real world' perf test.
Software-update: TrueNAS 12.0 - Computer - Downloads - Tweakers
Hearing real world results from others who can (using the Mellanox cards), would be nifty. :)
#29 Updated by aurf alien over 4 years ago
Well, not really, it's only about 60 nearline SAS drives in 10xRaidZ2 striped volume over NFSv3/v4.
We needed something that could push slightly over 10Gb but I was also running out of 10Gb ports.
I have several 40Gb ports available though.
However I can publish some real world numbers next week from our Avid/Premier/DaVinci editors as well as our Flame artists.
Our editors are 10Gb were as our Flame artists are 56Gb.
#30 Updated by Justin Clift over 4 years ago
#31 Updated by aurf alien over 4 years ago
Curious what you think about the following.
I am on the 9.10 nightlies train. Can I assume that the latest nightlies as of now is the latest stable?
I noticed there hasn't been any 9.10 nightlies since the 9.10 stable was updated.
#32 Updated by Justin Clift over 4 years ago
Hmmm, where are you looking?
Scanning this quickly, it seems like there are new nightlies at least today, yesterday, the day before... and a few days before that (but patchy):
I think that's the right place to be looking... ?
#33 Updated by Justin Clift over 4 years ago
Thinking about that more... there does seem to be a 'STABLE' release from 8th June. I think that's after when the Mellanox drivers were added, so that might have the needed support. Haven't tested it here though.
#34 Updated by aurf alien over 4 years ago
Doh, I was using the update feature within in the UI which said no update servers could be reached.
Since I can't seem to change to the 9.10 stable train, I'll back up my config which I do nightly, install the 9.10 latest stable and restore my config.
#35 Updated by Justin Clift over 4 years ago
Ugh, yeah, I just looked at the update options here too:
You're not allowed to change away from the nightly train, it is considered a downgrade.
If you have an existing boot environment that uses that train, boot into it in order to upgrade that train.
A bit inconvenient, especially for people who aren't blindly using every nightly and for whom the latest stable wouldn't be a downgrade.
But, it's not a a killer. Personally I'll just keep on the current nightly for now. :)
Please let me/us know if that latest stable does have the Mellanox ethernet driver support. It'll be a useful data point. :D
#36 Updated by aurf alien over 4 years ago
Seems like the latest download of the stable is borked. The CD boots and I can create a FreeNAS USB but as soon as I try to boot off it, the system cannot find a bootable partition.
So I downloaded the initial 9.10 release, upgraded to the latest release and restored my config, all good now.
I should have performance numbers within 2 weeks for you.
#37 Updated by Justin Clift over 4 years ago
#38 Updated by Vaibhav Chauhan over 4 years ago
- Target version changed from 261 to 9.10-STABLE-201606270534
#39 Updated by Vaibhav Chauhan over 4 years ago
- Status changed from Ready For Release to Resolved
#40 Updated by aurf alien over 4 years ago
Hi,
Well, I spent a lot of time on this. While I could replicate ~65TB of data over the 40Gb Mellanox connection, when browsing random directories, my NFS clients would lock up.
I tried various things to see if I could find an error somewhere. Nothing in the sysctl counters, no error logs, my tcpdumps looked identical between the 10Gb SolareFlare and the 40Gb Mellanox as I could trigger the lock in specific directories at will.
I tried various NFS mount options, disabled TSO/LRO on the server etc...
I even tried assiging interupts and cores to the Mellanox card via there performance tuning guide.
While NFSv4 yielded slightly better behavior, the clients would quickly become very slow.
Upon switching to Samba and maintaining the Mellanox 40Gb connection, things worked seamless. Samba is not an option as we are a Linux/MacOSX house running NFS only.
This looks to be a combination of NFS+Mellanox 40Gb.
I have a full system to test with if some one feels like helping. I didn't reach out to Mellanox although I did contact a rather senior sales engineer over there. He didn't have much to offer. I think his focus is Linux.
At any rate, perhaps when FreeNAS 10 rolls out, I will have better luck. But when booting and at the CLI, it looks like FreeNAS 9.10 is FreeBSD 10.3?
#41 Updated by Justin Clift over 4 years ago
Aurf, that's really, really welcome testing. :)
I've only been using Samba here with my Mellanox cards, so I have Windows clients needing access. :/
Would you be ok to email me? I don't have/remember your email address, and would like to get this info in front of the Mellanox FreeBSD dev's with you in CC, for their recommendations of next steps.
My email is justin@postgresql.org, if that helps. :)
#42 Updated by aurf alien over 4 years ago
I'm very appreciative that you would be willing to test and see whats going on, that's very very cool of you all.
I'll email you, it will be from brian@krusic.com.
#43 Updated by Justin Clift over 4 years ago
As a data point, the IB code in the FreeNAS's OS repo seems to be kept fairly well in sync with upstream FreeBSD.
For example, the sys/ofed/ directory:
- FreeNAS 9.10-STABLE : https://github.com/freenas/os/commits/9.10-STABLE/sys/ofed
- FreeBSD 10-STABLE : https://github.com/freebsd/freebsd/commits/stable/10/sys/ofed
There's only one obvious patch difference, with a patch from 3 days ago in the FreeBSD source not FreeNAS. Guessing that'll be applied soon too.
The contrib/ofed/ directories are aligned too:
- FreeNAS 9.10-STABLE : https://github.com/freenas/os/commits/9.10-STABLE/contrib/ofed
- FreeBSD 10-STABLE : https://github.com/freebsd/freebsd/commits/stable/10/contrib/ofed
#44 Updated by Justin Clift over 4 years ago
Forgot to add: 'So, the problem doesn't seem to be from badly outdated IB code in the FreeNAS repos.' ;)
#45 Updated by Justin Clift over 4 years ago
@aurf alien, would you have time to try out the FreeNAS 9.10.1 release from last month, and see if the problem still shows up?
#46 Updated by aurf alien almost 4 years ago
- Priority changed from Nice to have to Important
Hi,
Unfortunately the behavior is the same after updating to 9.10.2-U1.
Would any one have a good way to debug this? I've no way to see what's going on other then behavior. At a total loss.
If I switch to 10 nightlies, will it be possible to go back to 9.10 stable at any time?
#47 Updated by Dru Lavigne almost 4 years ago
aurf alien: please open a new ticket that contains a debug (made from System -> Advanced -> Save Debug) so that the dev can check the current state against 9.10.2-U1.
#48 Updated by aurf alien almost 4 years ago
BTW, this is fixed with the very latest 9.10.2-U2. I know this has been marked as resolved but it really is resolved now. Just wanted to make a note for others who may go the 40Gb Mellanox route.
FreeNAS 11.1 Is Now Available For Download! - IXsystems, Inc ...
Also available in: AtomPDF