Setup:

2 x c7000 Enclosure, 4 x VC FlexFabric 10Gb/24 Modules Stacked

OA are v4.01, VC's are v4.10

Blades are all BL460c Gen8 with 2 x Port Emulex FlexFabric 554FLB mapped to VC's in Bays 1-2, plus 4 x Port Intel 366M Mezz Card mapped to GbE2c Layer 2/3 Switches Bays 5-8.

All Blade Firmware up to SPP 2014.06

Using ESXi 5.5 u2 Novemeber 2014 HP Build

OA's / VC's all upgraded recently to firmware above. Upgarde went fine. No issues until I enabled the new VC v4.10 feature 'Hide FlexNIC's'  in a server profile.

This correctly hid the unused FlexNIC's for some servers. On others it did nothing and did not hide the unused FlexNIC's. On other servers it bizzarely hid some of the Intel 366M ports instead, sometimes one or two of them, sometimes all of them, completely disabling the card altogether which then no longer appears in the adapter list in ESX 5.5 u2.

Unfortunately, switching off the 'Hide FlexNIC' option in the profile does not result in these incorrectly hidden NIC's reappearing, though on one or two servers some of the NIC's have returned after a number of reboots, though not all.

I have tried this on a Windows OS as well and the result is the same. I have tried a fresh build of ESXi 5.5 u2 (i.e with no pre-existing network configuration), and the missing NIC's are still not detected so they have definately been unpresented. I have tried creating a new Server Profile in VC and applying this to an affected server - something this makes no difference; sometimes it results in one or two of the missing NIC's returning. I have tried upgrading one of the Blades to SPP 2014.09, but no difference in behaviour has been seen.

VC itself also shows some confusing information for the affected server blades (see attached). For example, Port d6 shows a 'Hide FlexNIC' enabled profile applied to a server. The display would seem to indicate that all ports have been "Administratively Hidden", including active ones with active networks. However, on the actual server they are still visible and operating normally. Port d7 below shows a server without 'Hide FlexNIC's' applied to the profile for comparison.

The conclusion would seem to be that there is a massive bug in the VC firmware that is incorrectly disabling FlexNIC's and Mezz NIC's on these blades when 'Hide FlexNIC's' is enabled, a bug that persists after the change is reverted, even on non-FlexNIC's. Has anyone witnessed a similar such effect with this feature, or am I alone?

If "reset port-protect" doesn't work, power off server, unassign the profile, apply and reeassign it to a slot again, apply.

Hi,

We are trying to create port mirroring between 2 VMs that are located on different hosts. Both ESXs are located on the same enclosure and are connected to an HP Flex-10.

Is it possible to create "Remote Port Mirroring Source/Destination" when the RSPAN vlan will be a normal vlan that is localy defined only on the Flex-10?

We started testing it a few days ago (we need this mirror in order to do voice recording) and for some reason when we opened wireshark at the destination VM we only saw SIP packets and not RTP packets.

Anyone got any expirience with this issue?

Thanks.

Dear readers,

Sometime ago we implemented a SCOM2012 R2 enviroment. At this point we want to monitor our C7000 Blade systems all concerning systems. We installed the DMS/DMC on our Reporting server and the monitoring of the OA seems to be working fine. However, we added and removed several VFlex-Connects. Two of the four are working fine.

When we try to add the other ones, an eventID 400 is logged in the DMS log and DMC says "Unable to add your device". Somehow, in the time we installed DMS/DMC etc. we probably tried a couple things and these two VFlex's aren't showing up in the DMC, but they are registered with it.

Is there anyway we can fix this? Many thanks in advance!

Bastiaan, Groningen, The Netherlands

Steve had a Virtual Connect stacking question:

************

Have an enclosure with VC Flex-10 modules in Bays 1 and 2 and VC-FF in 5 and 6. VC Management is on the VC-FF modules. There is no vertical stacking between modules 1 and 5 and 2 and 6. Is it supported to do vertical stacking connections between different types of VC modules? They are running FW 4.20

************

Reply from Vincent:

**********

Are you sure VC management is on the VC-FF modules ? Normally VC Manager runs on the modules in the lower-numbered bays, in your case Flex-10 in bays 1 and 2, you don't get to choose.

Anyway, not only it is supported but it is mandatory to have vertical stacking links between all VC Ethernet modules in the domain, at least with VC firmware < 4.30. VCM 4.30 introduces the partial stacking feature.

Regards.

*********

And from Mark:

************

A case where they built the Domain with 5/6 and installed 1/2 afterwards, because like you said you can’t choose 5/6, it defaults to lowest pair.

*********

Other comments?

I think it has more to do with NLB than with your Flex setup. You can configure the MS NLB with multicast or unicast and I think you have your default ESX settings which will not allow for promiscuous mode. Therefor your MAC adresses will be in trouble with multicast NLB.

We had more or less the same issues with MS NLB and we had to manually associate the NLB IP address with the generated MAC address in our firewall (or coreswitch, whereever it needs to be). Also we had to setup the cluster in Unicast mode instead of multicast mode.

We have two SUS (one per VC module) in our latest C7000 and we configured two links to our (different) coreswitches and they are active/active. I think your issue is that you have configured LACP on the switch side. You shouldn't do that. That way it will be active/active and the VirtualConnect will handle STP etc.

Try this in a Lab first though :)

It depends a bit on what goals you wish to achieve and how much redundancy and bandwith you need. I suggest you start with reading the Virtual Connect Cookbook, which is an excellent guide and it sums up a couple default configurations in a minimal amount of time. You can find it here. This little booklet has allready saved me loads of headaches :)

A customer question from David that involves VC Stacking:

*****************

Hi Experts,

I am working with a customer who is reviewing their Virtual Connect configuration after some outages.

My question which is slightly related:

In a scenario where a customer removes the Stacking Links between live enclosures and the Server Profiles are designed to only utilise the uplinks in their local enclosure, will this affect traffic flow in each of the local enclosures?

I am assuming not but as I haven’t actually tried this before, I wanted to validate with the community that there isn’t a requirement somewhere that I was unsure about that causes downtime when stacking links are removed.

As a background, the customer believes that removing stacking links in future designs is the way to go as this reduces the risk associated with firmware updates and I am exploring the benefits/drawbacks of this approach.

************

Input from Fred:

*************

It’s not possible to remove the stacking links between enclosures in a multi-enclosure stacked domain. You’d need to decompose the MES and then configure individual domains for each of the enclosures. In a MES, the VCM instance managing all of the enclosures is running on the base enclosure. Simply removing the links between the enclosures would leave the remote enclosures w/o management.

*************

Comments?

I'm having trouble connecting my VC FlexFabric to my Brocade DS-5100b SAN switch.

On the VCM I get "Disabled - Incompatible" at the SFP status.

Is there compatibility issues with the specific SFP (AJ718A) with the switch?

Do I have to use the other type SFP (HP 8Gb Shortwave B-series Fibre Channel SFP+ AJ716B) to get this resolved?

I do have 8GB SFP+ GBICs in my SAN switch, so that is not the problem.

It should work as far as I can see. We use exactly the same GBICs in our blades and SAN switches (connected to a 3Par). We also use the A series in our VirtualConnect, so either way, it should work.

Have you tried upgrading all the firmwares? VFlex, OA, SAN switch etc.?

-little correctie, we use the 716A ones in our VirtualConnect, I can't physically inspect the VFlex ones, but I assume they are the same. I'm still thinking firmware issues :)

Stefan was working on a customer issue:

*************

Have a customer with both vSphere 5.1 and 5.5 environments (Using vDS)

c7000, Gen8 blades (530FLB) and Virtual Connect. Everything updates to latest SPP + latest HP Custom VMware Images.

They are seeing that on their G7 servers, LLDP works just fine, however no info shows up on Gen8 servers.

Any points/ideas?

*************

Reply from David:

************

First, ask them to change LLDP to “Listen” mode.

“Advertise” and “Both” can create performance problems

http://h41302.www4.hp.com/km/saw/view.do?docId=mmr_kc-0105247&hsid=63251888&sz=5645

Second, you need to enable LLDP on 530FLB cards. It is not enabled by default:

http://h41302.www4.hp.com/km/saw/view.do?docId=emr_na-c04174689&hsid=63251918&sz=10644

**************

Other comments?

Rob had a VC SAN port question:

************

My customer has 2 enclosures, both using VC 8Gb 24-port FC cards going to 8G SAN switches, then to a 3PAR Array. When we look at the switchshow on the 8GB SAN switches, the ports show “1 N Port + 1 NPIV public”. Currently the servers being used only have the VSphere ESX client installed; no Windows or Linux O/S yet. Networking isn’t up and running yet, either.

At what point (O/S or networking install) will we see the port WWN’s on the switches so we can start zoning?

Regards;

*************

Reply from Stefan:

**********

“1 N Port + 1 NPIV public” means that one of the blade is already logged into the fabric via this port. You will see “both” (the one of the blade + the one of virtual connect module port) WWNs with portshow command.

WWNs should logon to fabric during driver initialization or with boot from san while booting.

The WWNs you could also find in Server Profile in VirtualConnect Mannager/OneView if you use virtual WWNs. Otherwise in OA->DeviceBay->Information

**************

Comments?

Patrick had a firmware upgrade question:

***********

Looking at the Release Notes of 4.30 or 4.40, the table indicates that a “reset” is required for the VC-FC Module. However, as mentioned below by Mark, this issue is resolved in 4.20.

So, do we need to do a reset or not if we are upgrading to 4.31 or 4.40?

***********

Input from Mark:

**********

I have asked the Release Notes be updated to reflect this, we will see if that happens soon.

Soft Reset – Reboot of the Kernel & Application – (Outage)

Hot Reset – Reboot the Application only – (No Outage)

VC-FC – Term used to describe the Firmware package for VC 4/20 and 8/20 modules

VC-FC2 – Term used to describe the Firmware package for VC 8/24 and 16/24 modules

1. The statement  “* Always reset the module after the upgrade or downgrade.” should be removed as VCSU automatically does “soft reset” when necessary. No manual reset required.
2. VC 4/20 & 8/20 – Any downgrade of Firmware causes “Soft Reset” – Upgrades to VC 4.20 (VC-FC Version 2.01) requires a “Soft Reset” – All other Firmware upgrades are a “Hot Reset”
3. VC 8/24 – Any downgrade of Firmware causes “Soft Reset” – Upgrades to VC 4.40 (VC-FC2 Version 3.00) requires a “Soft Reset” – All other Firmware upgrades are a “Hot Reset”
4. VC Flexfabric would also have Fabric outage as Ethernet controller inside FF will be rebooted which impact FCoE functionality as well.
5. Remember the “Activation order” by default is Serial for FC Modules. So VCSU (if performing a “Soft Reset”, never does more than one module at a time).

*************

Comments?

A performance and/or sizing, and/or capacity  planning question from Reginald:

**********

Team,

                Working with a customer on a storage issue using our blades and VC FlexFabric. They are using two ports per module  with traffic load balance across all ports. What metric should we be looking at in the log file to determine if the FC ports are saturated or congested? At what point or metric should we add more ports per module?

Thanks!

************

Reply from Lionel:

************

Statistics are not an absolute indication that you have a congestion somewhere but they can be used as one indication to help identifying that something is wrong. One recommendation with VC FC is to rather use advanced tools provided by the upstream Fabric switch, like Brocade Fabric Watch, like the Brocade FOS Bottleneck Detection capability that was introduced by Brocade to enhance the detection of congestion somewhere in a Fabric. Otherwise the metric you would monitor the most are the Buffer-to-Buffer Credit failures (fcBBCreditFrameFailures and fcBBCreditRRDYFailures) that shows the number of time a port was unable to process frames because of lack of buffer but according to the statistics you provided, you are not facing the problem.

Port monitoring using SNMP is I think a better method to help identifying port congestion as you can evaluate the data through graphics, get port trends over long period of time, etc. The counters customers usually monitor the most are :

.iso.org.dod.internet.experimental.fcmgmt.statSet.connUnitPortStatTable.connUnitPortStatEntry.connUnitPortStatCountBBCreditZero

Count of transitions in/out of BBcredit zero state. Increasing numbers show that the other side is not providing any credit.

.iso.org.dod.internet.experimental.fcmgmt.statSet.connUnitPortStatTable.connUnitPortStatEntry.connUnitPortStatCountInputBuffersFull

Count of occurrences when all input buffers of a port were full and outbound buffer-to-buffer credit transitioned to zero. There is no credit to provide to other side.

************

Reply from Reginald:

******

Thanks! Question, you reference using SNMP for monitoring, the counters mentioned below. Where can we find them and/or are they MIB definitions?

**********

From Lionel:

*************

For FC, VC uses the Fibre Alliance MIB (also known as FCMGMT-MIB, RFC 4044).

You can download the Fibre Alliance MIB from the Fibre Alliance website at the following link: http://www.fibrealliance.org/fb/mib_intro.htm

Then using a tool like Getif, you can browse the OIDs and get the definitions.

*************

Comments?

A SAN fabric question from Reginald:

*************

Good day! Can someone tell me, if you create SAN fabrics under Virtual Connect on your first setup and set the fibre channel “Login Re-Distribution” to manual but need to change it later to “Automatic”, can this be done on the fly? Will making this change in a production environment be disruptive or non-disruptive?

************

Input from Andrea:

**********

Yes, on the fly is safe.

Changing the parameter does nothing itself in the immediate state.

Only affects redistributions in case of cable failure and return to functionality.

Of course if you have a pregressive unbalanced connections, the change of automatic redistribution of logins will rebalance server NPIV over FC uplinks after the defined wait time.

So if the FC uplinks are unbalanced, change the parameter to auto with some delay from a SAN Fabric to another, more than the wait time you have in general options "link stability" fabric settings, normally 30sec.

**********

Comments?

Hi!

We had to also include the VC FC adapter's WWN in the zone as well. Before we added it the VMAX storage did not see any of the server HBA's.

Regards,

Louis

Aditya had a Jumbo Frame question from a customer:

**********

Hello,

We have a customer deployment, where we would like to enable Jumbo Frame support on the TOR and the individual blades. The current enclosure is a c7000 with Oneview 1.20 as the management utility. The blades are are BL 460C G8.

I would like to understand if there is there is a specific configuration that we would have to do at the VC module level to achieve this?

Does Oneview support Jumbo Frames?

Has anyone tried enabling Jumbo Frames in their customer environments before?

*************

Input from Kevin:

The VC switch fabric has jumbo frames enabled by default. OneView is a management appliance and basically has nothing to do with the capability of jumbo frames.

From Hongjun:

Did anyone verify this in the lab? I always thought that’s the case but my test couldn’t prove it.

I didn’t see my ping test go above 1500 bytes MTU. I made sure all NICs are mapped to the same virtual connect module to rule out TOR MTU issue.

I tried one CentOS VM(vsphere DVS MTU already raised) and a windows OS blade to ping another windows blade in the same enclosure. All NICs involved are mapped to the 1^st VC module.

Both my linux VM and windows source blade couldn’t ping above 1500 MTU ( with ICMP payload max at 1472 bytes +8 bytes of ICMP header + 20 bytes IP header).

Reply from Kevin:

We get this case asked of us at least 3 times a week in the response center. We have proven to the customer several times it is on by default. Remember jumbo frames if set has to be set within the physical path of two peers to work properly so there is no fragmentation.

Reply from Hongjun:

Thanks Kevin!

I forgot to increase windows driver side MTU size, after doing that, large packets can go through. My bad.

*************

Comments?

Santosh was looking for help to configure sFlow on Virtual Connect:

************

Dear Team,

Request your valuable information to fix the below mentioned issue in configuring sFlow on VC Flex Fabric 10Gb / 24 port.

Background information:

• To analyze the network performance and utilization we have a software module that can monitor the device (IMC-NTA module)

• On the VC flex, need to configure sFlow to capture samples of datagrams on ingress and egress interfaces and send them to IMC-NTA module (installed on a server)

• Current version on VC is 4.21

• This sFlow configuration can be done either via virtual connect manager or through CLI.

Problem Description:

Virtual Connect Manager:

• Added sflow receiver server details and enabled it

• When I tried to add bay-1 module, it was prompting to enter IP address, subnet mask and default gateway
• Am I supposed to enter actual bay IP details or else what other IP details need to be entered ?
• When I enter bay IP that is actually assigned via OA, it gave an error saying duplicate address and this is already assigned to Ethernet module

Normally on a Network switch, we configure sFlow Agent, sFlow ports and sFlow receiver. Where sFlow agent is switch itself so its own IP address is configured.

Kindly request your inputs on this issue, Thank You !

************

Input from Aaron:

************

I believe it’s an IP address on the VLAN/network you are sourcing your sFlow packets from.  Since the VC box is not a Layer 3 device, you have to specify an IP address on the source VLAN/network.

It’s not well explained in the VC User Guides, but you can deduce this is by the way it’s written.

**********

And from Mark:

**********

The error you are receiving is normal. You can use the same IP Address of the module (eth0) as the sflow interface. We don’t use the eth0 interface to send sFlow packets, eth0 is connected to the OA/Management network. When you enable sflow, a new interface (call it knt2) is created and that interface is what you are assigning a IP Address to either manually or via DHCP. It appears as a regular network interface to theLinux kernel, but it is a pseudo device created by Broadcom kernel driver. No incoming traffic is allowed on that interface other than the obvious (DCHP).

sflow traffic uses the network selected in the configuration to send datagrams to the receiver. If the receiver is on the same subnet, no Gateway address is necessary.

************

Other input?

A problem being faced by Esa:

************

Hi all

We have one customer with c7000 full of BL460c Gen9 servers with 536FLB adapters

Virtual Connect FlexFabric 20/40 F8 modules on bay 1 and 2

3Par StoreServ SAN Array as direct connected to FlexFabric module

They are not able to see 3PAR disk and 3PAR does not see any WWNs

Question:

Is 536FLB  a valid FCoE adapter at all??

***********

Input from Eirik:

*************

Yes, but you need to enable the FC portion of the adapter.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034702

***********

And from Dan:

************

Esa, trust what Eirik sent you.

It’s what I have been sending people for 14 months now based on not only the 536FLB Adapter but also the 534FLB Adapter on Gen8 blades.

The way it works with FC (and FCoE) is that you won’t see any WWNs show up on your Brocade or 3PAR until the FC Port is properly initialized and a FC Login occurs (called a FLOGI).

There are 2 main things that will initialize the port.

1. The driver in the OS loading and scanning for LUNs.
2. Enabling the FC BIOS on the Card during POST.

Note: This is no different on FCoE CNAs than it has been on FC HBAs for a decade.  I did this as a customer 7 years ago with Rackmounts and Windows and EMC Storage.

Because of the design of the Broadcom (now Qlogic) cards, they show up in VMware more like a Software FCoE adapter rather than a Hardware one.

VMware calls this a “Dependent” Adapter, as in its dependent on the OS to make it work (but still does full FCoE offload in the CNA).

The Emulex 553/554 Adapters we all grew up on was an “Independent” adapter, which means there was no OS side config that was needed for it to show up.

The OS Side config will expose the vmhba function you are looking for (VC/OneView already configured it like you said, but its hidden in the OS), which then initializes the FCoE driver, which then causes a FLOGI to the fabric which is what makes the WWN show up on your Brocade or 3PAR in the list of unconfigured WWNs.

Eirik sent you the VMware KB explaining the steps

In your case, I would follow the VMware KB since its fairly simple and I know it works.

************

Comments?