If an NVIDIA NDR link is down after repatching, read its physical and logical states before swapping another cable. Physical Down calls for checks on the changed MPO-12/APC path, cable map and disturbed interfaces. Physical LinkUp with InfiniBand still in INIT calls for subnet-manager checks. A clean-looking connector and a link-up light each answer only part of the problem.
This guide covers 400G NDR branches on the MMA4Z00-NS twin-port multimode OSFP, including MFP7E10 point-to-point and MFP7E20 splitter arrangements. For other optics, start with their own interface specifications; our general MPO link-down troubleshooting guide explains that wider selection process.
Field sequence: Read the port state → trace the repatch → confirm APC, pinning and channel map → inspect, dry-clean and inspect again → compare per-lane diagnostics and counter deltas. Physical LinkUp with INIT takes the subnet-manager branch below. Leave an undisturbed, healthy sibling connection seated.
In this guide
- Link down or INIT? Run the IB checks
- Locate the changed MPO path
- Identify point-to-point and splitter cables
- Check APC, pinning and fibre order
- Inspect and clean the disturbed interfaces
- Read mlxlink output and error counters
- Use numerical loss, BER and delay references
- Specify the replacement assembly
- Ten troubleshooting FAQs
1. NDR Link Down or INIT? Start With InfiniBand Status
On a Linux host with the RDMA diagnostic tools installed, identify the HCA and inspect the affected port. These are query examples; replace mlx5_0, port numbers and uppercase placeholder names with the actual identifiers. MFT device names used later are a different namespace.
ibstat -l ibstat mlx5_0 1 sminfo -C mlx5_0 -P 1 iblinkinfo -C HEALTHY_HCA -P HEALTHY_PORT --down perfquery -C HEALTHY_HCA -P HEALTHY_PORT SWITCH_LID SWITCH_PORT
ibstat reports the local state, physical state, rate, Base LID and SM LID. sminfo queries the subnet manager; use it without state or priority setters. Fabric discovery and remote perfquery need a reachable management path and suitable permissions, so use an Active management HCA if the affected port cannot reach the fabric. Tool syntax is documented in source 7.
| Observed state | Interpretation | Next action |
|---|---|---|
| Disabled port or unrecognised optic | The host has not established the intended port/module configuration | Check administrative state, installed PID and host support |
| Physical Down / Polling after repatching | The physical path has not trained | Check the changed route, interfaces, polarity and end faces in sections 2–5 |
| Physical LinkUp; logical INIT | Physical training has succeeded; fabric activation is incomplete | Check Base LID / SM LID, query the SM from a reachable port, and check discovery or isolation in UFM/OpenSM |
| Active with a downshift or increasing errors | Service exists, but the link needs investigation | Compare rate/width, per-lane diagnostics and time-based counters at both endpoints |
For example, State: Initializing, Physical state: LinkUp and Base lid: 0 direct you toward SM discovery and address assignment. A failed sminfo query can also reflect a reachability or permission problem. If an Active management port reaches the master SM but this port stays in INIT, inspect the fabric's discovery, isolation and activation logs. Keep the optics under investigation if the physical state also flaps.
Figure 1. Start with the observed port state. The arrows show suggested next checks, not failure probabilities. [7]
2. Locate What Changed During the Repatch
Trace the affected branch from its switch MPO socket to the far-end HCA. List the equipment cords, adapters, trunks and cassettes actually in that route. A new panel can change pinning, mapping and loss even when both equipment leads stayed in place. Use the panel, enclosure and cassette guide to distinguish the hardware.
One failed branch after one cable replacement points first to that changed path. Two failures after a shared-module event also call for module recognition and mode checks. Look for tight bends, trapped boots and strain introduced when the drawer closed; the fault may appear only in its final position.
Outage scope: Disconnect only the branch being tested. Keep an untouched healthy branch connected and compare its live status. A shared-module replacement interrupts both branches; schedule that step accordingly. Retain the before/after readings and the part that changed.
3. Match the MPO Cable to the NDR Topology
Read the optic PID, cable label and connection drawing together. For MMA4Z00-NS, each MPO socket serves a separate four-lane 400G port.
| Configuration | Required distinction | Expected endpoint arrangement |
|---|---|---|
| MMA4Z00-NS | Twin-port multimode OSFP with two MPO-12/APC sockets; the thermal-shell variant must suit the host | Two independent 400G optical ports |
| MFP7E10-Nxxx | Point-to-point MPO-12/APC cable with Type-B crossover | One 400G endpoint per cable; two cables serve two 400G endpoints |
| MFP7E20-Nxxx | MPO-12/APC to 2×MPO-12/APC splitter | Two 200G endpoints per splitter; two splitters serve four 200G endpoints |
NVIDIA requires both twin-port sockets to use the same cable form: point-to-point or splitter. The term straight cable in this topology discussion refers to the point-to-point form; its fibre mapping is still crossover Type-B. The two cables may have different lengths. See source 3, source 4 and source 5.
Figure 2. Point-to-point and splitter configurations allocate the same nominal source capacity differently. Cable form and fibre polarity are separate fields. [2] [4] [5]
4. Check APC, Pinning and the Complete Fibre Map
Confirm the mating pair at every junction
Use MPO-12/APC at this optical interface. An MPO-12/UPC lead from an older 100G or 200G deployment is incompatible with the specified APC mating surface. Connector colour helps locate parts, but the label and drawing establish the polish. Check an intermediate adapter from both sides.
The transceiver has alignment pins; the cable end mating to it needs alignment holes. Each intermediate MPO pair also needs valid pinned/unpinned mating. Two unpinned equipment leads need an appropriate mating arrangement when joined. A connector that resists normal seating needs investigation before force is applied.
Trace the active positions from transmitter to receiver
The MFP7E10 map connects 1→12, 2→11, 3→10 and 4→9, with the opposite-direction channels on positions 9–12. Positions 5–8 are marked Not Connected. That describes working-channel allocation; an unused position need not be visibly empty. Read both connector faces in the drawing's stated viewing orientation.
Type-A is straight-through mapping; Type-B reverses the array. A direct MFP7E10 link needs the crossover. For a route built solely from conventional A/B mapping elements, an odd number of B reversals produces a net crossover. Cassettes or regrouping harnesses require their actual maps. Our MPO patch-cord and channel-polarity guide explains this channel-level check. The NVIDIA cable map is in source 2.
5. Inspect and Clean the MPO Interfaces After Repatching
For a physical link down after repatching, inspect the interfaces that were exposed or disturbed once the cable identity and map are correct. A particle can obstruct an optical path or interfere with ferrule contact. Repeated mating can transfer it to the opposite face, so cleaning only the loose patch cord leaves part of the fault path unchecked.
Start with the symptom and the affected lane
Before disconnecting, save the affected branch's module identity, link state and available per-lane receive-power readings. A low or missing receive lane helps narrow the search to its transmitter, fibre path and receiver. Similar symptoms can come from mapping, a damaged fibre or an optical engine, so use inspection to test the contamination hypothesis.
If the link comes up when the drawer is open and fails when it closes, check routing and strain as well as cleanliness. An undisturbed sibling branch provides a live comparison without opening another pair of connectors.
Inspect the cable and the recessed receptacle
Isolate the affected optical path before opening it. Use a video inspection probe and the correct MPO/APC tip for the cable and recessed module interface. Follow laser-safety and ESD procedures; a connector microscope must be suitable for the inspection task.
Check all working fibre positions and the rectangular ferrule's contact surface. A view of the centre fibres alone misses the outer working positions. Look for loose debris, films, particles around guide holes or pins, and persistent scratches or chips. For automated pass/fail analysis, select the applicable connector and multimode inspection profile.
Match a weak telemetry lane to the connection drawing before choosing the fibre to test. Device lane indices and ferrule positions use different numbering conventions. Include transmit/receive direction and connector viewing orientation; a diagnostic label such as lane 0 does not by itself identify MPO position 1.
IEC 61300-3-35:2022 includes the whole contact surface when inspecting rectangular ferrules for contamination. Visual inspection complements attenuation and return-loss measurements. See source 10.
Use a dry-clean, re-inspect, mate sequence
- Inspect both mating surfaces and identify the contaminated part. Keep nearby disconnected ends protected.
- Use a dry optical connector cleaner intended for the specific MPO/APC interface. Check that its tip reaches the contact surface and use the tool manufacturer's stroke or click sequence.
- Inspect again with the same probe settings. If debris remains or moves, continue with the approved dry-cleaning method; do not mate the dirty end to a clean receptacle.
- When both surfaces meet the applicable inspection criteria, remove protection only as needed and mate promptly. Keep the ferrule clear of fingers, loose wipes and surrounding surfaces.
- Restore the affected branch and compare its state, per-lane readings and counter deltas under the same test conditions.
NVIDIA's instructions for this module prohibit applying liquids; its cable handling guidance also keeps the assembly away from liquids. Use that product restriction when choosing the cleaning method. See source 1, source 3 and our connector inspection and cleaning guide.
Use the response to choose the next test
If inspection passes and the same lane remains weak, measure the passive path or substitute one known-correct, inspected assembly. A problem that follows the removed lead points toward that lead or its mating history. A problem that stays with the port leaves the receptacle, local optical engine, far-end transmitter and configuration in scope.
At a patch panel, include the rear trunk connection in the inspection scope when it was disturbed. Cleaning the front equipment lead cannot remove contamination on a rear mating surface. A known-correct direct cable can help isolate the panel route, provided its mapping and topology suit both endpoints.
A scratch that persists after cleaning needs assessment against the applicable defect criteria and optical results. Identify whether it belongs to the cable or the module before selecting a replacement. If contamination cannot be removed with the supported method, stop mating that interface and refer it for further cleaning or repair.
After recovery, check the final routed position. Stable LinkUp, InfiniBand Active and a quiet error interval provide stronger evidence than a single link-up transition.
6. Link Down or Unstable? Read mlxlink and Error Counters
Use MFT for physical-layer and module queries at both endpoints. Select the affected logical port, especially on a twin-port module. Quantum-2 uses cage/port addressing, for example -p 15/2; a configured split port can require another suffix. These commands query status; option support depends on the installed MFT, firmware and platform. Source 6 documents the queries.
mlxlink --help mlxlink -d MFT_DEVICE -p PORT mlxlink -d MFT_DEVICE -p PORT --show_module mlxlink -d MFT_DEVICE -p PORT --show_counters
The following reduced display is a synthetic reading example for a 400G branch, not a field capture. Output names and formatting vary by version.
Operational Info State : Active Physical state : LinkUp Speed : IB-NDR Width : 4x Physical Counters and BER Info Time Since Last Clear [Min] : 15.0 Effective Physical Errors : 0 Raw Physical BER : 1E-7
Active, LinkUp and IB-NDR/4x identify the expected 400G branch in this example. A nonzero raw BER can coexist with FEC-corrected operation. Its value needs the platform's BER criteria and trend; the example's 1E-7 is not a universal acceptance limit.
| Readout | Useful comparison | Action |
|---|---|---|
| Raw Physical BER / per-lane raw errors | Compare the same fields, interval and operating conditions before and after the repair | Investigate a rising trend or a persistently worse lane; raw errors alone do not identify contamination |
| Effective Physical Errors | Take two readings after training; target Δ = 0 for the initial screening interval | New errors require continued investigation |
| LinkDownedCounter / recovery counters | Compare deltas after the deliberate disconnect and retraining have finished | New transitions indicate instability |
| Receive power per lane | Check the module range and compare with the earlier reading | An out-of-range lane needs diagnosis; an in-range lane still needs state and error checks |
As an initial screen, take two snapshots 60 seconds apart after the port stabilises, with representative traffic where available. Use UFM or supported monitoring for longer observation; NVIDIA describes mlxlink as a diagnostic tool rather than a continuous polling service. A 60-second screen does not establish a long-term BER guarantee.
Use perfquery twice for IB port-counter deltas. Its basic and extended counter sets differ; -x does not simply add every error counter. Do not equate a basic SymbolErrorCounter with UFM's SymbolErrorsExt without checking the counter definition. If physical LinkUp remains stable but INIT persists, return to the SM checks in section 1.
7. Numerical References: Loss, Return Loss, BER and Skew
Apply each number to the component or test it describes. A connector specification, a module conformance test and a live-service counter threshold answer different questions.
| Quantity | Published value or screening rule | Scope and interpretation |
|---|---|---|
| MFP7E10 insertion loss | IL ≤ 0.35 + 0.0004 × L dB; L in metres | NVIDIA labels this as connector-end-face IL. Calculated values: 0.362 dB at 30 m; 0.370 dB at 50 m. It is not the allowance for a multi-panel channel. [2] |
| MFP7E10 return loss | RL ≥ 35 dB | Connector-end-face specification. Higher positive RL means less reflected power; it is separate from insertion loss. [2] |
| MMA4Z00-NS receive power | −6.3 to +4.0 dBm per lane | Average-power range; the minimum is informative, and power inside the range does not ensure compliance. [1] |
| Receiver conformance BER | 2.4 × 10−4 | Specified for the stressed-receiver test signal at TP3. It is not a post-FEC service target or a generic raw-BER pass mark. [1] |
| Initial service screen | Δ effective errors = 0; Δ link-down transitions = 0 over 60 s after stabilisation | A diagnostic screening recommendation. Extend testing to the deployed platform's validation procedure. |
| Optical reach | OM3: 30 m; OM4: 50 m; nominal 850 nm | MMA4Z00-NS limits. A low measured loss does not extend the published reach. [1] |
Calculate the installed-path loss separately
For each working fibre, sum the specified losses of the mated junctions, fibre and any splices, then compare that total with the supported channel allowance. NVIDIA's 50 m application assumes two patch panels and four optical connector junctions. Additional connections need a fresh budget.
For example, if a project assigns 0.35 dB to each of four junctions and 0.18 dB to the fibre, its estimated path loss is 4 × 0.35 + 0.18 = 1.58 dB before any reserved margin. These are stated planning inputs, not measurements or an NVIDIA channel limit. The remaining margin is the validated channel allowance minus 1.58 dB. Do not turn the informative launch/receive average-power endpoints into a guaranteed loss budget.
Measure the disconnected path with suitable multimode MPO/APC equipment at 850 nm, using a documented reference setup and defined endpoints. A factory cable result excludes connections added later. Glory's data center cabling range describes the passive components for which assembly test reports can be requested.
Use a time window for NDR error validation
NVIDIA's UFM PDR deterministic procedure gives an operational NDR example: SymbolErrorsExt deltas are compared with 2.88 over 12 minutes and 3 over 125 minutes at 400 Gb/s, corresponding to its 10−14 and 10−15 rate calculations. The comparisons are strictly greater-than, so integer deltas of 3 and 4 respectively cross those boundaries. This procedure uses its own telemetry and sampling rules; see source 8. Apply the deployed UFM version's procedure rather than substituting mlxlink Raw Physical BER into those formulas.
Separate 4.5 ns/m propagation delay from lane skew
NVIDIA uses approximately 4.5 ns/m for fibre propagation delay: 50 m contributes about 225 ns one way. Lane skew is the arrival-time difference within a bonded lane group, max(t_i) − min(t_i). A 1 m lane-length mismatch would contribute approximately 4.5 ns of differential delay; that is a calculation, not a permitted skew limit.
The public MFP7E10 table cited here does not publish a lane-skew acceptance ceiling. Obtain that value for the installed optic/cable qualification if lane alignment is suspected. The two independent 400G branches may have different lengths; their difference is not automatically intra-port skew. The propagation reference is source 9.
Figure 3. Product-specific reach and connector-loss values. The calculated IL values are specification examples, not customer measurements or full-channel limits. [1] [2]
8. Specify a Replacement MPO-12/APC Assembly
Request the exact cable form, fibre grade and length, both-end APC polish and pinning, connector viewing orientation, working-channel map, loss specification and test report. Include both optic PIDs and the full route. A replacement for a direct 400G branch must preserve its crossover and endpoint arrangement.
Review Glory's MTP/MPO assemblies and custom cable options. Send the endpoint PIDs, route drawing and required test data through the project enquiry form for a configuration review.
Frequently Asked Questions
NDR link down after repatching: what should I check first?
Read the physical and logical port states. Physical Down directs you to the changed cable path, interfaces and cleanliness. Physical LinkUp with INIT directs you to subnet-manager discovery and activation. Use ibstat and the section 1 decision table before swapping another part.
Can I use MPO-12/UPC instead of MPO-12/APC?
The specified NVIDIA interface requires APC. A UPC lead is incompatible with that mating surface, even if the housing appears to fit. Verify polish from the part specification and check both sides of any intermediate connection.
Why is InfiniBand in INIT when the physical link is up?
The physical path has trained, but logical fabric activation is incomplete. Check Base LID and SM LID, query the subnet manager from a reachable management port, and review discovery or isolation logs. Stable physical LinkUp with INIT alone does not identify a dirty MPO connector.
Can the two sockets use a point-to-point cable and a splitter together?
NVIDIA requires the same cable form on both twin-port sockets: both point-to-point or both splitter. MFP7E10 and MFP7E20 serve different endpoint arrangements. The two branches may have different lengths within their specified reach.
Should I clean the patch cord or the transceiver receptacle?
Inspect both mating surfaces on the disturbed connection. Clean contamination using a tool suited to the MPO/APC interface, then inspect again before mating. Follow NVIDIA's no-liquid restriction for this module and leave a healthy, undisturbed sibling branch connected.
Does a nonzero raw BER mean the link has failed?
Raw errors can be corrected by FEC. Compare the trend, effective errors, link transitions and expected rate under the same conditions. The 2.4 × 10−4 stressed-receiver test condition is not a universal field acceptance limit for Raw Physical BER.
Is 4.5 ns/m the allowable MPO lane skew?
It is an approximate propagation-delay value. Skew is the difference in arrival time between lanes within one bonded group. Use the installed product's qualification limit; do not treat propagation delay per metre as a skew pass/fail threshold.
Can a passing insertion-loss result prove the link will work?
It establishes the loss of the tested path under its stated method and limit. It does not establish the required fibre map, APC compatibility, port configuration or stable service. Compare the test endpoints with the complete installed route.
Does NVIDIA's "straight cable" mean Type-A polarity?
In the point-to-point versus splitter description, straight names the cable form. MFP7E10 uses Type-B crossover mapping. Specify physical form and the end-to-end fibre map as separate fields.
What if the link stays down after cleaning and mapping checks?
Substitute one known-correct, inspected assembly or measure the passive path. If the fault stays with the port, investigate the local optical engine, remote optic and host configuration. A shared-module substitution affects both branches and needs a suitable outage.
Technical Sources
- NVIDIA MMA4Z00-NS specifications: reach, optical power, receiver test condition and handling.
- NVIDIA MFP7E10 specifications: IL/RL values and fibre-position map.
- NVIDIA MFP7E10 application guidance: APC, pinning, paired cable form and panel assumptions.
- NVIDIA MFP7E20 ordering information: splitter form and endpoints.
- NVIDIA twin-port connectivity overview: point-to-point and splitter operation.
- NVIDIA MFT v4.35.0 mlxlink utility: module and physical-counter queries.
- Linux RDMA project's diagnostic manuals: ibstat, sminfo, iblinkinfo and perfquery syntax.
- NVIDIA UFM PDR deterministic plugin: NDR counter definitions and time-window procedure.
- NVIDIA 100G-PAM4 transceiver and fibre details: 4.5 ns/m propagation reference.
- IEC 61300-3-35:2022 official summary: contact-area inspection and optical-performance scope.
