1. Symptoms
A certain shipping company started a network upgrade project three weeks ago to meet the growing business demands. They planned to upgrade their 10BaseT network to a 100BaseTX Ethernet network without making any changes to the cable system. After completing the equipment installation and configuration yesterday, network cutover operations began early this morning. All workstations were equipped with 100BaseTX network cards and gradually connected to the network. At this point, the engineering team noticed some strange occurrences, such as some workstations being unable to connect to the network, some workstations connecting successfully initially but losing the connection after a while, and some workstations experiencing intermittent connection or data errors. The systems integrator initially thought it was a platform installation issue, so they reinstalled the system platform and reinstalled the system and application software on the problematic workstations, but it didn’t resolve the issues. After almost a full day of troubleshooting, they couldn’t provide services to users, and the business operations were almost entirely disrupted.
2. Diagnostic Process
Upon receiving the report, I immediately went to the “site of the incident.” I activated all system components, including the problematic workstations, for network operation. Using the F683 network “multimeter,” I conducted a standard health test on the faulty network. After one minute, the test results showed: Network utilization 1.3% (as all employees had already left for the day), collision rate 8%, error rate 11%, and broadcast rate 9%. Clearly, the network collision rate and error rate were relatively high. I then opened the error diagnostics feature, which displayed FCS frame errors, local collisions, and fragment errors. This indicated the possibility of problems related to network card malfunctions, cable system issues, interference, or grounding circuits. Upon checking the specific FCS frame error test results, it was apparent that many workstations were sending erroneous FCS data frames. In general, the simultaneous failure of multiple network cards is unlikely. Most of these FCS frame errors are likely due to cable issues, particularly involving cables with excessive length. The users explained that their system used Category 5 cable, consisting of 270 workstations divided into six segments, with one private network router and one public network router. Prior to the upgrade, they had been using a 10BaseT Ethernet network, and except for one department that occasionally reported slow network speeds and occasional connection issues, other departments had no issues (the problematic department had the highest workload).
After the upgrade began, some workstations started experiencing the various issues mentioned earlier. The affected area consisted of approximately one-third of the workstations. I inquired whether the user had previously conducted tests on the cabling system. They responded, “We’ve only tested for continuity because we’ve always had a working 10BaseT Ethernet network, so we assumed there wouldn’t be any problems with the cabling system.”
To quickly diagnose the problem, I randomly selected ten problematic cables for testing. The results showed that eight had one-to-two socket faults, three had Category 3 connector errors, and four failed the PS NEXT (Pair-to-Pair Near-End Crosstalk) test. The conclusion was that there were significant issues with the cabling system.
3. Conclusion
The poor quality of network cabling is a significant concern in comprehensive cabling projects. Although there are mature testing standards and methods available, many users are unaware or don’t demand on-site certification testing according to these standards. In this case, cable faults resulted from a variety of causes, all stemming from irregularities in engineering design, construction, and acceptance. The issues included:
a) Faulty one-to-two sockets: Caused by wiring mistakes. Users did not consider the need for expansion when designing, so they used these non-standard one-to-two sockets where one socket could connect two PCs. In theory, this kind of configuration is generally workable, but it requires connecting lines 1-2/3-6 to one PC and lines 4-5/7-8 to another PC. However, the actual test results showed that the configuration was 1-2/3-6 and 4-5/3-6, which incorrectly treated lines 3-6 as the direct “shared media detection bus.” In a 10BaseT network, this improper connection could still work. While it caused a considerable number of data frames in the entire network traffic to contain errors, it went largely unnoticed because most existing networks had low utilization rates. However, in a 100BaseTX network, this incorrect configuration would lead to connection difficulties after the upgrade.
b) The system underwent a user-initiated expansion without proper planning, and fake Category 5 connectors (which were, in fact, Category 3) were used. In a 10BaseT network, Category 3 connectors would not interfere with normal network operation. However, after the upgrade, parameters like PS NEXT became critical and severely affected the connection between workstations and the network. This misusage of Category 3 connectors without proper testing could remain undetected for a long time.
c) Due to the use of one-to-two sockets, testing for high-end parameters, including PS NEXT, was necessary for cable testing. This test is especially important for cable configurations that employ one-to-two sockets. As a result, connections on cable configurations that failed the PS NEXT and other high-end parameters would become unstable or lead to difficulties connecting workstations.
4. Diagnostic Recommendations
It is recommended that cabling systems (cables and fiber optics) undergo certification testing before being put into operation. There are many standards to choose from, but TSB-67 or ISO11801 are popular international standards for testing. Only testing for physical continuity and assuming that the link is definitely usable is a very one-sided and harmful approach. Cable configurations that employ one-to-two sockets must undergo testing for high-end parameters like PS NEXT. Using Cat5e standards for certification testing is strongly encouraged. We suggest that the shipping company conduct testing and maintenance on all cabling links overnight and label the links that still fail high-end parameters after cleaning to facilitate future replacement.
5. Afterword
The next morning, all testing was completed. A total of 88 one-to-two sockets with connection errors were discovered (all were incorrectly connected), along with 54 instances of incorrect use of Category 3 connectors. After rectifying all the socket errors and addressing the misuse of Category 3 connectors, almost all workstations were back to normal operation. At lunchtime, after over a day of disruption, business operations were fully restored. Subsequent network performance assessments also mostly passed.
For individual links with high-end parameter failures, we advised network administrators to either replace the links or temporarily disable the one-to-two connection method to ensure the correctness of data transmission.