Essential Routing Troubleshooting Techniques for Network Engineers

1. Introduction to Routing Troubleshooting Techniques for Network Engineers

For network engineers, mastering routing troubleshooting techniques is essential to resolve network issues effectively. Familiarity with these techniques not only minimizes downtime but also ensures network stability. Despite the necessity, many novice network professionals still struggle with troubleshooting, which can lead to persistent issues.

However, it is necessary. There are still many novice network workers who don’t have a clear grasp of it or are not familiar with it, which causes them a lot of trouble.

For experienced network engineers, if they have mastered the general troubleshooting ideas, they will generally have no problems in handling the problem. However, they can still improve their troubleshooting skills.

2. How to Troubleshoot a Closed Router External Network Port

(1) Network environment

A certain unit uses a Cisco router and rents a 30MB local access and a 10MB education network dual-line Internet access from China Telecom. The network has been running stably for two years and the router has not malfunctioned.

As the number of Internet users increased, the original 30MB of Telecom could no longer meet the demand, so it was decided to rent 100MB of Telecom to solve the bandwidth problem.

After the telecommunications company uses optical fiber to access the unit’s computer room, it uses a 100M optical-to-electrical converter to convert the data and connect it to the router’s external network port via a twisted pair cable. The router uses a 1000M electrical port as its external network port. Since the optical-to-electrical converter is only 100MB, the speed of the port is displayed as 100MB after connection.

(2) External network port traffic is zero

After running for a few days, the administrator found that when the traffic at the router’s external network port exceeded 50Mbps/s every day, “Receive Errors” would appear on the port, indicating excessive traffic and a large number of error messages.

Then I could not access the external network. I Telnetted to the router and found that there was no traffic on the external network port corresponding to China Telecom. The status was displayed as UP, and other ports on the router worked normally.

The first reaction was that there was a problem on the telecommunications side, so we called the telecommunications department to check it out. They quickly responded that there was no problem and asked whether the photoelectric converter had crashed.

So the administrator restarted the photoelectric converter, but the problem still persisted. There was no other way, so he had to restart the router to troubleshoot the problem.

But less than an hour later, the fault reappeared.

After Telnetting to the router and executing shutdown and undo shutdown on the external network port, the fault was eliminated. However, after applying all virus-related security policies to the port and changing tcp mss to 2o48 (the manufacturer’s default is 1460), the fault still occurred.

(3) Fault analysis

The administrator found that when the fault occurred, the CPU was 23% and the Memory was 33%, which were not too high. The key was that other interfaces were working normally, and it seemed that the problem still occurred on this port.

But this port has been used for two years. Before the upgrade and expansion, there was no problem with the port not being able to communicate normally. There must be some problem with the port hardware.

Through the network management software to detect the traffic before the port is closed, it is found that there is a lot of traffic passing through the port before it is closed (over 80Mbps/s), and there are many error messages on the display port. After analysis, it is found that it is probably caused by too much network traffic and too high utilization.

When the traffic exceeds 80%, the port cannot function properly. If the port can work in Gigabit mode, the 100MB bandwidth only uses 10% of the port, so the port can handle it easily.

(4) Solution

After finding the problem, the recommended solution is to purchase a Gigabit optical-to-electrical converter to replace the original 100M equipment, which is also relatively cheap.

However, in order to ensure the stability of network operation, the unit decided to directly purchase a gigabit optical port routing module to directly use optical fiber for communication and reduce network delay.

Telecom uses port speed limit to ensure the provision of 100M bandwidth. After a period of operation, it was found that except for a small number of error messages, the port has never been closed for no reason.

3. Ouch, why does the router always fail to send packets?

During the configuration of the router, you often encounter the following problems:

The network communication is normal, and the router can successfully route data packets to the target network. However, the data packets sent from the router fail to be transmitted. The fault is manifested as the router failing to ping the target network.

(1) Description of the phenomenon

After the network configuration of a certain unit was completed, the administrator found that when testing the network connectivity, when sending a Ping from the PC (6.159.245.195) to the target network (6.159.245.65/26), the router R1 could successfully forward the data packet. However, when sending a Ping from R1 to the target network (6.159.245.65/26), the Ping failed.

(2) Troubleshooting process

First, trace the path that the ping takes. Check the routing table of R1, and the destination address 6.159.245.65 can be matched with 0.0.0.0/0 in the routing table. Check the routing tables of R2, R3, and R4, and you can find routing table entries that match the destination address.

Then, trace the path that the ICMP echo reply packet takes. To complete this step, you need to identify the source address of the echo reply packet. When the PC sends a ping, the destination address of the echo reply packet is 6.159.245.195.

When router R1 sends a ping, the destination address of the reply packet is 71.170.0.146.

By comparing R4’s routing table, a routing table entry matching 6.159.245.195 is found, but no routing table entry matching the destination address 71.170.0.146 is found.

It seems that the ICMP echo reply packet is discarded when it is processed by R4, so when sending a ping from R1 to the target network R4 (6.159.245.65/26), a pmg failure occurs.

(3) Solution

Add a static route to 71.170.0.144/30 on router R4, with the next hop address being 71.170.0.214. After completion, when R1 sends a ping to R4, everything is normal.

Although this type of network failure will not affect the normal communication of the network and the troubleshooting process is also very simple, we must consider the complete communication process when analyzing and troubleshooting network failures.

4. What should I do? It is so difficult to upgrade the IOS of Cisco routers

A certain school started to build a campus network in 2003. In recent years, the number of Internet users has continued to increase, making the original Cisco 2621 far from meeting the needs of the network.

Moreover, we are going to use the 0A office system recently, and we need to add a VPN device for off-campus users to access the on-campus 0A system.

For economic reasons, they wanted to upgrade the idle Cisco 2621 router to do VPN. However, they encountered some problems during the IOS upgrade.

(1) Garbled characters appear when logging into the HyperTerminal

I took out the router from the warehouse in the computer room and powered it on. I connected it through the console port and found that some garbled characters appeared on the super screen. Could it be that the console port was broken?

Analysis shows that if the console port of a Cisco device is broken, it will usually output a lot of garbled characters on the HyperTerminal screen. However, this time, the garbled characters appeared on the screen only after the Enter key was pressed, which may be due to the incorrect transmission rate per second.

The administrator changed the default value of 9600 to l15200. The router started successfully. After the router started, the console rate was indeed found to be l15200 using show run.

(2) Upgrade failed due to insufficient memory

The Cisco 2621 router to be upgraded has VPN function. The original IOS version is C2600-i-mz.122-8.T4.bin. I learned from the Internet that only the K8 and K9 series IOS of Cisco 2621 can support VPN.

So the administrator downloaded the new IOS c2600-ik9o3s3-mz.123-22.bin from the Internet, which was 15MB in size.

The upgrade process is as follows:

(1) Configure the IP address of the router Interfast 0/0, first use the “copy flash:tftp” command to back up the original IOS, and then upload it using the “copy tftpd flash” command.

(2) Restart the router and find the following error message, which means there is not enough memory to run IOS:

Error:memory requirements exceed available memory Memory required:0x0284A0BC

I checked the Cisco official website and found that the 10s image c2600-ik9o3s3-mz.123-22.bin requires the router to have 64MB of memory and 16MB of Flash. From the boot information above, I can see that the router has 32MB of memory, so it can’t boot.

Later, I bought a 128MB memory stick online and replaced it. After increasing the memory, the router started successfully.

5. IOS upload verification failed via TFTP in ROM mode

Since the router IOS upgrade failed, I want to restore the original IOS. After the Cisco IOS upgrade fails, there are two ways to restore the IOS: FTP and Xmodem. TFTP has a faster transmission speed, while Xmodem has a slower transmission speed.

In ROM mode, use TFTP to upload IOS. The process is as follows:

(1) Configure the IP address on interfast 0/0 and use the set command to view the configuration. By default, the IP address configured in ROM mode is under interfast 0/0. The configured IP address should be in the same network segment as the TFTP server.

(2) Download using tftpdnld. TFTP ServerMg initially used Cisco’s TFTP, but it timed out halfway through the transfer.

After transferring with 3Cdaemon, the following warning was found:

TFTP flash C0PY: Warning, ChecksSum comparison failed.
Restart the router, but the router cannot start and prompts IOS verification error.

I thought it might be an error when downloading IOS, but after re-downloading c2600-ipbase-mz.123-6c.bin, it still didn’t work, so it didn’t seem to be an IOS problem. Later, I changed the network cable and TFTP software, but it still didn’t work.

Solution

Use Xmodem to transfer. To make the transfer faster, we should change the Xmodem transfer speed to ll5200. After the IOS is transferred via Xmodem, restart the router and the router will be bootable.

6. Experience summary, the best of the best is here

(1) When uploading IOS using TFTP, if a verification error is displayed, you should consider using Xmodem to upload.

(2) The IP address of the TFTP server must be on the same network segment as the Ethernet port of the router.

(3) When uploading IOS using Xmodem, it is best to use the HyperTerminal that comes with Windows.

(4) TFTP (Trivial File Transfer Protocol) supports transferring files up to 32MB. If the IOS is larger than 32MB, you can consider using third-party TFTP software, such as 3Cdaemon.

Another way is to use FTP transfer command.

Router failures are complex and the troubleshooting methods are different, but the troubleshooting ideas and techniques are still common.

When encountering routing failures, you should analyze calmly, practice boldly, and be good at summarizing, so as to improve your troubleshooting capabilities, right?