Understanding the Magic Number: A Key Component in File Structure and Packet Capture

Background:

1. Lately encountered a problem in a project, tracing the issue of packet loss in the WIFI module. There are many stages where packet loss can occur.
2. All platforms used in my scenario: Main control (running LWIP protocol stack) + SDIO wifi.

• In the above scenario, there are many cases where packet loss may occur:
  1. The wifi module did not receive network packets (packet loss in the air).
  2. The wifi module did not successfully send network packets (packet loss in the air).
  3. Data transmission between main control and wifi dropped packets (SDIO transmission packet loss).

1. On the device side, viewing packet loss through serial printing is very cumbersome; there are many network packets, and it is difficult to identify whether packets have been lost.
2. Capturing network packets via Wireshark can clearly view packets, but it cannot determine whether the wifi received the network packets nor if SDIO had packet loss.
3. Both points 3 and 4 have drawbacks. Serial can know the device-end packets but cannot discern network packets. Wireshark can clearly identify packets but cannot know the device-side situation. If combined, each can complement the other and achieve the clarity of device-end network packets.
4. The above operations come from my supervisor–Old Wu, many thanks. He gave me this idea, and I implemented this method through Python.
5. This capture tool is named: SP2WS (serial packet to Wireshark) captures device-end network packets via the serial port, then displays them in Wireshark.
   • GitHub: https://github.com/RiceChen/SP2WS.git

Implementation Process of serial packet to Wireshark:

• The principle of SP2WS is very simple, implemented by Python to capture device-end network packets through a serial port, then package the network packets into data packets recognizable by Wireshark, and use Wireshark’s named pipe to capture, to allow checking device-end packets in Wireshark. For further understanding of Wireshark’s named pipes, you can directly check detailed instructions on the official website.
• Official website link: https://wiki.wireshark.org/CaptureSetup/Pipes
• Before describing the implementation principle, several concepts need to be understood:

Named Pipe

• The official method for implementing named pipes for Wireshark offers multiple implementations. Our SP2WS is implemented using Python, and we refer to the following example:

Python Environment

• This tool is implemented using Python3, so the PC needs to have the Python3 development environment installed (remember, it’s Python3; for Python2, you need to modify it yourself).
• Since SP2WS uses serial capture, if you encounter the following error while running my tool:

1.  

Traceback (most recent call last):  File ".\SP2WS.py", line 5, in     import serialModuleNotFoundError: No module named 'serial'

• Solution:

pip install pyserial

1.  

Traceback (most recent call last):  File ".\SP2WS.py", line 1, in     import win32pipe, win32fileModuleNotFoundError: No module named 'win32pipe'

• Solution:

pip install pypiwin32

Wireshark Packet Format

1. Wireshark has multiple file saving formats, and pacp is a relatively simple format. My tool also uses this format for capture.
2. The Wireshark packet format includes a file header, followed by a packet header in front of each packet.

• pacp header format:

struct pcap_file_header{  uint32_t magic_number;  /* magic number */  uint16_t version_major; /* major version number */  uint16_t version_minor; /* minor version number */  int32_t  thiszone;      /* GMT to local correction */  uint32_t sigfigs;       /* accuracy of timestamps */  uint32_t snaplen;       /* max length of captured packets, in octets */  uint32_t linktype;      /* data link type */}

Field	Size(Byte)	Description
magic_number	4	Indicates the start of the file (value: 0xA1, 0xB2, 0xC3, 0xD4)
version_major	2	Main version number of the current file (value: 0x00, 0x02)
version_minor	2	Minor version number of the current file (value: 0x00, 0x04)
thiszone	4	Standard local time (value: 0)
sigfigs	4	Accuracy of timestamps (value: 0)
snaplen	4	Max length of captured packets (value: 0)
linktype	4	Link type (value: 1) 0: BSD loopback devices, except for later OpenBSD 1: Ethernet, and Linux loopback devices 6: 802.5 Token Ring 7: ARCnet 8: SLIP 9: PPP 10: FDDI 100: LLC/SNAP-encapsulated ATM 101: “Raw IP”, with no link 102: BSD/OS SLIP 103: BSD/OS PPP 104: Cisco HDLC 105: 802.11 108: later OpenBSD loopback devices (with AF_value in network byte order) 113: Special Linux “cooked” capture 114: LocalTalk

• pcaprec header format:

struct pcaprec_hdr {  uint32_t ts_sec;        /* timestamp seconds */  uint32_t ts_usec;       /* timestamp microseconds (nsecs for PCAP_NSEC_MAGIC) */  uint32_t incl_len;      /* number of octets of packet saved in file*/  uint32_t orig_len;      /* actual length of packet */};

Field	Size(Byte)	Description
ts_sec	4	High part of the timestamp, accurate to seconds (value is counted since January 1, 1970 00:00:00 GMT)
ts_usec	4	Low part of the timestamp, accurate to microseconds (the number of microseconds when the packet was captured, offset from ts_sec)
incl_len	4	Length of the current data block, i.e., the length of the data frame captured, from which the position of the next data frame can be obtained.
orig_len	4	Actual length of the data frame in the network, generally not greater than caplen, mostly equal to Caplen.

• packet: Network packet (referred to in the tool as the network packet captured on the device side through the serial port)

Implementation:

1. Device-side implementation: The code is relatively simple, only needing to add a header and footer to the network packet. Where parameters: buf–the network packet received on the device side, len–the length of the network packet received on the device side.

#define PIPE_CAP_HDR_SIZE       3#define PIPE_CAP_TAIL_SIZE      3static void pipe_cap_dump(char *buf, int len){  char pipe_hdr[PIPE_CAP_HDR_SIZE] = {114, 116, 116};  char pipe_end[PIPE_CAP_TAIL_SIZE] = {101, 110, 100};  char *pipe_data = (char *)malloc(len + PIPE_CAP_HDR_SIZE + PIPE_CAP_TAIL_SIZE);  if(pipe_data != NULL)  {    memcpy(pipe_data, pipe_hdr, PIPE_CAP_HDR_SIZE);    memcpy(&pipe_data[PIPE_CAP_HDR_SIZE], buf, len);    memcpy(&pipe_data[PIPE_CAP_HDR_SIZE + len], pipe_end, PIPE_CAP_TAIL_SIZE);    uart_send(pipe_data, len + PIPE_CAP_HDR_SIZE + PIPE_CAP_TAIL_SIZE);    free(pipe_data);    pipe_data = NULL;  }}

1. Implementation of the SP2WS tool: Mainly consists of two classes, sp2ws_serial and sp2ws_pipe.

• sp2ws_serial class: Primarily handles the opening, closing, reading, and writing of the serial port.

class sp2ws_serial():    def __init__(self, port, baudrate):        self.port = port        self.baudrate = baudrate    def open(self):        self.ser = serial.Serial(self.port, self.baudrate, timeout = 0.5)    def close(self):        self.ser.close()        def read(self, length):        return self.ser.read(length)    def write(self, buff):        self.ser.write(buff)

• sp2ws_pipe class: Primarily handles the creation and connection of pipes (on successful connection, it sends the pacp header to the pipe), and pipe data transmission (adds a packet header before transmitting packets).

class sp2ws_pipe():    def __init__(self, pipe_name):        self.magic_num = 0xa1b2c3d4        self.major_ver = 0x02        self.minor_ver = 0x04        self.link_type = 1        self.pipe_name = pipe_name        def create_pipe(self):        self.pipe = win32pipe.CreateNamedPipe(                    self.pipe_name,                    win32pipe.PIPE_ACCESS_OUTBOUND,                    win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,                    1, 65536, 65536,                    300,                    None)        def connect_pipe(self):        win32pipe.ConnectNamedPipe(self.pipe, None)        # File header        global_header = struct.pack('IHHiIII',            self.magic_num,     # magic number 4            self.major_ver,     # major version number 2            self.minor_ver,     # minor version number 2            0,                  # GMT to local correction 4            0,                  # accuracy of timestamps 4            0,                  # max length of captured packets, in octets 4            self.link_type      # data link type 4        )        win32file.WriteFile(self.pipe, global_header)    def write_pipe(self, packet):        packet_len = len(packet)        if packet_len <= 0:            return        # Packet header        packet_header = struct.pack('IIII',            int(time.time()),                       # timestamp seconds            datetime.datetime.now().microsecond,    # timestamp microseconds            packet_len,                             # number of octets of packet            packet_len                              # actual length of packet        )                win32file.WriteFile(self.pipe, packet_header)        win32file.WriteFile(self.pipe, packet)

• Business logic implementation of the SP2WS tool:
  1. Create a serial port object.
  2. Create a pipe object.
  3. Connect the pipe.
  4. Read serial port data, capture effective network packets.
  5. Send the network packets to the pipe.
  6. The code is as follows:

if __name__ == "__main__":    if len(sys.argv) < 3:        print("SP2WS.py com[port_num] wifi_master [ssid] [pwd]")        print("SP2WS.py com[port_num] wifi_slave")        print(r'\\.\pipe\wifi_master')        print(r'\\.\pipe\wifi_slave')    port = sys.argv[1].upper()    if sys.argv[2].find("wifi_master") > -1:        ssid = sys.argv[3]        password = sys.argv[4]        wifi_connect = r'wifi join ' + ssid + ' ' + password    pipe_name = r'\\.\pipe' + '\\' + sys.argv[2]    # Create a serial port object.    sp2ws_serial = sp2ws_serial(port, 115200)    sp2ws_serial.open()    # Create a pipe object    sp2ws_pipe = sp2ws_pipe(pipe_name)    sp2ws_pipe.create_pipe()    print("Start connecting pipe...")    # Connect the pipe    sp2ws_pipe.connect_pipe()    print("Pipe connection successful!!!")    raw_buf = bytearray()    start = 0    init = 0    if sys.argv[2].find("wifi_master") > -1:        sp2ws_serial.write(bytes(wifi_connect + "\r\n", encoding='utf-8'))        sp2ws_serial.write(bytes("pipe_start\r\n", encoding='utf-8'))    while True:        # Read serial port data        raw = sp2ws_serial.read(1024)        raw_len = len(raw)        if raw_len > 0:            raw_buf = raw_buf + raw            # Capture effective network packets            while True:                raw_len = len(raw_buf)                # Find packet header                for index in range(raw_len):                    if (index + 2) < (raw_len - 1):                        if raw_buf[index] == 114 and raw_buf[index + 1] == 116 and raw_buf[index + 2] == 116:                            start = index + 3                            break                        start = 0                    else:                        start = 0                        break                if start == 0:                    break                                # Find packet tail                for index in range(start, raw_len):                    if (index + 2) < (raw_len - 1):                        if raw_buf[index] == 101 and raw_buf[index + 1] == 110 and raw_buf[index + 2] == 100:                            end = index                            break                        end = 0                    else:                        end = 0                        break                if end == 0:                    break                frame = raw_buf[start : end]                # Send the network packets to the pipe                sp2ws_pipe.write_pipe(frame)                end += 3                raw_buf = raw_buf[end : ]