TROUBLESHOOTING SIMPLE NETWORK PROBLEMS IN TORNADO 2.0 ------------------------------------------------------ If you are new to Tornado 2.0, or you are booting a target for the first time, or if the target booted successfully, but does not respond to pings, please read basicNetConfiguration article. basicNetConfiguration contains instructions on how to configure a kernel with a target shell and network debugging libraries. It contains information to avoid errors like: rpccore backend client RPC: Timed out Network interface unknown muxDevLoad failed for device entry wdbConfig: error configuring WDB communication interface Error loading file: errno = 0xd0003 If target responded to pings initially, and after running your socket application (or a library that uses the network such as the FTP server), applications sputter or hang for a while and then resume, please read netPerformance article. It gives an overview of vxWorks network internals and configuration issues that impact performance. This article assumes that a target has booted successfully and responds to pings initially. You are either running demo code from WindSuf->Demo Code->Networking or your own application. At some point, the target does not respond to pings anymore, or network performance is slow. At this point from the target shell, verify that tNetTask is still running at priority 50. If i shows tNetTask as suspended, call tt tNetTask. There might be a reference to a null pointer somewhere, or an unitialized interrupt. If this error occurs with a default configuration, you have not added third party network drivers, or software, your application is not running, and you are just using WRS software on a supported BSP, please contact: support@windriver.com. If application runs fine, but as you increase the number of sockets or the packet rate, the application sputters or hangs, run the show functions described below from the target shell AFTER the target boots and AFTER the problem occurs. ->netStackSysPoolShow Network stack system memory pool status ->netStackDataPoolShow Network stack data memory pool status ->i which tasks are running, show priorities ->inetstatShow shows sockets and their receive queues ->ipstatShow show IP protocol statistics ->tcpstatShow show TCP protocol statistics ->udpstatShow show UDP protocol statistics ->icmpstatShow show ICMP protocol statistics -->mRouteShow show route table, including masks ->routeShow show route table without masks ->iosFdShow show file descriptors ->arpShow show ARP table contents ->hostShow show host table content >ifShow show interfaces attached to IP. This function will show END and BSD44 drivers. ->muxShow show END drivers successfully loaded in the MUX ->endPoolShow("name", 0) END driver memory pool status of driver "name" For example: endPoolShow("fei", 0) endPoolShow is not part of the API. This utility is found in utilities.c SYMPTOMS: A. ENOBUFS SOCKET APPLICATION SPUTTERS If the network stack or driver memory pool sizes are too small, applications will appear to hang and recover within minutes as resources are freed. Errors like ENOBUFS, or S_netBufLib_NO_POOL_MEMORY may occur. The problem is most likely insufficient network stack and driver buffer pool. Call netStackSysPoolShow and netStackDataPoolShow. Increase the default allocation if there are 0 free buffers available. Increase the driver's buffer pool, and IP configuration parameters as indicated in netPerformance. B. TARGET STOPS RESPONDING TO PINGS. RPC CORE BACKEND TIMEOUT REPORTED IN TARGET SERVER WINDOW 1. APPLICATION/DRIVER BUFFER LOANING PROBLEMS If sockets buffers are not being read, driver buffers will be depleted. As long as the driver is out of buffers, the driver will not be able to receive or send packets. Pings will be unanswered. Target server will be unable to contact the target. muxShow displays END drivers only. If the boot device (or the other parameter if boot device is floppy, ata...) is listed in muxShow, the driver version used is END. ifShow displays both END and BSD 4.4 drivers. endPoolShow displays END driver's buffer pool status. If there are 0 free buffers, use inetstatShow to determine which socket has data in the receive queue. This information can be used to identify which task is not reading its sockets. See netPerformance for an explanation of network driver buffer loaning scheme. 2. DRIVER PROBLEMS If inetstatShow shows no data in the receive queue, the problem may be due to a faulty driver: If you are using the END driver version, try the BDS44 version (if available), and vice-versa. If the BSP supports more than 1 network card type, try a different card. If muxShow does not show any drivers, BSD44 drivers are configured. BSD44 drivers are configured if EITHER T101 build method is used, and INCLUDE_END is undefined in config.h OR in T2 Project facility: network components->network devices->End attach interface and End interface support components are excluded. See basicNetConfiguration document for information on configuring drivers. If changing drivers works, report the problem to support@windriver.com. Check WindSurf->Resolve to check Problem Lists and Keyword Search. If there are no other driver versions you can use, use the loopback driver. Run both the server and the client code in the target. Send to IP address 127.0.0.1. C. TARGER RESPONDS TO PINGS, BUT DATA TRANSFER STOPS If the target responds to pings, but data transfer stops, call inetstatShow on the target and the corresponding function (usually netstat) on the other host. If there is data backed up in the send side and on the receive side on the other side, most likely there is a deadlock situation within the client/server application code. Check if deadlock does not occur if you reduce the message size below 1460 bytes. (For TCP). TCP is a stream of bytes. Loops are required to read or write, since TCP can return partial reads and partial writes. D. APPLICATION WORKS BUT NETWORK PERFORMANCE IS SLOW See netPerformance article. E. PROTOCOL PROBLEMS Check the before and after protocol error statistics show routines for clues as to which error is increasing. Please read Section 1.4.1 of NetPerformance article for an explanation on UDP full sockets. F. PRIORITY PROBLEMS Make sure that tNetTask has a higher priority than application tasks depending on the network. Please read Section 1.2 of netPerformance article. G. HARDWARE OR DRIVER MISCONFIGURATION PROBLEMS 1. Try changing the network card or using a different card. Examine hub's LED. Verify that the LED shows activity for the driver. 2. Configure the BSD44 driver version, if available. Call ifShow to determine if there are collisions, CRC ... The end driver does not report this information. If the driver reports excessive collisions make sure that another node in the network is not in full duplex mode. 3. If there are broadcast storms, verify that ifShow does not show driver is in PROMISCUOUS mode. ------_extPart_000_01C0FFDE.1E130740 Content-Type: application/octet-stream; nametilities.c" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filenametilities.c" /*********