SENS/TORNADO 2.0 NETWORK STACK CONFIGURATION/PERFORMANCE ISSUES -------------------------------- 1. INTRODUCTION 1.1 NETWORK STACK VERSIONS 1.2 NET POOLS 1.2.1 CLUSTERS 1.2.2 CLUSTER BLOCKS 1.2.3 MBLOCKS 1.3 RECEIVING: DATA FLOW FROM DRIVER TO APPLICATION LEVEL 1.4 SENDING: DATA FLOW FROM APPLICATION LEVEL TO DRIVER 1.5 SOCKET BUFFERS 1.5.1 UDP SOCKET BUFFERS 1.5.2 TCP SOCKET BUFFERS 2. NETWORK MEMORY RESOURCES SUMMARY 2.1 MEMORY POOLS 2.1.1 NETWORK STACK SYSTEM POOL 2.1.2 NETWORK STACK DATA POOL 2.1.3 NETWORK DRIVER POOLS 2.2 RECEIVE QUEUES 2.2.1 IP PROTOCOL RECEIVE QUEUE 2.2.2 IP FRAGMENT REASSEMBLY QUEUE 2.2.3 ARP RECEIVE QUEUE 2.2.4 TCP REASSEMBLY QUEUE 2.2.5 SOCKET RECEIVE QUEUES 2.3 SEND QUEUES 2.3.1 SOCKET SEND QUEUES 2.3.2 IP NETWORK INTERFACE SEND QUEUES 3. ESTIMATING NETWORK RESOURCES REQUIREMENTS 3.1 SOCKETS AND ROUTES 3.2 SENDING AND RECEIVING 3.2.1 LARGE SOCKET RECEIVE BUFFERS 3.2.2 TCP SMALL PACKETS 3.2.3 SENDING 3.2.4 RECEIVING 3.2.5 APPLICATION ERRORS 4. CONFIGURATION 4.1 NET POOL CONFIGURATION 4.1.1 NETWORK STACK MEMORY POOLS 4.1.2 NETWORK DRIVER MEMORY POOLS 4.2 PROTOCOL RECEIVE QUEUES 4.3 IP SEND QUEUES 5. REFERENCES FIGURE 1. MEMORY POOLS AND QUEUES DEFAULT VALUES 1. INTRODUCTION ------------ This article provides background and guidelines for configuring and performance-tuning the VxWorks network stack. This article applies to the Tornado 1.0.1 stack with SENS installed, and to the Tornado 2.0 stack with or without the Network Protocol Toolkit product installed. The assumed network programming interface is the BSD sockets interface; the zbuf sockets interface is mentioned only in passing. Tornado/setup.log file shows which version of Tornado and which components are installed. The Tornado 2.0 VxWorks Network Programmer's Guide, Section 4.6.3, has the following comments regarding network stack memory configuration: "The values shown above are reasonable defaults, but the network requirements for your system could be radically different. For example, your use of the network stack could require more clusters of a particular size. Making such a determination is a matter of experimentation and analysis.", "Change these constants after you fully understand what they do. Setting inappropriate values can make the TCP/IP stack inoperable.", and "Carefully planned changes can significantly improve performance, but reckless changes can significantly reduce performance." As many users find out, the default values for the network stack configuration parameters are adequate to download a file, open a few sockets, add a few routes to the routing table, and run simple client/server applications. Anything beyond that makes the stack hang for a while and resume a short time later. The network system requires clusters (blocks of memory) of various sizes from different pools, used to store system data structures, incoming data, and outgoing data. Clusters carrying data are stored in various holding areas (queues) along the path from application level to the network driver and vice-versa. The number of clusters of each size and the capacity of the holding areas must be configured to suit the application needs. It is also necessary to ensure that the clusters are processed regularly so that they will be available to carry new data. If clusters are held in any area for too long, the system's throughput will decrease. If the accumulation continues, data flow will eventually stop until clusters are freed up. Analyzing application needs requires basic knowledge on how the network system works. This paper will describe: 1. How data flows from application level to network interface driver and vice-versa, for TCP and UDP; 2. The role of tNetTask; 3. How the configuration of network stack and driver memory pools, application socket buffer sizes, protocol input queues, and interface output queues interact with network performance. Understanding these factors would help the reader set the network stack configuration values appropriately. Tuning configuration parameters for performance is in essence estimating the resources needed to sustain data flowing at a certain rate. As in any other inventory situation, requirements must be estimated taking into account fluctuations in demand and delays. Having enough capacity to handle the worst case is the only sure way to ensure reliability. This approach may not be cost effective in many cases, but may be the only alternative for some. Only users can make that determination based on their application needs. Section 1 presents background information on basic network stack versions and terms, followed by a description of data flow for receiving and sending, and socket buffers. Section 2 presents a summary of the memory resources used by the network system and a description of the various queues where these resources wait is presented next. Sections 3 and 4 present some guidelines for estimating requirements and configuration information. Section 5 lists resources where additional information can be found in vxWorks Documentation, reference books and articles from: WindSurf->Technical Solutions II. 1.1 NETWORK STACK VERSIONS -------------------------- The network stack in Tornado 1.0.1 is based on BSD4.3, which is essentially the network protocol IP over the link layer protocol Ethernet. BSD4.3 Ethernet network interface drivers do not support multicasting, or polled Ethernet for system mode debugging. Under T101 the SENS component was optional. Under Tornado 2.0, SENS is the one and only network stack. SENS features: a. BSD4.4 network stack. BSD4.4 has enhancements at the network stack level such as support for CIDR (variable length subnet masks), multicasting, and various protocols such as DHCP, and RIP. b. MUX The MUX interface facilitates connecting END drivers to other user-ported network protocols besides IP. END drivers can be written for link layer protocols besides Ethernet. c. BSD4.4 and END drivers for supported Ethernet network cards. BSD4.3 drivers were modified slightly to work with the BSD4.4 network stack. Either BSD4.4 or END drivers can be used for booting. END drivers are required for multicasting and system mode debugging over Ethernet. The SENS release provided END Ethernet drivers for some BSPs. The Tornado 2.0 release provides END Ethernet drivers for all supported BSPs. The free Network Protocol Toolkit (NPT) source product contains updated documentation for porting to the END driver model and source for the MUX and supporting network libraries. In addition, NPT contains information for porting to the NPT driver model which facilitates porting non-Ethernet drivers. Network drivers developers should get the Network Protocol ToolKit product from their sales representative. The outdated BETA User's Guide is still available from WindSurf->Documentation->Tornado 2.0 Documentation. For sending, the path from IP to the physical medium can take the BSD4.4 driver path, or the END driver path. Similarly, for receiving, the BSD4.4 driver and the END driver path both converge at the IP level. The details are hidden in the initialization process. The driver’s initialization process installs the proper function pointers in the ifnet structure that IP keeps for every configured interface. For example, ipAttach will install ipOutput as the if_output function pointer of the ifnet structure for an END driver. Similarly, ether_attach will install ether_output for a BSD4.4 driver. IP selects an interface to send a packet, based solely on the routing table. IP calls the if_output function of the selected interface, so IP does not need to know what kind of driver it is. The setup.log file found in the Tornado base directory shows which network stack version is installed: SENS 1.0: T101 + WindNet SENS installed under SETUP 1.4 SENS 1.1: T101 + WindNet SENS installed under SETUP 1.5 Tornado 2.0: SETUP 2.0. The word SENS does not appear since SENS is not an optional component anymore. Note: T101 ARM BSPs have SENS 1.0 installed by default. There was no SENS 1.1 patch update for ARM BSPs. 1.2 NET POOLS --------- Tornado 1.0.1 (without the SENS component installed) had a memory pool consisting of fixed data structures, BSD style mbufs, used both to store small stack data structures such as socket addresses, and for packet data. BSD mbufs were designed to facilitate passing data between network drivers and the network stack, and contain pointers that can be adjusted as protocol headers are added or stripped. BSD mbufs contain space within them to store small amounts of data. Larger amounts of data were stored in fixed-sized clusters (typically 2048 bytes), which could be referenced and shared by more than one mbuf. In T101 + SENS and in Tornado 2.0, the BSD network buffer implementation has been changed somewhat. Slightly modified BSD mbufs are retained, but always reference external clusters rather than carrying data directly; they are now called "mblocks." Pools of clusters of multiple sizes are used, and a new structure called the "cluster block" was added, supporting the zbuf sockets interface and multiple network pools in addition to cluster sharing. The file target/h/netBufLib.h has the definitions of the new memory system. The basic structure is the NET_POOL, which contains mblocks, cluster blocks, and clusters of one or more sizes. The network stack and some network drivers create different NET_POOLs: a) The network stack DATA pool is used for actual data and headers transferred between the application, the stack, and the network interface. b) The network stack SYSTEM pool is used to allocate clusters for holding network stack control structures such as sockets, protocol control blocks, routes, and so forth. c) Network interface pools are used by (some) network drivers to hold clusters for receiving packets off the wire, and for staging packets to be transmitted by the target. Users can create pools for their own purposes as well. The header target/h/net/mbuf.h maps the Berkeley style mbuf definitions to the new style. 1.2.1 CLUSTERS -------- A NET_POOL may contain several cluster pools, each one holding clusters of a single size. Valid cluster sizes range from 64 (2 to the 6th) to 65536 (2 to the 16th). However, not all conceivable combinations of cluster pool sizes are allowed within a net pool. The NET_POOL structure contains an array of pointers to cluster pools. Each row in the array corresponds to an allowed cluster size range, measured in bytes, starting at a power of 2. The array has room for a maximum of 11 entries, with indices ranging from 0 (6 - minimum cluster size log 2 = 6 - 6) to 10 (16 - 6). When configuring a NET_POOL, choose at most one cluster size from within the range of each row, as given by this table: Row 0: 64 <= cluster size <= 127 Row 1: 128 <= cluster size <= 255 Row 2: 256 <= cluster size <= 511 Row 3: 512 <= cluster size <= 1023 Row 4: 1024 <= cluster size <= 2047 ... Row 9: 32768 <= cluster size <= 65535 Row 10: 65536 = cluster size If 2 cluster sizes fall in the same row, the pointer to the second cluster will replace the first. 1.2.2 CLUSTER BLOCKS -------------- A cluster block is a control structure pointing to a cluster. It contains a count of the mblocks referencing the cluster, as well as an optional function pointer and three function arguments. If the function pointer (pClFreeRtn field) is non-null, the function is called with the arguments when the cluster is freed; this is used to support the zbuf sockets interface. Users can provide their own functions as well. If the pClFreeRtn function pointer is null, the network stack returns the cluster to the appropriate cluster pool. The address of the cluster pool is found by subtracting 4 bytes from the clNode.pClBuf member of the cluster block. One cluster block is required for each cluster. 1.2.3 MBLOCKS ------- Mblocks are control structures which point to a section of data within clusters, and link them together into packets and chains of packets. Mblocks also contain other information about the type of data contained in the clusters they reference. There should be at least as many mblocks as there are cluster blocks. Some extra mblocks are needed for the network stack DATA pool. The stack uses DATA pool mblocks to reference data in the clusters without copying. A TCP socket's send buffer is a chain of mblocks linking together clusters containing data which has not yet been acknowledged by the socket's peer endpoint. Some of this data is also referenced by additional mblocks used to form the TCP segments which are passed to the network driver for transmission. In UDP, additional DATA mblocks are used to store source address and port information for received UDP packets. A rough estimate (sometimes erroneously treated as a hard rule) is that there should be two times as many mblocks in the stack DATA pool as cluster blocks. This "rule" certainly does not apply to the stack SYSTEM pool or to network interface pools. If the Network Protocol Toolkit product has not been installed, the IP protocol driver allocates just two mbufs from the net pool of each END interface it attaches to. Only the network driver allocates from an END or NPT driver pool if NPT has been installed. BSD 4.4 drivers may allocate mbufs from the DATA pool. When freed, mblocks, clusters, and cluster blocks are returned to the pool from which they were allocated. 1.3 RECEIVING: DATA FLOW FROM DRIVER TO APPLICATION LEVEL ----------------------------------------------------- Network drivers, and protocols like TCP and IP, schedule tNetTask to do task level processing. IP and TCP schedule tNetTask to process timers: to drop IP fragments that timed out before being reassembled, and to send TCP delayed data acknowledgements, or retransmissions. Drivers schedule tNetTask to process received packets, and sometimes to perform miscellaneous other jobs. While processing received packets, protocols may need to send output messages (ICMP errors, TCP immediate ACKs or fast retransmits, TCP data sent in response to peer's window openings, ARP replies, and so forth). Together with timer-triggered TCP retransmissions or resets, these are cases when sending is done in the context of tNetTask. tNetTask is not involved when applications call output functions like sendto() or write(). Such processing is done within the context of the application task. (However, TCP data placed into a socket send buffer by an application's write() call may be later transmitted in the context of tNetTask when the send window opens.) Below is a overview of the receive processing done by tNetTask: When a packet is received, within the driver's interrupt service routine, netJobAdd() is called to add the driver's task level receive routine to a ring buffer of jobs (fixed capacity = 85 jobs) for tNetTask. This is done to minimize the time spent in the ISR with interrupts locked. If the job is successfully added, netTaskSemId is given to awaken tNetTask. If not, a ring buffer overflow log message appears. When tNetTask (priority 50) runs, it will first process all the jobs in the ring buffer. If any IP datagrams have been queued, it then calls ipintr() to process them, and finally calls arpintr() if any ARP packets are queued. Each time tNetTask runs a job from the ring buffer, and again before calling ipintr() and arpintr(), it takes the splSemId semaphore by calling splnet() to lock out other network stack tasks. The driver's task-level receive function calls the stack receive function. The driver lends the buffer to the stack, if possible. The stack strips the packet's link layer header to determine who should process the packet next. (NPT drivers do the header "parsing" for the stack; for other drivers, the stack must do so itself.). For END drivers, the MUX hands the packet to the appropriate protocol input routine; for ARP and IP packets, this is ipReceiveRtn(). BSD 4.4 drivers call do_protocol (ether_input). ipReceiveRtn and ether_input in turn call function do_protocol_with_type(), which places the packet in the appropriate protocol queue (IP = type 0x800), (ARP = 0x806), if there is room. The default size for both the IP and ARP input queues is 50 packets. The packet is dropped if the corresponding queue is full. IP determines if the packet arrived in good condition and whether it is a fragment or a full datagram. If it is a datagram, or if it is a fragment completing a datagram, and the datagram specifies a known transport-level protocol, IP then calls the appropriate protocol receive function (e.g. tcp_input, udp_input, icmp_input ...). The receiving protocol processes the packet. If there is room in the socket receive buffer, the data is appended to the socket's receive buffer. If a task was blocked waiting to read data from the socket, it is given a semaphore; the task is then READY to run. When the priority level is such that the task runs and reads from the socket, the socket layer copies the data to the buffer provided by the user. When all the data from a cluster has been copied to the application's buffers, the cluster is returned to the driver's pool, or to the network stack DATA pool if a BSD4.4 driver had to borrow from the network stack DATA pool. (If the zbuf interface is used, the cluster is not returned to its pool until the received zbuf is deleted.) Some output may occur during processing of received packets. IP may need to send various ICMP messages. If the received packet is a UDP packet, and there is no socket bound to the destination port, UDP will call icmp_error to send an ICMP Unreachable error message. For TCP, tcp_output() may be called during reception for a number of reasons, for example, to send an immediate ACK if an out-of-sequence segment is received, to do a fast retransmit, or to send more available data, if the application is sending data, in response to a window opening. (If the application is receiving data, window updates may be sent to the peer after the application reads data from the socket buffer, but this does not happen in the context of tNetTask.) The function ip_output(), and ultimately the driver's output function are eventually called, still in the context of tNetTask. 1.4 SENDING: DATA FLOW FROM APPLICATION LEVEL TO DRIVER --------------------------------------------------- In most cases transmission of application data to the network occurs in the context of application tasks. tNetTask is involved in just a few cases: tNetTask may transmit TCP application data, both during retransmission, and when data earlier placed in the socket buffer by the application is later sent in response to the send window opening; and tNetTask may transmit any IP datagram held earlier pending an ARP reply. When send(), sendto(), or write() is called at the application level, the following happens: splSemId semaphore is taken by calling splnet to lock out other network stack tasks. If there is room in the socket send buffer, the socket layer copies data from application level buffers to clusters allocated from the network stack DATA memory pool (_pNetDpool). For example, if an application writes 1460 bytes, the socket layer will request a buffer big enough to contain the data plus some space for link layer headers. The system will return the closest size available. The data will be copied to a single buffer (2048 bytes, in the default configuration), if available, or to a chain of smaller buffers, otherwise. If there are no buffers, the system attempts to find buffers, and if it is not successful sendto, write ... will return ENOBUFS for non-blocking AND blocking sockets. The socket layer then calls the protocol user request function for output, which in turn calls the actual protocol output function (e.g., udp_output(), tcp_output()) to process the data. After the appropriate protocol processes the data and adds its header, one or more packets may be passed to IP. Based on the destination IP address, IP locates the appropriate route and calls the output function for the outgoing interface corresponding to that route. If the packet is too large for the outgoing interface's MTU, IP will fragment it before calling the interface output function. The output function will resolve the address to obtain a hardware address, if necessary. It will then add the packet to the send queue for the appropriate interface, if there is room. (When a network interface is attached to IP, the default size for the send queue is set at 50 packets.) ENOBUFS is returned if there is no room in the send queue, and the packet is dropped. The appropriate if_start() function is called to pass every packet in the interface send queue to the driver's transmit function. After all packets have been passed to the driver (or the driver has stopped accepting packets until a transmit restart) splx() is called to release splSemId. Each driver (END or BSD) is different. Some END drivers have clusters for sending, others use different data structures called various names such as descriptors or frames. Some BSD drivers have 1 transmit frame, others have many. Whether the driver uses clusters or frames, it still has to find a free one. In general for END DRIVERS, using clusters: Within the driver's output function, the driver takes its own transmit semaphore which is SEM_INVERSION_SAFE The driver attempts to allocate a buffer from its pool to copy the packet passed by the stack. (The copy is done for two reasons: the hardware may not support gather-write, and the stack buffers containing the data may not be cache-coherent.) If the driver does not have enough buffers, it returns END_ERR_BLOCK. The stack will prepend the packet to the send queue and try again the next time a packet is queued for the interface, or after the driver calls the MUX transmit restart routine. If the driver has enough buffers, the driver frees the network stack DATA buffer. It then transmits the packet. The driver gives its transmission semaphore. UDP frees the network stack DATA buffers as soon as the message is sent. TCP, on the other hand, holds the DATA buffers for possible retransmission until it receives an acknowledgement from the peer TCP. 1.5 SOCKET BUFFERS -------------- Socket buffers are used to store messages and other information required by the protocols. As compiled, the maximum size for a socket receive or send buffer is 258,111 bytes. When a new UDP socket is created, its receive and send buffer are set to the value of the global variables udp_recvspace and udp_sendspace respectively. Similarly, TCP sockets buffers are set to the value of the global variables tcp_recvspace and tcp_sendspace. The values of the global variables are: -> tcp_sendspace tcp_sendspace = 0xf9e00: value = 8192 = 0x2000 -> tcp_recvspace tcp_recvspace = 0xf9e04: value = 8192 = 0x2000 -> udp_sendspace udp_sendspace = 0xfec74: value = 9216 = 0x2400 -> udp_recvspace udp_recvspace = 0xfec78: value = 41600 = 0xa280 Another value set when a socket is created, or changed when setsockopt is called with options SO_SNDBUF/SO_RCVBUF, is the maximum number of mbufs (in bytes) that a socket may hold. mbufs is the same as clusters. The limit is set to avoid having a single socket consume too many mbufs and is calculated as follows: max number of mbufs = minimum ( 5* maximum buffer size, 256*1024) (in bytes) For both UDP and TCP, space in the send or receive buffer is calculated using the formula below prior to sending or receiving: space = min (buffer maximum size - total bytes in buffer), (max number of mbufs - total mbuf bytes in buffer) To change the default buffer sizes for all sockets, change the value of the global variables from the shell or programmatically. Use setsockopt to change the value for an individual socket. Calling setsockopt to set the maximum size for buffer space just declares what the limit will be. No memory is actually allocated until a packet is sent or received. 1.5.1 UDP SOCKET BUFFERS ------------------ For sending, if the message size is larger than the socket send buffer, EMSGSIZE error is returned. For receiving, since UDP is connectionless, UDP stores the message source IP address and port in the receive buffer. Therefore, an additional 16 bytes per message are needed for the receive buffer. If there is no space, as determined by the space calculation above, received packet is dropped and the error is reported by udpstatShow as full sockets error. For example, for a 172 byte UDP message received over Ethernet: The 172 byte message is within a buffer of size 1520. The address and port information is stored in a separate 128 byte buffer. Each buffer requires a control data structure, an mblock (size = 32 bytes): # mbuf bytes = packet mbuf bytes + address information mbuf bytes # mbuf bytes = 1520 + 32 + 128 + 32 = 1712 bytes For a 2000 byte UDP message, since the maximum UDP message size over Ethernet is 1472 bytes (1500 - 28 byte header), 2 packets are received: # mbuf bytes = (1520 + 32) * 2 + 128 + 32 = 3264 bytes 1.5.2 TCP SOCKET BUFFERS5 ------------------ For TCP sockets, the socket receive/send buffer sizes determine the maximum send and receive window sizes for the connection. setsockopt must be called before the listen call on the server, and before the connect call on the client, since maximum window sizes are among the parameters exchanged when the connection is first established. Increasing the buffers later has no effect. Periodically TCP sends window updates informing the peer how much data it can receive. The advertised window size is the available room in the receive buffer. The window size cannot exceed the maximum set when the connection was established, but can decrease to 0 if the socket is not read, and the receive buffer becomes full. To improve TCP performance, Richard Stevens (UNIX Network Programming Volume 1, Second edition, Page 192), suggests that: 1. socket buffer sizes should be at least 3 times the MSS. 2. buffer sizes should be an even multiple of the MSS for the connection and 3. Buffer sizes should be at least as large as the product of the bandwidth (bytes/sec) and the round trip time to keep the "pipe" full and get the expected performance. Round trip times (RTT) are obtained by using ping. For example, if the bandwidth is 1000 bytes per second, and the average round trip time is 2 seconds, buffer sizes would have to be at least 2000 bytes. For Ethernet, and an average RTT = 50 msecs: 10000000/8 bytes * 0.050 seconds = 62500 bytes which is approximately 42.8 * MSS. Following 1 and 2 above, socket buffer size should be 64240 (44*1460). On an Ethernet which is not overloaded, RTT is much faster than 50 msecs. The window size field in the TCP header is an unsigned short (maximum = 65535 bytes). The Tornado 2.0 stack implements RFC1323, the window scale option (See netLib.h), which is designed to get around the 65535 limit for high speed connections (ATM) or slow media (satellite) which require large window sizes. 2. NETWORK MEMORY RESOURCES SUMMARY -------------------------------- There are 3 memory pools consisting of clusters and their control structures, which are allocated at initialization time. These pools cannot be increased later: NETWORK STACK SYSTEM POOL NETWORK STACK DATA POOL NETWORK DRIVER POOLS There are 7 types of queues (linked lists) that hold data waiting to be processed. The data is contained in clusters previously allocated from the DATA or the DRIVER pools. Adding a cluster to a queue is just chaining the cluster to the queue's linked list. The stack sets default values for the maximum length of some queues, which can be changed after initialization: RECEIVE QUEUES: 1. IP PROTOCOL RECEIVE QUEUE 2. IP FRAGMENT REASSEMBLY QUEUE 3. ARP RECEIVE QUEUE 4. TCP REASSEMBLY QUEUE 5. SOCKET RECEIVE QUEUES SEND QUEUES: 1. SOCKET SEND QUEUES 2. IP NETWORK INTERFACE SEND QUEUES 2.1 MEMORY POOLS ------------ 1. NETWORK STACK SYSTEM POOL ------------------------- This pool is used to store data structures such as sockets and routes. The default values are: ------ CLUSTERS REQUIRED ----- CLUSTER SIZE TOTAL PER TCP PER UDP PER ROUTE BYTES SOCKET SOCKET ARP MAX ** NUM_SYS_64 40 0 0 2 3 NUM_SYS_128 40 1 1 0 0 NUM_SYS_256 40 1 0 1 2 NUM_SYS_512 20 1 1 0 0 **: Route requirements vary. For example, the first time a network route is added with a new gateway, (Example; routeAdd("147.11.48.0", "147.11.41.254") 3 64 and 2 256 clusters (3,0,2,0) are allocated. If another route is added using the same gateway, (Example: routeAdd("147.20.30.40", "147.11.41.254") (1,0,1,0) clusters are allocated. If an ARP entry is added with arpAdd, (2,0,1,0) clusters are allocated. routeAdd, arpAdd themselves open a raw socket (0,1,0,1) temporarily to pass information to the stack. 2. NETWORK STACK DATA POOL ----------------------- This pool is used to copy data passed by applications (sendto, write...) from application level buffers to network stack clusters. The clusters are passed to the protocols and ultimately to the network drivers. Protocols (ICMP, TCP, RIP, ...) send messages too. The stack allocates a cluster big enough to hold the message plus some extra space for protocol headers. Smaller clusters would be allocated and chained if the right size is not available. CLUSTER SIZE TOTAL BYTES NUM_64 100 NUM_128 100 NUM_256 40 NUM_512 40 NUM_1024 25 NUM_2048 25 For example, if an application calls "write" to send one 1460 byte message for a TCP socket, 1 2048 byte cluster would be allocated from the pool. TCP would hold this cluster in the socket send queue until the recipient notifies that the data was received. Clusters used by UDP are freed as soon as the driver copies the data to its own clusters. 3. NETWORK DRIVER POOLS -------------------- Tornado 2.0 offers 2 driver versions for each supported network card: BSD 4.4 and END. Either type can be used, but END drivers are required for multicasting and for system mode debugging over Ethernet. END drivers are configured by default in Tornado 2.0. muxShow displays configured END drivers. ifShow displays both END and BSD4.4 drivers. Drivers set up their own pools of non-cacheable memory and allocate buffers from it which are used for sending and receiving. Buffers can be clusters or other data structure types called frames or descriptors. Drivers copy data passed by the stack to their own buffers for transmission. Devices receive packets in the driver's buffers. END and BSD4.4 drivers (some, not all) lend buffers containing received packets to the network stack to avoid copying. Buffers have to be big enough to receive or transmit the maximum packet size allowed by the link layer. For Ethernet 10 Base T, the maximum size is 1518 bytes. Two extra bytes are required to align the IP header on a 4 byte boundary for incoming data. The default size of end driver cluster pools vary, but the number could be as few as 40 1520 clusters total. In this article, when it comes to drivers, clusters are a generic name for buffers: descriptors, frames, or actual cluster data structures. Drivers differ: END DRIVERS ----------- Some END drivers have frames for transmission and clusters for reception. Some END drivers have clusters for both. END network drivers lend all their clusters. BSD 4.4 DRIVERS --------------- BSD drivers have frames. BSD drivers borrow mblocks and cluster blocks from the network stack DATA pool when they lend their frames to the stack. BSD drivers may borrow clusters as well if they have lent the configured maximum number of LOAN frames to the stack. Some BSD drivers always borrow clusters. 2.2 RECEIVE QUEUES -------------- 1. IP PROTOCOL RECEIVE QUEUE ------------------------- This queue stores DRIVER or DATA cluster containing incoming packets passed to IP from all configured interfaces. IP processes the packets and passes them to the protocols (TCP, UDP...). Protocols process the packets and queue data in the destination socket receive buffers. If packets come in bursts and the queue is full, the excess is dropped. The default value for this queue is 50 packets. 2. IP FRAGMENT REASSEMBLY QUEUE ---------------------------- This queue stores DATA or DRIVER clusters that contain fragments of a larger IP datagram waiting to be reassembled. The maximum size for an IP packet is 65535 bytes. Packets exceeding the maximum transmission unit (MTU) for a link layer protocol must be fragmented. For example, for Ethernet 10BaseT, the maximum packet size is 1518 bytes. Subtracting the Ethernet header and checksum (18 bytes), the IP header (20 bytes), and the UDP header (8 bytes) leaves 1472 bytes. A UDP message consisting of 1473 bytes arrives in 2 separate IP packets. There is a default timeout to drop the fragments if they cannot be reassembled in time. There is no explicit queue size. IP fragments are dropped also when a pool is out of clusters and the system drains the protocols. 3. ARP RECEIVE QUEUE ----------------- This queue holds DATA or DRIVER buffers containing ARP packets which are used for hardware address resolution. If packets come in bursts and the queue is full, the excess is dropped. The default value for this queue is 50 packets. 4. TCP REASSEMBLY QUEUE -------------------- This queue contains DATA or DRIVER clusters containing TCP segments that arrive out of order. For example, the maximum TCP message size over Ethernet 10BaseT is 1460 bytes (assuming no TCP options). TCP avoids IP fragmentation, so each segment is 1460 bytes maximum. The socket receive/send buffer sizes determine the maximum send and receive window sizes for the connection. If socket receive buffers are large, the peer TCP will be able to send a large number of segments within the window. TCP has to pass data to the application in the order in which it was sent. When packets arrive out of order, or are lost, TCP stores these segments in a reassembly queue until the missing segment arrives. There is no limit on the queue size nor is there a timeout period to drop the fragments. 5. SOCKET RECEIVE QUEUES --------------------- Each socket's receive buffer stores the DATA or DRIVER clusters until the application reads the socket, at which point, the clusters would be freed. This is important: Driver or DATA clusters are freed IF AND WHEN applications read the sockets. If they don't, data flow may stop eventually. The default value is 8192 bytes for TCP sockets and 41600 bytes for UDP sockets. 2.3 SEND QUEUES ----------- 1. SOCKET SEND QUEUES ------------------ Each TCP socket's send buffer holds DATA clusters containing packets waiting to be acknowledged. The default value for TCP sockets send queues is 8192 bytes. UDP sockets send queues do not hold any data, but the stack checks that the message size does not exceed the send socket buffer size (9216 bytes). A DATA cluster is freed as soon as the driver copies the data to its own clusters prior to transmission. 2. IP NETWORK INTERFACE SEND QUEUES -------------------------------- IP has a send queue for each configured interface which stores the outgoing packets, DATA buffers, that IP needs to pass to the driver for transmission. If a driver is busy transmitting, and this queue is full, the packet is dropped and ENOBUFS error is returned to the application. The default value is 50 packets. FIGURE 1: MEMORY POOLS AND QUEUES DEFAULT VALUES -------------------------------------- RAW SOCK SEND BUFF SYSTEM POOL ****************** ****************** * DATA *-----------> * SOCKETS * * CLUSTERS * | * ROUTES * * 8192 bytes * | * PCBs, TCPCB * ****************** | ****************** IP SEND QUEUE UDP SOCK SEND BUFF FOR "FEI" END DRIVER ****************** ****************** * DATA CLUSTERS *--->* DATA CLUSTERS *----------> * 9216 bytes * * 50 PACKETS * | ****************** ****************** | | IP SEND QUEUE | TCP SOCK SEND BUFF FOR "ELT" BSD 4.4 DRIVER | ****************** ****************** | * DATA CLUSTERS *--->* DATA CLUSTERS *----------|-------- * 8192 bytes * * 50 PACKETS * | | ****************** ****************** | | | | ******************** | | * ARP RECEIVE QUEUE*<-<--------|-------|< **** 50 PACKETS *** | | || | V || UDP SOCK RCV BUFF IP FRAGMENT QUEUE | FEI END DRIVER || ****************** *************** | ************** || * DRIVER, DATA * * DRIVER, DATA* |<*END DRIVER * || * 41600 bytes *<---*CLUSTERS/FRAM* * * || ****************** *************** *RECV: * || | ^ --*CLUSTERS: 64* || <----------------- | | *CLBLKs: 64 * || | | | *MBLK: 128 * || RAW SOCK RECV BUFF IP RECEIVE QUEUE | *TX FRAMES:32* || ****************** *************** | ************** || * DRIVER, DATA * *DRIVER,DATA *<--| || *CLUSTERS/FRAMES *<---*CLUSTERS/FRAM* || * 8192 bytes * * 50 PACKETS *<-- ELT BSD4.4 DRIVER|| ****************** *************** | ************** || | | | *BSD44 DRIVER* || <--------------- | | | * * || V V | *FRAME * || TCP SOCK RECV BUFF TCP REASSEMBLY Q. |-*DESCRIPTORS:*<- | ****************** *************** * Rx: 20 * | * DRIVER, DATA *<---* DRIVER,DATA * * Tx: 1 *---> * 8192 bytes * *CLUSTERS/FRAM* * Loan: 16 * ****************** *************** ************** 3. ESTIMATING NETWORK RESOURCES REQUIREMENTS Network stack memory exhaustion problem is not unique to vxWorks. TCP and UDP do not prioritize sockets. There are no facilities to identify sockets which could be considered less important and which could be targeted for selective drops, or even closing, when resources are limited. Packets will be dropped during peak times if there are not enough driver buffers to lend to the stack to support the sum of the UDP and TCP socket receive buffers for the expected number of concurrent socket connections. This sum could be a very large number. Even if there is a single socket, there could be still be high demand. For example, applications expecting to receive bursts of 100 UDP packets even if they carry only one byte of data will still be consuming 100 driver buffers. Similarly, applications will hang and resume after a while if there are not enough SYSTEM clusters to open sockets or create routes for applications that generate a high demand for them. 3.1 SOCKETS AND ROUTES ------------------ If the stack does not have enough SYSTEM clusters, applications like FTP and Web servers can hang for a while when there is not enough memory to open sockets. iosFdShow displays open sockets. inetstatShow displays socket status. When TCP on the target closes a socket, the socket is removed from the file descriptor table, but the memory used by that socket is not de-allocated until TIME_WAIT time elapses (about 1 minute). The TCP protocol needs this time to minimize the chance that duplicate packets belonging to an earlier connection will be delivered to a new TCP connection. inetstatShow displays the socket states. If a routing protocol like RIP is configured, or if, for example, the target is not given an explicit subnet mask, when necessary, the route table can get very large since the target is really listening to ARP broadcasts for all subnets. A message may appear on the on the console: arpresolve: can't allocate llinfo netStackSysPool can be used to tune the SYSTEM pool. See 5.1.1 NETWORK STACK MEMORY POOLS for details. 3.2 SENDING AND RECEIVING --------------------- Some conditions that would generate a high demand for driver buffers: 3.2.1 LARGE SOCKET RECEIVE BUFFERS ---------------------------- A UDP or TCP message consisting of a single byte of user data will be received in a whole driver buffer. Large socket receive buffers can contain many driver buffers. UDP messages larger than the MTU (For Ethernet = 1472 bytes) are fragmented by IP. IP on the receiving side will have to reassemble the fragments. Each fragment will be received on a separate driver or DATA buffer. TCP does not send packets larger than the MTU. However, if socket receive buffers are large, the peer TCP will be able to send a large number of segments within the window. Since TCP has to pass data to the application in the order in which it was sent, when packets arrive out of order, or are lost, TCP stores these segments in a reassembly queue until the missing segment arrives. TCP has no provision to drop packets in the reassembly queue, so driver buffers can sit there for a long time or indefinitely. TCP on the receiver side sends ACKS for the missing segment every time it gets an out of sequence packet. TCP on the sender side retransmits the missing segment, or even starts over again retransmitting the first unacknowledged segment if it does not get any response. TCP on the sender side retries a number of times but eventually it gives up. This situation could happen if all of the driver buffers are held up in the TCP reassembly queue, and there are no available driver buffers for transmission or to receive the missing segment. 3.2.2 SMALL TCP PACKETS ----------------- TCP normally does not send small packets, but many users enable TCP_NODELAY to force TCP to do so. Enabling TCP_NODELAY on the sender side (See entry for setsockopt in vxWorks Reference Manual) prevents TCP from waiting to collect small packets until an MSS sized packet can be sent. For TCP over Ethernet: MSS = 1460 bytes. TCP on the sender side still is subject to the constraint that it must not send more than the peer's advertised window. It should be noted that enabling TCP_NODELAY on the receiver side will not have a direct effect on how fast the receiver side acknowledges data. When an application reads a socket, the TCP on the receiver's side calculates whether it should tell the peer that is OK to send more. It does so by sending ACKS. This ACK is not immediate. The following calculation takes place: Compare available window to amount of window known to peer. If the difference is at least two maximum size segments (For TCP = 1460 bytes) or at least 50% of the maximum possible window, then send a window update to peer. Otherwise, the ACK gets delayed until the FAST TIMER goes off about 200 msecs later. TCP works this way following RFCs to avoid contributing to network congestion. If the application message size is small (less than 1460 bytes), reducing the socket receive buffer size would make ACKS come faster. If the application message size is larger than MSS (1460 bytes), you should test various message sizes (1 MSS, 2 MSS, ...) to find a level that gives good performance. Surprisingly, large buffers are not always better. TCP ACK-every-other-segment feature improves performance since the 200 msec delay would not occur as often. It should be noted that TCP was developed to maximize reliability, not performance. TCP wants to be efficient also, so it tries not to add unnecessary packets on the network thereby increasing congestion. The THEORETICAL maximum throughput, under ideal conditions, for a single TCP socket connection on 10 BASET, is 1,183,667 bytes/second. In reality, there are delays. Achieving 90% of this maximum is considered very good. TCP can be tuned to give a reasonable performance, but the reader should understand its limitations by consulting TCP/IP reference books. 3.2.3 SENDING ------- Sending a lot of packets requires the following steps: STEP 1: INCREASE THE NETWORK STACK DATA POOL ------------------------------------ netStackDataPoolShow displays the status of the DATA pool. If there are no free buffers or if the statistics for the number of times failed to find space is greater than 0, increase the appropriate cluster pools based on the application message size. Detailed instructions can be found in Section 4.1.1 NETWORK STACK MEMORY POOLS. STEP 2: INCREASE THE NETWORK DRIVER BUFFER POOL --------------------------------------- See Section 4.1.2 NETWORK DRIVER MEMORY POOLS. STEP 3: INCREASE THE NETWORK INTERFACE SEND QUEUE ----------------------------------------- See Section 4.3 IP SEND QUEUES 3.2.4 RECEIVING --------- Receiving lots of packets for many socket connections requires the following steps: EITHER STEP 1: INCREASE THE NETWORK DRIVER BUFFER POOL --------------------------------------- If an application will be using the network intensively, performance will be enhanced if the network interface driver is configured with a sufficient number of buffers so that packets are not dropped unnecessarily. See Section 4.1.2 NETWORK DRIVER MEMORY POOLS. A utility like endPoolShow, found in attached utilities.c can help the user find the optimal number for END drivers. For BSD4.4 drivers, there is no specific function that can show how many times the driver was out of its own buffers and had to borrow from the network stack DATA pool. However, netStackDataPoolShow can be helpful. Both endPoolShow and netStackDataPoolShow would display if buffers were exhausted at any time. STEP 2: INCREASE IP INPUT QUEUE ----------------------- See Section 4.2 PROTOCOL RECEIVE QUEUES. OR STEP 1: DECREASE SOCKET RECEIVE BUFFERS STEP 2: DECREASE THE NUMBER OF CONCURRENT SOCKET CONNECTIONS STEP 3: ADJUST APPLICATION MESSAGE SIZE AS NECESSARY TO BE CLOSER TO THE MTU TO IMPROVE EFFICIENCY. 3.2.5 APPLICATION ERRORS ------------------ The network driver buffer mechanism can be disrupted if tNetTask cannot process packets, or applications errors have deadlock conditions which prevent the sockets from being read. The priority of applications that need the network should be lower (>50) than tNetTask's (= 50). User's applications usually run at priority 100. Call i from the shell to display priorities for running tasks. tNetTask takes splSemId which is SEM_INVERSION_SAFE. Do not create and then take SEM_INVERSION_SAFE semaphores before making a socket call or your task could be promoted to run at tNetTask level. See priority inversion discussion in the vxWorks Programmer's Guide (Section 2.4.3). If priority inversion occurred, the command i would show it as 0+I: ->i tNetTask netTask 807c9750 0+I PEND Application may have deadlock conditions which prevent them from reading sockets. If inetstatShow (or equivalent in other systems) displays data backed up on the send side and on the receive side of the peer, most likely there is a deadlock situation within the client/server application code. TCP is a stream of bytes. Loops are required to read or write, since TCP can return partial reads and partial writes. Running both server and client in the target by sending to 127.0.0.l or to the target's own IP address is a good way to detect this kind of problem. 4. CONFIGURATION See WindSurf->Technical Articles II: How to build images with network support in Tornado 2.0 which contains instructions for building a kernel with network show libraries and a target shell whose output goes to serial console. The target shell can be used to debug network problems when network problems occur. The attached utilities.c file contain code for some utilities which are not part of the API: endPoolShow protocolQValuesShow changeProtocolInputQMax ifQValuesShow changeIfSendQMax 4.1 POOL SHOW FUNCTIONS ------------------- Call pool show functions to find the level where resources are adequate: netStackSysPoolShow netStackDataPoolShow endPoolShow (found in utilities file). Pool show functions display: number of times failed to find space: If not 0, pool is too small. number of times waited for space: 0 number of times drained protocols for space: IP fragments are dropped when drivers or the stack are out of clusters. Usage: the cumulative number of times clusters of each size have been allocated. Clusters: Total clusters: Free + clusters in queues. Mbufs and their use. Increase clusters, cluster blocks and mblocks as necessary until there are no shortages. 4.1.1 NETWORK STACK MEMORY POOLS -------------------------- Increase the default values as follows: 1. INCREASE NUM_FILES The socket library uses NUM_FILES to configure the size of the socket file descriptor table. The default value for NUM_FILES is 50 sockets. 2. TCP SOCKETS For every TCP socket, the system allocates 3 clusters (1 128 byte, 1 256 byte, and 1 512 byte clusters) used for the generic protocol control block (inpcb), TCP protocol control block, and the socket structure respectively. 3. UDP SOCKETS For every UDP socket, the system allocates 2 clusters (1 128 byte, 1 512 byte) used for the inpcb and socket structures respectively. 4. ROUTES For every link layer route (ARP), 3 clusters (2 64 byte, and 1 256 byte) clusters are allocated. BOOTROM IMAGE MACROS -------------------- The macros with MIN suffix (i.e. NUM_SYS_64_MIN... and NUM_64_MIN...) are used for BOOTROM images which usually need to be small. The values for the MIN macros are usually adequate to download the runtime image and do not normally need to be changed. RUNTIME IMAGE MACROS -------------------- The network stack SYSTEM POOL macros are NUM_SYS_64, NUM_SYS_128, ...etc. The network stack DATA pool MACROS are NUM_64, NUM_128...etc. TORNADO 101 BUILD METHOD ------------------------ a. Increase NUM_FILES in target/config/all/configAll.h. b. Increase the default network stack memory sizes in netBufLib.h. PROJECT FACILITY BUILD METHOD ------------------------------ a. Increase NUM_FILES Right Click on network components-> networking protocols-> BSD SOCKET to bring up the properties window. Click on Params window to change NUM_FILES. b. Change values for the runtime image in the Params window for network buffer initialization component: network components-> basic network initialization components-> network buffer initialization-> 4.1.2 NETWORK DRIVER MEMORY POOLS --------------------------- The size of a driver's pool is determined by the parameters passed to the driver, during initialization time (See the VxWorks Reference entry for the driver, since each one is different. Drivers have a ring of buffers for transmission and reception. For some drivers, the ring size may be limited by the hardware to a maximum. BSD 4.4 DRIVERS --------------- The LOAN parameter allows having extra buffers for loaning so the ring always has enough buffers. END DRIVERS ----------- END drivers lend all their clusters. If END driver's cluster pools are depleted, reception will stop until the clusters are freed by the network stack or the applications IF and WHEN socket buffers are read. Transmission may stop as well if the driver uses clusters for transmission. For end drivers, target/config/bsp/configNet.h is the place to change the default parameters in the driver's load string. For BSD4.4 drivers, the parameters are passed to the attach function. See basicNetConfiguration document for more details. The number of buffers is typically the number of transmit descriptors plus the number of receive descriptors plus the number of loan buffers. If the loan parameter is not part of the initialization string, then changing the LOAN parameter in the driver's header file will have no effect unless you have source for the driver and can recompile the object. Some drivers just make loan buffers equal to the number of receive or send descriptors. Example: PPC target uses dc END driver. In configNet.h /* :::::::: */ #ifdef ULTRA # define DEC_LOAD_STRING "0x81020000:0x80000000:0x09:0x09:-1:-1:-1:0:0x0" #else # define DEC_LOAD_STRING "0x81020000:0x80000000:0x0a:0x0a:-1:-1:-1:0:0x0" #endif >From dec21x40End manual entry: The default -1 means use 32 buffers So I changed the default numbers to 128 each: #define DEC_LOAD_STRING "0x81020000:0x80000000:0x0a:0x0a:128:128:-1:0:0x0" NUM_LOAN is declared in dec21x40End.h as 16. The cluster pool available to the driver will be: numRds + numTds + NUM_LOAN = 128 + 128 + 16 = 272 The receive and transmit numbers are specified separately, but the clusters in the pool are in fact used for both sending and receiving, and all buffers are loaned. -> endPoolShow("dc",0) type number --------- ------ FREE : 510 DATA : 2 HEADER : 0 SOCKET : 0 PCB : 0 RTABLE : 0 HTABLE : 0 ATABLE : 0 SONAME : 0 ZOMBIE : 0 SOOPTS : 0 FTABLE : 0 RIGHTS : 0 IFADDR : 0 CONTROL : 0 OOBDATA : 0 IPMOPTS : 0 IPMADDR : 0 IFMADDR : 0 MRTABLE : 0 TOTAL : 512 number of mbufs: 512 number of times failed to find space: 0 number of times waited for space: 0 number of times drained protocols for space: 0 __________________ CLUSTER POOL TABLE _________________________________________________________ size clusters free usage 1520 272 144 450 --------------------------------------------------------- 4.2 PROTOCOL RECEIVE QUEUES ----------------------- There are 2 global variables that control the queue length for the ARP queue and the IP queue: arpintrq, and ipintrq. The queues hold the unprocessed ARP and IP packets. Packets are dropped if the queue length is at the maximum level. The default value for each is 50. No error statistics are increased. If building the Tornado 101 way, change the default value for ipintrq.ifq_maxlen by changing the value for IP_QLEN_DFLT in target/h/netLib.h. If building using the Tornado 2.0 GUI: network components networking protocols core TCP/IP components Right Click on IPv4-> Properties-> Parameters The default values and drop statistics are displayed by protocolQValuesShow. Change the default value for ARP or IP at run time calling changeProtocolInputQMax. Neither is an API function. See utilities.c. 4.3 IP SEND QUEUES -------------- When an END, BSD4.4 driver, or the loopback driver is attached to IP, one of the parameters set by IP is the size of the maximum send queue length. On sending, ipOutput for END drivers or ether_output for BSD4.4 drivers attempts to add a packet to interface's send queue, one of the fields of the ifnet structure that each driver creates (See target/h/net/ if.h). Default queue size is 50 packets. Packet is dropped if queue is full, the ifq_drops statistics increases and the socket call returns ENOBUFS. Call ifQValuesShow to display the current values and drop statistics. Use changeIfSendQMax to increase the default maximum, if an application will be sending more than 50 packets in rapid succession. ifQValuesShow and changeIfSendQMax utilities can be found in the code section. 5. REFERENCES ---------- 5.1 VXWORKS DOCUMENTATION --------------------- The Tornado 2.0 Documentation Collection has the most up-to-date manual versions. Manuals are available from the Help Menu in Tornado and also from WindSurf. In addition to manuals, WindSurf has Patches, Topic keyword search engine, and Networking Demo Code among other things. www.windriver.com -> Support -> WindSurf Access to WindSurf requires a password. Register at: http://www.windriver.com/corporate/support/private/validate.html Network documentation is distributed in the following manuals: TORNADO USER'S GUIDE Chapter 1: Overview. Description of Tornado IDE. Chapter 2: Setup and Startup. Host and target setup. Chapter 4: Projects. Project Facility Build Method. Chapter 5: Target Server. Tools/target interface. Chapter 6: Target Shell. Host-based shell: WindSh. Appendix F: FTP Server. Host system FTP server setup. TORNADO RELEASE NOTES Section 7: Host Development Environment Enhancements Section 9.1: Enhanced Network Stack Section 11: Backwards Compatibility Issues VXWORKS PROGRAMMER'S GUIDE Chapter 8: Configuration and Build. Tornado 101 Build Method. Chapter 9: Target Shell. VXWORKS NETWORK PROGRAMMER'S GUIDE Chapter 13: Booting over the Network. Chapter 14: Upgrading 4.3 BSD Network Drivers. Network drivers developers should get the Network Protocol ToolKit product from their sales representative. NPT is the most up to date manual on this subject. The BETA User's Guide is available from WindSurf which is an updated version of the SENS for Tornado manual. NPT is free and comes with some network libraries source. There is a patch also in source on WindSurf->Download->Patches for SPR# 30894. 5.2 TCP/IP REFERENCE BOOKS ---------------------- TCP/IP Illustrated Volume 1, by Richard Stevens ----------------------------------------------- This book is an overview of TCP/IP protocols. TCP/IP Illustrated Volume 2, by Richard Stevens ----------------------------------------------- The Tornado 2.0 network stack is based on BSD4.4 Lite. This book explains BSD4.4 code, function by function, line by line. UNIX Network Programming: Networking APIs: Sockets and XTI Volume 1, Second Edition, by Richard Stevens. ---------------------------------------------------------- This book summarizes the TCP/IP Illustrated Volume 2 information and provides lots of client/server code examples which illustrate common programming errors. Effective TCP/IP Programming 44 Tips to Improve Your TCP/IP Programming by Jon C. Snader ----------------------------------------------------------- High-Speed Networks TCP/IP and ATM Design Principles by William Stallings. ---------------------------------------------------- This book discusses TCP/IP performance and memory resource issues. 5.3 WindSurf->Technical Solutions II -------------------------------- How does Tornado 2.0 (SENS) support END and BSD44 drivers at the same time How do END driver pass link layer information to the MUX Network Driver Porting Issues ------_extPart_000_01C0FFDE.1E130740 Content-Type: text/plain; nameroubleshooting.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filenameroubleshooting.txt" TROUBLESHOOTING SIMPLE NETWORK PROBLEMS IN TORNADO 2.0 ------------------------------------------------------ If you are new to Tornado 2.0, or you are booting a target for the first time, or if the target booted successfully, but does not respond to pings, please read basicNetConfiguration article. basicNetConfiguration contains instructions on how to configure a kernel with a target shell and network debugging libraries. It contains information to avoid errors like: rpccore backend client RPC: Timed out Network interface unknown muxDevLoad failed for device entry wdbConfig: error configuring WDB communication interface Error loading file: errno = 0xd0003 If target responded to pings initially, and after running your socket application (or a library that uses the network such as the FTP server), applications sputter or hang for a while and then resume, please read netPerformance article. It gives an overview of vxWorks network internals and configuration issues that impact performance. This article assumes that a target has booted successfully and responds to pings initially. You are either running demo code from WindSuf->Demo Code->Networking or your own application. At some point, the target does not respond to pings anymore, or network performance is slow. At this point from the target shell, verify that tNetTask