Contents - Fakernet =================== 1. Purpose 2. License 3. Overall design 4. VHDL interface 4a. PHY interfacing 4b. UART SLIP interface 5. UDP protocol 5a. C helper library interface 5b. Control program 6. Performance 7. Compilation and testing 8. Testing on hardware 8a. Digilent Arty A7-35 9. Acknowledgements (referencing) 10. Contact 1. Purpose ========== Fakernet is a minimalistic VHDL implementation to allow an FPGA with a physical Ethernet interface (PHY chip) to send data over a TCP/IP connection, as well as allow control access over a UDP register interface. A typical use case is for data acquisition purposes. No special OS/kernel drivers are needed for the connected computer. For FPGA boards with no PHY interface, a serial interface can also be used with SLIP. 2. License ========== Fakernet is free software; distributed under the 3-clause BSD license. For details, see the accompanying LICENSE file. Some example board support files have other licences. They are not needed for Fakernet as such. Details are in the respective sub-folders. 3. Overall design ================= The Fakernet name comes from the minimalist-pragmatic approach of only implementing the minimum features that are needed to interoperate with common OS stacks for UDP and TCP/IP networking, in controlled environments. Packets are limited to have an even number of octets (bytes), and bandwidth bottle-necks should be avoided. The TCP data connection is established from the controlling PC side. The TCP stream can only transmitt data from the Fakernet node. Resetting the TCP connection is handled via the UDP control interface. Thus no connection timeouts need to be handled in the Fakernet VHDL circuit. Fakernet manages an internal memory buffer for data to be sent over TCP. Data can be written (and overwritten) in arbitrary order at offsets relative to the current end of the buffer. Once the newly written part has been committed, it will be transmitted. The handling is such that the part of the buffer that is under transmission (i.e. has not yet been acknowledged by the receiving end) cannot be modified, even by mistake. (Committing data beyond the overflow protection is a user bug and causes the buffer to not accept any further data, requiring a TCP reset.) The control interface over UDP is realised as a register access interface. Each request packet contains a number of addresses that shall be accessed, together with a read/write marker, and the data value. For writes, the data value provided is used, and for reads, the data value is returned. In order for the response packets to always have the same length as the requests, the data values are included as place-holder dummy values in the respective other direction. By the use of a sequence counter, the UDP register access interface is made reliable, i.e. such that for a sequence of several access packets, each request is only performed once. This even if some UDP packets are lost, either on the way to Fakernet, or from it. The necessary timeouts and repeats are all handled on the controlling PC side. (The general design is that except for TCP data packets, Fakernet only generate (one) packet in response to each successfully parsed incoming packet.) Fakernet responds to ARP requests, thus providing the mapping between MAC and IP address as usual. Is is necessary to provide the VHDL circuit with these identifiers. Fakernet also responds to ICMP echo (ping) packets. For every fourth ARP packet, Fakernet generates a RARP request to get a dynamic IP from its MAC address. (This is generated instead of the ARP response.) Received RARP responses that provide an IP number for the MAC address are used to configure a second IP address. Fakernet answers to packets for both the IP address provided to the VHDL circuit and this second dynamic address. Thus the circuit-provided address can be e.g. 0.0.0.0. To use this, some computer on the network segment needs to respond to RARP requests, e.g. by running 'rarpd'. 4. VHDL interface ================= The Fakernet VHDL circuit presents itself as a single entity to the user, with some configuration using a generic map. The other entities are solely for internal use. All identifiers that are exposed through work.* have fnet_ prefixes, to avoid clashing with other user code. The interface: entity fakernet_module is generic (data_bufsize_addrbits : natural; test_regs : boolean := true; lcl_data_gen : boolean := true; limit_payload_window : boolean := true; debug_regs : boolean := true; debug_counters : boolean := true); port (clk : in std_logic; cfg_macaddr : in std_logic_vector (47 downto 0); cfg_ipaddr : in std_logic_vector (31 downto 0); cfg_fixed_ip : in std_logic := '1'; cfg_dyn_ip : in std_logic := '0'; cfg_gen_rarp : in std_logic := '0'; cfg_gen_bootp : in std_logic := '0'; cfg_gen_dhcp : in std_logic := '0'; dyn_ip : out std_logic_vector (31 downto 0); dyn_ip_set : out std_logic; in_word : in std_logic_vector(15 downto 0); in_got_word : in std_logic; in_new_packet : in std_logic; out_word : out std_logic_vector(15 downto 0); out_ena : out std_logic; out_payload : out std_logic; out_taken : in std_logic; mdc_out : out std_logic; mdc_ena : out std_logic; mdio_in : in std_logic; mdio_out : out std_logic; mdio_ena : out std_logic; reg_addr : out std_logic_vector(24 downto 0); reg_data_wr : out std_logic_vector(31 downto 0); reg_data_rd : in std_logic_vector(31 downto 0); reg_write : out std_logic; reg_read : out std_logic; reg_done : in std_logic; reg_cnt : out std_logic_vector(3 downto 0); data_word : in std_logic_vector(31 downto 0); data_offset : in std_logic_vector; data_write : in std_logic; data_commit_len: in std_logic_vector; data_commit : in std_logic; data_free : out std_logic; tcp_reset : out std_logic; slow_clock_tick : in std_logic; timeout_tick : in std_logic; ); end; clk The clock network signal that drives Fakernet. cfg_* Address configuration. macaddr The MAC address to be used. ipaddr The IP address to be used. fixed_ip Accept and respond on IP addressed given above. dyn_ip Accept and respond with IP address retrieved with RARP. cfg_gen_rarp Generate RARP requests and handle RARP responses. cfg_gen_bootp Generate BOOTP requests and handle BOOTP responses. cfg_gen_dhcp Generate DHCP requests and handle DHCP responses. (The three above to get dynamic IP from MAC). dyn_* ip Dynamic obtained IP address. ip_set Dynamic IP address is set (/ in use). in_* Input from the PHY interface. word 16-bit input word. got_word Set to '1' each cycle a new word is delivered. new_packet Set to '1' to mark the start of a new packet. To be given before the first word, i.e. at the start of frame delimiter (SFD). out_* Output to the PHY interface. word 16-bit output word. ena The data word should be transmitted, i.e. is either preamble or data, but not inter-packet gap. payload Marks that the data word is payload (not preamble). taken Set to '1' to mark that the data word is consumed by the PHY interface. A new data word will be available next cycle. mdc_/mdio_* MDIO interface. out Output signal. end Enable signal (if 0, then tri-state). in Input signal. Note: MDIO interface is only used for debugging so far. It has not been needed to set used boards up. reg_* Register access interface. addr 25-bit address. data_wr Data value for write access. data_rd Return data value for read access. write Access is a write access. read Access is a read access. done Return value, set to '1' when access has been performed. This must be set within ~10 cycles (16 cycles as seen by the Fakernet register access control, minus pipeline stages). cnt A counter giving the number of cycles since the access was requested. (Can be useful for pipelines.) data_* Buffer interface for TCP output data. word 32-bit data word to write to the buffer. offset Location of the data to write. write Set to '1' to write a word. commit_len Amount of data to mark as ready for transmission. Further writes will write data relative to the end of this area. commit Set to '1' to commit. free Flag indicating that enough buffer memory is free, such that a new commit group can be started and written without overflowing the buffer. tcp_reset The TCP connection has been reset, which includes the buffer being cleared. slow_clock_tick This signal should be delivered ~2-20 times per max length packet send period. I.e. period of ~ 500-5000 ns @ 1 Gbps. Any convenient power of two will suffice. The signal is used to drive the TCP round-trip time (RTT) estimator and the re-transmit timer. timeout_tick After two timeout ticks, unused UDP connections can be reset. Suggested to be on the order of a second. Any convenient power of two will suffice. The generic map give control over some configurable items. Unless severe resource constraints suggest otherwise, it is recommended to let the ancillary functionality be activated for debugging purposes. data_bufsize_addrbits Number of bits of address in the TCP data buffer. This determines the size of the buffer. test_regs Internal test registers, which can be used to control both a dummy data generator, and artificially limit the TCP packet payload size, and TCP window. lcl_data_gen Internal data generator. limit_payload_window Artificially limit the TCP packet payload size, and TCP window. debug_regs Debug registers, giving information about the internal state of Fakernet (mostly TCP parameters). debug_counters Counters giving information about the internal behaviour of Fakernet (mainly about the incoming packet parsing, and sent packets). 4a. PHY interface ================= Circuits to interface with the PHY chip is outside the scope of Fakernet, since they depends on the actual PHY interface in use. When available, such interface modules are placed in the sub-folder extra_vhdl/. 4a. UART SLIP interface ======================= Circuits to interface using a 3-wire serial UART interface (SLIP) are also available. 5. UDP protocol =============== The UDP protocol consists of request and response packets with the same layout. Some items are only used in one direction. The length of the packets are given in the UDP header as such, and thus not repeated. The structure is described in client/fakernet.h as fakernet_reg_access, first static header items: uint16_t status_udp_channels uint16_t status_tcp uint16_t sequence_request uint16_t sequence_response status_udp_channels Bit-mask of which UDP control channels are in use and active. Bits in the he low byte indicate all used channels, while bits in the high byte mark the recently active channels (which cannot be reset). status_tcp Gives the status of the TCP connection. This generally need not be inspected, except for debugging. sequence_request Sequence number of the request. Flags to request that the sequence number is reset, to establish an access channel. sequence_response Flags to give responses to the requests above, as well as returning the access sequence number. This header is followed by 0 or more register access requests: uint32_t addr uint32_t data addr 25-bit address of the access. The high bits mark if the request is for a read or write, and also response bits to mark successfully executed accesses. data 32-bit data value. For further details, see the header file, the C helper library or the VHDL sources. 5a. C helper library interface ============================== A small library with utility functions to perform the UDP access requests from the PC side is provided in client/fnet_client.c (with header fnet_client.h). In particular, this handles timeout and re-sending the requests when responses are not received, as well as initiating the sequence number when establishing an access channel. The functions use return values together with errno to indicate error conditions. More descriptive error strings can also be obtained, since the possible errno codes are not easy to associate with the possible underlying issues. The functions are described in detail in the header file client/fnet_client.h. The most important: fnet_ctrl_connect() Establish a connection with an access channel. fnet_ctrl_close() Disconnect the access channel. fnet_ctrl_get_send_recv_bufs() Retrieve pointers to the memory areas used for the register access requests. fnet_ctrl_send_recv_regacc() Perform an access request. fnet_ctrl_open_tcp() Open the TCP connection. fnet_ctrl_reset_tcp() Reset the TCP connection. fnet_ctrl_get_sockaddr_in() Retrieve the address to use for a TCP connection. fnet_ctrl_last_error() Get a string describing the most recent error. 5b. Control program =================== A command-line utility to check the functionality of Fakernet systems is also provided, as client/fnetctrl. It is not needed for general use - but can be useful for debugging purposes. Usage: fnetctrl HOSTNAME HOSTNAME is the name or IP-address of the Fakernet system. Options: --stat Print statistics of the Fakernet system, updating once per second. This uses the debug registers and counters. --tcp-reset Reset the TCP connection. --tcp Reset the TCP connection, and connect a TCP socket, reading data. --tcp-payload=N Set the artificial TCP payload limit. --tcp-windows=N Set the artificial TCP window size. --udp-flood=N Send UDP register access requests as quickly as possible. Each request packet with N accesses. --btn2led With the Arty A7-35 board test gateware: read the status of the user switches and push-buttons on the board, and control some LEDs based on the number of buttons pressed. --help Display list of options. 6. Performance ============== The VHDL code allows for timing in excess of 100 MHz even on 10-year old FPGAs. Together with handling data as 16-bit words, this allows operation to saturate 1 Gbps links (which would only require 67.5 MHz operation). 7. Compilation and testing ========================== Usage of the code as such does not involve any separate compilation steps. The code is intended to be included into other projects either by symlinking or copying. The distribution includes a machinery to test the routines using a few packet samples, as well as determining approximate minimum clock periods achievable. The test machinery will build and test the VHDL circuit using ghdl (http://ghdl.free.fr/). To perform the tests: make short Wireshark (www.wireshark.org) can be used to view the packet exchanges of the various tests. If available, the text2pcap program (part of Wireshark) will generate .pcap files in the tb_vhdl/ folder. Determining approximate FPGA clocking capabilities requires that the development chain from the manufacturer can be accessed from the command line. To test with Xilinx tools ('xst' must be accessible): make time_virtex4 make time_spartan6 To test with Intel/Altera tools ('quartus_map/fit/sta') must be accessible): make time_max10 make time_cyclonev To build all of the above timing tests: make timing -j 10 The flag '-j 10' sets the number of jobs to run in parallel. This may be omitted or changed, depending on the number of threads available on the testing machine. To compile the fnetctrl program: make client 8. Testing on hardware ====================== To test the Fakernet circuit on actual hardware, a board with a PHY chip connected directly to the FPGA is needed. This since the Fakernet circuit by design include the MAC layer functionality. 8a. Digilent Arty A7-35 ======================= The Digilent Arty A7-35 has a 10/100 Mbps PHY with RJ45 connector. Programming and power is provided over a micro-USB connector. The board-specific files are in the boards/Arty_A7-35/ folder. To synthesise the project and create a bit-stream (the Xilinx tool-chain 'vivado' is required): cd boards/Arty_A7-35/ ./build.sh To program the FPGA of a connected board: ./prgm_fpga.sh Note: to install/activate the Vivado USB programming interface, run: Vivado/20xx.x/data/xicom/cable_drivers/lin64/install_script/install_drivers/install_drivers The two first switches on the Arty board select the two low bits of the Fakernet IP address. It is in the range 192.168.1.192-195. They switches are continuously sampled. The 4 user LEDs display a 4-bit counter of incoming packets. To communicate with the board, it must be network-wise connected to a computer configured with access to the 192.168.1.x subnet. It would then e.g. respond to ping 192.168.1.192 # or .193-.195, as selected The internal debug registers can be displayed using the fnetctrl program: cd ../../ client/fnetctrl 192.168.1.192 --stat To read a stream of TCP data: client/fnetctrl 192.168.1.192 --tcp 9. Acknowledgements (referencing) ================================= The recommended way to refer to Fakernet, when used for work that is published in a research article, is to cite the following paper H. T. Johansson, A. Furufors and P. Klenze, Fakernet - small and fast FPGA-based TCP and UDP communication, (2020), eprint arXiv:2003.12527 10. Contact =========== Håkan T. Johansson e-mail: f96hajo@chalmers.se Subatomic, High-Energy and Plasma Physics Department of Physics Chalmers University of Technology 412 96 Göteborg Sweden Project homepage: http://fy.chalmers.se/~f96hajo/fakernet/