Readme.md 23 KB
Toshiba Electronic Devices & Storage Corporation TC956X PCIe Ethernet Host Driver
Release Date: 26 Dec 2023
Release Version: V_01-03-59 : Limited-tested version
TC956X PCIe EMAC driver is based on "Fedora 39, kernel-6.6.1".
Compilation & Run: Need to be root user to execute the following steps.
By default, DMA_OFFLOAD_ENABLE is enabled. Execute following commands:
#make clean
#make
If IPA offload is not needed, disable macro DMA_OFFLOAD_ENABLE in common.h. set DMA_OFFLOAD = 0 in Makefile and execute following commands:
#make clean
#make
To compile driver with load firmware header (fw.h) use the below command #make TC956X_LOAD_FW_HEADER=1
In order to compile the Driver to include the code for applying Gen3 setting, execute Make with below argument #make TC956X_PCIE_GEN3_SETTING=1
Please note, incase both fw.h and Gen3 settings are needed, then both arugments need to be specified.
Load phylink module
#modprobe phylink
Load the driver
#insmod tc956x_pcie_eth.ko pcie_link_speed=X
In the module parameter pcie_link_speed, X is the desired PCIe Gen speed. X can be 3 or 2 or 1. Passing module parameter (pcie_link_speed=X) is optional. If module parameter is not passed, by default Gen3 speed will be selected by the driver.
Please note that driver should be compiled using below command to use this feature: #make TC956X_PCIE_GEN3_SETTING=1
Remove the driver
#rmmod tc956x_pcie_eth
Note:
Use below commands to advertise with Autonegotiation ON for speeds 10Gbps, 5Gbps, 2.5Gbps, 1Gbps, 100Mbps and 10Mbps as ethtool speed command does not support.
ethtool -s advertise 0x7000 autoneg on --> changes the advertisement to 10Gbps
ethtool -s advertise 0x1000000006000 autoneg on --> changes the advertisement to 5Gbps
ethtool -s advertise 0x800000006000 autoneg on --> changes the advertisement to 2.5Gbps
ethtool -s advertise 0x6020 autoneg on --> changes the advertisement to 1Gbps
ethtool -s advertise 0x6008 autoneg on --> changes the advertisement to 100Mbps
ethtool -s advertise 0x6002 autoneg on --> changes the advertisement 10Mbps
Use the below command to insert the kernel module with specific modes for interfaces:
#insmod tc956x_pcie_eth.ko mac0_interface=x mac1_interface=y
argument info:
mac0_interface: For PORT0 interface mode setting mac1_interface: For PORT1 interface mode setting x = [0: USXGMII, 1: XFI (default), 2: RGMII (unsupported), 3: SGMII, 4: 2500Base-X] y = [0: USXGMII (unsupported), 1: XFI (unsupported), 2: RGMII, 3: SGMII(default), 4: 2500Base-X]If invalid and unsupported modes are passed as kernel module parameter, the default interface mode will be selected.
Regarding the performance, use the below command to increase the dynamic byte queue limit
$echo "900000" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:03.0/0000:05:00.0/net/enp5s0f0/queues/tx-0/byte_queue_limits/limit_min
900000 is the random value chosen. It needs to adjust this value on their system and check "0000:00/0000:00:01.0/0000:01:00.0/0000:02:03.0/0000:05:00.0/" value can be obtained from the "lspci -t" command
The debug counters to check the interrupt count is available.
"#ethtool -S " needs to be executed and sample output is as below
total_interrupts: 120109 lpi_intr_n: 0 pmt_intr_n: 0 event_intr_n: 0 tx_intr_n: 120000 rx_intr_n: 51 xpcs_intr_n: 0 phy_intr_n: 46 sw_msi_n: 12
With V_01-00-07, when IPA API start_channel() is invoked for Rx direction, MAC_Address1_High is updated with 0xBF000000. This register setting is almost similar to promiscuous mode. So please install appropriate FRP instructions.
From V_01-00-08 onwards, Port0 ethernet interface will not be created only if there is no ethernet PHY attached to it
Enable TC956X_PHY_INTERRUPT_MODE_EMAC0 macro for supporting PORT0 Interrupt mode. Disable the macro if the phy driver supports only polling mode. Enable TC956X_PHY_INTERRUPT_MODE_EMAC1 macro for supporting PORT1 Interrupt mode. Disable the macro if the phy driver supports only polling mode.
Change below macro values for configuration of Link state L0 and L1 transaction delay. /* Link state change delay configuration for Upstream Port */ #define USP_L0s_ENTRY_DELAY (0x1FU) #define USP_L1_ENTRY_DELAY (0x3FFU)
/* Link state change delay configuration for Downstream Port-1 */ #define DSP1_L0s_ENTRY_DELAY (0x1FU) #define DSP1_L1_ENTRY_DELAY (0x3FFU)
/* Link state change delay configuration for Downstream Port-2 */ #define DSP2_L0s_ENTRY_DELAY (0x1FU) #define DSP2_L1_ENTRY_DELAY (0x3FFU)
/* Link state change delay configuration for Virtual Downstream Port */ #define VDSP_L0s_ENTRY_DELAY (0x1FU) #define VDSP_L1_ENTRY_DELAY (0x3FFU)
/* Link state change delay configuration for Internal Endpoint */ #define EP_L0s_ENTRY_DELAY (0x1FU) #define EP_L1_ENTRY_DELAY (0x3FFU)
Formula:
L0 entry delay = XXX_L0s_ENTRY_DELAY * 256 ns L1 entry delay = XXX_L1_ENTRY_DELAY * 256 ns XXX_L0s_ENTRY_DELAY range: 1-31 XXX_L1_ENTRY_DELAY: 1-1023To check vlan feature status execute: ethtool -k | grep vlan
To enable/disable following vlan features execute:
(a) rx-vlan-filter: ethtool -K <interface> rx-vlan-filter <on|off> (b) rx-vlan-offload: ethtool -K <interface> rxvlan <on|off> (c) tx-vlan-offload: ethtool -K <interface> txvlan <on|off>Use following to configure VLAN:
(a) modprobe 8021q (b) vconfig add <interface> <vlanid> (c) vconfig set_flag <interface>.<vlanid> 1 0 (d) ifconfig <interface>.<vlanid> <ip> netmask 255.255.255.0 broadcast <ip mask> upDefault Configuraton:
(a) Rx vlan filter is disabled. (b) Rx valn offload (vlan stripping) is disabled. (c) Tx vlan offload is enabled.Please use the below command to insert the kernel module for passing pause frames to application except pause frames from PHY:
#insmod tc956x_pcie_eth.ko mac0_filter_phy_pause=x mac1_filter_phy_pause=x
argument info:
mac0_filter_phy_pause: For PORT0 mac1_filter_phy_pause: For PORT1 x = [0: DISABLE (default), 1: ENABLE]If invalid values are passed as kernel module parameter, the default value will be selected.
Use below commands to check WOL support and its type: #ethtool
WOL command Usage : #ethtool -s wol .
Supported WOL options and meaning:
Option Meaning p Wake on phy activity g Wake on MagicPacket(tm) d Disable (wake on nothing). (Default) Example - To wake on phy activity and magic packet use : ethtool -s eth0 wol pg
Please use the below command to insert the kernel module to enable EEE and configure LPI Auto Entry timer:
#insmod tc956x_pcie_eth.ko mac0_eee_enable=X mac0_lpi_timer=Y mac1_eee_enable=X mac1_lpi_timer=Y
argument info:
mac0_eee_enable: For PORT0 mac1_eee_enable: For PORT1 X = [0: DISABLE (default), 1: ENABLE] This module parameter is to Enable/Disable EEE for Port 0/1 - default is 0. If invalid values are passed as kernel module parameter, the default value will be selected. mac0_lpi_timer: For PORT0 mac1_lpi_timer: For PORT1 Y = [0..1048568 (us)] This module parameter is to configure LPI Automatic Entry Timer for Port 0/1 - default is 600 (us). If invalid values are passed as kernel module parameter, the default value will be selected.In addition to above module parameter, use below ethtool command to configure EEE and LPI auto entry timer. #ethtool --set-eee eee tx-timer
Use below command to check the status of EEE configuration #ethtool --show-eee
Please use the below command to insert the kernel module for RX Queue size, Flow control thresholds & TX Queue size configuration.
#insmod tc956x_pcie_eth.ko mac0_rxq0_size=x mac0_rxq0_rfd=y mac0_rxq0_rfa=y
mac0_rxq1_size=x mac0_rxq1_rfd=y mac0_rxq1_rfa=y mac0_txq0_size=x mac0_txq1_size=x mac1_rxq0_size=x mac1_rxq0_rfd=y mac1_rxq0_rfa=y mac1_rxq1_size=x mac1_rxq1_rfd=y mac1_rxq1_rfa=y mac1_txq0_size=x mac1_txq1_size=xargument info:
mac0_rxq0_size: For PORT0 RX Queue-0 mac0_rxq1_size: For PORT0 RX Queue-1 mac1_rxq0_size: For PORT1 RX Queue-0 mac1_rxq1_size: For PORT1 RX Queue-1 mac0_txq0_size: For PORT0 TX Queue-0 mac0_txq1_size: For PORT0 TX Queue-1 mac1_txq0_size: For PORT1 TX Queue-0 mac1_txq1_size: For PORT1 TX Queue-1 x = [Range Supported : 3072..44032 (bytes)], default is 18432 (bytes) mac0_rxq0_rfd: For PORT0 Queue-0 threshold for Disable flow control mac0_rxq1_rfd: For PORT0 Queue-1 threshold for Disable flow control mac0_rxq0_rfa: For PORT0 Queue-0 threshold for Enable flow control mac0_rxq1_rfa: For PORT0 Queue-1 threshold for Enable flow control mac1_rxq0_rfd: For PORT1 Queue-0 threshold for Disable flow control mac1_rxq1_rfd: For PORT1 Queue-1 threshold for Disable flow control mac1_rxq0_rfa: For PORT1 Queue-0 threshold for Enable flow control mac1_rxq1_rfa: For PORT1 Queue-1 threshold for Enable flow control y = [Range Supported : 0..84], default is 24 (13KB)If invalid values are passed as kernel module parameter, the default value will be selected for Queue Sizes and for Flow control 80% of Queue size will be used.
Note:
- Please configure flow control thresholds (RFD & RFA) as per Queue size (Default values are for Default Queue size which is 18KB).
Please use the below command to insert the kernel module for counting Link partner pause frames and output to ethtool:
#insmod tc956x_pcie_eth.ko mac0_en_lp_pause_frame_cnt=x mac1_en_lp_pause_frame_cnt=x
argument info:
mac0_en_lp_pause_frame_cnt: For PORT0 mac1_en_lp_pause_frame_cnt: For PORT1 x = [0: DISABLE (default), 1: ENABLE]If invalid values are passed as kernel module parameter, the default value will be selected. Note: It is required to enable kernel module parameter "mac0_filter_phy_pause/mac1_filter_phy_pause" along with this module parameter to count link partner pause frames.
Please use the below command to insert the kernel module for power saving at Link Down state:
#insmod tc956x_pcie_eth.ko mac_power_save_at_link_down=x
argument info:
mac_power_save_at_link_down: Common for both PORT0 & PORT1 x = [0: DISABLE (default), 1: ENABLE]If invalid values are passed as kernel module parameter, the default value will be selected.
Debufs directory will be created for port specific in debug path of kernel i.e. "/sys/kernel/debug" in x86 Linux platfrom. Under port specific debugfs directory (tc956x_port0_debug/tc956x_port1_debug), module specific files are created to get dump of debug information related to module. Example: config_stats --> Registers related to CONFIG module mac_stats --> Registers related to MAC block mtl_stats --> Registers related to MTL block dma_stats --> Registers related to DMA block m3_stats --> Debug information related to M3 Firmware interrupt_stats --> Registers related to MSI & INT blocks other_stats --> Information related to Driver & Firmware, TAMAP, Flexible Receiver Parser, mmc counters reg_dump --> Dumps all registers of MAC, MTL, DMA and CNFG modules
Information will be printed to "dmesg" console, when files related to specific module are invoked.
debugfs file can be invoked by using "cat" command. Example: cat /sys/kernel/debug/tc956x_port0_debug/config_stats
Use the below command to insert the kernel module for SW reset during link down.
#insmod tc956x_pcie_eth.ko mac0_link_down_macrst=x mac1_link_down_macrst=y
argument info:
mac0_link_down_macrst: For PORT0 x = [0: DISABLE, 1: ENABLE (default)] mac1_link_down_macrst: For PORT1 y = [0: DISABLE (default), 1: ENABLE]If invalid values are passed as kernel module parameter, the default value will be selected.
- Initial Version
- Used Systick handler instead of Driver kernel timer to process transmitted Tx descriptors.
- XFI interface supported and added module parameters for selection of Port0 and Port1 interface
- kernel_read API replaced with kernel_read_file_from_path API
- sprintf, vsprintf APIs replaced with vcnsprintf or vcnsprintf APIs
- API to print IPA DMA channel statistics supported
- Correction of print statement about selection of C45 PHY for Port0 interface
- XFI interface supported through compile time macro.
- Removed module parameters for selection of Port0 and Port1 interface
- Debugfs support for IPA statistics
- Debugfs not supported for IPA statistics
- Default Port1 interface selected as SGMII
- Module parameters for selection of Port0 and Port1 interface
- Dynamic CM3 TAMAP configuration
- Add support for contiguous allocation of memory
- Add support to set MAC Address register
- Store and use Port0 pci_dev for all DMA allocation/mapping for IPA path
- Register Port0 as only PCIe device, in case its PHY is not found
- PHY interrupt mode supported through .config_intr and .ack_interrupt API
- TC956X_PCIE_GEN3_SETTING macro setting supported through makefile. By default Gen3 settings will not be applied by the Driver as TC956X_PCIE_GEN3_SETTING is not defined.
- TC956X_LOAD_FW_HEADER macro setting supported through makefile. By default, TC956X_LOAD_FW_HEADER macro is disabled. If FIRMWARE_NAME is not specified in Makefile, the default value shall be TC956X_Firmware_PCIeBridge.bin
- Platform APIs supported.
- Modified PHY C22/C45 debug message.
- Configuration of Link state L0 and L1 transaction delay for PCIe switch ports & Endpoint. By default maximum values are set for L0s and L1 latencies.
- Reverted changes related to usage of Port-0 pci_dev for all DMA allocation/mapping for IPA path
- Synchronization between ethtool vlan features "rx-vlan-offload", "rx-vlan-filter", "tx-vlan-offload" output and register settings.
- Added ethtool support to update "rx-vlan-offload", "rx-vlan-filter", and "tx-vlan-offload".
- Removed IOCTL TC956XMAC_VLAN_STRIP_CONFIG.
- Removed "Disable VLAN Filter" option in IOCTL TC956XMAC_VLAN_FILTERING.
- Updated RX Queue Threshold limits for Activating and Deactivating Flow control
- Filtering All pause frames by default.
- Capturing RBU status and updating to ethtool statistics for both S/W & IPA DMA channels
- Added check for Device presence before changing PCIe ports speed.
- Configuring pause frame control using kernel module parameter also forwarding only Link partner pause frames to Application and filtering PHY pause frames using FRP.
- Returning error on disabling Receive Flow Control via ethtool for speed other than 10G in XFI mode.
- Added M3 SRAM Debug counters to ethtool statistics.
- Added MTL RX Overflow/packet miss count, TX underflow counts,Rx Watchdog value to ethtool statistics.
- Added support for GPIO configuration API
- Added PM support for suspend-resume.
- Added WOL Interrupt Handler and ethtool Support.
- Updated EEE support for PHY and MAC Control. (EEE macros are not enabled as EEE LPI interrupts disable are still under validation)
- Added separate control functions for MAC TX and RX start/stop.
- Stopped disabling/enabling of MAC TX during Link down/up.
- Disabled link state latency configuration for all PCIe ports by default
- Skip queuing PHY Work during suspend and cancel any phy work if already queued.
- Restore Gen 3 Speed after resume.
- Single port Suspend/Resume supported
- Restricted MDIO access when no PHY found or MDIO registration fails
- Added mdio lock for making mii bus of private member to null to avoid parallel accessing to MDIO bus
- Runtime configuration of EEE supported and LPI interrupts disabled by default.
- Module param added to configure EEE and LPI timer.
- Driver name corrected in ethtool display.
- Print message correction for PCIe BAR size and Physical Address.
- Added PHY Workqueue Cancel during suspend only if network interface available.
- Free EMAC IRQ during suspend and request EMAC IRQ during resume.
- Resetting SRAM Region before loading firmware.
- Max C22/C45 PHY address changed to PHY_MAX_ADDR.
- Added error check for phydev in tc956xmac_suspend().
- Added module parameters for Rx Queue Size, Flow Control thresholds and Tx Queue Size configuration.
- Renamed all module parameters for easy readability.
- Support for link partner pause frame counting.
- Module parameter support to enable/disable link partner pause frame counting.
- Support for eMAC Reset and unused clock disable during Suspend and restoring it back during resume.
- Resetting and disabling of unused clocks for eMAC Port, when no-found PHY for that particular port.
- Valid phy-address and mii-pointer NULL check in tc956xmac_suspend().
- Null check added while freeing skb buff data
- Code comments corrected for flow control configuration
- During emac resume, attach the net device after initializing the queues
- Fixed phy mode support
- Error return when no phy driver found during ISR work queue execution
- IRQ device name modified to differentiate between WOL and EMAC interrupt IRQs
- Skip resume_config and reset eMAC if port unavailable (PHY not connected) during suspend-resume.
- Restore clock after resume in set_power.
- Shifted Queuing Work to end of resume to prevent MSI disable on resume.
- Set Clock control and Reset control register to default value on driver unload.
- Debug dump API supported to dump registers during crash.
- Tx Queue flushed and checked for status after Tx DMA stop.
- DMA channel status cleared only for SW path allocated DMA channels. IPA path DMA channel status clearing is skipped.
- Ethtool statistics added to print doorbell SRAM area for all the channels.
- Reset assert and clock disable support during Link Down.
- Supported GPIO configuration save and restoration.
- XPCS module is re-initialized after link-up as MACxPONRST is asserted during link-down.
- Disable Rx side EEE LPI before configuring Rx Parser (FRP). Enable the same after Rx Parser configuration.
- Handling of Non S/W path DMA channel abnormal interrupts in Driver and only TI & RI interrupts handled in FW.
- Reading MSI status for checking interrupt status of SW MSI.
- PCI bus info updated for ethtool get driver version.
- Disable MSI and flush phy work queue during driver release.
- Dynamic MTU change supported. Max MTU supported is 2000 bytes.
- Ignoring error from tc956xmac_hw_setup in tc956xmac_open API.
- Perform platform remove after MDIO deregistration.
- Checking for DMA status update as stop after TX DMA stop.
- Checking for Tx MTL Queue Read/Write contollers in idle state after TX DMA stop.
- Triggering Power saving at Link down after releasing of Offloaded DMA channels.
- Added kernel Module parameter for selecting Power saving at Link down and default is disabled.
- Added Lock for syncing linkdown, port rlease and release of offloaded DMA channels.
- Added debugfs support for module specific register dump.
- For IPA offload path, disable RBU interrupt when RBU interrupt occurs. Interrupt should be enabled back in IPA SW.
- Fix for configuring Rx Parser when EEE is enabled and RGMII Interface is used
- 2500Base-X support for line speeds 2.5Gbps, 1Gbps, 100Mbps.
- MDIO registration failures treated as error of type "ENODEV"
- Update of fix for configuring Rx Parser when EEE is enabled
- Support for SW reset during link down.
- Module parameters introduced for the control of SW reset and by default SW reset is disabled.
- Module parameters for SW reset (during link change) enabled by default for Port0.
- Automotive AVB/TSN support
- Default Port 0 interface is XFI and Port1 interface is SGMII
- IPA (macro TC956X_DMA_OFFLOAD_ENABLE) enabled by default
- Kernel timers are used to process transmitted TX descriptor. Systick timers are not used.
- Dynamic change of MTU not supported. Max MTU supported is 9000.
- Port 1 supports USXGMII, XFI and 25000Base-X interface also.
- Kernel 6.1.18 Porting changes
- TC956x switch to switch connection support (upto 1 level) over DSP ports
- Kernel 6.6.1 Porting changes
- Added the support for TC commands taprio and flower
tx_intr_n = No of. Tx interrupts originating from eMAC sw_msi_n = No. of SW MSIs triggered by Systick Handler as part of optimized Tx Timer based on Systick approach. So total number of interrupts for Tx = tx_intr_n + sw_msi_n Please note that whenever Rx interruts are generated, the Host ISR will process the Tx completed descriptors too.