last modified: 24 June 2003
The Linux-USB kernel code is complex enough to need some focused testing efforts, and this web page tries to give an overview of the key ones. There are basically two things to tests: hosts like desktop PCs or other "USB masters", and devices, peripherals, or whatever you want to call the gadgets that act as "USB slaves" and implement the device function the USB host is accessing.
Although most of these tests will be of interest to folk maintaining USB system software like device drivers or firmware, some may be useful to sysadmins or end users that suspect they may have flakey USB hardware.
You're likely to be interested in this if you're maintaining a USB Host Controller Driver (HCD), especially if it's one that's not widely available on PCI hardware; or if you're using Linux as a host when testing some kinds of product. Such tests help serve as driver regression tests, so they're good to have as arrows in the test case quiver of a Linux distributor. These are also the tests that might be helpful in turning up hardware problems with some USB configurations. You may even notice interesting performance characteristics.
Other than the old usbstress 0.3 software (which wasn't widely used), the primary effort here started with the 2.5.42 kernel release. It consists of:
Assuming you have a recent Linux kernel (such as 2.5.72) you will already have the kernel source code for the tests, so the first question is how to get the other two pieces. (The answer to the second question is: yes, you can add new test cases. Please do!)
You won't notice issues with class or vendor-specific functionality with that kind of test setup.
At this writing, there are many devices known to work with this testing software. Such "bulk sink" and "bulk source" functionality is also supported by most device firmware development kits.
The first type of device is anything using a full speed Cypress EZ-USB chip. (Cypress FX2 development kits include similar drivers.) The devices use GPL'd firmware written by Martin Diehl, instead of whatever they might normally be using. (The source for that is in the firmware/ezusb/testing area in CVS for the Linux-Hotplug project.) Many products (notably, many types of serial adapters) use those devices internally, and rely on device drivers (or fxload) to download the firmware. You can disable their "official" device drivers and then use them for testing. Store the test firmware into the /etc/hotplug/usb directory so that you can download it with 'fxload' from http://linux-hotplug.sourceforge.net. Then enable the usbtest kernel driver, and install this /etc/hotplug/usb/usbtest driver setup script, you should be able to run these tests with very little trouble, either in a formal "test this now" mode or as background tasks in parallel with other activity (including other USB activity).
The second type of device uses the Linux-USB Gadget driver framework API. That's standard in Linux 2.5, and is also in use on Linux 2.4 kernels. It includes a Gadget Zero driver. To use it, you need a hardware-specific driver to make your USB device controller implement that API. High speed USB devices can work, as well as full and low speed ones. (The collection of supported hardware is beginning to grow.)
Hey! Intel sells official USB2 compliance testing devices! PDF at around $100 each. Maybe you can help make these work with "usbtest" or related code.
The testusb program just issues ioctls to perform the tests implemented by the kernel driver. Run it like this:
[root@krypton misc]# testusb must specify '-a' or '-D dev' usage: testusb [-a] [-D dev] [-n] [-c iterations] [-s packetsize] [-g sglen] [root@krypton misc]# |
Use 'usbtest -a' to test all recognized devices in parallel (one thread per device). I did this on a uniprocessor, for two high speed devices. That's with lots of I/O parallism, so likely these would be good SMP test modes too:
[root@krypton misc]# usbtree /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-hcd/5p, 480M |__ Port 1: Dev 2, If 0, Class=vend., Driver=usbtest, 480M |__ Port 2: Dev 3, If 0, Class=vend., Driver=usbtest, 480M [root@krypton misc]# [root@krypton misc]# testusb -a unknown speed /proc/bus/usb/001/003 unknown speed /proc/bus/usb/001/002 /proc/bus/usb/001/002 test 0 took 0.000011 sec /proc/bus/usb/001/003 test 0 took 0.000006 sec /proc/bus/usb/001/003 test 1 took 0.201934 sec /proc/bus/usb/001/002 test 2 took 0.226852 sec /proc/bus/usb/001/003 test 3 took 0.211918 sec /proc/bus/usb/001/002 test 4 took 0.222404 sec /proc/bus/usb/001/003 test 5 took 2.137454 sec /proc/bus/usb/001/002 test 6 took 2.133821 sec /proc/bus/usb/001/003 test 7 took 2.125387 sec /proc/bus/usb/001/002 test 8 took 2.115402 sec [root@krypton misc]# |
What did the tests do? UTSL; current versions add at least control message tests (covering many "chapter 9" spec behaviors, and unlink testing), or for the short version:
[root@krypton misc]# dmesg | tail -10 usbtest.c: 02:00.2-1 TEST 0: NOP usbtest.c: 02:00.2-1 TEST 2: read 512 bytes 1000 times usbtest.c: 02:00.2-2 TEST 0: NOP usbtest.c: 02:00.2-2 TEST 1: write 512 bytes 1000 times usbtest.c: 02:00.2-2 TEST 3: write 0..512 bytes 1000 times usbtest.c: 02:00.2-1 TEST 4: read 0..512 bytes 1000 times usbtest.c: 02:00.2-2 TEST 5: write 1000 sglists, 32 entries of 512 bytes usbtest.c: 02:00.2-1 TEST 6: read 1000 sglists, 32 entries of 512 bytes usbtest.c: 02:00.2-2 TEST 7: write 1000 sglists, 32 entries 0..512 bytes usbtest.c: 02:00.2-1 TEST 8: read 1000 sglists, 32 entries 0..512 bytes [root@krypton misc]# |
If you're testing a device with Linux, you can use module options to make it use the "usbtest" driver, and then the "testusb" program can talk to it with "test 9" and "test 10". Those are chapter 9 tests (control traffic) that every USB device should be able to pass.
As of 2.5.44, the three main HCDs (EHCI, OHCI, UHCI) seemed to pass those basic tests on at least some basic hardware configurations, on runs of a few hundred iterations. That's clearly a good milestone, but it certainly shouldn't be the last one! (Some host controllers have run these tests for weeks without significant problems; and since 2.5.44, more test cases have been added.)
You're likely to be interested in this if you're developing a USB device, and want to make it support Linux hosts. You can implement the device by embedding Linux, using the Gadget driver framework, or with some other OS. Gadget drivers are written to a hardware-neutral API, which can support both generic (class style) functionality or vendor-specific functionality. If you're using Linux, the Gadget Zero driver can be very useful when you're bringing up new USB controller hardware, or as a model when you're writing a new gadget driver. It uses all code paths the hardware-specific controller driver needs for "chapter 9" conformance, and basic bulk transfer streams.
There are several levels of testing that a Linux host can perform to your device. The basic test is whether Linux-USB can enumerate your device and parse its descriptors; that's basic plugfest style testing. Any device driver supports this, gadget zero or otherwise. (Look at /proc/bus/usb/devices after Linux enumerates the device, and verify that the descriptors are displayed correctly.) There are also some chapter 9 tests that your device should handle; if it handles the analagous USB-IF tests (from a Windows host), it should pass these with little trouble. For full Linux support you must make sure Linux host side applications can use your device, using some kind of device driver. (User mode drivers can sometimes work here.)
The http://linux-usb-test.sourceforge.net documentation is a resource for interoperability testing that you may find helpful.
You should do all such basic testing with all major Linux host configurations: the three primary types of host controller (EHCI, OHCI, and UHCI) with their drivers, and with both USB 1.1 and USB 2.0 hubs (with transaction translators).
You can't do thorough testing without both kinds of external hub, and you probably need an add-on PCI host controller card to make sure you have the other kind of USB 1.1 controller (OHCI or UHCI) and/or a USB 2.0 controller (EHCI). Or at least borrow the use of a system with such hardware, if you don't want to own it yourself. Although it's a goal to minimize differences in how the different USB host controllers behave on Linux, they can't all report the same status codes given the same errors.
High-speed capable devices must be tested both at high speed, connecting to EHCI directly or through a USB 2.0 hub, and at full speed. Most other devices run only at full speed (or sometimes low speed), so they won't need as much testing.
Full-speed (or low speed) tests connect devices in one of two ways.
The "testusb" program gives you access to two basic kinds of "chapter 9" tests. Test 9 just makes sure a number of required operations are handled correctly; no device should ever fail it. Test 10 is reasonably aggressive, and tests things like queueing, protocol stalls, short reads, and handling of consecutive faults (where it's easy for devices and hosts to misbehave). Of course, if you're testing a device you'll also want to be sure it passes the USB-IF tests (after paying a MSFT license tax, since that software only runs on Windows hosts). Any device running under Linux should pass all of those tests, regardless of how much additional Linux integration is done.
The more interesting level is whether your Linux-using customers can use your USB device through host-side applications. Some devices can work through 'usbfs' with user mode device drivers; those devices tend to rely only on simple half-duplex protocols. (Some very useful USB-IF class specifications are half duplex...) Otherwise you'll need some kernel device driver. The Linux kernel community strongly prefers GPL'd device drivers, which can safely be merged into kernel distributions. Closed-source drivers are undesirable, and can't usually be bugfixed. If you feel you must close your source, do it in user mode applications.
If your device works with Linux, presumably you'll have a set of application level tests to verify higher level functionality. Here's a short checklist of other things you need to support from those host side drivers:
Of course, there are also the usual sort of driver portability issues. You can often expect other people in the Linux community to help with those issues, if your software is clean and portable. Once you have the basics working (plug/unplug, all host controllers, and your driver functionality meeting current application requirements), users should be able to submit patches for the rest.