Even until several years ago, most of WLAN application is tied up with PC and laptop and the capable throughput and expected is not that high. The range of the testing were not that wide either. Now at the age of Smart phone and many other electronics supporting WLAN functionality, the range of the testing WLAN has been drastically widened. Followings are some of the testing items poping up my mind as of now and the list will get longer and longer as we evolve to 5G, in which WLAN would play even bigger roles in whole communication system.
The diagram below puts WLAN Testing in the centre and eight test areas around it. Six green items form the inner ring, from PHY/RF to Throughput, and item 4 also names SIP Call over ePDG. Two purple items, WLAN HotSpot and WiFi Direct, sit outside the ring. Those two use WLAN in a different role, as the access point of a phone or as a direct link between two devices.
The areas also differ in who tests them. PHY/RF belongs to the chipset and module makers, while the other areas belong to the device makers and the network operators. Items 4, 5 and 7 connect WLAN with the cellular network, so they need test equipment that simulates both sides at the same time.
- 1. PHY/RF
- 2. User Interface
- 3. Authentication/Security Mechanism
- 4. ePDG/IKE
- 5. Access Selection between WLAN and Cellular Network
- 6. Throughput
- 7. WiFi HotSpot
- 8. WiFi Direct
- Reference
1. PHY/RF
PHY/RF testing is mainly for testing RF transimission signal quality, reciever sensitity. These are usually tested by WLAN Chipset vendor or WLAN Module maker. It has its own list of test items mainly defined in IEEE specification.
Following is an example of PHY/RF testing system from Anritsu, named MT8862A.
In the setup below, a PC Controller drives the MT8862A through a Router, over a Remote Ethernet link. The MT8862A acts as an Access Point or as a Station, and it talks over the air to the DUT, a WiFi Device. The lower half shows four result windows on the PC: a spectrum, a power versus time trace, an IQ constellation and a table of numerical results.
Ping Request and the reply : the tester sends a Ping Request and the DUT answers. The arrow for the answer is labelled Packet Replay. The ping traffic makes the DUT transmit real packets while it is connected, so the tester can measure them.Spectrum and power versus time : these show the shape of the transmitted signal in frequency and the burst structure in time.IQ Constellation : the demodulated symbols. A tight cloud around each point means a clean transmitter.Numeric Result : the table lists EVM, Center Frequency Tolerance, Symbol Clock Frequency Tolerance and IQ Imbalance, each with average, maximum and minimum values.
So the tester measures the DUT while the DUT runs its normal protocol, instead of a special test mode. The same setup can also check the receiver. The tester can lower its output power, and the ratio of lost packets shows where the DUT's receiver sensitivity ends.
Item 2~6 are usually tested by company who provide WLAN as a complete service like PC maker, Mobile Phone Maker or Network Operators. Usually all of these items are consolidated into a single test package.
Transmitter quality is a set of numbers : EVM, frequency tolerance, symbol clock tolerance and IQ imbalance describe how clean the DUT's signal is.A connected DUT is enough : ping traffic over a normal connection gives the tester packets to measure.
2. User Interface
In case of usual laptops/PCs that does not have any cellular communication capability, user interface for WLAN would not be that complicated. Probably only Authentication setup GUI is enough. But in case of mobile phone (SmartPhone), there are many different ways for wireless communication in addition to WLAN. So there are many options for communications and there are more options for WLAN as well comparing to PC/laptops. In addition, the behvaior of WLAN related User Interface may act differently depending on the settings of User Interface of other communication technology. So all the possible combination of WLAN User Interface setting and verification of their behavior become important test items. You would see a lot of test cases just related to this kind of User Interface. Following is one example of User Interface directly or indirected related to WLAN on a mobile phone. You may see different interface on different mobile phones. This is just an example.
Connections screen, on the left : Wi-Fi, Bluetooth, Data usage, T roaming, More networks, NFC and S Beam sit on one screen, and each one is a setting the tester can combine with Wi-Fi.Wi-Fi screen, in the middle : the Smart network switch option, a list of SSIDs with their security, and the Scan and Wi-Fi Direct buttons.Mobile networks screen, on the right : Data network settings, Data roaming, Access Point Names and Data network mode. These cellular settings decide what happens when the phone leaves Wi-Fi.
Even though this is only User Interface testing, you may need a test system that can support both Cellular technology and WLAN technology because some user interface operations requires the completion of a certain radio protocols.
Test combinations, not single switches : a Wi-Fi setting can change its behaviour when a cellular setting such as Data roaming changes.
3. Authentication/Security Mechanism
As WiFi technology evolves, the techniques that tries to break into the security protection mechanism also evolves. As a result, you would see more and more different types of Security mechanisms are used. Since a mobile phone (or laptops) is expected to work with WiFi network anywhere, you should verify that your device can work with all different types of Security system. Following is just an example of WiFi access points that my mobile phone can detect in my office (I intentionally changed the SSID name for information protection purpose :) and you will see various different types of Security algorithms are used ( the security method is indicated on the yellow line. The 'empty' space on the yellow line indicate 'Open' connection (basically No Security at least on the Access point). Of course, this is just an example and you may see even more diverse method where you are.
Sign-in is required, SSID 1 : usually an open network with a login page. The phone connects without a key, and the user signs in through a browser.No text, SSID 2 and SSID 3 : open networks with no security on the access point.Secured with WPA/WPA2, SSID 5 : a shared passphrase that every user of the network types in.Secured with 802.1x, SSID 4 and SSID 6 : each user authenticates to a server with EAP. The dialog on the right shows the settings for SSID 6: the EAP method PEAP, Phase 2 authentication, a CA certificate and an Identity.
To test these method, you would need a test equipment that supports all of these security methods and user data connectivity, or you may use your live Access Point.
The 802.1x case is where WLAN security meets the cellular world. The same EAP framework also carries SIM-based methods. For example, TS 33.402 uses EAP-AKA' for trusted non-3GPP access, so the phone authenticates to the WLAN with its USIM and the user types nothing. A complete test plan therefore covers each 802.1x method the device supports, plus the certificate settings, because a wrong CA certificate setting fails in a different way from a wrong password.
Four security types in one office : open, open with a sign-in page, WPA/WPA2 with a passphrase, and 802.1x with EAP.802.1x hides a second choice : the EAP method and its Phase 2 setting must match the server, not only the password.
4. ePDG/IKE
ePDG is a special mechanism to connect a WiFi network (Untrusted Network Component) to a Cellular network at IP data layer. To connect this kind of untrusted component to Cellular network, a special type of authentication and security algorithm is required and in most case IKEv2 are used. I wouldn't go through the details of how ePDG works. If you want to know further details, refer to WiFi Offload page and IKE page.
As far as I experienced, this started being developed in mobile phone industry almost two years now (as of Jun 2015), but only recently a few network operators started seriously testing in their lab. To test this functionality, you would need an equipment which has at least following components that can simulate both cellular network and ePDG/AAA Servre (IKE) as illustrated below. If you are a person who is involved in this kind of test, refer to WiFi Offload Check List page first.
3GPP Access and PDN GW : two network simulators take these roles. The PDN GW has to be shared by LTE and WiFi, or a handover cannot keep the IP address.ePDG and 3GPP AAA Server : two PCs run these functions, joined on SWm by the red authentication path.Un Trusted Non 3GPP Access : an access point. The pink common path runs from the phone to it, and the green data path runs on to the ePDG and the PDN GW.
Each box in the picture has to behave as the specification says, and each one can fail on its own. The ePDG runs IKEv2 on SWu and relays EAP-AKA on SWm, as in TS 33.402. The AAA Server needs the same subscriber keys as the USIM in the phone. The PDN GW must anchor both accesses, so that the UE keeps its IP address after the move. For a SIP call over the ePDG, the test system also needs an IMS server that the UE can reach through the tunnel.
The test system needs both worlds : an LTE simulator alone cannot test ePDG, and a WLAN tester alone cannot either.Keys must match end to end : the USIM, the AAA Server and the HSS function must share the same credentials, or IKE_AUTH fails.
5. Access Selection between WLAN and Cellular Network
In case of all the mobile phone and some PCs that support both WiFi and Cellular technology, the device has to make a complicated decision whether it should connect to Cellular Network or WiFi network. If the device is connected to both network simultaneously, it has to make another decision when user is trying to make a voice call.. it has to determine whether it initiate call using cellular network or WiFi network.
A certain level of decision is made by the special settings that a user has configure and some other level of decision is made by signal strength between cellular network and WiFi network. But the detailed selection criteria is specified by each service provider (Network Operator) and these selection algorithm is a very important part of testing in most network operators.
Following is one example of UE setting that directs UE on how to select network between cellular and WiFi. This is one of the simplest configuration and you would see much complicated setting if your device support not only data call over WiFi but also voice call over WiFi.
The checkbox, on the left : Smart network switch sits at the top of the Wi-Fi screen, and it is unchecked in the left screen.The dialog, on the right : turning it on connects the device to a mobile network automatically when the Wi-Fi connection is unstable, and the dialog warns that this may cost extra.
A setting like this is a UE decision. The operator can also steer the choice from the network side, and 3GPP defines two ways. ANDSF can send the UE policies such as ISMP, ISRP and the WLAN Selection Policy, as described in TS 23.402. From Release 12, E-UTRAN can also provide RAN rules with thresholds for LTE and WLAN. TS 24.302 then defines which of the two rule sets controls WLAN selection and traffic routing. So a full test covers the user setting, the operator policy and the radio thresholds, and it checks that the device does not move back and forth between the two accesses near a threshold.
Three inputs decide the access : the user setting, the operator policy from ANDSF or RAN rules, and the measured radio conditions.Test the edges : the most useful test cases sit near the thresholds, where the device changes its decision.
6. Throughput
Conceptually throughput test is simple to understand, but as far as I experienced performing throughput test in WiFi is more difficult than cellular technology because WiFi PHY/MAC scheduling is not as reilable as celluar technology. Because WiFi is scheduling data transfer based on CSMA/CA and random backoff, the throughput test result would be fluctuating drastically if there are any interference in the channel.
At the initial phase of throughput test, we usually test about maximum throughput under ideal channel condition, but recently I saw many company is putting tougher criteria in terms of throughput test condition. The most common criteria about throughput test you may see as of now would be as follows :
- Max throughput under ideal channel condition
- Throughput vs Range (distance between the device and AP)
- Throughput vs SNR
- Throughput vs Frame Fragmentation
Let's see what each criterion tells you. The maximum throughput shows how close the device gets to its PHY rate. The PHY rate depends on the modulation and coding, the number of spatial streams and the channel width. The measured throughput is always lower, because contention, acknowledgements and headers take part of the air time. Throughput versus range and throughput versus SNR both test rate adaptation, which is how quickly and how correctly the device lowers its rate as the signal gets weaker.
Frame fragmentation needs a word of its own. When a frame is split into fragments, each fragment carries its own header and needs its own acknowledgement. So fragmentation lowers throughput on a clean channel, but on a noisy channel a lost small fragment costs less than a lost large frame. Also, a TCP throughput and a UDP throughput differ even on the same link, so the test report has to say which one it used.
Throughput is always below the PHY rate : the MAC overhead of CSMA/CA, acknowledgements and headers takes a share of the air time.Range and SNR tests measure rate adaptation : a good device steps down its rate smoothly instead of losing packets.
7. WiFi HotSpot
WiFi HotSpot is a technology which can translate WiFi protocol into another type of wireless technology (e.g, WCDMA/HSPA or LTE). In some case, we use live network or lab network but in most case we use a network simulator as shown below. In this case, we assume that Cellular part network simulator is performing ideal max throughput but as you might have experienced there is no such a thing like 'ideal'. At the early stage of test system setup, you would need to spend a lot of time and effort to make it sure that the cellular network simulator is working as expected. Once this part is verified, the remaining test method and criteria on WiFi part is similar to 'Throughput' test described in previous section.
In the setup below, a Cellular Network simulator for WCDMA/HSPA or LTE connects to the phone by a cable, and that link is labelled Celluar Protocol. The phone works as a Mobile Phone as a HotSpot, and a Data Client laptop connects to it with the WiFi Protocol. A Data Server laptop sits behind the simulator, so the test data flows from the client, through the phone and the simulator, to the server.
The network list : the Data Client shows AndroidHotspot5873 as Connected, among other SSIDs in the area.No Internet access : the client reports this for the hotspot. The test network most likely has no route to the Internet, so the laptop's connectivity check fails, even though traffic to the Data Server works.
The phone in this test does two jobs at once. It is a UE on the cellular side and an access point on the WiFi side, and it forwards IP packets between them. So a throughput problem can come from either side, or from the forwarding inside the phone. That is why the cellular side has to be verified first, as described above, before the WiFi side is measured.
The hotspot throughput is limited by the slower side : the end-to-end rate cannot exceed the cellular rate or the WiFi rate.A client warning is not always a failure : No Internet access on a lab network can simply mean that there is no Internet behind the simulator.
8. WiFi Direct
WiFi Direct is a mechanism by which a device can communicate with another device directly without going through AP (Access Point). If you are not faimilar with this functionality, see this video : How to Use Wifi DIrect. If you are more interested in the technical details, refer to WiFi Direct page.
If you want to perform very strict test about this, you would need special equipment that can simulate WiFi device, but I haven't seen this kind of equipment yet. There might be this kind of equipment which I don't know of.
The picture below shows the practical alternative, a test between two real devices from different vendors. Each phone has its Wi-Fi Direct screen open, lists its own device name under My device name, and lists the other phone under Available devices.
Left phone : its own name is Jaeku Ryu (SHV-E330S), and it lists BLACKBERRY-2B5E with Tap to connect.Right phone : its own name is BLACKBERRY-2B5E, and it lists Jaeku Ryu (SHV-E330S) the same way.Direct Data Transfer Over WiFi : after the connection, data goes between the phones with no AP in between.
A test like this checks the steps a user sees. First, each device has to discover the other. Then one device has to become the group owner, which plays the AP role for the pair. Finally, the pair has to transfer data. Devices from different vendors are the most useful pair, because each vendor may implement the optional parts differently.
Discovery works both ways : each device must list the other, as both screens show here.Cross-vendor pairs find more problems : two devices of the same model share the same implementation and hide interoperability issues.