Since I have been in the area of Test/Verification for the longest time of my career, there might be the largest amount of things to say in this area. However, there has been the least amount of the contents in this area in ShareTechnote.

I have long been planning to start posting on this area but I didn't have chance to trigger yet. Sometimes, I found it difficult to say even a single word when there are too many things to say :)

Recently I got suggested/pushed to say something on this :), I thought this is a good chance to start talking on this. I am not sure how long (months ? years ?) to complete posting a critical mass of the contents, but I am starting anyway.

Let me set the direction of this page first. It starts with the words that testers use, then places each test level on the V model. After that it goes down to the smallest unit and back up to the full system. The examples come from an LTE modem, but the method applies to any layered product.

• Basic Terminologies
• V Model for Testing Cellular Technology
• Unit to System
• What is Unit Test ?
• Integration Test/System Test/Function Test
• Reference

Basic Terminologies

As in any area of the technology, Test/Verification is also a technology with wide spectrum and with a long history. In any of those technologies, you would see a lot of terminologies which in many cases does not sound clear and confusing to you. Following is the list of the terminology at the highest level (not even getting into any details) in random order. Do all of these sound clear to you ? or still confused ?

Don't get disappointed too much even if you are still confused.. I am still confused as well :).  In many cases, these terminlogies are not used in clear manner and in many cases they use the same words for different things and some other times they use different words for the same thing. It happens in any language and any dialog.

So my recommendation is 'Don't focus on the definition of these terminologies. Just focus on the details of what you have to do in your testing/verification task'. Any test/verification document with quality are very clear on exact procedure/detailed step-by-step procedure for each of the test. (I will creat another document describing on which one is a good test document and which one is bad document later when I have chance... too many things to talk on this :))

• IODT
• Integration Test
• Function Test
• Unit Test
• Component Test
• System Test
• Smoke Test
• Sanity Test
• Usability Test
• Regression Test
• Performance Test
• Stress Test
• End-to-End Testing
• Pre-Certification Test
• Certification Test
• Conformance Test
• Accepatance Test
• Whitebox Test
• Blackbox Test

Even so, it helps to see that the list mixes several different questions. The first question is how much of the product a test covers. Unit Test and Component Test cover one block, Integration Test covers several blocks together, and System Test and End-to-End Testing cover the whole product.

The second question is who writes the procedure. Conformance Test, Pre-Certification Test and Certification Test follow procedures from 3GPP and the certification bodies. Acceptance Test follows the procedure of the customer, often an operator. IODT, interoperability development testing, runs a device against another vendor's equipment while both are still in development.

The third question is when you run the test. Smoke Test and Sanity Test are quick checks on a new build, and Regression Test repeats earlier tests after a change. The fourth question is what you check. Function Test and Usability Test check behaviour, while Performance Test and Stress Test check limits. The last question is how much of the inside you can see. Whitebox Test uses the internal structure, and Blackbox Test uses only the external interfaces.

• Scope terms say how much is under test : Unit, Component, Integration, System and End-to-End form a ladder from one block to the whole product.
• Ownership terms say who wrote the procedure : Conformance, Certification and Acceptance tests follow documents written outside your team.
• Timing terms say when a test runs : Smoke and Sanity tests gate a new build, and Regression tests guard old features after a change.
• The same word can mean different things : so read the procedure of a test before you trust its name.

V Model for Testing Cellular Technology

If you got trained formally in Testing area or looked into formal textbook, you might have seen the concept of 'V' model. 'V' model is a kind of representation technique to align planning (defining the specification) process on the left and testing process on the right in the form of the letter 'V'.  The steps in the order of time (chronoligical order) is from the top left of the V and slide down to the bottom and then proceed upwards to the top right of the V.

Each of the blocks on the path would vary case by case. If you google 'V model' image, you would see a lot of example and you would see none of the examples are exactly the same.

Following is my version of V Model in Cellular technology. You may or may not agree with the details.. you may have your own version. Whatever it is, it would be good to draw this kind of model to have a clear big picture of the process of your own.

• Left arm, top to bottom : Acceptance Test Docs, Certification Test Docs, System Requirement and API/Layer Specification.
• Bottom : Code /Hardware, where the left arm turns into the right arm.
• Right arm, bottom to top : Unit Test, Integration Test, System Test, Pre-Certification Test, Certification Test and Acceptance Test.
• Green arrows : API/Layer Specification feeds Unit Test, System Requirement feeds Integration Test and System Test, Certification Test Docs feed Pre-Certification Test and Certification Test, and Acceptance Test Docs feed Acceptance Test.
• Colours : the orange boxes on the right are the test levels with procedures written by others, and the teal boxes are the levels you design yourself.

In case of Cellular technology, workload on the left side would be relatively small since most of the job have been done by international standard organization like 3GPP. Probably the specification right on top of Code/Hardware implementation (API or Unit specification) would only be your own.

In this page, I would mainly talk about the right low part of the V model (Unit Test, Integration Test, System Test). I would not touch much on the right top part of the V model either since it is usually defined by 3GPP and implemented by test equipment vendor. There are not much room for you to put much of your own idea on those steps.

• Each test level checks one document on the left : a failed test points back to the specification level it was written against.
• In cellular, 3GPP writes most of the left arm : only the API or unit specification is usually your own.
• The lower right is where test design matters : Unit, Integration and System tests are the levels you have to plan yourself.

Unit to System

In testing procedures (right parts of V model) in cellular technology, you don't have to worry much about the parts marked in orange (Pre-Certification and above) in terms of test strategy/methodology because the detailed test procedure/methodology is already established by somebody else (mostly international standard organization like 3GPP), but you you may need to establish your own strategy/methodology for the tests at the lower half (e.g, Unit Test, Integration Test, System Test) if you are involved in the test at early stage of development cycle.

Practically and logically, you can illustrate overall development process and testing as shown below. If your company is working on celluar modem chipset (wether for modem for mobile phone or modem for eNB), you might have independent development team for each layer. And there would be another team that is in charge of integrating all of the layers into a whole system.

• Left column : each layer team, from PHY at the bottom to RRC at the top, sees its own layer as a System made of many Units.
• Middle column : the integration team sees each whole layer, PHY, MAC, RLC, PDCP and RRC, as one Unit.
• Right side : PHY and MAC meet in the first Integration step. RLC joins at the second step and PDCP at the third, and RRC joins at the final step into the SYSTEM bar.

In terms of testing, the first thing you have to figure out (or define) is to define what is the fundamental unit (the smallest component to be tested) and what is your system. This definition is very important. However, the definition would be different depending on which specific area (scope) your testing would work.

For example, if you are working at PHY development team and setup a test procedure of PHY layer only, Your test Unit would be various basic functions (e.g, API or FPGA blocks performing each steps in various channel coding process) and your test System would be the whole integrated PHY layer. I will describe more on this in following sections.

If you are working at inegration team of the full protocol stack, your test Unit can be the whole PHY or whole MAC etc and your System would be the whole modem with the full stack.

So the words Unit and System describe your position, not the block itself. The PHY layer is a System for the PHY team and a Unit for the integration team. Write down both definitions at the start of a project, because each team plans its tests against its own definition.

• Unit and System depend on the scope of your team : the same block can be a system for one team and a unit for another.
• Define the unit and the system before writing test cases : the definition decides where one test ends and the next one starts.
• A layered modem integrates bottom up : PHY and MAC come first, and the upper layers join afterwards.

What is Unit Test ?

As I mentioned above, the definition of Unit depends the scope of overal testing that you are working on. However, the whatever your scope is, it would be the most important to clearly define each and every possible Units.

One of the most common example of test Unit is a API function (if the test system is a software) or the minimum hardware component to which you can connect measurement probes and measures various quantities (e.g, current, voltage or digital pulses).

Regardless of whether it is hardware or software, a unit has well defined input and output as illustrated below. The functionality of the unit is to covert the given inputs into the expected outputs.

you need to be able to clearly define what are the inputs and what are the outputs of the unit. If you are working on a software project like developing a spreadsheet(e.g, Excel). Each of the mathematical function can be a test Unit. For example, if you have a function named 'Div(a,b)'. The unit can be illustrated as shown below.

Once you defined a unit as shown above, the example of Unit test can be as follows. In most case, it is impossible to perform the unit test for all the possible combinations of input and output. So the job of the experts in test/verification team is to figure out the best input/output combination sets that can verify the unit in minimum effort (minimum try). In many cases, we test with some of the typical cases and some of corner cases. 'Corner case' refer to the case that may not happen often but it is highly likely to create the problem if it happens. In case of our div(a,b) example, Test 1,2 may belong to the typical cases and Test 3,4 may belong to Corner cases. Usually it would be relatively easy to define typical cases, but it tend to be difficult to define Corner cases.

• Test 1 : A = 4 and B = 2. A typical case with an exact result.
• Test 2 : A = 4 and B = 3. Still typical, but the result does not end, so the test checks the precision of the output.
• Test 3 : A = 4 and B = 0.000000123. A corner case, because a very small divisor gives a very large result that can overflow or lose precision.
• Test 4 : A = 4 and B = 0. A corner case, because division by zero has no result, and the unit has to define what it returns.
• The output of every test is drawn as a question mark. The test designer has to write the expected output down before running the test, otherwise the test cannot fail.

Now let's look into a more practical cases in wireless communication. Following shows the CFI channel process (I picked this as an example since this is much more simpler comparing other channels like PDSCH, PUSCH). The left most track shows the CFI coding process depicted in 3GPP specification. How do you implement this specification in your protocol stack. Of course, there can be almost infinite number of different ways to implement this. I put two typical examples of implementations : Case 1 and Case 2. Logically and personally I prefer Case 1, but depending on your hardware architecture you may need to implement it like Case 2 (As you might see, you might need a lot of memory copies or data transfer between each function in Case 1 unless you come out with special techniques to reduce this kind of data transfer process).

• Left track : the five steps of the specification. Channel Coding in 36.212 5.3.4 turns the CFI into 32 bits b₀ to b₃₁, and Scrambling in 36.211 6.7.1 keeps 32 bits.
• Left track, continued : Modulation in 36.211 6.7.2 gives 16 QPSK symbols d₀ to d₁₅, Layer Mapping/Precoding in 6.7.3 gives y₀ to y₁₅, and Resource Element Mapping in 6.7.4 places them as four quadruplets z₀ to z₃.
• Implementation Case 1 : one function per step, ChanCode(), Scrambling(), Modulation(), PreCode() and RE_Map(). Each function passes an array to the next one.
• Implementation Case 2 : a single function ProcessCFI(CFI,AryZ[]) that takes the CFI and returns the mapped symbols.

How can we define the 'Unit' in your test plan ? The answer would be different depending on how you implemnted this process. If you implemented this process as in Case 1, your 'Unit' in the test procedure would be ChanCode(), Scrambling(), Modulation(), PreCode(), RE_Map(). If you implemented this process as in Case 2, your 'Unit' is only one, i.e, ProcessCFI().

Now let's see how the specification helps you choose the test inputs. ChanCode() is the easiest unit, because 36.212 Table 5.3.4-1 fixes a 32 bit code word for each of CFI 1, 2 and 3, and CFI 4 is reserved. Three inputs therefore test ChanCode() completely, and the expected outputs come straight from the table.

The other units have inputs that the drawing does not show. Scrambling() needs the slot number and the physical cell identity, because 36.211 initialises the scrambling sequence with both at the start of each subframe. PreCode() needs the number of antenna ports. RE_Map() needs the physical cell identity and the downlink bandwidth, because they decide which resource element groups carry the PCFICH. So a complete unit definition lists these hidden inputs too, and the corner cases often sit in them, for example the largest cell identity or the smallest bandwidth.

This also shows the cost of Case 2. ProcessCFI() takes every hidden input at once, so a single test has to vary the CFI, the cell identity, the slot, the antenna ports and the bandwidth together. When such a test fails, the test cannot tell you which of the five steps is wrong.

Personally I think Unit definition and Unit testing is the most important step, but in reality I think this is frequenly the least performing steps in many project (especially in many failing project I have observed).

Why do I think the Unit test is the most important step ? There are couple of reasons as follows.

• i) Technically/Theoretically, you need to prove that your unit works as expected at least for a couple of typical inputs.
• ii) Practically, well defined unit test will be the best tools for troubleshoot at later step (i.e, in integration test or in system test)

I want to put some more comments on item ii). As far as I have experienced, I am almost 100% sure that you will come across a situation where you have some problems with the system and the trouble should be investigated in a very low level. Usually the problem caused by the low layer issues is very hard to troubleshoot. Sometimes some simple trace logging is not easy because it may cause another performance issue if you enable all the loggings at the very lower layer.

However, if you have well-defined sets of Unit test, you may be point out at least an handful of units that MAY be the root cause of the problem and you can just run those unit test to duplicate the problem. Once you are able to duplicate the problem, fixing the problem is relatively easy.

In many cases (especially in many failing project), I have seen the cases where there is not much of the well-defined Unit test. In later steps, some problem happens and even a single problem may delay the whole project in days (if you are lucky) and in weeks or even in month (if you are not lucky).

Then your next question would be 'why they fail to define/implement a good unit test cases in some project ?'.

Based on my experience/observation, it is more on human factor and management issue rather than pure technical issues. Followings are my observation.

• i) most of the unit test (unit definition as well) requires very detailed domain knowledge which is almost same level as developers. It is hard to find verification engineers with such level of knowledge (it may be almost impossible).
• ii) So, in many cases Unit Test (Unit definition) are supposed to be done by developers, but developers does not have enough time to do this while they are busy with the development. Also, not so many developers like this kind of documentation and test case creation job.
• iii) Management does not realize that this is an important step. So they does not allocate enough human resource and time for this step and just expect the whole bunch of system level test cases to be passed. In many case, management person tend to focus only on the number of test title in the spreadsheet and doesn't care much on how to achieve the testing goal.

I think the critical factors for the success of Unit test stage is the verification engineer with the detailed domain knowledge that is at least good enough to understand what the R&D engineer is talking about and understand the industry specification document in depth.

• A unit is defined by its inputs and outputs : list every input, including the ones that are only configuration.
• Typical cases prove the unit works, and corner cases find the bugs : division by zero is the corner case of Divide A by B.
• Take expected outputs from the specification : the CFI code words of 36.212 Table 5.3.4-1 are a complete test set for ChanCode().
• Small units are easier to test and to debug : a failure in Case 1 points to one function, and a failure in Case 2 points to all five steps.

Integration Test/System Test/Function Test

Literally speaking, you can say 'Integration Test' is the test at each integration step (mostly before all the units are combined to a complete system) and 'System' test is the test after all the units are assembled(Integrated) into a complete system. But in reality, the boundary between Integration test and System Test may not be as clear as it sound.

In theory, integration of a system would goes with adding one unit (or one layer) at a time as illustrated as below.

But in reality, this kind of "one layer at a time" integration might not be easy for various reason. According to my experience with supporting many project, it seems that the integration goes more like as shown below. Usually PHY and MAC integration goes one by one, but once PHY/MAC get integrated all the remaining stack gets integrated at once. In this case, it is hard to define clear distinction between integration and system test. After PHY/MAC integration, you may say Integration test and System Test would be almost same.

• Case 1 : PHY and MAC meet in one Integration step. RLC, PDCP and RRC then join that result together, in a single step into the SYSTEM bar.
• Case 2 : the MAC is split into a Hi Unit and a Lo Unit. PHY integrates with the Lo Unit first. The Hi Unit, RLC, PDCP and RRC then join the SYSTEM together.
• In both cases only the lowest step is a real integration step. Everything above it is effectively a system test.

It is also hard to draw clear boarder line between System Test and Function Test because in most case Function test is done after full system integration and done at the system level (not at Unit level).

In practice, when you call 'Function test', it usually defines several very rough functional items without defining detailed intermediate process. For example, if you are testing on Mobile phone and is doing a 'Browsing' test as a functional test level, the test case would go as follows.

• i) Turn on DUT and camp on to Test Equipment or Live Network
• ii) Open a browser in DUT (Mobile phone)
• iii) type in www.google.com at URI input
• iv) check if UE displays the contents of www.google.com

As you see here, the procedure describes each step at very high level (almost only at User interface level) and rarely defines any detailed procedure (e.g, detailed radio protocol and IP layer protocol). So even when a multiple DUTs passed the same function test, the detailed procedure (e.g, detailed protocol sequence) may not be same.

In addition, even though a DUT has passed this function test, you cannot guarantee that it went through the specific protocol sequence as you might expected.

If you have very well implemented system and all the function test cases are passed without any failure, you are lucky. But if the system integration is not so perfect (as in most case), you would see issues (failure) during the test. It tend to be very difficult to troubleshoot based on function test case document.  For any kind of troubleshoot, you need to have very clear specification for expected behavior and internal sequence (e.g, detailed protocol sequence). Otherwise, you don't even know what to fix.

So if you have only a function test document and does not have detailed knowledge of internal process, troubleshooting is extremly difficult. Unfortunately, I come across this kind of situation too often while I am working on many project.

This is where the earlier levels pay back. A function test tells you that the browsing failed, but not where. The unit tests and the layer logs tell you where. So let's keep a simple rule. When a function test fails, first repeat it with protocol logging on and find the first message that differs from the expected sequence. Then run the unit tests of the layer that sent that message. The function test finds the problem, and the lower level tests locate it.

• Integration and system test overlap in a modem : after PHY and MAC are integrated, the rest of the stack usually joins at once.
• A function test describes the user view only : two devices can pass it with different protocol sequences.
• Troubleshooting needs the expected internal sequence : without it you cannot tell which layer to fix.
• Use each test level for its own job : function tests find problems, and unit tests and logs locate them.

Reference

• 3GPP TS 36.212 v19.3.0 : Evolved Universal Terrestrial Radio Access (E-UTRA); Multiplexing and channel coding. Clause 5.3.4 Control format indicator and Table 5.3.4-1 CFI code words.
• 3GPP TS 36.211 v19.3.0 : Evolved Universal Terrestrial Radio Access (E-UTRA); Physical channels and modulation. Clause 6.7 Physical control format indicator channel.