Ask TechDesign PM: Testing
Q: How can I improve app testing for my consumer IoT product?
A: Based on consumer reviews for IoT products, I would estimate that over 70% of complaints were related to app quality and bugs. With a complicated system like IoT that involves product-level integration with hardware, software and cloud, the risks are high for potential bugs to devastate your customer’s user experience. To prevent this from happening, a solid testing flow is a must. It should at least consist of (1) a complete set of test cases that simulates how a regular user would use it, and (2) a bug severity rating and tracking mechanism.
For (1), it is useful to build two sets of test cases – one for rapid testing with only the major function flows (e.g., device setup, data dashboard display, OTA, factory reset) and the other for full testing. They serve different purposes – with major bugs in need of urgent patching, you might want to release an update ASAP with only the rapid test done and run the full test later. To ensure the app’s functionality under different hardware environments, app testing using various smart devices (e.g., smartphone, tablet) will also be required. iOS is simpler with fewer device models, but you still need to decide up to which iOS version your app and hardware will be compatible with. The testing environment for Android is much more complicated with countless brands and models on the market in various screen sizes. While there are online app testing platforms that emulate how your app would look like running on different devices, our recommendation would be to study which smart devices are widely used in your target market first, and to obtain physical devices of brands which accounted for at least 90% of coverage in terms of your customer demographics. Another tip is to make use of software such as Firebase or Instabug in understanding the type of device models your customers are using.
For (2), having an assessment mechanism in place to determine the handling priority of a bug is essential – as there is no such thing as a bug-free software, you want your engineering resources to be allocated wisely. In most cases, it can be calculated based on the level of disturbance and occurring frequency of the bug in a matrix form. For bug tracking, software such as JIRA or Mantis are already equipped with all the necessary functions for recording, managing and sharing of bug information across your teams. Coupled with a handling priority mechanism, you can build a testing flow to decide whether an update should be released based on the testing results. For instance, if two major or four middle-class bugs were identified during rapid testing, the app update should be postponed until a full test is carried out. Finally, since IoT product development is often cross-disciplinary, it might be a good idea to host QA meetings on a regular basis with app, cloud, and firmware/hardware engineers gathered to figure out the bug’s root cause and required countermeasures more efficiently.
Q: How should I prepare my software testing plan?
A: If you have built-in, own-developed software and plan to outsource your hardware development, you will need to prepare a testing plan to make sure all functionalities perform as expected upon delivery. In general, manufacturers and product design companies know how to perform the typical hardware function testing in hardware development but need a testing plan to experiment with your applications when resting the working prototype. Here are some tips to create a dynamic testing plan:
- Prepare the test documents – manufacturers’ QA teams usually have no idea on the entire scope of your functions and what the test objectives are. Therefore, you should include all elements of applications and functions, tools, test cycle, defect management and set-up environment in detail.
- Create the test scenarios – for example, if you develop a smart office access control system that uses facial recognition, there can be numerous different scenarios that show alerts, sound alarms or display messages. Aligning the testing document with the different test cases can help the test team better find out if something exceeds the scope of the design.
- Create a new image in-line with the software version – imagine hundreds or thousands of devices on the production line waiting to install software updates, how long might it take and cost for this long and complicated setup process? You can facilitate the installation process by building your software into the docker image. The tamperproof docker image allows manufactures to optimize your production flow and save you a lot on test charges!
Q : How to guarantee the quality of your PCB?
A: After designing the circuit, the next step is to make the circuit into a PCB (printed circuit board). The PCB is the basic element of the entire electronic machine, and its good design and manufacturing will affect the quality of the entire product. It is important to ensure that the PCB design accounts for the convenience of machine production, meets safety requirements, uses materials that meet the service life requirements of the product, etc.
The following is a description of the testing required to guarantee the quality of your PCB:
1. Functional test – for open & short circuits – this checks whether the PCB traces made are correct
2. Electrical characteristics – check the electrical characteristics of the design and of the PCB materials. The general test items are i) Characteristic impedance ii) Withstand voltage iii) Dielectric insulation resistance iv) Surface insulation resistance
3. Reliability – Check whether the design, materials and processes can be used in different environments. The general test items are i) Tensile test ii) Thermal stress iii) High humidity iv) Thermal shock v) Corrosion resistance.
4. Mechanical – Can be understood as the degree of damage that can be sustained under assembly or disassembly conditions. The general test items are i) flexibility ii) Impact test.
You won’t need to conduct every test for every product so choose the appropriate test according to your product’s attributes.
Q: What tests are needed after circuit board production?
A: During both design and manufacture phases, many tests are required to ensure product quality and reliability, without which could result in RMA (Return Merchandise Authorization) issues. Many malfunctions or manufacturing defects occur during the SMT process often due to unreliable solder connections such as non-wetting issues (when there is no solder on either the lead or the pad, e.g., from solder skip, cold soldering, solder short, solder insufficient, etc.) or components missed/shifted. Systematic testing after PCB Assembly is critical to reduce manual debug effort and increase defect coverage.
What are the tests required after PCB assembly to screen and inspect for defects?
- AOI (Automated-Optical-Inspection) – Scan each PCBA to detect any faults that can be found by visual inspection. This reduces labor costs, while the portability of the program makes it flexible enough to be used anywhere in the line. It is also sufficiently accurate to detect small deviations in position
- X-Ray (AXI) – Similar to AOI, x-ray imaging can identify hidden solder joints, component shields, and high-density double-sided boards. Suitable for inspection of complicated boards, especially ones with BGAs, CGAs, CSPs or components that are under RF shields
- FPT (Flying-Probe Test) – This test requires no test fixtures or NRE but has long test times and high unit test costs. It is suitable for products that are in the early stages of development, have low-volume orders or have high complexity boards
- ICT (In-Circuit-Test) – This test requires NRE for developing extra fixtures, but can quickly perform electrical logic functionality testing, which means a low unit test cost. It is suitable for products planning to do mass production
- FVT (Function Verification Test) – This is a power on test to validate the software is functioning properly, usually tested before enclosure assembly
AOI & ICT tests are typically applied in normal electronics products. AXI is usually combined with FPT &FVT for high-end and low volume products.
Q: What are the test plans for the certification of RF products?
A : Any product with Radio functionality will need to obtain a different certification for each country it will be sold in. No matter if it’s Bluetooth, Wifi or 4G LTE, all need to pass national radio certification. For example, FCC is needed for North America and CE for Europe. Although every protocol has slightly different test items, the main test plans are the same and shown as below:
1. Conducted Emissions Test: We need to make sure all the emissions from the EUT (equipment under test) meet the standards of conducted measurement without antenna. Usually RF cables are used to connect the EUT to measure emission parameters.
2. Radiated Emissions Test: We need to make sure all the emissions from the EUT meet the standards of radiated measurement. The EUT is set 3m away from the receiving antenna which is set at varying heights of 1m to 4m. Then a spectrum analyzer will scan different frequencies to find the highest power emission power and ensure it does not exceed the standard limit.
3. Occupied Bandwidth Test: The test ensures that the transmitter filter is well designed and the clock of the EUT is working properly. If the clock rate is too high, this may result in a wide occupied bandwidth (OBW) and malfunction of the EUT.
4. Maximum Peak Output Power Test: To ensure the peak output power is limited and doesn’t interfere with other devices, the maximum peak power must not exceed the standard limit.
5. Band Edge Test: In order to avoid interference between different bands, radiated emissions which fall in the restricted bands, must also comply with the radiated emission limits.
6. Power Density: In order to avoid damage to people or other devices, the radio power strength per unit square (mW/cm2) is tested at several specific frequencies and distances to ensure it does not exceed the standard limit.
7. Antenna Port Emission: To ensure spurious noise does not exceed the standard limits under RF peak output power, the conducted spurious emissions are tested using a spectrum analyzer at low, middle, and high channels (frequency range will depend on the protocol).
Q: How do I decide what reliability testing items are needed for my device level product?
A: The purpose of reliability testing is to maintain product quality over a given lifecycle. Also, reliability testing can reduce the risk of having to recall your product from the market which is not only costly but very damaging to your reputation. Based on your product category, the reliability test items will vary accordingly. Make sure you understand your product well and set the corresponding test items.
Reliability tests usually take place in the stage of DVT (Design Verification Test). So you’ll need to prepare your reliability test plan before DVT stage. Use DFR (Design For Reliability) concept employing a joint design engineering perspective (across different engineering disciplines) to develop your reliability strategy. Device level reliability will need to consider technical viewpoints from layout, component, system, manufacturing, and thermal perspectives. The most common reliability tests include environment test, mechanical test, and package and delivery test. Detailed test items range from cold/ heat test, temperature cycling test, HALT& HASS, container compression test, drop test, vibration test, just to name a few. Your final test plan must of course fit your product.