Ask TechDesign PM: Testing

Q: How can I improve app testing for my consumer IoT product?

A: Based on consumer reviews for IoT products, I would estimate that over 70% of complaints were related to app quality and bugs. With a complicated system like IoT that involves product-level integration with hardware, software and cloud, the risks are high for potential bugs to devastate your customer’s user experience. To prevent this from happening, a solid testing flow is a must. It should at least consist of (1) a complete set of test cases that simulates how a regular user would use it, and (2) a bug severity rating and tracking mechanism.

For (1), it is useful to build two sets of test cases – one for rapid testing with only the major function flows (e.g., device setup, data dashboard display, OTA, factory reset) and the other for full testing. They serve different purposes – with major bugs in need of urgent patching, you might want to release an update ASAP with only the rapid test done and run the full test later. To ensure the app’s functionality under different hardware environments, app testing using various smart devices (e.g., smartphone, tablet) will also be required. iOS is simpler with fewer device models, but you still need to decide up to which iOS version your app and hardware will be compatible with. The testing environment for Android is much more complicated with countless brands and models on the market in various screen sizes. While there are online app testing platforms that emulate how your app would look like running on different devices, our recommendation would be to study which smart devices are widely used in your target market first, and to obtain physical devices of brands which accounted for at least 90% of coverage in terms of your customer demographics. Another tip is to make use of software such as Firebase or Instabug in understanding the type of device models your customers are using.

For (2), having an assessment mechanism in place to determine the handling priority of a bug is essential – as there is no such thing as a bug-free software, you want your engineering resources to be allocated wisely. In most cases, it can be calculated based on the level of disturbance and occurring frequency of the bug in a matrix form. For bug tracking, software such as JIRA or Mantis are already equipped with all the necessary functions for recording, managing and sharing of bug information across your teams. Coupled with a handling priority mechanism, you can build a testing flow to decide whether an update should be released based on the testing results. For instance, if two major or four middle-class bugs were identified during rapid testing, the app update should be postponed until a full test is carried out. Finally, since IoT product development is often cross-disciplinary, it might be a good idea to host QA meetings on a regular basis with app, cloud, and firmware/hardware engineers gathered to figure out the bug’s root cause and required countermeasures more efficiently.


Q: How should I prepare my software testing plan?

A: If you have built-in, own-developed software and plan to outsource your hardware development, you will need to prepare a testing plan to make sure all functionalities perform as expected upon delivery. In general, manufacturers and product design companies know how to perform the typical hardware function testing in hardware development but need a testing plan to experiment with your applications when resting the working prototype. Here are some tips to create a dynamic testing plan:

  • Prepare the test documents – manufacturers’ QA teams usually have no idea on the entire scope of your functions and what the test objectives are. Therefore, you should include all elements of applications and functions, tools, test cycle, defect management and set-up environment in detail.
  • Create the test scenarios – for example, if you develop a smart office access control system that uses facial recognition, there can be numerous different scenarios that show alerts, sound alarms or display messages. Aligning the testing document with the different test cases can help the test team better find out if something exceeds the scope of the design.
  • Create a new image in-line with the software version – imagine hundreds or thousands of devices on the production line waiting to install software updates, how long might it take and cost for this long and complicated setup process? You can facilitate the installation process by building your software into the docker image. The tamperproof docker image allows manufactures to optimize your production flow and save you a lot on test charges!


Q : How to guarantee the quality of your PCB?

A: After designing the circuit, the next step is to make the circuit into a PCB (printed circuit board). The PCB is the basic element of the entire electronic machine, and its good design and manufacturing will affect the quality of the entire product. It is important to ensure that the PCB design accounts for the convenience of machine production, meets safety requirements, uses materials that meet the service life requirements of the product, etc.

The following is a description of the testing required to guarantee the quality of your PCB:
1. Functional test – for open & short circuits – this checks whether the PCB traces made are correct
2. Electrical characteristics – check the electrical characteristics of the design and of the PCB materials. The general test items are i) Characteristic impedance ii) Withstand voltage iii) Dielectric insulation resistance iv) Surface insulation resistance
3. Reliability – Check whether the design, materials and processes can be used in different environments. The general test items are i) Tensile test ii) Thermal stress iii) High humidity iv) Thermal shock v) Corrosion resistance.
4. Mechanical – Can be understood as the degree of damage that can be sustained under assembly or disassembly conditions. The general test items are i) flexibility ii) Impact test.

You won’t need to conduct every test for every product so choose the appropriate test according to your product’s attributes.


Q: What tests are needed after circuit board production?

A: During both design and manufacture phases, many tests are required to ensure product quality and reliability, without which could result in RMA (Return Merchandise Authorization) issues. Many malfunctions or manufacturing defects occur during the SMT process often due to unreliable solder connections such as non-wetting issues (when there is no solder on either the lead or the pad, e.g., from solder skip, cold soldering, solder short, solder insufficient, etc.) or components missed/shifted. Systematic testing after PCB Assembly is critical to reduce manual debug effort and increase defect coverage.

What are the tests required after PCB assembly to screen and inspect for defects?

  • AOI (Automated-Optical-Inspection) – Scan each PCBA to detect any faults that can be found by visual inspection. This reduces labor costs, while the portability of the program makes it flexible enough to be used anywhere in the line. It is also sufficiently accurate to detect small deviations in position
  • X-Ray (AXI) – Similar to AOI, x-ray imaging can identify hidden solder joints, component shields, and high-density double-sided boards. Suitable for inspection of complicated boards, especially ones with BGAs, CGAs, CSPs or components that are under RF shields
  • FPT (Flying-Probe Test) – This test requires no test fixtures or NRE but has long test times and high unit test costs. It is suitable for products that are in the early stages of development, have low-volume orders or have high complexity boards
  • ICT (In-Circuit-Test) – This test requires NRE for developing extra fixtures, but can quickly perform electrical logic functionality testing, which means a low unit test cost. It is suitable for products planning to do mass production
  • FVT (Function Verification Test) – This is a power on test to validate the software is functioning properly, usually tested before enclosure assembly

AOI & ICT tests are typically applied in normal electronics products. AXI is usually combined with FPT &FVT for high-end and low volume products.

Leave a Reply

Your email address will not be published. Required fields are marked *