Results of Stray Event Scanner evaluation at a customer site
As I mentioned, about a week ago I visited a customer site to find out if our Stray Event Scanner tool actually did the trick, on their codebase. That customer has kindly said I can post some results here.
Results were mixed!
| Proportion of this customer’s codebase scanned | 5% (estimate). | |
|---|---|---|
| False positives reported by the tool | 47 | |
| Of which | are things I might be able to fix | 37 |
| are intrinsic limitations | 8 | |
| are already fixed in a later development version of the tool | 2 | |
| Interestingly, 35 of these false-positives were in one smallish area of code. It seems that some Symbian OS code conforms to the norms expected by the tool, whilst some code really gets is unnecessarily overexcited. All the rest of the code we scanned only produced 12 false-positives. | ||
| Real bugs found by the tool | 14 | |
| Of which | tool was precise about the problem | 6 |
| tool was wrong, or vague, about the specific problem but did identify the right bit of faulty code | 8 | |
| Actual bugs filed in this customer’s defect-tracking system | 10 (because the other four were duplicates) | |
Conclusions. Good things first…
- As expected, the tool found plenty of bugs. Nearly all of these bugs were real problems which could cause E32USER-CBase 46 panics under some circumstances, which are awful to diagnose.
- In fact one of them was a duplicate of an existing E32USER-CBase 46 bug, which they’d been searching for for some time, but had been unable to find. Result!
- The tool broadly worked as planned. There were no hopeless incompatibilities with their source code.
- With the exception of that one component, there were far fewer false-positives than I realised.
Bad things…
- There were lots of false-positives in that one component. Many of these will be impossible to remove using the current analysis method.
- In some cases the tool was not very accurate about specifying exactly what was wrong with the code. It would identify the right area of code, but a lot of examination and thought was required to identify whether there really was a problem there. Again, there’s not much that can be done about this: stray event problems are intrinsically complicated and somebody technically experienced will be required to analyse the results.
Next steps…
- See how many of those false-positives I can remove easily
- Fix a few other minor bugs that cropped up
- Decide whether the tool is ready to leave the beta stage!