Commit | Line | Data |
---|---|---|
94dab75b MD |
1 | * Test Anti-Patterns |
2 | ||
3 | OK, there are a few patterns that have been found over and over in the | |
4 | testing code base which makes the tests flaky. Here is an incomplete | |
5 | list. Don't do that. | |
6 | ||
7 | 1) Using pidof to wait for a background application (by name) to | |
8 | disappear. | |
9 | ||
10 | Why is it flaky ? | |
11 | ||
12 | The application may be delayed after being forked, but not executed | |
13 | yet. Therefore, pidof will not find it. Use "wait" instead. | |
14 | ||
15 | 2) Using sleep as delay-based optimistic synchronization technique. | |
16 | ||
17 | Why is it flaky ? | |
18 | ||
19 | Everything that needs to happen before/after other things need to | |
20 | be explicitly synchronized using e.g. a file (used as a flag). | |
21 | Sleep is just an indicator of a minimum arbitrary delay, but | |
22 | machine load and scheduling can actually mess up the real delay | |
23 | between applications. Use explicit synchronization points. Never | |
24 | sleep. | |
25 | ||
26 | 3) Using killall on a background application. | |
27 | ||
28 | Why is it flaky ? | |
29 | ||
30 | Similarly to pidof, killall may run before the background application | |
31 | executes, thus failing to find it. Store the application PID after it | |
32 | it launched in background into a temporary variable for later use | |
33 | by kill and wait. | |
34 | ||
35 | 4) Using wait ${!} to wait for completion of many background | |
36 | applications. | |
37 | ||
38 | Why is it flaky ? | |
39 | ||
40 | It just waits for the last application put in background. Use | |
41 | "wait" to wait for all background applications. | |
42 | ||
43 | 5) Forgetting wait at the end (or error return path) of a test phase | |
44 | that has background applications. | |
45 | ||
46 | Why is it flaky ? | |
47 | ||
48 | Those application may interact with the following testing phases, | |
49 | thus skewing the results. | |
50 | ||
51 | 6) Not grepping into the entire code base for similar patterns. | |
52 | ||
53 | When you find a problematic coding pattern, chances are it appears | |
54 | elsewhere in the testing code base. Please fix it everywhere! | |
55 | ||
56 | 7) Introducing a utility abstraction without changing all open coded | |
57 | similar code path. | |
58 | ||
59 | When an abstraction for e.g. starting and stopping the session daemon | |
60 | is introduced as a utility (e.g. utils.sh), future changes will | |
61 | assume that all the testing code base is using this abstraction. | |
62 | Leaving a few custom open-coded sites of duplicated code around is a | |
63 | good way to make it a pain to update the abstraction in the future. |