Automate or Not? — Test Automation Decision Tool
🤖 AI-powered · Free · No signup
Know what to automate<br>in seconds.
Drop in a test case and let AI evaluate it against a research-backed<br>decision tree. One click — get an Automate or Keep Manual answer instantly.
⚡ Answer in seconds
🤖 AI reads your test case
Backed by research
⚡ AI Evaluation<br>✏️ Answer manually
📋
Drop a test case here — CSV , XML , JSON or .feature , or paste below
✨ No test case handy? Try an example
✨ Evaluate with AI →
AI pre-fills every factor it can — you just confirm the rest.
Name your test case
Start Evaluation →<br>Walk the decision tree yourself, one factor at a time.
🌳 See how the decision tree works
Reading your test case…
Your test case is being routed through the decision tree below.
AutomateOrNot.dev
Test Automation Decision Tool
About This Tool
TitleEconomics of Test Automation – Test case selection for automation
AuthorDavid Lindholm
UniversityLinköping University — Dept. of Computer and Information Science
Year2019 · Master's thesis, 30 ECTS
SupervisorAzeem Ahmad
Industrial partnerSectra Imaging IT Solutions Ltd
What the research did
The thesis developed and evaluated a structured method for selecting which test cases<br>to automate. Starting from Garousi & Mäntylä's 43-factor checklist, David conducted<br>industry interviews to refine it into a 23-factor decision tree with 8 decision points,<br>validated on 12 regression test cases at Sectra. The ROI was then calculated for three<br>automated test cases to confirm economic benefit.
Key findings
65% of the original 43 factors were confirmed as important by industry practitioners.
Automated tests selected by the decision tree reached positive ROI in 0.5–4 years.
3 organisational benefits were confirmed: less human testing effort, cost reduction, and shorter release cycles.
8 of 12 regression test cases were recommended for automation; 3 were not; 1 partially.
How this tool works
The 23 factors are grouped into 8 decision points (F1–F8), ordered by their mean importance<br>score from industry interviews. At each point you evaluate its factors; their majority result<br>(Agree or Disagree) routes you down the tree exactly as in Appendix 7.E of the thesis. You only<br>ever see the decision points on your path, and the tool stops the moment a verdict is reached —<br>for example, disagreeing at F1 (the test oracle) ends immediately with "Don't Automate", since a<br>test whose result a computer can't reliably judge should not be automated at all.
🌳 View the full decision tree
📄<br>Read the full thesis (DiVA)
🔗<br>View David Lindholm's Portfolio
Connect on LinkedIn
How the Decision Tree Works
Start at F1 . At each decision point, your answers give an overall<br>✓ Agree or ✗ Disagree,<br>which routes you along the tree until you reach a verdict:<br>Automate or Keep Manual.<br>You only ever walk the points on your path.
Why is the tree structured this way?
The tree is not arbitrary — the 8 decision points are ordered by their<br>mean importance score from interviews with industry practitioners<br>at Sectra Imaging IT Solutions. Points that experts consistently rated as most<br>important appear first; lower-ranked points only appear if the earlier gates don't<br>produce a verdict.
F1
Test oracle & determinism is the hard gate.<br>If a computer cannot reliably evaluate the test result — because outcomes are<br>non-deterministic, require human judgement, or are full of false positives —<br>automation is fundamentally broken regardless of any other factor. The tree<br>stops here immediately with "Don't Automate".
F2
Strong business value is sufficient on its own.<br>Once the oracle is sound, clear economic benefit plus high-risk/important coverage<br>is the clearest signal in the research to automate. If practitioners agree,<br>the tree goes straight to "Automate" without investigating ease or product traits.
F3–F6
When business value is weak, the left branch looks for other justifications.<br>F3 asks if automation is easy (low maintenance cost). If so, the tree pivots right<br>to check product characteristics (F7/F8). If not, it continues down:<br>F4 covers high human-error risk, F5 a knowledge gap, F6 whether it's a<br>performance/load test — each a standalone reason to automate even without<br>strong business value.
F7–F8
Product context and test type are the tiebreakers.<br>Frequent releases, many configurations, or a long-lived product (F7) justify<br>the long-term investment. Smoke and build-verification tests (F8) run on every<br>build — their repetition alone makes automation worthwhile even when no other<br>factor clearly favours it.