Automated Software Testing Adoption and Trends
Automated software testing reduces human error and speeds up e icient delivery, but how integrated is it in modern software engineering organizations, and which trends stand to impact it? Find out.
One minute insights:
Respondents commonly automate API testing and use automated testing continuously throughout the development cycle
Many respondents’ organizations chose to automate testing to improve product quality and increase deployment speed
Respondents report higher test accuracy after automating testing, but some struggle with implementation
Tech leaders predict generative AI will impact the future of automated testing and that we will see a change in QA headcount and daily responsibilities
A range of automated software testing types and cadences are in place
API testing (56%), integration testing (45%) and performance testing (40%) were among the most common types of automated software testing currently in use in respondents’ organizations.
What type(s) of automated software testing does your organization currently use? Select all that apply.
Graphical user interface (GUI) testing 30% | Accessibility testing 29% | Regression testing 27% | Data-driven testing 25% | Unit testing 24% | Black-box testing 20% | Security testing (e.g., penetration testing) 19% | Scalability testing 12% | Smoke testing 9% | Stability testing 9% | Recovery testing 9% | Gray-box testing 8% | White-box testing 6% | Can not say 6% | Type(s) of automated software testing not listed here 4% | Keyword-driven testing 3% | Don’t know 3% | Other 0%
n = 248
40% of respondents automate software testing continuously during their development cycle. Other common timing options included at set milestones (38%), like prior to 40% a commit, and at set intervals (32%), such as a specific number of days.
When during your development cycle do you automate software testing? Select all that apply.
Can not say 5% | Don’t know 2% | Cadence not listed here 2% | Other* <1%
*Other includes: “On new sprint”
n = 248
Please join or sign in to view more content.
By joining the Peer Community, you'll get:
- Peer Discussions and Polls
- One-Minute Insights
- Connect with like-minded individuals
Question: What are your final thoughts on automated software testing?
Automated testing is necessary to find bugs and security issues in a timely, efficient manner. We would be much slower to release new software without it.
This technology is still in its infancy. I’m anxiously looking forward to the coming advances in automated testing in the coming months and years.
Organizations automate software testing to improve product quality and increase deployment speed
60% of respondents said improving product quality was among their organization’s reasons to automate software testing. 58% said their 60% organization was influenced by a desire to increase deployment speed.
Other common reasons for automating testing included improving departmental agility (41%), facilitating CI/CD (36%) and alleviating the QA team’s workload (29%).
What are the reasons your organization decided to automate software testing? Select all that apply.
Address bugs/product issues 26% | Follow industry standards 23% | Free up resources to expand product 19% | Reduce engineering spend 18% | Reduce headcount 18% | Improve cybersecurity 15% | Can not say 3% | Don’t know 3% | Reason(s) not listed here 1% | Other* <1%
*Other includes: “Reduce human errors”
n = 248
Question: What are your final thoughts on automated software testing?
[Automated software testing is] essential to ensure software development can scale, that changes can be released rapidly and to be sure many tests are covered despite human restrictions on headcount, working hours and accuracy.
It’s easily one of the most important hygiene practices in any Software Development Environment. Without some form of automated testing, you’re essentially flying completely blind to your most important workflows breaking.
Leaders report higher test accuracy after automating testing, but some struggle with implementation
Higher test accuracy (43%), increased agility (42%) and wider test coverage (40%) are among the most significant benefits respondents have seen at their organizations since automating testing.
What are the most significant benefits you have seen in your organization since automating software testing? Select up to three.
Fewer bugs in final product 27% | Better scalability 25% | Better adherence to CI/CD 19% | More positive developer experience 18% | Simplified SDLC 15% | Improved cybersecurity 14% | Better ROI on software products 12% | I haven’t seen any benefits 6% | Don’t know 3% | Can not say 2% | Other* 1% | Benefit(s) not listed here <1%
n = 248
*Other includes: “Faster TTM [time to market],” “Too soon to say”
When asked about the most significant challenges to automated software testing deployment in respondents’ organizations, struggles with implementation (36%), automation skill gaps (34%) and high upfront costs (34%) were among the most commonly reported.
What are the most significant challenges you have experienced with automated software testing deployment in your organization? Select up to three.
Hard-to-define ROI 23% | Lack of trust in automation tools/vendors 21% | Product complexities make it hard to automate testing 19% | High maintenance costs 16% | I have not experienced any challenges 10% | Lack of executive buy-in 7% | Cybersecurity problems 6% | Lack of employee buy-in 4% | Challenge(s) not listed here 2% | Can not say 2% | Don’t know 2% | Other* <1%
*Other includes: “Firewall or internal security preventing execution of automated script”
n = 248
Question: What are your final thoughts on automated software testing?
There is an increasing need to have automated testing. The challenge is the upfront investment to implement such framework and to maintain it. AI can positively impact the capability of automated software testing.
The practice has lost a lot of its sophistication and e ective tooling over the past decade or two. The push towards developer-led testing has reduced the availability of qualified resources who can help implement truly effective testing practices. We are struggling to find people and even find ideas of how to implement effective testing.
Tech leaders predict generative AI will impact the future of automated testing and that we will see a change in QA headcount and daily responsibilities
Respondents predict that, in the next three years, generative AI (69%) and low-code/no-code (53%) will impact automated software testing.
In the next three years, what industry trends do you believe will impact automated software testing? Select all that apply.
Citizen development 18% | Composability 13% | Industry trend(s) not listed here 6% | Don’t know 4% | Can not say 3% | Other 0%
n = 248
Thinking only of generative AI’s potential impacts on automated software testing in the next three years, respondents believe it will predict common issues or bugs (57%), analyze test results (52%) and suggest error solutions (46%).
In the next three years, how do you think generative AI will impact automated software testing the most? Select up to three.
Keep tests up to date 28% | Create tests 27% | I don’t think it will 5% | Impact(s) not listed here 4% | Don’t know 3% | Can not say 2% | Other 0%
n = 248
Considering automated software testing’s potential impact on the QA department or roles, respondents believe that, in the next three years, we will see a reduction in QA headcount (40%) and a fundamental change to QA’s daily responsibilities (40%).
In the next three years, how do you believe automated software testing will impact the QA department/roles? Select all that apply.
Elimination of QA department/role 23% | I don’t believe it will 7% | Can not say 3% | Don’t know 3% | Impact(s) not listed here 2% | Other 0%
n = 248
Question: What are your final thoughts on automated software testing?
The [automated software testing] process still has a way to go to completely replace individuals, but within 5 years and certainly 10 it may completely take over the function entirely.
Automated testing is going to fundamentally change the role of QA testing in [the] SDLC process and make the process a lot more robust and agile in the short run. In the long run, in combination with Generative AI, the level of automation may not be only limited to testing execution but will expand to script generation and test result analysis.
Want more insights like this from leaders like yourself?
Click here to explore the revamped, retooled and reimagined Gartner Peer Community. You'll get access to synthesized insights and engaging discussions from a community of your peers.
Respondent Breakdown
Note: May not add up to 100% due to rounding
Respondents: 248 IT and software engineering leaders involved in managing and/or making decisions related to their organization’s automated software testing