Testing
The Testing Research Group is one of the largest groups of its kind in the UK developing innovative approaches to software testing and quality assurance.
Drawing equally on theory and practice, the aim of the research group is to produce practical software engineering solutions for industry, improving the productivity of developers and testers, and improving the reliability of software systems.
The group investigates a number of areas, including model-based testing, search-based testing, security testing, reverse engineering, model-driven engineering, multi-agent modelling and XML data processing.
Crossing the boundaries between the recovery and verification of designs, to the generation of economical test-suites that exercise software completely, our research has created innovative tools for industry, including: EvoSuite, IGUANA, JWalk, Broker@Cloud, HOLTestGen (test generation), StateChum, SUMO (reverse engineering) and FLAME (agent-based modelling). Our work is funded by the EU, EPSRC, InnovateUK and industry.
Research themes
-
Model-based Testing
-
This formal testing method generates complete functional test suites from a model/specification expressed as an extended finite state machine; and offers guarantees of correctness once testing is complete. The work has been used at Daimler for testing automotive systems designed using Harel Statecharts, applied to testing the gate-logic on low-power ARM chips, used for unit testing Java and also used to test software services for SAP and SingularLogic in the Cloud. Several software testing tools have grown out of this work, including StateTest, JWalk and Broker@Cloud.
Leaders: Prof Hierons, Dr Simons, Dr Bogdanov
-
Search-based Testing
-
Search-based testing uses a problem-specific fitness function to guide a search process (such as a genetic algorithm) to automatically generate test cases or select from a given set of test cases. Fitness is judged by instrumenting the tested code to measure what is covered by the tests. The approach has been applied to large C programs and also to object-oriented programs in Java. Several software tools have grown out of this work, including EvoSuite for Java, and IGUANA for C.
Leaders: Prof Hierons, Dr Walkinshaw, Prof McMinn
-
Property-based Testing
-
Property-based testing provides a high-level approach to testing and is a complementary technique to model-based testing. Rather than focusing on individual test cases to encapsulate the behaviour of a system, in property-based testing this behaviour is specified by properties, expressed in a logical form. The system is then tested by checking whether it has the required properties for randomly generated data, which may be inputs to functions, sequences of API calls, or other representations of test cases. This work has resulted in a code coverage tool (Smother) and a mutation testing tool (Mu2) for the Erlang language.
Leaders: Prof Derrick, Dr Bogdanov, Dr Taylor
-
Security Testing
-
Security encompasses information security, software engineering, security engineering, and formal methods. Our research in this area investigates all security aspects of distributed and service-oriented systems. This includes applied security aspects, such as access control or business-process modelling, as well as fundamental aspects, such as novel static and dynamic approaches for ensuring the security of applications. We participate in the development of interactive theorem proving environments for Z (HOL-Z) and UML/OCL (HOL-OCL, which is integrated into a formal MDE tool-chain) and a model-based test-case generator (HOL-TestGen).
Leaders: Professor Clark
-
Model-Driven Engineering
-
The aim of this approach is to create systems not by writing program code, but rather by generating it from high-level abstract models that are closer to end-user requirements. What initial models and languages should be chosen? How should models be checked? How should they be folded together to create more detailed system specifications? How should the transformation rules be verified? Our work so far has investigated dependently-typed languages as a means of verifying model transformations, has generated simple information systems from requirements and has also generated platform-specific test suites for SOAP or REST-based software services in the Cloud.
Leaders: Dr Simons, Prof Derrick
-
Reverse Engineering
-
This novel approach recovers specifications from legacy software systems. The reverse engineering method collects traces of the system's execution and performs grammar inference on the traces, to detect behavioural regularity. From this, finite state models are constructed, which allow further hypotheses about the specification to be generated and tested. The approach has been applied to recover both flat and nested state specifications, and used for the supervised re-modularisation of software systems. Software tools include StateChum (for reverse engineering) and SUMO (for supervised re-modularisation).
Leaders: Dr Walkinshaw, Dr Bogdanov, Dr Hall, Dr Taylor
-
XML Data Processing
-
The rise of distributed data processing in the Cloud has led to a resurgence of non-relational key-value and tree-structured data formats like XML. This has opened up new research areas in XML data compression, distribution and storage, with associated issues of data indexing and query-processing. We have developed algorithms for fast searching in distributed XML databases using sparse binary matrix indexing; and for trust-based access control to XML data with dynamic learning. The technology has been applied in Botswana for the distributed mobile phone hosting of compressed XML databases.
Leaders: Dr North
-
Software Engineering
-
Why do software development projects frequently fail, and what can be done about it? This strand of our research has investigated both traditional and agile software development methods, to see how they work in practice. We have published internationally-famous critiques of the UML notation and associated development process, and also of the Agile method known as XP (eXtreme Programming). We have collaborated with work psychologists to monitor the behaviour and effectiveness of developer teams in our own software company Genesys Solutions, revealing how the constitution of teams affects how well they work.
Leaders: Dr Cowling, Dr Simons
Core members
Academic staff
- Prof. Phil McMinn - Suite Health Testing, Flakiness Testing and Search-based Software Engineering (Head of Group)
- Dr Kirill Bogdanov - Reverse Engineering
- Prof. John Derrick - Property-based Testing
- Prof. Rob Hierons - Software Testing
- Dr Siobhán North - Evaluation of Algorithms
- Dr José Miguel Rojas Siles - Automated Software Testing and SE Education
- Dr Anthony Simons - Model-Driven Engineering and Testing
- Dr Neil Walkinshaw - Exploratory Black-Box Testing
- Dr Ramsay Taylor - Testing Embedded Systems
- Dr Donghwan Shin - Testing ML-enabled Systems
Research staff
- Mr Islam Elgendy (Teaching and Research Assistant)
- Dr Michael Foster (Research Associate in CITCOM)
- Dr Maciej Gazda (Research Associate)
- Mr Owain Parry (Research Associate in Flaky Software Tests)
- Sanjeetha Pennada (Research Associate in Testing for Machine Learning Autonomous Systems)
- Dr Yasmeen Rafiq (Research Associate in Verifiability Node)
- Mr Areeb Sherwani (Teaching and Research Assistant)
- Affiliated academics
-
- Prof Graham Birtwistle (University of Calgary)
- Eur Ing Dr Anthony J Cowling (Honorary)
- Dr Dimitris Dranidis (International Faculty)
- Dr George Eleftherakis (International Faculty)
- Prof Gordon Fraser ( University of Passau)
- Prof W Michael L Holcombe (Honorary)
- Prof Petros Kefalas (International Faculty)
- Dr Mariam Kiran (Lawrence Berkeley National Laboratory, California)
- Dr Raluca Lefticaru
- Dr Pietro S Oliveto (EPSRC Early Career Fellow)
- Dr Ioanna Stamatopoulou (International Faculty)
- Dr José Miguel Rojas (University of Leicester)
Publications
- Academic articles
-
Here you can find research publications for the Testing Research Group, listed by academic. The head link navigates to the official web page for the relevant academic (with highlighted favourite publications). The remaining links navigate to their DBLP author page, their Google Scholar citations page and optionally a self-maintained publications page.
Academic staff
Affiliated academics
Research staff
Dr Raluca-Elena Lefticaru DBLP GoogleScholar Dr Krishna Patel DBLP Affiliated researchers
Dr Mike Croucher GoogleScholar Dr Paul Richmond DBLP GoogleScholar Web Visiting academics
Dr Mathew Hall - Research theses
-
Here you can find recently-published PhD (and MPhil) theses, which have been deposited in the White Rose eTheses Online repository. Follow links to the abstract, and then to the full thesis (if public) or to a request form (if a time-embargo restriction has been placed on public release).
Recently published theses
Seminars
The Testing Research Group seminar series is a public-facing forum which gives details of forthcoming seminars by members of the group and by our research visitors.
This forum may be viewed by the general public, but only authorised members of the Testing Research Group may post new messages. Messages emailed to the testing-seminar-group@sheffield.ac.uk alias will appear in the newsfeed below. Authorised members can also interact with the newsfeed directly to post new messages or extend existing threads with new content.