prodei 29/2/2008 temas para doutoramento: projecto amber-itest (an automated model-based user...

ProDEI 29/2/2008

Temas para doutoramento: Projecto Amber-iTest (An Automated Model-Based User Interface Testing Environment)

Ana Paiva & João Pascoal Faria{apaiva,jpf}@fe.up.pt

Grupo de Engenharia de Softwarewww.fe.up.pt/~softeng

Departamento de Engenharia Informáticawww.fe.up.pt/si/unidades_geral.visualizar?p_unidade=151

Faculdade de Engenharia da Universidade do Portowww.fe.up.pt

2Ana Paiva & João Pascoal Faria, ProDEI – 29/2/2008

Research interests

Software Quality – our society depends ever more on software, so its crucial to assure its quality through various means

Product quality – testing and certification of software products (the customer perspective) particularly model-based user interface testing partnerships: MSR/FSE, etc.

Process quality – improving the software development process to assure the production of high-quality software in a predictable and cost-effective manner (the supplier perspective) particularly the combination of CMMI/PSP/TSP/Agile

methods for fast process improvement partnerships: SEI, etc.

Formal methods – mathematically based software specification and verification methods particularly lightweight and automated methods integrated

with commonly used development environments partnerships: UM, etc.


Model-based User Interface Testing

UI testing is a fundamental but currently very time consuming and poorly systematized V&V activity

Microsoft: number of testers ≈ number of developers

Model-based testing can help to systematize and automate the UI testing process (through automatic test case generation from formal models), but poses specific challenges:

Test case explosion problem: how to generate a test suite of manageable size that still assures adequate UI testing?

How to bridge the gap between the model and the implementation?

How to model GUIs? How to reduce the effort in the construction of the model?


An example of a MBUIT environment

Model-based testing tool: Spec Explorer (FSE/MSR) Automates test case generation and execution for API

conformance testing Test sequences are derived from a FSM which is generated

by bounded exploration of the model

Formal modelling language: Spec# (FSE/MSR) Extends C# with contracts and high-level constructs Can be used as modelling or programming language

GUI testing extensions (from previous work at FEUP) GUI modelling techniques, promoting abstraction and reuse FSM reduction tool for hierarchical GUIs, to handle the

state/test case explosion problem GUI mapping tool, to bridge the gap between the model

and its implementation


GUI modelling with Spec#: example

class FindDialog {

string findWhat = "";bool matchCase = false; UpDown direction = Down;

[Action] void SetFindWhat(string value)requires IsEnabled("FindDialog"); { findWhat = value;}

[Action] void FindNext()requires IsEnabled("FindDialog") && findWhat != ""; { Notepad.Find(findWhat, direction, matchCase);}

[Action] void Cancel() requires IsEnabled(“FindDialog”){ WindowManager.RemoveWindow("FindDialog");}…

}

(imaginar em inglês)

Janela -> classe

Botão ou opção menu -> acção

Controlo com estado -> var. estado + acções


Current issues

Time and effort required to build the GUI modelsReluctance of GUI testers and modellers in writing textual formal specifications (they strongly prefer graphical notations with which they are familiar)

UML models better accepted by testers, but the detail and rigour of UML models is insufficient for being used as test oracles

Effort required to configure the test generation process to guarantee the quality of the test cases generated.

There is the need for coverage criteria best adapted for GUI testing, such as coverage of navigation maps

With Spec Explorer, the bounded exploration process is based on parameter domain values provided by the tester, but there is no feedback mechanism to evaluate the quality of the test cases obtained based on appropriate test adequacy criteria for GUIs


AMBER-iTest project Goals

Develop a set of tools and techniques to automate specification based GUI testing, solving the shortcomings found in our previous work, so that it can be used in industrial environments. Industrial partner: Critical SoftwareDuration: January 2008 – December 2010


Amber-iTest overview

SW a testar

Modelo do SW

Modelação

Geração automática de casos de teste

Casos de teste

(abstractos)

Execução automática dos testes

Lista de erros

3. Engenharia Reversa

1. Modelação visual e teste

Microsoft Research:

• Spec Explorer

• Spec#/AsmL

≈30% linhas de código

Requisitos

2. Ferramenta para mapear acções do modelo com objectos de interacção da GUI

4. Definição de critérios de cobertura para teste de GUIs

5. Validação industrial da abordagemNModel

Mapear acções

do modelo

Informação de mapeamento


Theme/Task 1. Development of a visual GUI modelling front-end

study different graphical notations used to model GUIs, and adapt (extend or combine) one notation or a combination of notations according to the requirements already described;

since it is not a goal of the project to develop a new visual editor to support the edition of the graphical notation, a freely available visual editor will be selected and adapted to support the graphical view of the GUI model;

definition of a set of rules for the translation between the graphical notation and the formal textual notation;

construction of a tool for the automatic translation and round-trip engineering between the two notations.

Preliminary results: Paper MBT’07


Visual to formal model translation

Visual GUI modelling front-end overview

Spec# formal model

(implicit spec)

UML visual model

Visual modellin

g

Refinement

m()prepost

new

Spec# formal model

(explicit spec)

m()prepost{body}

Test coverag

e analysis

new

Coverage informatio

n

Test suite

(abstract)

Test Executio

n GUI Applicatio

nUnder Test

Test results

Spec Explorer

Spec Explorer

GUI mapping

code & data (adapter)

........

......

...

Test casegeneratio

n

Model to implementatio

n mapping

ora

cle


Theme/Task 2. Development (generalization) of a model to implementation mapping tool

support the majority of the Windows controls used in Windows applications;

support Java SWT applications;

support Web applications (HTML, DHTML, etc.);

support a one-to-many mapping between logical model actions and physical GUI actions, so that a model action can correspond to a sequence of physical actions; the sequence of physical actions will be captured in way similar to the recording mechanism of capture-replay testing tools.

Previous work: Paper ICFEM’05


Theme/Task 3. Development of a GUI reverse engineering tool

In a first phase, discover the navigation map of the application and represent it by a state machine.

In a second phase, the goal will be to discover state variables within each window or page and effects of user actions on state variables.

In a third phase, it will be introduced the ability to mix automatic exploration with manual exploration, in order to be able to access parts of the application functionality that are protected by a key or that depend on specific input data (this feature represents an improvement to the state-of-art).

The reverse engineering tool will generate, at the same time, the mapping information between the model and the implementation that is needed for test execution (see task 2).

Preliminary results: Paper FMICS’07


Reverse engineering example// Main window

[Action] void AddressBook_NewContact()

requires IsEnabled(“AddressBook”);

ensures IsEnabled(“Contact”);

{ AddWindow(“Contact”); }

//Contact dialog

[Action] void SetLastName(string str)

requires IsEnabled(“Contact”);

{ LastName = str; }

//…

// Find dialog

[Action] void SetFindWhat(string str)

requires IsEnabled(“Find”);

ensures FindButton.IsEnabled();

{ FindButton.Enabled = true; }

// RECORD MODE[Action(Kind=ActionAttributeKind.Scenario)])void PasswordScenario()requires IsEnabled(“Password");{ TextBox = “mypassword"; Ok(); }


Theme/Task 4. Development of GUI test coverage analysis and enforcement tools

identification of test coverage criteria that are best suited for model-based GUI testing, and that can be used to evaluate the quality and guide the generation of test cases;

techniques and tools to determine the degree of coverage reached by a given test suite, according to the criteria previously identified;

techniques and tools to control and guide the generation of test cases in order to meet specified test coverage goals:

stop the generation of test cases (or the intermediate FSM) as soon as the coverage goals are met;

remove redundant test cases (or redundant parts of the FSM) with respect to the coverage goals specified, extending our previous work presented at ASM´05;

automatically generate domain values that allow achieving the coverage goals specified (e.g., using constraint satisfaction algorithms);

Previous work: paper ASM’05


Amber-iTest - Vantagens

Projecto financiado pela FCT2 ofertas de bolsa ainda em aberto, duração 2 anosJá existe trabalho feito em muitas das tarefasEnvolvimento com a indústriaProjecto na área da qualidade de sw (área em expansão no mundo empresarial)Interacção/integração com/no Grupo de Engenharia de Software (investigação e ensino)

Obrigada pela vossa atenção!

Mais informação: http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=projects:amber_itest:start


References and further information“Reverse Engineered Formal Models for GUI Testing”. Ana C. R. Paiva, João C. P. Faria, Pedro M. C. Mendes. 12th International Workshop on Formal Methods for Industrial Critical Systems (FMICS 2007). July 1-2, 2007, LNCS vol. 4916, Berlin, Germany."Towards the Integration of Visual and Formal Methods for GUI Testing" Ana C. R. Paiva, João C. P. Faria, Raul F. A. M. Vidal. MBT'06 - Third Workshop on Model Based Testing, 31 March - 1 April, 2007, Braga, Portugal"A Model-to-implementation Mapping Tool for Automated Model-based GUI Testing". Ana C. R. Paiva, João C. P. Faria, Nikolai Tillmann, Raul F. A. M. Vidal.ICFEM 2005 - Seventh International Conference on Formal Engineering Methods, 1-4 November 2005, Manchester, UK. "Modeling and Testing Hierarchical GUIs". Ana C. R. Paiva, João C. P. Faria, Raul F. A. Moreira, Nikolai Tillmann. ASM 2005 - 12th International Workshop on Abstract State Machines, March 8-11, 2005, Paris, France. Automated Specification-Based Testing of Graphical User Interfaces, PhD Thesis, Ana Cristina Paiva Pimenta, FEUP, November 2006 AMBER iTest Project: http://paginas.fe.up.pt/~softeng/wiki/doku.php?id=projects:amber_itest:start

prodei 29/2/2008 temas para doutoramento: projecto amber-itest (an automated model-based user...

Documents