using models to test web service-oriented applications: an ... · a test execution environment...

Pontifícia Universidade Católica do Rio Grande do SulFaculdade de Informática

Programa de Pós-Graduação em Ciência da Computação

Using models to test web service-oriented applications:an experience report

Andre Takeshi Endo∗, Maicon Bernardino da Silveira†, Elder Macedo Rodrigues†,Adenilso Simao∗, Flavio Moreira de Oliveira†, Avelino Francisco Zorzo†

∗ Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo (USP),PO Box 668, 13560-970 São Carlos, SP, Brazil

†Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS),Avenida Ipiranga, 6681 - Prédio 32, 90619-900 Porto Alegre, RS, Brazil

Technical Report No 067

Porto Alegre, September, 2012

Abstract

Service-oriented architectures and web services have been widelyadopted by companies to pervade integration among software systems. Asreliable services are essential to assure that these systems work correctly,formal and systematic testing should be performed. This document reportsthe application of a model-based approach to test web services in the con-text of real-world applications of an IT company. The employed approachis called ESG4WSC in which an event-driven model is provided to supportthe test modeling and generation, as well as an environment to support thetest concretization and execution.

1 IntroductionService-Oriented Architecture (SOA) is an architectural style in which func-

tionalities are decomposed into distinct units named services [11]. These services,distributed over a global or internal network, can be reused and combined to cre-ate new and more complex enterprise applications. In this process, a compositeservice is developed by integrating the functionalities of partner services. Thereexist various languages and specifications that have been used to describe or exe-cute service compositions, such as WS-CDL [10] and BPEL [4]. Service-orientedapplications have been mainly developed using a set of XML standards (WSDL,SOAP, and UDDI), known as Web Services technology, that is implemented byvarious programming languages.

Service testing has been investigated, since SOA was proposed as an architec-tural style and web services as its main implementation technology [3, 2]. Thereis research effort on proposing formal testing approaches that support the verifica-tion of single and composite web services [7]. Among them, model-based testing(MBT) has been advocated as a promising approach to provide more formal sys-tematic and automated testing. In MBT, a model of a system under test (SUT) isdesigned in order to derive test cases automatically. Endo and Simao [6] proposean MBT process for testing service-oriented applications, describing the artifacts,tools and revisiting different steps. A more specific MBT approach to test webservice compositions is presented in [1]. The authors propose a modeling tech-nique, named ESG4WSC (event sequence graph for web service composition),that is able to represent communication between the partner services and the com-position. Tools are also presented to support modeling and generation, as well asa test execution environment supported by an Enterprise Service Bus (ESB).

2

In this document, we report an experience of applying a model-based testingapproach in the context of service-oriented applications developed within an ITcompany. The ESG4WSC approach [1] was adopted to model and test the webservices, as well as a testing process was followed as in [6]. First, a set of BPELservices were modeled and test suites generated to evaluate the practical applica-tion of the test modeling and generation. Second, the complete approach, frommodeling to execution, was applied in an ongoing project. We show the achievedresults, as well as the lessons learned during this project.

The remainder of this report is organized as follows. Section 2 briefly de-scribes the MBT process adopted and presents the modeling technique appliedto design the test models. Section 3 reports the study configuration and adoptedtools. Section 4 shows the results, observations and lessons learned. Finally, Sec-tion 5 shows the conclusion and discusses future work.

2 Testing ApproachIn this study, an MBT process was employed to test service-oriented applica-

tions, which can be divided into four main steps: (i) test modeling, (ii) test gener-ation, (iii) test concretization, and (iv) test execution. We adopted the ESG4WSCapproach to model and generate the test suites [1].

ESG4WSC model is a directed graph in which nodes are events and edgesrepresent valid sequences of events. Figure 1 shows the ESG4WSC model for theBCS-05 Service 1. In web service composition, request messages are representedas light gray circles and response messages as dark gray circles. The messages ofthe composition itself are represented with a bold line. These events are referredto as public events, while the messages exchanges with the partner services arereferred to as private events. There are two special nodes, ’[’ and ’]’, that representthe entry and exit nodes, respectively.

A sequence of nodes that is connected by edges is called event sequence. Anyevent sequence that starts with ’[’ is called partial event sequence (PES). If a PESends with ’]’ is called complete event sequence (CES). There are two assump-tions in the graph: (i) every event must be reachable, through an event sequence,from the entry node, and (ii) the exit node must be reachable, through an eventsequence, from any event.

1We replaced the original labels by generic ones along the text.

3

Figure 1: ESG4WSC model.

It is possible to associate decision tables (DTs) with public request eventswhen input parameters may cause different events. In Figure 1, eventCS01:operation01 has two next events that are provoked according to aninput parameter. Events with associated DTs are double circled. Table 2shows a simple DT for event CS01:operation01. It contains a constraintconstraint01 =′ smd′, which can be evaluated as true or false, and two rules (R1and R2). R1 means that if constraint constraint01 =′ smd′ is evaluated as true,the next event should be PS01:operation02_smd. The ESG4WSC modelhas other features, such as refined events, associated refining ESG4WSCs, andparallel representation. See [1] for more details on the model.

Table 1: Decision table for event CS01:operation01.rules

constraints R1 R2constraint01 =′ smd′ True False

next actions (events)PS01:operation02 X

PS01:operation02_smd X

4

2.1 Positive TestingTo generate the positive test cases, a test suite composed of CESs is produced

automatically to cover all edges (event pairs) in the test model. The constraintsin the DTs are also solved and new edges are added and covered if necessary [1].Using the model in Figure 1, 14 test cases were generated, as shown in Table 2.1.

Table 2: Positive test cases for the BCS-05 service.〈 CS01:operation01(constraint01=’smd’), PS01:operation02_smd, PS01:operation02Fault, PS05:operation06 〉〈 CS01:operation01(constraint01!=’smd’), PS01:operation02, PS01:operation02Fault, PS05:operation06〉

〈 CS01:operation01(constraint01=’smd’), PS01:operation02_smd, PS01:operation02Response, PS02:operation03,

PS02:operation03Response_na, PS05:operation06〉

〈 CS01:operation01(constraint01=’smd’), PS01:operation02_smd, PS01:operation02Response_na, PS05:operation06〉


PS02:operation03Fault, PS05:operation06〉


PS02:operation03Response, PS03:operation04, PS03:operation04Response_na, PS05:operation06〉


PS02:operation03Response, PS03:operation04, PS03:operation04Fault, PS05:operation06〉


PS02:operation03Response, PS03:operation04, PS03:operation04Response_d〉


PS02:operation03Response, PS03:operation04, PS03:operation04Response_on_t, PS04:operation05,

PS04:operation05Response, PS05:operation06, PS06:operation07〉


PS02:operation03Response, PS03:operation04, PS03:operation04Response_nt, PS05:operation06〉


PS02:operation03Response, PS03:operation04, PS03:operation04Response_on_f〉



PS04:operation05Fault, PS05:operation06〉



PS04:operation05Response_s, PS06:operation07〉

〈 CS01:operation01(constraint01!=’smd’), PS01:operation02, PS01:operation02Response, PS05:operation06〉

Figure 2 shows a representation for the first test case in Table 2.1 as an XMLfile.

5

Figure 2: XML file for a test case #1.

2.2 Negative TestingUsing the same model presented in Figure 1, test cases can also be generated

for undesirable situations. These cases are referred to as negative test cases whichcan be applied to public or private events. In public negative testing, unexpectedcases are tested between public events that were not predicted in the model. Thetest cases generated for this case is called public faulty event sequence (PubFES).For the private negative testing, different fault classes are defined and simulatedduring the tests. Table 2.2 shows the fault classes we used, in which type of eventis applied, and the number of negative test cases generated for the BCS-05 ser-vice. The test cases generated for this case is called private faulty event sequence(PrivFES). It is important to emphasize that new fault classes can be defined andintegrated in the approach.

Table 3: Number of negative test cases.Fault Class Event #Test cases

no response (NR)request 8missing service (MS)

unexpected fault (UF)Longtime response (LR)

response 16Wrong XML schema (WSc)Wrong XML syntax (WSy)

Right Schema, wrong data (WD)

Figures 3 and 4 show examples of XML files for negative test cases appliedon request events and response events, respectively.

6

Figure 3: XML files for faults in request events.

Figure 4: XML files for faults in response events.

7

3 Study ConfigurationThis study was conducted in cooperation with an IT company that provided

access to its applications, as well as technological support and documentation.The communication with the IT company happened through on-line means (e-mail, instant messaging) and meetings with the development team. Overall, alldata presented in this document was produced and collected in one month.

We adopted a set of tools in order to support the conduction of the studyand automation. The Test Suite Designer (TSD) tool was used to design theESG4WSC model and generated the test cases [1]. The Event Runner for TestExecution (ERunTE) tool was adopted to support the test concretization and ex-ecution [1]. ERunTE is integrated with mule-ESB [12] to record and control allmessages produced during the tests. SoapUI [8] was employed to mock some ser-vices that have security issues and create initial stub messages. Finally, Eclipse[5] and JUnit [9] were used to support the development of adaptors and the testexecution.

4 Analysis of ResultsThis study was divided into two parts. First, we focused on the modeling and

test generation over BPEL composite services. Second, we analyzed the approachin the context of the ABC project, emphasizing the test concretization and execu-tion.

4.1 Test Modeling and GenerationIn this part, we used 23 composite services specified in BPEL to evaluate the

ESG4WSC modeling capabilities. Then, we evaluated the test suites generatedfrom these models. The models were designed in an exploratory way that is di-vided into two steps:

1. Exploratory modeling: an initial ESG4WSC model is designed, identifyingrequest and response events. The order among them is also modeled. Dur-ing this step, the global communication is prioritized and branches and DTsare put aside. Generic events and comments were used to recall that theseissues need to be handled in future.

8

2. Test-driven modeling: using the model designed in previous steps, a moredetailed analysis was conducted to identify and model DTs (constraints,rules, and actions). Generic events and comments were removed andbranches along the model were solved. The goal of this step is to set upa model that is adequate to generate test cases.

This configuration of steps was intuitively conducted during the modeling ofthe first four services. After a phase of identification, all services were modeledin two steps and therefore with two model versions. The sources of informationabout the 23 services were the BPEL specification itself, WSDL files, logs of theBPEL engine, and talks to the developers.

Table 4.1 shows the test model’s characteristics for each of the modeled ser-vices. Observe that all services are asynchronous (request-only) since the num-ber of public response events is zero in all lines. The number of public requestevents is low, most of them with one or two requests. Only services BCS-03and BCS-20 have more public requests. The number of private request and re-sponse events reflects the complexity of communication with the partner services.The overall complexity of the test models can be summarized by the number ofevents and edges. Clearly, BCS-11 has the large test model with 447 events and501 edges, followed by BCS-22, BCS-09, and BCS-20. In services BCS-11,BCS-20, and BCS-22, refined events and refining ESG4WSCs were adopted todeal with the complexity of many events and edges. ESGs in parallel were notused in these models. Most of the branches in the models are caused by differenttypes of responses, instead of input parameters. This is observed by the numberof DTs (and their elements, constraints, actions, and rules) that is low. BCS-20has the highest number of DTs which is consistent with its public request events;BCS-10 has the highest number of constraints for one DT. Ten out of 23 servicesdo not have associated DTs.

For the test models, positive and negative test suites were generated. Table 4.1shows the test suite’s information divided by type of testing. For each type of testsuite, the number of executed events is also shown as a measure of cost. All testsuites were automatically generated using the TSD tool. The cost of positive testcases is highly dependent on number of events and edges in the model. BCS-11has the highest number of test cases (40) and BCS-01, BCS-15, BCS-17, andBCS-23 has the lowest number of test cases (2).

For PubFESs, the cost is related to the number of public request and responseevents. BCS-03 and BCS-20 have the highest number of test cases and cost, 49test cases executing 98 events and 25 test cases executing 222 events, respectively.

9

Table 4: Test model information.

Sixteen out of 23 services have only one test case.The PrivFESs are divided in accordance with the fault classes described in

Section 2.2: no response (NR), missing service (MS), unexpected fault (UF), long-time response (LR), wrong XML schema (WSc), wrong XML syntax (WSy), andright schema wrong data (WD). The test suites for the fault classes NR, MS, andUF are dependent on the number of private request events. They have differentcosts because each fault class has its own characteristics during the execution. Forinstance, in class NR, the affected request event happens and no response is pro-duced, while in class UF a response event is provoked. BCS-11, BCS-09, andBCS-22 have the highest number of test cases.

The test suites for the fault classes LR, WSc, WSy, and WD are dependenton the number of private response events. BCS-11, BCS-09, and BCS-22 alsohave the highest number of test cases. BCS-03, BCS-08, and BCS-23 have no

10

test case for these fault classes since their models do not contain private responseevents (as observed in Table 4.1).

The last two columns show the total number of PrivFESs, including all sevenfault classes. As the negative testing for private events produces test suites thatcover all request and response events in combination with the fault classes, a highnumber of negative test cases is generated. For all models, its cost exceeds thecost of positive testing.

Table 5: Test suite information.

Test model information gives an idea on the human effort that would be spentsince the tester is supposed to design it manually. Although the modeling effortis directly related to the application’s size and complexity, we observe that otherfactors may also influence on it. The restricted access to sources of informationand unclear test purposes may increase the cost during the modeling. We do notinvestigate this hypothesis, but to design two or three models for different testpurposes instead of a large one (as in BCS-11) may be more cost-effective.

The cost of generating the test suites is low since it is automatically performedby the tool. For the largest model, the tool took less than 11 seconds to produce

11

the positive and negative tests. We have provided the test suite information as anadditional measure to predict the cost of execution. It is important to emphasizethat a development effort is also needed to implement adaptors during the testconcretization. This topic was not investigated in this part of the study. We providefurther discussion on it in Section 4.2.

During the modeling of the 23 services, limitations were identified on theESG4WSC model and tool. Although these limitations can be handled from apractical point of view, we describe them as a contribution for future improve-ments in the modeling technique and tool:

• BPEL-ForEach(parallel=yes) and OracleBPEL-flowN: the activity ForEach(with parallel=yes) from BPEL 2.0 and flowN extension of Oracle BPELengine introduce, e.g., the possibility of executing n request events in paral-lel. As the parallelism is implicitly introduced and only decided in runtime,ESGs in parallel are not able to directly represent this case in the ESG4WSCmodel. The tester needs to define the number of instances before the testgeneration and replicate the ESGs in parallel.

• Private request/response events within loops: this case is similar to the pre-vious one, as the number of iterations is decided in runtime. The testershould identify the number of iterations and repeat the instances in themodel before generating the tests.

• Global/internal variables: BPEL engines have the concept of global vari-ables that are independent and assigned outside the composition However,they can be referred within the BPEL and define different branches. In-ternal variables are more common in composite services implemented intraditional programming languages, instead of BPEL. These composite ser-vices tend to interact with databases and modify the workflow depending oninternal variables. These variables cannot be represented in the ESG4WSCmodel and, in some way, introduce preconditions in the test cases.

• Event branch solved by distant predecessor event or parameter: when someevent branch is solved by some event or parameter (DT) that happens previ-ously in the workflow. In the current solution, the intermediate path betweenthe solving event and the branch needs to be replicated.

12

4.2 Test Concretization and ExecutionIn this part, we applied the ESG4WSC approach in the ABC appli-

cation, specifically in its composite service ABCService (ABCS). Thisservice interacts with three other services: PartnerService01 (PS01),PartnerService02 (PS02), and PartnerService03 (PS03). Table 4.2shows the tested services, if they are composite or not, their total number of oper-ations, and number of operations involved in the tests.

Table 6: Information about services.Service name composite service? #operations #involved operationsABCService yes 8 2

PartnerService01 no 26 2PartnerService02 no 12 1PartnerService03 no 10 1

Based on performance test scripts and on meetings with development team, thetest model shown in Figure 5 was designed. It focuses on the flow of messagestriggered by the operations operation01 and operation02 of ABCService.This model was augmented with SOAP messages for the public request eventsand private response events. These messages are necessary to provoke some se-quences.

Figure 5: ESG4WSC model for ABC application.

The four services involved in the tests were deployed in the mule-ESB. Thus,

13

all messages produced during the tests will be passed through the bus and con-trolled by the module ERunTE-esbcomp. As services PartnerService01,PartnerService02, and PartnerService03 have some security issues,we used SoapUI to mock them. Simple modifications were performed inABCService, modifying the original service endpoints to the ones provided bythe ESB. Thus, all messages produced during the tests are controlled by the ESB.

Using the model in Figure 5, test cases were generated using the TSD tool.Table 4.2 summarizes the test suites generated for positive and negative testing.In total, 68 test cases were generated.

Table 7: Number of positive and negative test cases.Positive testing

Test Suite #Test cases Execution time#CESs 7 ≈ 13s

Negative testingTest Suite #Test cases Execution time#PubFESs 4 ≈ 5s

#PrivFESs (NR) 7 ≈ 75s#PrivFESs (MS) 7 ≈ 515s#PrivFESs (UF) 7 ≈ 13s#PrivFESs (LR) 9 ≈ 96s

#PrivFESs (WSc) 9 ≈ 12s#PrivFESs (WSy) 9 ≈ 5s#PrivFESs (WD) 9 ≈ 12s

The next step was the concretization of the tests. By doing so, two adap-tors were implemented, PublicEventAdaptor and MessageCheckingAdaptor.PublicEventAdaptor implements the calls for public events, i.e., invoking thecomposite service, and checks its responses. Each event has an associated methodthat is annotated with the event name. MessageCheckingAdaptor implementsindividual verifications for all events. Its purpose is, after executing a test case, toverify if all messages were produced and in the defined order. Moreover, a coupleof additional classes were implemented to configure and run the tests. The mod-ules ERunTE-runner and ERunTE-service were used to support the test executionwith the developed adaptors. Table 4.2 shows the number of lines of code andcyclomatic complexity for the two adaptors and the entire project. The metrics

14

were collected using Eclipse Metrics plugin 2.

Table 8: Code metrics for the test project (adaptors, setup code).Lines of Code average cyclomatic complexity

PublicEventAdaptor 94 1.5MessageCheckingAdaptor 231 1.8

Entire test project 900 1.6

All test cases were successfully executed; the approximate execution time isalso presented in Table 4.2. The test suite for fault class MS took more time thanothers (around 515 seconds. This happened due to a limitation in the ERunTEtool that requires more time to simulate a missing service. The test suites for faultclasses NR and LR also took more time since the ERunTE tool simulates timeoutsfor these tests.

During the tests, no fault was found in the SUT. This can be explained by theapplication’ stability. The application has been released for more than two yearsand many cycles of testing/maintenance were performed. Although most of thegenerated test cases (scenarios) were likely covered by previous tests (performedby the development team), there is a lack of automated solutions for testing webservices. Our impressions are that the MBT approach can be useful in scenarios,similar to the presented one, that have complex workflows and mocking differentservices and sequences of messages are too complex and error-prone. However,more robust and automated tools would be essential to a large scale adoption.

In this context, we observed that there is still room for improvements in theapproach automation. In the adaptors development, MessageChekingAdaptormay use the XML schema in the WSDL files to support automatic verification ofmessages. Most of the written code can be generated automatically. Moreover,XPath queries that currently are evaluated in the adaptor code can be includeddirectly in the model (and in the XML test cases as well). Thus, ERunTE-runnerwould be in charge of reading XPath queries and evaluated them, working as a testoracle. ESB configurations may also be automatically performed and integratedwith the development environment.

We also noticed opportunities for supporting performance testing. For in-stance, the module esbcomp (integrated with the ESB) of ERunTE could be ex-tended to capture metrics used to evaluate and monitor performance, such as re-sponse time and throughput. Extensions could also be carried out in the model

2http://metrics.sourceforge.net/

15

in a way that performance testing information is introduced and used to guide thegeneration of model-based performance tests.

5 ConclusionIn this technical report, we have presented an experience report on applying

a model-based approach to test web services. A set of real-world applications ofan IT company was used in the study. In the first part, the results on modelingand test generation of 23 composite services have been described. In the secondpart, we have provided more details on the test concretization and execution forthe ABC application.

The experience reported in this document has given evidences that model-based testing, more specifically the ESG4WSC (event sequence graph for web ser-vice composition) approach, is applicable to test service-oriented applications inreal and less controlled scenarios within an IT company. A set of lessons learnedand limitations has been identified in order to improve the approach and tools.More investigation on how to deploy the approach in an ongoing project and itscost-effectiveness is still necessary. Another topic is to research the automaticgeneration of partial event-driven models out of BPEL specifications. This willreduce the initial effort to produce test models to verify this kind of applicationsand/or to maintain legacy systems.

AcknowledgmentsAndre Takeshi Endo is financially supported by FAPESP/Brazil (grant

2009/01486-9) and CAPES/ Brazil (grant 0332-11-9). In this study, he was alsopartially supported by the project PROCAD/ CAPES 191/2007 – “Integrandoe aprimorando atividades de pesquisa, ensino/treinamento e transferência tec-nológica em teste e validação de software”. Study partially developed by theResearch Group of the PDTI 001/2012, financed by Dell Computers of BrazilLtd. with resources of Law 8.248/91. We thank CNPq/Brazil, CAPES/Brazil,INCT-SEC, and Dell for the support in the development of this work.

16

6 AgradecimentosAvelino F. Zorzo possui bolsa de produtividade CNPQ/Brasil e

CAPES/PROCAD. Elder M. Rodrigues possui bolsa CAPES e é pesquisadorintegrante do INCT-SEC.

References[1] F. Belli, A. T. Endo, M. Linschulte, and A. Simao. Model-based testing

of web service compositions. In IEEE 6th International Symposium on Ser-vice Oriented System Engineering (SOSE 2011), pages 181–192, Irvine, CA,USA, 2011. IEEE Computer Society.

[2] M. Bozkurt, M. Harman, and Y. Hassoun. Testing and verification in service-oriented architecture: a survey. Software Testing, Verification and Reliability,pages n/a–n/a, 2012.

[3] G. Canfora and M. Di Penta. Service-oriented architectures testing: A sur-vey. In Software Engineering: International Summer Schools (ISSSE), pages78–105, Berlin, Heidelberg, 2009. Springer-Verlag.

[4] D. Jordan et al. OASIS web services business process execution language(WSBPEL) v2.0, 2007.

[5] Eclipse.org. Eclipse, 2012. [9 August 2012].

[6] A. T. Endo and A. Simao. Model-based testing of service-oriented applica-tions via state models. In IEEE International Conference on Services Com-puting (SCC 2011), pages 432–439, Washington, DC, USA, 2011. IEEEComputer Society.

[7] A. T. Endo and A. S. Simao. A systematic review on formal testing ap-proaches for web services. In 4th Brazilian Workshop on Systematic andAutomated Software Testing (SAST), pages 89–98, Natal, Brazil, 2010.

[8] Eviware. soapUI, 2012. [9 August 2012].

[9] junit.org. JUnit, 2012. [9 August 2012].

17

[10] N. Kavantzas, D. Burdett, G. Ritzinger, T. Fletcher, Y. Lafon, and C. Barreto.Web services choreography description language version 1.0, 2005.

[11] C. M. MacKenzie, K. Laskey, F. McCabe, P. F. Brown, R. Metz, and B. A.Hamilton. OASIS reference model for service oriented architecture 1.0,2006.

[12] MuleSoft. Mule ESB: Open source ESB and integration platform. [9 August2012].

18

using models to test web service-oriented applications: an ... · a test execution environment...

Documents