ffc/proposta_fernando_pos_entrega_v2.… · web view“an embedded software component quality...
TRANSCRIPT
Pós-Graduação em Ciência da Computação
“An Embedded Software Component Quality Verification Framework”
Fernando Ferreira de Carvalho
PHD THESIS PROPOSE
Universidade Federal de [email protected]
www.cin.ufpe.br/~posgraduacao
RECIFE, 11/2008
UNIVERSIDADE FEDERAL DE PERNAMBUCO
CENTRO DE INFORMÁTICA
PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO
Fernando Ferreira de Carvalho
“An Embedded Software Component Quality Verification Framework"
ESTE TRABALHO FOI APRESENTADO À PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO DO CENTRO DE INFORMÁTICA DA UNIVERSIDADE FEDERAL DE PERNAMBUCO COMO REQUISITO PARCIAL PARA OBTENÇÃO DO GRAU DE DOUTOR EM CIÊNCIA DA COMPUTAÇÃO.
A PHD. THESIS PRESENTED TO THE FEDERAL UNIVERSITY OF PERNAMBUCO IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF PHD. IN COMPUTER SCIENCE.
ADVISOR: Silvio Lemos Meira
RECIFE, NOVEMBER/2008
R ESUMO
Um dos maiores desafios para indústria hoje é fornecer produtos com alto nível de qualidade e funcionalidade com um baixo custo e curto tempo de desenvolvimento. Os requisitos de custo e tempo de desenvolvimento têm sido abordados com bastante êxito pela engenharia de software baseada em componentes (CBSE). A tecnologia baseada em componentes é amplamente utilizada nas indústrias mecânica e de componentes eletrônicos, entre outras áreas, mas na indústria de software embarcados ainda não alcançou o nível desejado. É de conhecimento geral que a força de um corrente está intimamente ligada com a força de cada elemento que a compõe. Da mesma forma, a qualidade dos atributos de um sistema embarcado, como seu desempenho, sua confiabilidade e segurança, dependem do nível de qualidade de cada componente que integra o sistema. Por este motivo, a abordagem baseada em componentes, embora atraente para muitas razões, é difícil utilizar em domínios nos quais atributos de qualidade são de vital importância. Em CBSE, utilização de mecanismos apropriados de pesquisa, seleção e avaliação do processo de componentes, é considerada o ponto chave para o desenvolvimento de qualquer sistema baseado em componentes. Por este e outros motivos, esta tese propõe um Framework para Verificação da Qualidade de Componentes de Software Embarcados, que prove mecanismos de avaliação dos atributos de qualidade do componente de software embarcado, baseando-se módulos bem definidos que se complementam uns aos outros em busca da qualidade dos componentes embarcados, de forma consistente. O sonho do desenvolvedor de sistemas
embarcados é que a avaliação dos atributos de qualidade de seus componentes alcance níveis semelhantes ao dos componentes utilizados nos sistemas mecânicos. Pela introdução de verificação e certificação da qualidade de componentes de software no projeto de sistemas embarcados, esta tese é um pequeno passo para alcançar este sonho.
A bstract
One of the major challenges to industry today is to provide products with high degrees of quality and functionality at low cost and short time to market. The cost and time to market requirements have quite successfully been addressed by the component-based software engineering (CBSE). Unfortunately, satisfactory solutions for handling quality are not yet available. Component technologies are widely used in the electronics and mechanics industries, but software industry has not reached the desired level. It is well known that the strength of a chain is proportional to the strength of its interconnected elements. Similar to this concept, the quality attributes of the final embedded system, such as its performance, scalability or reliability, depends on the quality level of the components that integrate the system. For these reasons, the component-based approach, although attractive for many reasons, is difficult to apply in domains in which quality attributes are of primary importance. In CBSE, the proper search, selection and evaluation process of components is considered the corner stone for the development of any effective component-based system. For these and other reasons, this thesis proposes an Embedded Software Component Quality Verification Framework that provides mechanisms for assessing quality attributes of embedded software components, based upon well-defined modules that complement each other looking for the embedded component quality in a consistent way. Having the means to reason about the qualities of an embedded software design in the same way as one can reason about the
qualities of a mechanical design is a dream of embedded design. By introducing component quality verification and certification in embedded systems design, this thesis is a small step towards fulfilling this dream.
L ist of Figures
Figure 1.1. The Framework for Software Reuse...............................18Figure 1.2. The Component Certification Process was divided into two parts:...........................................................................................19Figure 1.3. Embedded Software Component Quality Verification Framework.........................................................................................20Figure 2.1. An embedded system encompasses the CPU as well as many other resources........................................................................31Figure 2.2. Component technology for small embedded Systems....33Figure 2.3. The results. Y-axis: priority of quality attributes in a scale 1 (highest),.........................................................................................41Figure 3.1. Structure of Prediction-Enabled Component Technology (Wallnau, 2003)..................................................................................54Figure 3.2. SQuaRE Architecture......................................................63Figure 3.3. ISO/IEC 25040.................................................................68Figure 3.4. Process of obtaining, certifying and storing components...........................................................................................................71Figure 3.5. Research on embedded software component quality and certification timeline..........................................................................72Figure 4.1. Embedded Software Component Quality Verification Framework.........................................................................................78Figure 4.2. Relations among the quality model elements.................90Figure 4.3.1 A different evaluation level for each embedded component characteristics...............................................................100Figure 4.5.1. Component Certification Process...............................119Figure 4.5.2. Establish Evaluation Requirements activity..............120Figure 4.5.3. Specify the Evaluation activity...................................125Figure 4.5.4. Design the Evaluation activity...................................129Figure 4.5.5. Execute the Evaluation activity..................................133
L ist of Tables
Table 2.2. Summary of relevant characteristics in Crnkovic’s research.............................................................................................37Table 2.2.2. Priority classes used to classify the importance of the different quality attributes.................................................................40Table 3.1. Software quality standards...............................................58Table 3.2. IEC61131.........................................................................59Table 3.3. DO-178B............................................................................61Table 3.4. Characteristics and Sub-Characteristics in SQuaRE project................................................................................................67Table 4.1. Changes in the Proposed Embedded software Component Quality................................................................................................83Model, in relation to ISO/IEC 25010.................................................83Table 4.2. The Proposed Component Quality Model, with the sub-characteristics...................................................................................89divided into two kinds: runtime and life-cycle...................................89Table 4.2.4. Embedded software Component Quality Attributes for 91Sub-Characteristics those are observable at Runtime and Life-cycle............................................................................................................91Table 4.2.5 Additional Information....................................................98Table 4.3.1. Guidelines for selecting evaluation level.....................101Table 4.3.2. Embedded software component Maturity Model........105Table 4.3.3. Embedded Quality Attribute X Evaluation Techniques..........................................................................................................106Table 4.5.1. Example of Importance’s definition.............................123Table 4.5.2. Example of Quality Attributes Definition.....................125Table 4.5.3. Example of Define EMM..............................................126Table 4.5.4. Example of Tools Selection..........................................130Table 4.5.5. Example of Data Collection..........................................134Table 5.3. Cronogram......................................................................140
A cronyms
Terms - DescriptionsB2B - Business to BusinessCBD - Component-Based DevelopmentCBSE - Component-Based Software EngineeringCMU/SEI - Carnegie Mellon University’s Software Engineering InstituteCOTS - Commercial Off-The-SelfCBSD - Component-Based Software DevelopmentCOM - Component Object ModelCCM - CORBA Component ModelCMM - Capability Maturity ModelCMMI - Capability Maturity Model IntegratedEQM - Embedded software component Quality ModelEMM - Embedded software component Maturity ModelGQM - Goal Question Metric ParadigmISO - International Organization for StandardizationIEC - International Electro-technical CommissionPECT - Prediction-Enabled Component TechnologyPACC - Predictable Assembly from Certifiable ComponentsRiSE - Reuse in Software Engineering groupSPICE - Software Process Improvement and Capability dEterminationOMG - Object Management GroupXML - eXtensible Markup Language
C ontents
1 Introduction..................................................................111.1 Motivation...............................................................14
1.1.1 The needs and priorities in research.................................201.2 Problem formulation...............................................241.3 Proposed solution....................................................251.4 Out of Scope............................................................261.5 Statement of Contributions.....................................281.6 Organization of the Thesis......................................28
2 Embedded systems design...................................302.1 Basic concepts for Component-Based embedded systems............................................................................31
2.1.1 Component-based approach for small embedded systems322.1.2 Component-based approach for large embedded systems
332.2 Specific requirements for embedded systems........35
2.2.1 Industrial Automation.....................................................372.2.2 Automotive......................................................................382.2.3 Medical............................................................................412.2.4 Consumer Electronics.....................................................422.2.5 Other domains.................................................................43
2.3 The needs and priorities in research......................43
3 Embedded Software Component Quality and Certification: A Survey..............................................46
3.1 Failures in Software Component Certification.......563.1.1 Failure in National Information Assurance Partnership (NIAP)..........................................................................................573.1.2 Failure in IEEE................................................................57
3.2 Standardization: Software Quality and Certification57
3.2.1 ISO/IEC 61131................................................................593.2.2 RTCA DO 178B................................................................603.2.3 ISO/IEC 61508................................................................62
3.2.4 ISO/IEC 25000 (SQuaRE project)...................................623.3 Software Component Certification.........................693.4 Summary of the Study............................................723.5 Summary.................................................................73
4 Embedded software Component Quality Verification Framework............................................74
4.1 Overview of the Framework...................................754.2 Embedded software component Quality Model......78
4.2.1 The Embedded software component Quality Model (EQM) 804.2.2 Changes in relation to ISO/IEC 25010............................83
4.2.2.1 Quality sub-characteristics that were created from ISO/IEC 25010 844.2.2.2 Quality sub-characteristics that were removed from ISO/IEC 25010 864.2.2.3 Quality sub-characteristics that were renamed from ISO/IEC 25010 874.2.2.4 Quality characteristics that were extended from ISO/IEC 25010 88
4.2.3 Summary.........................................................................894.2.4 Embedded software Component Quality Attributes.......904.2.5 Other relevant Component Information.........................98
4.3 Maturity Level Evaluation Techniques...................994.3.1 Embedded software component Maturity Model (EMM)
1004.4 Metrics Approach..................................................109
4.4.1 Metrics to track the EMM Properties...........................1144.4.2 Metrics to track the Certification Techniques Properties
1154.4.3 Metrics to track the Certification Process Properties. .116
4.5 Component Certification Process.........................1174.5.1 The Component Certification Process..........................118
4.5.1.1 Establish Evaluation Requirements activity......................1194.5.1.2 Specify the Evaluation activity..........................................1244.5.1.3 Design the Evaluation activity...........................................1294.5.1.4 Execute the Evaluation activity.........................................1324.5.1.5 Process Summary..............................................................135
5 Conclusion and future works.............................1365.1 Contributions........................................................1375.2 Future Work..........................................................1395.3 Chronogram..........................................................140
6 Bibliographic References...................................141
Appendix A. Metrics Example..........................................155
11
1 Introduction
Embedded system is at the heart of many systems in use today. Added value to products is to a large extent provided by the software. Furthermore, production cost reduction is imperative and is often achieved by introducing software that permits the use of less complex hardware. Domains in which the use of software is now essential include, among others, the automotive, medical systems, process control, and manufacturing industries. Embedded systems are often systems consisting of software and hardware, the software part incorporates many software components that must cooperate without fail. It is platform-dependent, consequently difficult to port, upgrade, and customize, and offers limited opportunities for reuse, even within a single application domain.
The demands that companies in these electronic products must satisfy include low production costs, short time to market and high quality. The cost and time to market issue is addressed by means of the rapidly emerging Component-based Development (CBD) approach. Adding new functionalities to existing products must be done at low cost and high quality. In CBD, software systems are built as an assembly of components already developed and prepared for integration.
12
The potential benefits of component-based development are as attractive in the domain of embedded systems as they are in other areas of the software industry. These include reduction of costs, improved quality, specialization of expertise, effective management of complexity, reduced time to market, increased productivity, a greater degree of consistency, and a wider range of usability (Brown, 2000). Additionally, CBD offers an opportunity to increase productivity by providing natural units for reuse (components and architectures), by raising the level of abstraction for system construction, and by separating services from their configurations to facilitate evolution (Müller et al., 2002).
However, these benefits are much more difficult to realize in embedded systems development than in other areas of software engineering because the problems of composing extra-functional requirements such as quality, reliability and performance are much more acute. When building new applications from existing components it is not only necessary to ensure that they behave as expected, but also that these extra-functional properties are composed correctly. Until recently it was difficult to do this in an economically viable way, but recent trends have combined to make component-based software engineering as imperative for embedded systems development as it is in other domains. According to Atkinson (Atkinson at al., 2005), there are three important trends with embedded domain:
The shear growth in the number and ubiquity of embedded systems in the world around us;
The growth in the size and complexity of software in embedded systems.
The creation of larger product families in order to tailor products variants to the needs of specific customers and/or market segments.
13
New functionality is frequently added to products to meet market requirements. The new functionality is often introduced by adding components to the system. This is the reason why the component-based development approach is attractive to industry.
Different classes of components are used in different industrial settings, and different component technologies target different application domains. Industrial component technologies such as IEC 61131-3 (ISO/IEC 1131-3, 1995) and Koala (Van Ommering, 2002) are used to build applications. Other component classes are used to provide the infrastructure in systems for examples, databases, real-time operating systems, user interfaces and communication components. This diversity means that it is possible to use components at different levels in a system.
The quality aspects of software products are not, however, addressed adequately by component-based development. Component-based applications are sensitive to evolution of the system. As components and applications have separate lifecycles and different kinds of requirements, there is some risk that a component will not completely satisfy the particular requirements of certain applications or that it may have characteristics unknown to the application developer. When introducing changes on the application level (changes such as updating the operating system, updating other components, changes in the application), there is a risk that the change introduced will reduce the reliability of the system due to different unforeseen mismatches (Larsson, 2000).
Components are in general considered as black boxes with little or no information easily accessible. The information needed from the software components relates to their quality attributes. If the components’ attributes are not known, the attributes of the system of which they are part of is even more difficult to reason about. Industrial products are required to meet high functional and
14
quality standards. Customers often pay for the functionality they require and take the quality of the product for granted. This business model influences the way software is developed. Unfortunately, as software developers tend to focus on delivering the functionality without giving the quality requirements the same priority it is possible to ensure the quality characteristics and the functionality of a product by means of extensive testing, but this is generally an expensive and inefficient approach. A much more effective approach would be to ensure the quality attributes of the product on the basis of the quality of the components of which it is constructed.
A software program computes functions with certain functional attributes. There are other attributes that we care for more than the functional ones, these attributes are expressed in terms of quality attributes. A quality attribute is originally the characteristic of a system but today we also recognize quality attributes of software. Security, safety, performance and dependability are examples of system characteristics. Quality attributes are primarily attributes of systems which more and more are achieved by software.
Quality attributes are often referred to as properties, non-functional or extra-functional attributes because they relate to the quality of the component and not explicitly to the component functionality (Barbacci et al, 1995). The quality attributes describe the characteristics of a component. The properties of a component are the concrete accessible values that represent the characteristics. A component can express its quality attributes in terms of its properties.
1.1 Motivation
This creates a big challenge for embedded-software development. In the years to come, the key to success will be the
15
ability to successfully develop high-quality embedded systems and software on time. As the complexity, number, and diversity of applications increase, more and more companies are having trouble achieving sufficient product quality and timely delivery. To optimize the timeliness, productivity, and quality of embedded software development, companies must apply software engineering technologies that are appropriate for specific situations.
Unfortunately, the many available software development technologies do not take into account the specific needs of embedded-systems development. This development is fundamentally different from that of non embedded systems. Technologies for the development of embedded systems should address specific constraints such as severe timing constraints, limited memory and power use, predefined hardware platform technology, and hardware costs. Existing development technologies do not address their specific impact on, or necessary customization for, the embedded domain. Nor do these technologies give developers any indication of how to apply them to specific areas in this domain—for example, automotive systems, telecommunications, or consumer electronics. Consequently, tailoring a technology for a specific use is difficult. Furthermore, the embedded domain is driven by reliability factors, cost factors, and time to market. So, this embedded domain needs specifically targeted development technologies. In industry, the general feeling is that the current practice of embedded software development is unsatisfactory. However, changes to the development process must be gradual; a direction must be supplied.
Embedded systems comprise a scale from ultra small devices with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume
16
products in which the system dependability is the most important feature. All these different requirements have impact on feasibility, on use, and on approach in component-based development.
There is no universal definition of an embedded system. However, there is a certain consensus that the following features are common to most embedded systems (Camposano & Wilberg, 1996):
They are part of a larger system (host system), hence the term embedded, with which they continuously or frequently interact. Usually, the embedded system serves as a control unit inside the host system.
They have a dedicated functionality and are not intended to be reprogrammable by the end-users. Once an embedded system is built, its functionality does not change throughout its lifetime. For example, a device controlling the engine of a car will probably never be reprogrammed to decode Mp3s. A desktop computer, on the other hand, has a wide range of functionality, including web browsing, word processing, gaming, advanced scientific calculator, etc.
They have real-time behavior. The systems must, in general, respond to their environment in a timely manner.
They consist of both hardware and software components. In order to cope with the wide and unpredictable range of applications, the hardware of a general purpose computer has to be meticulously designed with the risk of wasting resources. However, since the set of applications to be run on an embedded system is known at design-time, including their performance requirements, the hardware can be tuned at design-time for best performance at minimal cost. Similarly, software must also be optimized to build a globally efficient HW/SW system.
17
There are many domains in which embedded systems are used extensively. Some of them are: Telecommunication, avionics and aerospace, transportation, computer games, home electronics, navigation systems, etc. While there are many similarities between these domains there are also very different requirements for their functional and extra-functional properties. The consequences are that the requirements for component-based technologies are different, and consequently it expects to have one component model that summarizes all quality attributes in common in different domains. The expectations are that only one component models will exist, some common characteristics, such as basic principles of component specifications through interfaces, basic composition and run-time services, certain patterns, and similar.
All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes (Crnkovic, 2005). These properties can be classified in run-time and life cycle extra-functional properties.
A common characteristic of all systems is the increasing importance of software. For example, software development costs for industrial robots today constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten to fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies
One of the most compelling reasons for adopting component-based approaches in embedded software design is the premise of reuse. The idea is to build software from existing components
18
primarily by assembling and replacing interoperable parts. The implications for reduced development time and improved product quality make this approach very attractive (Krueger, 1992).
Software reuse is not a new idea. Since McElroy’s pioneer work, “Mass Produced Software Components” (McElroy, 1968), the idea of reusing software components on a large scale is being pursued by developers and research groups. This effort is reflected in the literature, which is very rich in this particular area of software engineering.
McIlroy’s work does not consider an essential requirement for these systems: the assets certification. In a real environment, a developer that retrieves a faulty component from the repository would certainly lose his trust in the system, becoming discouraged to make new queries. Thus, it is extremely important to assert the quality of the assets that are stored into the repository before making them available for reuse. Despite this importance, the software engineering community had not explored these issues until recently. In this way, a new research area arose: components certification and quality assurance (Wohlin et al., 1994), (Mingins et al., 1998), (Morris et al., 2001), (Schmidt, 2003), (Wallnau, 2003). However, several questions still remain unanswered, such as:
(i) How certification should be carried out? (ii) What are the requirements for a certification process?
and, (iii) Who should perform it? (Goulão et al., 2002a).
This is the reason why there is still no well-defined standard to perform component certification (Voas et al., 2000), (Morris et al., 2001).
This thesis addresses embedded software component quality issues with a particular emphasis on component quality certification
19
for reuse proposes. It also covers issues related to design of component-based embedded systems. This thesis is investigating effective ways to demonstrate that embedded component certification is not only possible and practically viable, but also directly applicable in the embedded software design. Through certification, some benefits can be achieved, such as: higher quality levels, reduced maintenance time, investment return, reduced time-to-market, among others. According to Weber et al. (Weber et al., 2002), the need for quality assurance in software development has exponentially increased in the past few years.
Moreover, this thesis is a part of a big project that aims to develop a robust framework for software reuse (Almeida et al., 2004), in order to establish a standard for component development; to develop a repository system; and to develop a general software component certification process. This project has been developed in collaboration between the industry and academia (the RiSE group1 and three other universities), in order to generate a well-defined model for developing, evaluating quality, storing and, subsequently, making it possible for software factories to reuse software components.
The framework (Figure 1.1) that is being developed has two layers. The first layer (on the left) is composed of best practices related to software reuse. Non-technical factors, such as education, training, incentives, and organizational management are considered. This layer constitutes a fundamental step prior to the introduction of the framework in the organizations. The second layer (on the right) is composed of important technical aspects related to software reuse, such as processes, environments, tools, and a certification process, which is the focus of this thesis.
20
Figure 1.1. The Framework for Software Reuse.
However, to better support the embedded domain and our specific quality characteristics, the component certification process was divided into two parts. The first of these, focuses on general propose software component certification, in general, the software component used in IBM-PC compatible. The other, is a component certification processes that aligns to specific requirements and constraints to develop quality characteristics for embedded domain.
Figure 1.2. The Component Certification Process was divided into two parts:
i – General Propose Certification and ii - Embedded Component Certification.
The process of certifying components proposed is not a simple one. First, there should be an Embedded software component Quality Model (EQM). Differently from other software product quality models, such as (McCall et al., 1977), (Boehm et al., 1978), (Hyatt et al., 1996), (ISO/IEC 9126, 2001), (Georgiadou, 2003), this model should consider Component-Based Development (CBD) characteristics applied to embedded domain, and describe attributes
21
that are specific to the promotion of reuse. With the availability of an embedded software component quality model, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different level of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics is required, in order to track the components properties and the enactment of the certification process. These four main issues:
(i) Embedded software component Quality Model, (ii) Maturity Level Evaluation Techniques, (iii) Metrics Approach, and (iv) Certification Process,
Are the modules of an Embedded Software Component Quality Verification Framework (Figure 1.3) that is being investigated as a part of the RiSE project.
Figure 1.3. Embedded Software Component Quality Verification Framework.
The framework will allow that the components produced in a Software Reuse Environment are certified before being stored on a Repository System. In this way, software engineers would have a greater degree of trust in the components that are being reused.
1.1.1 The needs and priorities in research
22
The component-based approach on system level, where hardware components are designed with embedded software, has been successfully used for many years. Also large-grain generic components like protocol stacks, RTOSs, etc. have been used for a long time. In addition to this, technology supporting a component-based approach has been developed; either in the form of proprietary component models, or by using reduced versions of some widely used component models.
Still, there are needs for a number of improvements (Crnkovic, 2005). Some of them are the following (differently important for different domains):
There is a lack of widely adopted component technology standards which are suitable for embedded systems. For smaller-size embedded systems, it is important that a system composed of components can be optimized for speed and memory consumption, which is still missing in most of the component technologies available today.
There is also a need for interoperability between different component technologies. Specification technology is not sufficiently developed to guarantee a priori component interoperability.
Component certification: In order to transfer components across organizations, techniques and procedures should be developed for ensuring the trustworthiness of components.
Most current component technologies do not support important extra-functional properties.
There is a need for generic platform services, for, e.g., security and availability.
Tools that support component based development are still lacking.
There is a lack of efficient implementations of component frameworks (i.e., middleware), which have low requirements
23
on memory and processing power. Major needs for the further development of component technology for embedded systems are the following (Brinksma et al., 2001).
Need for adopted component models and frameworks for embedded systems. A problem is that many application domains have application-dependent requirements on such a technology.
Need for light-weight implementations of component frameworks. In order to support more advanced features in component-based systems, the run-time platform must provide certain services, which however must use only limited resources.
Obtaining extra-functional properties of components: Timing and performance properties are usually obtained from components by measurement, usually by means of simulation. Problems with this approach are that the results depend crucially on the environment (model) used for the measurements may not be valid in other environments, and that the results may depend on factors which cannot easily be controlled. Techniques should be developed for overcoming these problems, thereby obtaining more reliable specifications of component properties.
Platform and vendor independence: Many current component technologies are rather tightly bound to a particular platform (either run-time platform or design platform). This means that components only make sense in the context of a particular platform.
Efforts to predict system properties: The analysis of many global properties from component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.
Component noninterference: Particularly in safety-critical applications, there is a need to ensure separation and
24
protection between component implementations, in terms of memory protection, resource usage, etc.
Tool support: The adoption of component technology depends on the development of tool support. The clearly identified priorities of CBSE for embedded systems are:
Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc.
Development of widely adopted component models for real-time systems. Such a model should be supported by technology for generating necessary runtime infrastructure (which must be light-weight), generation of monitors to check conformance with contracts, etc.
Further, according to Voas (Voas, 1998), to foster an emerging software component marketplace, it must be clear for buyers whether a component’s impact is positive or negative. Ideally, buyers should have this information before buying a component. Component buyers could then choose an appropriate component according to its certification level. With this information, system builders could make better design decisions and be less fearful of liability concerns, and component vendors could expect a growing marketplace for their products.
The Carnegie Mellon University’s Software Engineering Institute (CMU/SEI) (Bass et al., 2000) studied industry trends in the use of software components from technical and business perspectives. A distinct set of inhibitors to adopting software component technology emerged from the conducted surveys and interviews of earlier adopters of software component technology.
25
From this data and from the interviews, the study concludes that the market perceives the following key inhibitors for component adoption, presented here in decreasing order of importance:
Lack of available components; Lack of stable standards for component technology; Lack of certified components; and Lack of an engineering method to consistently produce quality
systems from components.
The software engineering community is already attempting to reduce the gaps related to the two first inhibitors.
However, in relation to the third inhibitor, the community is still a fledgling. Further research is required in order to assure the production of certified components, especially when combined with the lack of component-based software engineering techniques that deliver predictable properties (the last inhibitor).
Important researches on software components, such as Heineman (Heineman et al., 2001), Councill in (Councill, 2001), Crnkovic (Crnkovic, 2005), Wallnau (Wallnau, 2003), Wallnau (Wallnau, 2004), Schneider & Han (Schneider & Han, 2004) and Andreou & Tziakouris (Andreou & Tziakouris, 2007) indicates that the future of software components is certification. These authors state that certification is a necessary precondition for CBSE to be successfully adopted and to achieve the associated economic and social benefits that CBSE could yield. With the success of CBSE, software developers will have more time to develop, instead of spending their time addressing problems associated with understanding and fixing someone else’s code. Certified components used during development will have predetermined and well established criteria, thus reducing the risk of system failure and
26
increasing the likelihood that the system will comply with design standards.
1.2 Problem formulation
The previous section stated that designers increasingly often build systems using reusable components due to the complexity of their designs. Therefore, there is an increasing need to efficiently and effectively qualify such systems. Quality assurance, in particular certification, which can effectively cope with this situation and take advantage of the component-based structure, need to be developed.
The evaluation of the quality of components (i.e. the assessment of their quality attributes) needs to be done by independent parties, at least until software vendors acquire the level of maturity that hardware vendors currently have. The software industry still far from counting with the hardware data sheets and catalogues available for hardware components that describe all their characteristics. However, it is necessary to have them for software components too if it wants to talk about a “real” Component-based Software Engineering. Many organizations struggle in their attempts to select and evaluate an appropriate component in their system. For this reason, a well-defined and consistent software component quality assurance is essential for the component market adoption (i.e. without an efficient mechanism to select/evaluate the component quality.
According to Morris et al. (Morris et al., 2001), there is a lack of an effective assessment of software components in general. Besides, the international standards that address the software products’ quality issues (in particular, those from ISO and IEEE) have shown to be too general for dealing with the specific characteristics of embedded components. While some of their characteristics are appropriate to the evaluation of components in embedded domain, others are not well suited for that task.
27
However, the main drawback of the existing international standards, in this case the ISO/IEC 25010, is that they provide very generic quality models and guidelines, which are very difficult to apply to specific domains such as embedded components and CBSE. Thus, the quality characteristics of this model should be analyzed in order to define and adequate to the component quality characteristics context.
A unified and prioritized set of CBSE requirements for trustworthy components is a challenge in itself (Schmidt, 2003). This fact may be due also to the relatively novelty of this area (Goulão et al., 2002a). Moreover, there is a lack of processes, methods, techniques and tools that support the component certification activity (Alvaro et al., 2005). The existent processes and/or methods dealing only with specific aspects of software component (Alvaro et al., 2005) were not evaluated into industrial scenarios. In fact, they are based on the researchers’ experience, and the real efficiency of evaluating software components using these process/methods remains unknown.
One of the keys to success in industry in the future will be the ability to develop high-quality embedded systems and their software on time in order cost reduced to remain competitive. Organizations whose aim is to construct software by integrating components – rather than developing software from scratch – will not be able to meet their objectives if they cannot find sufficient number of components and component versions that satisfy certain functional and quality requirements. Without a quality level, the component usage may have catastrophic results (Jezequel et al., 1997). However, the common belief is that the market components are not reliable and this prevents the emergence of a mature software component market (Trass et al., 2000). Thus, the components market quality problems must be resolved in order to increase the reliability,
28
and third-party certification programs would help to acquire trust in the market components (Heineman et al., 2001).
1.3 Proposed solution
This work defines an Embedded Software Component Quality Verification Framework, which includes the steps of definition of embedded software component quality, maturity level evaluation techniques, metrics and component certification, based on a set of activities, metrics and guidelines. Moreover, the process is based on the state-of-the-art in the area and its foundations and elements are discussed in details
In this way, the main goals of this thesis were to define Embedded Software Component Quality Verification Framework that is composed of four inter-related modules in order to assure the component quality degree. This framework was proposed with basis in the standards ISO/IEC 25010, ISO/IEC 9126, ISO/IEC 14598 adapted for component context and embedded domain. Different from other approaches in the literature (Goulão et al., 2002b), (Beus-Dukic et al., 2003), (Cho et al., 2001), (Gui and Scott, 2007) that provide only isolated aspects to assure the component quality, this thesis tries to investigate and make available a complete framework with all necessary mechanisms to execute the component evaluation activities.
Through these evaluations it is expected a continuous evolution of the whole framework in a way that the software industry can start to trust in it and evaluate its own software components.
In order to achieve the main objective, the process is based on the following foundations:
First, there should be an embedded software component quality model. With an embedded software component quality
29
model available, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different levels of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics are needed, in order to track the components properties and the enactment of the certification process. These four main modules:
Embedded software component Quality Model, Maturity Level Evaluation Techniques, Metrics Approach, and Certification Process.
1.4 Out of Scope
As the proposed reuse process is part of a broader context, a set of related aspects will be left out of its scope. Nevertheless, as these aspects were envisioned since the initial definitions of the process, they can be added in the future with some adjustments. Thus, the following issues are not directly addressed by this work:
Cost Model: Cost estimation is a key requirement for CBSD before the actual development activities can proceed. Cost is a function of the enterprise itself, its particular development process, the selected solution, and the management and availability of the resource during the development project (Cechich et al., 2003), (Mahmooda et al, 2005). A cost model is useful to help the software engineer during the analysis of the software component available to purchase (or to select or to evaluate). However, it makes sense when, first, you have defined processes, methods, techniques and tools to execute the selection and/or the evaluation task in order to identify the cost/benefit to purchase or to evaluate a component.
Formal Proof: Meyer (Meyer, 2003) and Karlsson (Karlsson, 2006) proposes a formal approach in order to acquire trust in
30
software components. His idea is to build or to use software components with fully proved properties or characteristics. The intention is to develop software components that could be reliable to the software market.
This thesis does not consider cost model because the whole framework that is the basis for embedded software component evaluation will be considered in this first moment and, after that a cost model to help organizations to define if it is viable evaluate certain kinds of components (or not) will be useful. The formal quality assurance is not considered mainly because the software component market is not inclined to formally specify their software components. This kind of approach is highly expensive, in terms of development time and level of expertise that is needed, and component developers still do not know if it is cost effective to spend effort in this direction without having specific requirements such as strict time constraints or high reliability. However, the Embedded software component Maturity Model (EMM) provides formal proof evaluation techniques that could be useful in some scenarios, depending of the customer’s necessities and the cost/benefit to do so;
1.5 Statement of Contributions
As a result of the work presented in this thesis, the following contributions can be enumerated:
An extensive study of the key developments in the field of quality and certification of embedded software component, in an attempt to analyze this research area and identify trends to follow;
A survey of the state-of-the-art of quality and certification of general propose software component in order to understand
31
and identify the weak and strong points of existing processes;
A development of an Embedded Software Component Quality Verification Framework aiming to provide a complementary mechanism to standards ISO/IEC to evaluate the component quality;
Definition of the Embedded software component Quality Model (EQM), based on the new standard, the Software Product Quality Requirements and Evaluation (SQuaRE) project (ISO/IEC 25000, 2005) and Component Quality Model (CQM) (Alvaro et al., 2006);
A development of the Embedded software component Maturity Model (EMM) in order to provide different thoroughness levels of evaluation techniques and a set of guidelines for selecting those evaluation levels;
A development of the Component Certification Process in order to provide a high quality and consistent evaluation process; and
A development of the Metrics Approach that is composed of a set of valuable measures to be considered as starting point during the component evaluations;
1.6 Organization of the Thesis
The remainder of this thesis is organized as follows.
Chapter 2 presents a brief overview of embedded system design, component-based development areas and requirements for embedded in different domain. The main concepts of these topics are considered.
Chapter 3 presents, in the first part, the survey of the state-of-the-art of the embedded software component quality and certification area that was performed. The failure cases that can be found in the literature are also described in this chapter.
32
Subsequently, in the final part, this chapter describes the aspects related to the software quality and certification concepts and standardization. The intention is to show that software reuse depends on quality.
Chapter 4 presents the Embedded Software Component Quality Verification Framework proposed and its related modules. Session 4.2 describes the proposed Embedded software component Quality Model (EQM), showing its characteristics, its sub-characteristics, the quality attributes and the metrics that are related to the model. Session 4.3 presents the Embedded software component Maturity Model (EMM) that is composed of a set of evaluation levels in order to provide flexibility to the component evaluation. Further, a set of guidance is delineated to direct the evaluation team during the selection of levels. Session 4.4 presents the Metrics Approach and the paradigm adopted to define the metrics is also presented. Some few examples of metrics usage are presented. Session 4.5 presents the Software Component Certification Process, describing all activities and steps that should be followed to execute the component evaluation activity in a more controllable way.
Chapter 5 summarizes the main contributions of this work, presents the related works, the concluding remarks and the future work that will be accomplished during next months.
33
2 Embedded systems
designEmbedded systems comprise a scale from ultra small devices
with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. An embedded system may be represented by a dedicated processor surrounded by dedicated hardware systems, performing very specific tasks (Carvalho & Junior et al., 2004). Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume products in which the system dependability is the most important feature. All these different requirements have an impact on feasibility, on use, and on approach in component-based development. A common characteristic of all systems is increasing importance of software. For example, software development costs for industrial robots currently constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten, fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies. In this chapter, a short overview of embedded systems design will be shown.
34
Figure 2.1. An embedded system encompasses the CPU as well as many other resources.
1.7 Basic concepts for Component-Based embedded systems
In classic engineering disciplines a component is a self contained part or subsystem that can be used as a building block in the design of a system. In software engineering, there are many different suggestions for precise definitions of components in component based software development. The best accepted understanding of component in the software industry world is based on Szyperski’s definition (Szyperski et al. 1998). From this definition it can be assumed that a component is an executable unit, and that deployment and composition can be performed at run-time.
In the domains of embedded systems this definition is largely followed, in particular the separation between component implementation and component interface. However the demands on the binary or executable from is not directly followed. A component can be delivered in a form of a source code written in a high-level language, and allows build-time (or design-time) composition. This more liberal view is partly motivated by the embedded systems context, as will be discussed below.
35
Many important properties of components in embedded systems, such as timing and performance, depend on the characteristics of the underlying hardware platform. Kopetz and Suri (Kopetz and Suri et al., 2003) propose to distinguish between software components and system components. Extra-functional properties, such as performance, cannot be specified for a software component in isolation. Such properties must either be specified with respect to a given hardware platform, or be parameterized on (characteristics of) the underlying platform. A system component, on the other hand, is defined as a self-contained hardware and software subsystem, and can satisfy both functional and extra functional properties.
1.7.1 Component-based approach for small embedded systems
The specific characteristics of embedded systems lead to specific requirements of component technologies. In particular the approaches in development process and component specifications using interfaces are different form those implemented in the component technologies widely used in other domains.
The component interface summarizes the properties of the component that are externally visible to the other parts of the system. As for embedded systems non-functional properties are as important as functional there is a tendency to include specification of extra-functional properties in the component interface (for example timing properties). This allows more system properties to be determined when the system is designed, i.e. such interface enables verification of system requirements and prediction of system properties from properties of components.
In general-purpose component technologies, the interfaces are usually implemented as object interfaces supporting polymorphism
36
by late binding. While late binding allows connecting of components that are completely unaware of each other beside the connecting interface, this flexibility comes with a performance penalty and increased risk for system failure. Therefore the dynamic component deployment is not feasible for small embedded systems.
Taking into account all the constraints for real-time and embedded systems, we can conclude that there are several reasons to perform component deployment and composition at design time rather than run-time (Crnkovic & Larsson et al., 2002): This allows composition tools to generate a monolithic firmware for the device from the component-based design and in this way achieve better performance and better predictability of the system behavior. This also enables global optimizations: e.g., in a static component composition known at design time, connections between components could be translated into direct function calls instead of using dynamic event notifications. Finally, verification and prediction of system requirements can be done statically from the given component properties.
This implies that the component-based characteristic is mostly visible at design time. To achieve an efficient development process tools should exist which will provide support for component composition, component adaptation and static verification and prediction of system requirements and properties from the given component properties.
There may also be a need for a run-time environment, which supports the component framework by a set of services. The framework enables component intercommunication (those aspects which are not performed at design time), and (where relevant) control of the behavior of the components. Figure 2.1 shows different environments in a component life cycle. The figure is adopted from (Crnkovic & Larsson et al., 2002).
37
Figure 2.2. Component technology for small embedded Systems
1.7.2 Component-based approach for large embedded systems
For large embedded systems the resource constraints are not the primary concerns. Complexity and interoperability play a much more important role. Also due to complexity, the development of such systems is very expensive and cutting the development costs is highly prioritized. For this reason general-purpose component technologies are of more interest than in the case for small systems.
The technology mostly used in large systems is Microsoft COM and recently .NET, and to a smaller extent different implementations of CORBA, although none of these technologies provides support for real-time. The systems using these technologies belong to the category of soft-real time systems. Often a component technology is used as a basis for additional abstraction level support, which is specified either as standards or proprietary solutions. The main reason for wide use of component-based technology is the possibility of reusing solutions in different ranges of products, efficient development tools, standardized specifications and interoperation, and integration between different products.
One successful example of the adoption of a component-based technology is the initiative OPC Foundation (OLE process control Foundation, www.opcfoundation.org), an organization that consists of more than 300 member companies worldwide, it is responsible for a specification that defines a set of standard interfaces based upon Microsoft’s OLE/COM and recently .NET technology. OPC consists of a standard set of interfaces, properties, and methods for use in
38
process-control and manufacturing-automation applications. OPC provides a common interface for communicating with diverse process-control devices, regardless of the controlling software or devices in the process. The application of the OPC standard interface makes possible interoperability between automation/control applications, field systems/devices and business/office applications.
Another example of a component-based approach is development and use of the standard IEC 61131 (ISO/IEC 61131-3, 1995). IEC 61131 defines a family of languages that includes instruction lists, assembly languages, structured text, a high level language similar to Pascal, ladder diagrams, or function block diagrams (FBD). Function blocks can be viewed as components and interfaces between blocks are released by connecting in-ports and out-ports. Function lock execution may be periodic or event-driven. IEC 61131 has been successfully used in development of industrial process automation systems for many years.
Large embedded systems that must fulfill real-time requirements usually do not use general-purpose component-based technologies. However in some cases, such as for ABB controllers, a reduced version of COM has been used on top of a real-time operating system (Lüders et al., 2002). The reused version includes facilities for component specification using the interface description language (IDL), and some basic services at run-time such as component deployment has been used. These services have been implemented internally. Different communication protocols and I/O drivers have been identified as components.
1.8 Specific requirements for embedded systems
Embedded systems vary from very small systems to very large systems. For small systems there are strong constrains related to different recourses such as power or memory consumption and others. In most of the cases, embedded systems are real-time
39
systems. For these as well as for large embedded systems the demands on reliability, functionality, efficiency and other characteristics that depends on domain or application. Finally, in many domains, the product life cycle is very long – in can stretch to several decades.
All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes. Unfortunately, the priority quality attributes vary according to domain application. These properties can be classified in run-time and life cycle extra-functional properties. In this way, research developed by Crnkovic (Crnkovic, 2005) lists four main characteristics that must be considered to embedded systems, they are listed below:
(i) Real-time properties: a violation of time requirements even of a proper functional response violates the system functionality. The real-time properties:
a - Response time or latency, b - Execution time, c - Worst case execution time, d - Deadline.
(ii) Dependability is defined as an ability of a system to deliver a service that can justifiably be trusted and an ability of a system to avoid failures that are more severe and frequent than is acceptable to the users (Crnkovic, 2005). The main means to attain dependability are related to avoidance of faults: fault prevention, fault tolerance, fault removal and fault forecasting and it is characterized by several attributes (Avižienis et al., 2001):
a - Reliability, b - Availability,c- Integrity,
40
d- Safety, e -Confidentialityf - Maintainability.
(iii) Resource consumption. Many embedded systems have strong requirements for low and controlled consumption of different resources. The reasons may be the size of the systems and/or the demands on lower production costs. Examples of such restrictions and constraints are:
a - Power consumption, b - Memory consumption,c - Execution (CPU) time, d - Computation (CPU) power.
(iv) Life cycle properties. In many domains the embedded systems have very long life time running round the clock year after year. During the lifetime of a system several generations of hardware and software technologies can be used. The long life systems must be able to cope with these changes introduced either into the surrounding environment or into the systems themselves.
In this way, the research concludes that many of the most important requirements of the embedded systems are related to extra-functional properties. This implies that development and maintenance of such systems are very costly. In particular activities related to verification and guaranteed behavior (formal verification, modeling, tests, etc.) and maintenance (adaptive maintenance, debugging, regressive testing, etc.) require a lot of effort. For these reasons the technologies and processes that lead to lower costs for these activities are very attractive and desirable.
Table 2.2. Summary of relevant characteristics in Crnkovic’s research.
Characteristics Sub-characteristicsReal-time properties Response time or
41
latencyexecution time worst case execution time deadline
Dependability ReliabilityAvailabilityintegrityconfidentialitysafety
Resource consumption
Power consumption
computation (CPU) powermemory Consumptionexecution (CPU) time,
Life cycle properties maintainability
1.8.1 Industrial Automation
Typical application domains of industrial automation are in control of industrial processes, power supply, and industrial robots. Industrial automation domain comprises a large area of control, monitoring and optimization systems. They typically include large pieces of software that have been developed over many years (often several decades). Most control systems are manufactured in rather large volumes, and must to a large extent be configurable to suit a variety of customer contexts. They can be classified according to different levels of control (Crnkovic & M. Larsson, 2002):
(i) Process level (for example, a valve in a water pipeline, a boiler, etc.),
(ii) Field level that concerns sensors, actuators, drivers, etc. (iii) Group control level that concerns controller devices and applications which control a group of related process level devices in a closed-loop fashion,
42
(iv) Process control level i.e. operator stations and processing systems with their applications for plant-wide remote supervision and control and overview the entire process to be controlled, (v) Production or manufacturing management level that includes systems and applications for production planning.
Notice that, even if the higher levels are not embedded, they are of uttermost importance as they need to be interoperable with the lower level which greatly influences the possible choices of the component model and in fine the design choices. The integration requirements have in many cases led to a decision to use component technologies which are not appropriate for embedded systems but provide better integration possibilities. Depending on the level, the nature of the requirements and the implementation will be quite different. In general, the lower the level, the stronger are the real-time requirements (including timing predictability) and the resource limitations. Also, the component based approach will include different concepts at different levels. The most important quality attributes in industrial automation, following the researchers, is:
(i) lowest levels:a - Availability, b - Timeliness, c - Reliability
(ii) higher levels: a - Performance, b - Usability, and c - Integrability.
1.8.2 Automotive
To provide a domain specific classification of the importance of quality attributes for software in vehicles, and discusses how the
43
attributes could be facilitated by a component technology. Åkerholm (Åkerholm et al., 2005) executed a research in main vehicle industries. The research method is divided into three ordered steps:
1 - During the first step a list of relevant quality attributes was gathered;
2 - In the next step technical representatives from a number of vehicular companies placed priorities on each of the attributes in the list reflecting their companies view respectively; 3 - Finally a synthesis step was performed, resulting in a description of the desired quality attribute support in a component technology for vehicular systems.
The list of quality attributes have been collected from different literature trying to cover qualities of software that interest vehicular manufactures. In order to reduce a rather long list, attributes with clear similarities in their definitions have been grouped in more generic types of properties, e.g., portability and scalability are considered covered by maintainability. Although such grouping could fade the specific characteristics of a particular attribute, it put focus on the main concerns. In the ISO/IEC 9126 standard (ISO/IEC 9126, 2001), 6 quality attributes (functionality, reliability, usability, efficiency, maintainability, and portability) are defined for evaluation of software quality. However, the standard has not been adopted fully in this research; it is considered too brief and does not cover attributes important for embedded systems (e.g., safety, and predictability). Furthermore, concepts that sometimes are mixed with quality attributes (for example fault tolerance) are not classified as quality attributes, rather as methods to achieve qualities (as for example safety). Finally, functionality is of course one of the most important quality attributes of a product, indicating how well it satisfies stated or implied needs. However, it focuses on quality attributes beyond functionality often called extra-functional or non-functional properties. The resulting list of quality attributes is
44
presented below. Having been presented with the basic characteristics of quality attributes related to component technologies tailored for vehicular systems below:
1. Safety 2. Reliability 3. Predictability 4. Usability 5. Extendibility 6. Maintainability7. Efficiency 8. Testability 9. Security10. Flexibility
Representatives from the technical staff of several companies have been requested to prioritize a list of quality attributes, reflecting each of the respective companies’ view. The attributes have been grouped by the company representatives in four priority classes as shown in Table 2.1. The nature of the quality attributes imply that no quality attributes can be neglected. It is essential to notice that placing an attribute in the lowest priority class (4) does not mean that the company could avoid that quality in their software, rather that the company does not spend extra efforts in reaching it. The following companies have been involved in the classification process:
Volvo Construction Equipment Volvo Cars Bombardier Transportation Scania ABB Robotics
Table 2.2.2. Priority classes used to classify the importance of the different quality attributes.
45
Priority
Description
1 very important, must be considered2 important, something that one should try
to consider3 less important, considered if it can be
achieved with a small effort4 Unimportant, do not spend extra effort on
this
As the last step the authors provide a discussion where we have combined the collected data from the companies with the classification of how to support different quality attributes in (Larsson, 2004). The combination gives an abstract description of where, which, and how different quality attributes should be supported by a component technology tailored for usage in the vehicular industry.
Figure 2.3. The results. Y-axis: priority of quality attributes in a scale 1 (highest), to 4 (lowest). X-axis: the attributes, with the highest
prioritized attribute as the leftmost, and lowest as rightmost. Each of the companies has one bar for each attribute, textured as indicated
below the X-axis.
1.8.3 Medical
46
Wijnstra (Wijnstra et al., 2001) described their experience with quality attributes and aspects in the development of a medical imaging product family. To meet such requirements for a range of medical imaging products, a product family has been defined. Wijnstra distinguished between quality attributes and aspects as views. A quality attribute of an artifact is an observable property of the artifact. A quality attribute can be observable at development-time or at run-time.
Following Wijnstra, quality attributes and aspects are used to add structure to the various phases of the development process. They form a supporting means for achieving completeness, i.e. have all relevant concerns been taken into account? In a product family context where the family members are constructed from a component-based platform, it is especially useful to achieve aspect-completeness of components, allowing system composition without worrying about individual aspects.
Software architecture is defined as “the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships among them”. The architecture must accommodate the quality attributes as imposed on the system, which can be handled via structures in the high-level architecture, aspects, or rules & guidelines.
The most characteristic and important quality attributes found in Wijnstra’s research for the medical imaging product family is given below:
1. Reliability2. Safety3. Functionality4. Portability5. Modifiability
47
a. Configurabilityb. Extensibility and Evolvability
6. Testability7. Serviceability
1.8.4 Consumer Electronics
Consumer electronics products, such as TV, VCR, and DVD, are developed and delivered in form of product families which are characterized by many similarities and few differences and in form of product populations which are sets of products with many similarities but also many differences. Production is organized into product lines -this allows many variations on a central product definition.
A product line is a top-down, planned, proactive approach to achieve reuse of software within a family or population of products. It is based on use of a common architecture and core functions included into the product platform and basic components. The diversity of products is achieved by inclusion of different components. Because of the requirements for low hardware and production costs, general-purpose component technologies have not been used, but rather more dedicated and simpler propriety models have been developed. An example of such a component model is the Koala component model used at Philips (Ommering et al., 2000), (Ommering, 2002). Koala is a component model and an architectural description language to build a large diversity of products from a repository of components. Koala is designed to build consumer products such as televisions, video recorders, CD and DVD players and recorders, and combinations of them.
1.8.5 Other domains
There are many domains in which embedded systems are used extensively. Some of them are: Telecommunication, avionics and
48
aerospace, transportation, computer games, home electronics, navigation systems, etc. While there are many similarities between these domains there are also very different requirements for their functional and extra-functional properties. The consequences are that the requirements for component-based technologies are different, and consequently it expects to have one component model that summarizes all quality attributes in common in different domains. The expectations are that only one embedded quality component models will exist, some common characteristics, such as basic principles of component specifications through interfaces, basic composition and run-time services, certain patterns, and similar.
1.9 The needs and priorities in research
The component-based approach on system level, where hardware components are designed with embedded software, has been successfully used for many years. Also large-grain generic components like protocol stacks, RTOSs, etc. have been used for a long time. In addition to this, technology supporting a component-based approach has been developed; either in the form of proprietary component models, or by using reduced versions of some widely used component models. Still, there are needs for a number of improvements. Some of them are the following (differently important for different domains):
- There is a lack of widely adopted component technology standards which are suitable for embedded systems. For smaller-size embedded systems, it is important that a system composed of components can be optimized for speed and memory consumption, which is still missing in most of the component technologies available today.
49
- There is also a need for interoperability between different component technologies. Specification technology is not sufficiently developed to guarantee a priori component interoperability.
- Most current component technologies do not support important extra-functional properties
- There is a need for generic platform services, for, e.g., security and availability.
- Tools that support component based development are still lacking
- There is a lack of efficient implementations of component frameworks (i.e., middleware), which have low requirements on memory and processing power.
Major needs for the further development of component technology for embedded systems are the following (Brinksma et al., 2001).
- Need for adopted component models and frameworks for embedded systems. A problem is that many application domains have application-dependent requirements on such a technology.
- Need for light-weight implementations of component frameworks. In order to support more advanced features in component-based systems, the run-time platform must provide certain services, which however must use only limited resources.
- Obtaining extra-functional properties of components: Timing and performance properties are usually obtained from components by measurement, usually by means of simulation. Problems with this approach are that the results depend crucially on the environment (model) used for the measurements may not be valid in other environments, and that the results may depend on factors which cannot easily be controlled. Techniques should be developed for overcoming these problems, thereby obtaining more reliable specifications of component properties.
- Platform and vendor independence: Many current component technologies are rather tightly bound to a particular platform (either
50
run-time platform or design platform). This means that components only make sense in the context of a particular platform.
- Efforts to predict system properties: The analysis of many global properties from component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.
- Component certification: In order to transfer components across organizations, techniques and procedures should be developed for ensuring the trustworthiness of components.
- Component noninterference: Particularly in safety critical applications, there is a need to ensure separation and protection between component implementations, in terms of memory protection, resource usage, etc.
- Tool support: The adoption of component technology depends on the development of tool support. The clearly identified priorities of CBSE for embedded systems are:
- Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc.
- Development of widely adopted component models for real-time systems. Such a model should be supported by technology for generating necessary runtime infrastructure (which must be light-weight), generation of monitors to check conformance with contracts, etc.
51
3 Embedded Software
Component Quality and Certification: A Survey
This Chapter presents a survey of cutting edge embedded software component quality and certification research, in an attempt to analyze the trends in CBSE/CBD applied embedded systems projects and to probe some of the component quality and certification research directions.
Some relevant research works explore the theory of component certification in academic scenarios, but the literature is not so rich in reports related to practical embedded software component certification experience. In this way, this chapter presents a survey of embedded software component quality and certification research, from the early 90s until today. Another relevant aspect observed is that in the pioneering works the focus was mainly on mathematical and test-based models while more recently researchers have focused on techniques and models based on predicting quality requirements.
Works related to certification process in order to evaluate embedded software component quality are surveyed, however the literature contains several works related to software component quality achievement, such as: component and contracts (Urting et al., 2001), Feature Modeling of CBD embedded (Kalaoja, 1997), Security as a new dimension in embedded (Kocher, 2004), Quality attributes of a Medical Family (Wijnstra, 2001), Usage-based software certification process (Voas, 2000), component testing
52
(Councill, 1999), (Beydeda & Gruhn, 2003), component verification (Wallin, 2002), (Reussner, 2003). Since the focus of this survey is on processes for assuring component quality, it does not cover these works, which deal only with isolated aspects of component quality.
The first research published about embedded software certification system was focused on mathematical and test based models. In 1993, Poore (Poore et al., 1993) developed an approach based on the usage of three mathematical models (sampling, component and certification models), using test cases to report the failures of a system later analyzed in order to achieve a reliability index. Poore et al. were concerned with estimating the reliability of a complete system, and not just the reliability of individual software units, although, they did consider how each component affected the system’s reliability.
One year later, in 1994, Wohlin et al. (Wohlin et al., 1994) presented the first method of component certification using modeling techniques, making it possible not only to certify components but to certify the system containing the components as well. The method is composed of the usage model and the usage profile. The usage model is a structural model of the external view of the components, complemented with a usage profile, which describes the actual probabilities of different events that are added to the model. The failure statistics from the usage test form the input of a certification model, which makes it possible to certify a specific reliability level with a given degree of confidence. An interesting point of this approach is that the usage and profile models can be reused in subsequent certifications, with some adjustments that may be needed according to each new situation. However, even reusing those models, the considerable amount of effort and time that is needed makes the certification process a hard task.
53
One different work that can be cited was published in 1994. Merrit (Merrit, 1994) presented an interesting suggestion: the use of components certification levels. These levels depend on the nature, frequency, reuse and importance of the component in a particular context, as follows:
• Level 1: A component is described with keywords and a summary and is stored for automatic search. No tests are performed; the degree of completeness is unknown;
• Level 2: A source code component must be compiled and metrics are determined;
• Level 3: Testing, test data, and test results are added; and• Level 4: A reuse manual is added.
Although simple, these levels represent an initial component maturity model. To reach the next level, the component efficiency and documentation should be improved. The closer to level four, the higher is the probability that the component is trustworthy and may be easily reused. Moreover, Merrit begins to consider other important characteristics related to component certification, such as attaching some additional information to components, in order to facilitate their recovery, defining metrics to assure the quality of the components, and providing a component reutilization manual in order to help its reuse in other contexts. However, this is just a suggestion of certification levels and no practical work was actually done to evaluate it.
Subsequently, in 1996, Rohde et al. (Rohde et al., 1996) provided a synopsis of in-progress research and development in reuse and certification of embedded software components at Rome Laboratory of the US Air Force, where a Certification Framework (CF) for software components was being developed. The purpose of the CF was: to define the elements of the reuse context that are important to certification; to define the underlying models and
54
methods of certification; and, finally, to define a decision-support technique to construct a context-sensitive process for selecting and applying the techniques and tools to certify components. Additionally, Rohde et al. developed a Cost/Benefit plan that describes a systematic approach of evaluating the costs and benefits of applying certification technology within a reuse program.
After analyzing this certification process, Rohde et al. found some points that require improved formulation in order to increase the certification quality, such as the techniques to find errors (i.e. the major errors are more likely to be semantic, not locally visible, rather than syntactic, which this process was looking for) and thus the automatic tools that implements such techniques. In summary, Rohde et al. considered only the test techniques to obtain the defects result in order to certify software components. This is only one of the important techniques that should be applied to component certification.
Two years later, Voas (Voas, 1998) defined a certification methodology using automated technologies, such as black-box testing and fault injection to determine if a component fits into a specific scenario.
This methodology uses three quality assessment techniques to determine the suitability of a candidate COTS component;
(i) Black-box component testing is used to determine whether the component quality is high enough;
(ii) System-level fault injection is used to determine how well a system will tolerate a faulty component;
(iii) Operational system testing is used to determine how well the system will tolerate a properly functioning component, since even these components can create system wide problems.ww.trusted-components.org
55
The methodology can help developers to decide whether a component is right for their system or not, showing how much of someone else’s mistakes the components can tolerate.
According to Voas, this approach is not foolproof and perhaps not well suited to all situations. For example, the methodology does not certify that a component can be used in all systems. In other words, Voas focused his approach in certifying a certain component within a specific system and environment, performing several types of tests according to the three techniques that were cited above.
Another work involving component test may be seen in (Wohlin and Regnell, 1998), where Wohlin and Regnell extended their previous research (Wohlin et al., 1994), now, focusing on techniques for certifying both components and systems. Thus, the certification process includes two major activities:
(i) usage specification (consisting of a usage model and profiles), and
(ii) certification procedure, using a reliability model.
The main contribution of that work is the division of components into classes for certification and the identification of three different ways of certifying software systems:
(i) Certification process, in which the functional requirements implemented in the component are validated during usage-based testing in the same way as in any other testing technique;
(ii) Reliability certification of component and systems, in which the component models that were built are revised and integrated to certify the system that they form; and,
56
(iii) Certify or derive system reliability, where the focus is on reusing the models that were built to certify new components or systems.
In this way, Wohlin and Regnell provided some methods and guidelines for suitable directions to support software component certification. However, the proposed methods are theoretical without experimental study. According to Wohlin et al., “both experiments in a laboratory environment and industrial case studies are needed to facilitate the understanding of component reliability, its relationship to system reliability and to validate the methods that were used only in laboratory case studies” (pp. 09). Until now, no progress in those directions was achieved.
Another relevant work was presented in 2000 by Jahnke, Niere and Wadsack (Jahnke, Niere and Wadsack, 2000). They developed a methodology for semi-automatic analysis of embedded software component quality. This approach evaluates data memory (RAM) utilization by the component. This approach is applied in Java technology development components. Java is a multiplatform language, so components developed from a project in a determinate platform could be easily ported to another platform. This technology employs intensive use of RAM, additionally a virtual machine to interpret the code is necessary. However, embedded systems usually have a small RAM capacity.
The work is restricted because it verifies the component quality from only one point of view, use of data memory, among many other characteristics to be considered. Furthermore, the language most commonly used for embedded development is C and C + +, Java is widely used for the development of desktops systems.
In 2001, Stafford et al. (Stafford et al., 2001) developed a model for the component marketplaces that supports prediction of system properties prior to component selection. The model is
57
concerned with the question of verifying functional and quality-related values associated with a component. This work introduced notable changes in this area, since it presents a CBD process with support for component certification according to the credentials, provided by the component developer. Such credentials are associated to arbitrary properties and property values with components, using a specific notation such as <property,value,credibility>. Through credentials, the developer chooses the best components to use in the application development based on the “credibility” level.
Stafford et al. also introduced the notion of active component dossier, in which the component developer packs a component along with everything needed for the component to be used in an assembly. A dossier is an abstract component that defines certain credentials, and provides benchmarking mechanisms that, given a component, will fill in the values of these credentials. Stafford et al. finalized their work with some open questions, such as: how to certify measurement techniques? What level of trust is required under different circumstances? Are there other mechanisms that might be used to support trust? If so, are there different levels of trust associated with them and can knowledge of these differences be used to direct the usage of different mechanisms under different conditions?
Besides these questions, there are others that must be answered before a component certification process is achieved, some of these apparently as simple as: what does it mean to trust a component? (Hissam et al., 2003), or as complex as: what characteristics of a component make it certifiable, and what kinds of component properties can be certified? (Wallnau, 2003).
58
In 2002, Comella-Dorda et al. (Comella-Dorda et al., 2002) proposed a COTS software product evaluation process. The process contains four activities, as follows:
(i) Planning the evaluation, where the evaluation team is defined, the stakeholders are identified, the required resources is estimated and the basic characteristics of the evaluation activity is determined;
(ii) Establishing the criteria, where the evaluation requirements are identified and the evaluation criteria is constructed;
(iii) Collecting the data, where the component data are collected, the evaluations plan is done and the evaluation is executed; and
(iv) Analyzing the data, where the results of the evaluation are analyzed and some recommendations are given. However, the proposed process is an ongoing work and, until now, no real case study was accomplished in order to evaluate this process, becoming unknown the real efficiency to evaluate software components.
With the same objective, in 2003 Beus-Dukic et al. (Beus-Dukic et al., 2003) proposed a method to measure quality characteristics of COTS components, based on the latest international standards for software product quality (ISO/IEC 9126, ISO/IEC 12119 and ISO/IEC 14598). The method is composed of four steps, as follows:
(i) Establish evaluation requirements, which include specifying the purpose and scope of the evaluation, and specifying evaluation requirements;
(ii) Specify the evaluation, which include selecting the metrics and the evaluation methods;
59
(iii) Design the evaluation, which considers the component documentation, development tools, evaluation costs and expertise required in order to make the evaluation plan; and
(iv) Execute the evaluation, which include the execution of the evaluation methods and the analysis of the results.
However, the method proposed was not evaluated in a real case study, and, thus its real efficiency in evaluating software components is still unknown.
In 2003, Hissam et al. (Hissam et al., 2003) introduced Prediction- Enabled Component Technology (PECT) as a means of packaging predictable assembly as a deployable product. PECT is meant to be the integration of a given component technology with one or more analysis technologies that support the prediction of assembly properties and the identification of the required component properties and their possible certifiable descriptions. This work, which is an evolution of Stafford et al.’s work (Stafford et al., 2001), attempts to validate the PECT and its components, giving credibility to the model, which will be further discussed in this section.
During 2003, a CMU/SEI’s report (Wallnau, 2003) extended Hissam work (Hissam et al., 2003), describing how component technology can be extended in order to achieve Predictable Assembly from Certifiable Components (PACC). This new initiative is developing technology and methods that will enable software engineers to predict the runtime behavior of assemblies of software components from the properties of those components. This requires that the properties of the components are rigorously defined, trusted and amenable to certification by independent third parties.
SEI’s approach to PACC is PECT, which follows Hissam et al.’s work (Hissam et al., 2003). PECT is an ongoing research project that
60
focuses on analysis – in principle any analysis could be incorporated. It is an abstract model of a component technology, consisting of a construction framework and a reasoning framework. In order to concretize a PECT, it is necessary to choose an underlying component technology, to define restrictions on that technology to allow predictions, and to find and implement proper analysis theories. The PECT concept is portable, since it does not include parts that are bound to any specific platform. Figure 3.1 shows an overview of this model.
Figure 3.1. Structure of Prediction-Enabled Component Technology (Wallnau, 2003).
A system built within the PECT framework can be difficult to understand, due to the difficulty of mapping the abstract component model into the concrete component technology. It is even possible that systems that look identical at the PECT level behave differently when realized on different component technologies.
Although PECT is highly analyzable and portable, it is not very understandable. In order to understand the model, the mapping to the underlying component technology must be understood as well.
This is an ongoing work in the current SEI research framework. This model requires a better maturation by the software engineering community in order to achieve trust in it. Therefore, some future works are being accomplished, such as: tools development to automate the process, the applicability analysis of
61
one or more property theories, non-functional requirements certification, among other remarks. Moreover, there is still the need for applying this model in industrial scenarios and evaluating the validity of the certification.
Another expressive work was proposed by Magnus Larsson in 2004 (Larsson, 2004), A predictability approach of the quality attributes, where one of the main objectives of developing component-based software systems is to enable integration of components which are perceived as black boxes. While the construction part of the integration using component interfaces is a standard part of all component models, the prediction of the quality attributes of the component compositions is not fully developed. This decreases significantly the value of the component-based approach to building dependable systems. The authors classify different types of relations between the quality attributes of components and those of their compositions. The types of relations are classified according to the possibility of predicting the attributes of compositions from the attributes of the components and according to the impacts of other factors such as system environment or software architecture. Such a classification can indicate the efforts which would be required to predict the system attributes that are essential for system dependability and in this way, the feasibility of the component-based approach in developing dependable systems.
According to composition principles we can distinguish the following types of attributes:
(i) Directly compassable attributes. An attribute of an assembly which is a function of, and only of the same attribute of the components involved.
(ii) Architecture-related attributes. An attribute of an assembly which is a function of the same attribute of the components and of the software architecture.
62
(iii) Derived attributes. An attribute of an assembly which depends on several different attributes of the components.
(iv) Usage-depended attributes. An attribute of an assembly which is determined by its usage profile. System environment context attributes. An attribute which is determined by other attributes and by the state of the system environment.
This work is very useful, because after the component composition we have to predict and assure the system’s quality, but before this, the component quality must be known.
Finally, in 2006 Daniel Karlson (Karlson et al., 2006) presented novel techniques for the verification of component-based embedded system designs. These techniques were developed using a Petri net based modeling approach, called PRES+. Two problems are addressed: component verification and integration verification.
With component verification the providers verify their components so that they function correctly if given inputs conforming to the assumptions imposed by the components on their environment. Two techniques for component verification were proposed:
The first technique enables formal verification of SystemC designs by translating them into the PRES+ representation.
The second technique involves a simulation based approach into which formal methods are injected to boost verification efficiency.
Provided that each individual component is verified and is guaranteed to function correctly, the components are interconnected to form a complete system. What remains to be verified is the interface logic, also called glue logic, and the interaction between
63
components. Each glue logic and interface cannot be verified in isolation. It must be put into the context in which it is supposed to work. An appropriate environment must thus be derived from the components to which the glue logic is connected. This environment must capture the essential properties of the whole system with respect to the properties being verified. In this way, both the glue logic and the interaction of components through the glue logic are verified. Finally, the author presents algorithms for automatically creating such environments as well as the underlying theoretical framework and a step-by-step roadmap on how to apply these algorithms.
This approach verifies the component from only one perspective: functionality, but there is another perspective that has to be considered. The other consideration is formal verification, because much time and financial effort are employed. So, formal verification is used only in few cases when it is mandatory.
1.10 Failures in Software Component Certification
This section describes two failure cases that can be found in the literature. The first failure occurred in the US government, when trying to establish criteria for certificating components, and the second failure happened with an IEEE committee, in an attempt to obtain a component certification standard.
1.10.1 Failure in National Information Assurance Partnership (NIAP).
One of the first initiatives attempting to achieve trusted components was the NIAP. The NIAP is an U.S. Government initiative originated to meet the security testing needs of both information technology (IT) consumers and producers. NIAP is collaboration between the National Institute of Standards and
64
Technology (NIST) and the National Security Agency (NSA). It combines the extensive IT security experience of both agencies to promote the development of technically sound security requirements for IT products, systems and measurement techniques.
Thus, from 1993 until 1996, NSA and the NIST used the Trusted Computer Security Evaluation Criteria (TCSEC), a.k.a. “Orange Book.”5, as the basis for the Common Criteria6, aimed at certifying security features of components. Their effort was not crowned with success, at least partially because it had defined no means of composing criteria (features) across classes of components and the support for compositional reasoning, but only for a restricted set of behavioral assembly properties (Hissam et al., 2003).
1.10.2 Failure in IEEE.
In 1997, a committee was gathered to work on the development of a proposal for an IEEE standard on software components quality. The initiative was eventually suspended, in this same year, since the committee came to a consensus that they were still far from getting to the point where the document would be a strong candidate for a standard (Goulao et al., 2002a).
1.11 Standardization: Software Quality and Certification
One of the main objectives of software engineering is to improve the quality of software products, establishing methods and technologies to build software products within given time limits and available resources. Given the direct correlation that exists between software products and processes, the quality area could be basically divided into two main topics (Pressman, 2005):
5 http://www.radium.ncsc.mil/tpep/library/tcsec/index.html6 http://csrc.nist.gov/cc
65
Software Product Quality: aiming to assure the quality of the generated product (e.g. ISO/IEC 9126 (ISO/IEC 9126, 2001), ISO/IEC 12119 (ISO/IEC 12119, 1994), ISO/IEC 14598 (ISO/IEC 14598, 1998), SQuaRE project (ISO/IEC 25000, 2005) (McCall et al., 1977), (Boehm et al., 1978), among others); and
Software Processes Quality: looking for the definition, evaluation and improvement of software development processes (e.g. Capability Maturity Model (CMM) (Paulk et al., 1993), Capability Maturity Model Integrated (CMMI) (CMMI, 2000), Software Process Improvement and Capability dEtermination (SPICE) (Drouin, 1995), among others).
Currently, many institutions are concerned with creating standards to properly evaluate the quality of the software product and software development processes. In order to provide a general vision, Table 3.1 shows a set of national and international standards in embedded domain.
Table 3.1. Software quality standards.Standards Overview
ISO/IEC 61131 component-based approach for industrial systems
RTCA DO 178B guidelines for development of aviation software
ISO/IEC 61508 Security Life cycle for industrial software
ISO/IEC 9126 Software Products Quality CharacteristicsISO/IEC 14598 Guides to evaluate software product, based on
practical usage of the ISO 9156 standardISO/IEC 12119 Quality Requirements and Testing for Software
PackagesSQuaRE project(ISO/IEC 25000)
Software Product Quality Requirements and Evaluation
IEEE P1061 Standard for Software Quality Metrics Methodology
ISO/IEC 12207 Software Life Cycle Process.NBR ISO 8402 Quality Management and Assurance.NBR ISO 9000- Model for quality assurance in Design,
66
1-2 Development, Test, Installation and ServicingNBR ISO 9000-3 Quality Management and Assurance. Application
of the ISO 9000 standard to the software development process (evolution of the NBR ISO 8402).
CMMI (CapabilityMaturity ModelIntegration)
SEI’s model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes.
ISO 15504 It is a framework for the assessment of software processes.
The embedded software market has grown in the decade, as well as the necessity of producing software with quality and trust. Thus, obtaining quality certificates has been a major concern for software companies. In 2002, Weber (Weber et al., 2002) showed how this tendency influenced the Brazilian software companies from 1995 until 2002.
The number of companies looking for standards to assure the quality of their products or processes has grown drastically in the recent past. The graph on the left shows this growth in relation to ISO 9000, which assures the Quality Management and Assurance. The graph on the right shows this growth in relation to CMM, which assures the software development processes quality.
However, there is still no standard or effective process to certify the quality of pieces of embedded software, such as components. As shown in Chapter 1, this is one of the major inhibitors to the adoption of CBD. However, some ideas of software product quality assurance may be seen in the SQuaRE project (described next), which will be adopted as a basis for defining a consistent quality framework for embedded software components.
1.11.1 ISO/IEC 61131
67
IEC 61131 was developed by the International Electro-technical Commission with the input of vendors, end-users and academics, to provide a generic programming environment for the industry. IEC 61131 consists of five parts:
Table 3.2. IEC61131Part Title
Part 1 General informationPart 2 Equipment and test
requirementsPart 3 PLC programming languagesPart 4 User guidelinesPart 5 Communications
IEC 1131-3 is the international standard for programmable controller programming languages. As such, it specifies the syntax, semantics and display for the following suite of PLC programming languages:
Ladder diagram (LD) Sequential function Charts (SFC) Function Block Diagram (FBD) Structured Text (ST) Instruction List (IL)
One of the primary benefits of the standard is that it allows multiple languages to be used within the same programmable controller. This allows the program developer to select the language best suited to each particular task.
1.11.2 RTCA DO 178B
68
A special committee (SC-145) of the Radio Technical Commission for Aeronautics (RTCA) convened in 1980 to establish guidelines for developing airborne systems. The report “Software Considerations in Air-borne Systems and Equipment Certification” was published in January 1982 as the RTCA Document Order (DO)-178 (and revised as DO-178A in 1985).
Due to rapid advances in technology, the RTCA established a new committee (SC-167) in 1989 with the objective of updating the DO-178A by focusing on five areas: documentation integration and production, system issues, software development, software verification, and software configuration management and software quality assurance. The resulting document, DO-178B, provides guidelines for applicants developing software-intensive airborne systems.
The targeted DO-178B certification level is either A, B, C, D, or E. Correspondingly, these DO-178B levels describe the consequences of a potential failure of the software: catastrophic, hazardous-severe, major, minor, or no-effect.
Table 3.3. DO-178BDO-178B Certification Requirements
All items are not required at all certification levels.DO-178B Documents: DO-178B Records:
Plan for Software Aspects of Certification (PSAC)
Software Verification Results (SVR)
Software Development Plan (SDP) Problem ReportsSoftware Verification Plan (SVP) Software Configuration
Management RecordsSoftware Configuration Management Plan (SCMP)
Software Quality Assurance Records
Software Quality Assurance Plan (SQAP)Software Requirements Standards (SRS)Software Design Standards (SDS)Software Code Standards (SCS)
69
Software Requirements Data (SRD)Software Design Description (SDD)Software Verification Cases and Procedures (SVCP)Software Life Cycle Environment Configuration Index (SECI)Software Configuration Index (SCI)Software Accomplishment Summary (SAS)
The DO-178B certification process is most demanding at higher levels. A product certified to DO-178B level A would have the largest potential market, but it would require thorough, labor-intensive preparation of most of the items on the DO-178B support list. Conversely, a product certifying to DO-178B level E would require fewer support items and be less taxing on company resources. Unfortunately, the product would have a smaller range of applicability than if certified at a higher level.
1.11.3 ISO/IEC 61508
The International standard IEC 61508 “Functional safety of electrical / electronic / programmable electronic safety-related systems (E/E/PES)” is intended to be a basic functional safety standard applicable to all kinds of industry. IEC 61508 defines functional safety as: “part of the overall safety relating to the EUC (Equipment Under Control) and the EUC control system which depends on the correct functioning of the E/E/PE safety-related systems, other technology safety-related systems and external risk reduction facilities.”
The standard covers the complete safety life cycle, and may need interpretation to develop sector specific standards. It has its origins in the process control industry sector.
70
The safety life cycle has 16 phases which roughly can be divided into three groups as follows: phases 1-5 address analysis, phase 6-13 address realization and phase 14-16 address operation. All phases are concerned with the safety function of the system. The standard has seven parts. Parts 1-3 contain the requirements of the standard (normative), while 4-7 are guidelines and examples for development and thus informative.
Central to the standard are the concepts of risk and safety function. The risk is a function of frequency (or likelihood) of the hazardous event and the event consequence severity. The risk is reduced to a tolerable level by applying safety functions which may consist of E/E/PES and/or other technologies. While other technologies may be employed in reducing the risk, only those safety functions relying on E/E/PES are covered by the detailed requirements of IEC 61508.
IEC 61508 has the following views on risks:
zero risk can never be reached safety must be considered from the beginning non-tolerable risks must be reduced (ALARP)
1.11.4 ISO/IEC 25000 (SQuaRE project)
The SQuaRE (Software Product Quality Requirements and Evaluation) project (ISO/IEC 25000, 2005) has been created specifically to make two standards converge, trying to eliminate the gaps, conflicts, and ambiguities that they present. These two standards are the ISO/IEC 9126 (ISO/IEC 9126, 2001), which define a quality model for software product, and ISO/IEC 14598 (ISO/IEC 14598, 1998), which define a software product evaluation process, based on the ISO/IEC 9126.
Thus, the general objective for this next series is to respond to the evolving needs of users through an improved and unified set of
71
normative documents covering three complementary quality processes: requirements specification, measurement and evaluation. The motivation for this effort is to supply those responsible for developing and acquiring software products with quality engineering instruments supporting both the specification and evaluation of quality requirements.
SQuaRE will also include criteria for the specification of quality requirements and their evaluation, and recommended measures of software product quality attributes which can be used by developers, acquirers and evaluators. SQuaRE consists of 5 divisions as shows in Figure 3.2.
Figure 3.2. SQuaRE Architecture
The Quality Requirements Division (ISO/IEC 2503n) (ISO/IEC25030, 2007) contains the standard for supporting the specification of quality requirements, either during software product quality requirement elicitation or as an input for an evaluation process:
Quality requirements and guide: to enable software product quality to be specified in terms of quality requirements;
The Quality Model Division (ISO/IEC 2501n) (ISO/IEC 25010) contains the detailed quality model and its specific
72
characteristics and sub-characteristics for internal quality, external quality and quality in use. This division includes:
Quality model and guide: to describe the model for software product internal and external quality, and quality in use. The document will present the characteristics and sub-characteristics for internal and external quality and characteristics for quality in use.
The Product Quality General Division (ISO/IEC 2500n) (ISO/IEC 25000, 2005) contains the unit standards defining all common models, terms and definitions referred to by all other standards in the SQuaRE series. Readers are reminded that the Quality Management theme will deal with software products, in contrast to the distinct processes of Quality Management as defined in the ISO 9000 family of standards. This division includes two unit standards:
Guide to SQuaRE: to provide the SQuaRE structure, terminology, document overview, intended users and associated parts of the series, as well as reference models;
Planning and management: to provide the requirements and guidance for planning and management support functions for software product evaluation.
The standards in the Quality Measures Division (ISO/IEC 2502n) (ISO/IEC 25020, 2007) were derived from ISO/IEC 9126 and ISO/IEC 14598. This division covers the mathematical definitions and guidance for practical measurements of internal quality, external quality and quality in use. In addition, it will include the definitions for the measurement primitives for all other measures. This theme will also contain the Evaluation Module to support the documentation of measurements. This division includes:
73
Measurement reference model and guide: to present introductory explanations, the reference model and the definitions that is common to measurement primitives, internal measures, external measures and quality in use measures. The document will also provide guidance to users for selecting (or developing) and applying appropriate measures;
Measurement primitives: to define a set of base and derived measures, being the measurement constructs for the internal quality, external quality and quality in use measurements;
Measures for internal quality: to define a set of internal measures for quantitatively measuring internal software quality in terms of quality characteristics and sub-characteristics;
Measures for external quality: to define a set of external measures for quantitatively measuring external software quality in terms of quality characteristics and sub-characteristics;
Measures for quality in use: to define a set of measures for measuring quality in use. The document will provide guidance on the use of the quality in use measures.
The Quality Evaluation Division (ISO/IEC 2504n) (ISO/IEC 25040) contains the standards for providing requirements, recommendations and guidelines for software product evaluation, whether performed by evaluators, acquirers or developers:
Quality evaluation overview and guide: to identify the general requirements for specification and evaluation of software quality and to clarify the generic concepts. It will provide a framework for evaluating the quality of a software product and for stating the requirements for methods of software product measurement and evaluation;
Process for developers: to provide requirements and recommendations for the practical implementation of software
74
product evaluation when the evaluation is conducted in parallel with development;
Process for acquirers: to contain requirements, recommendations and guidelines for the systematic measurement, assessment and evaluation of software product quality during acquisition of “commercial-off-the-shelf” (COTS) software products or custom software products, or for modifications to existing software products;
Process for evaluators: to provide requirements and recommendations for the practical implementation of software product evaluation, when several parties need to understand, accept and trust evaluation results;
Documentation for the evaluation module: to define the structure and content of the documentation to be used to describe an Evaluation Module.
The next section will present more close the Quality Model Division, Quality Evaluation Division and Quality Measurement Division. These three divisions are the basis of the SQuaRE project and contain the guidelines/techniques that guide this thesis during the software component quality framework proposal. It is important to say that those five modules from SQuaRE have been in its draft version and, probably, some modifications will be made until its final version. Thus, the idea has accomplished the following modification according to its evolutions.
3.1.1.1 ISO/IEC 2501n (Quality Model Division)
The ISO/IEC 2501n (ISO/IEC 25010) is composed of the ISO/IEC 9126 -1(ISO/IEC 9126, 2001) standard, which provides a quality model for software product. At the present time, this division contains only one standard: 25010 – Quality Model and guide. This is an ongoing standard in development.
75
The Quality Model Division does not prescribe specific quality requirements for software, but rather defines a quality model, which can be applied to every kind of software. This is a generic model that can be applied to any software product by tailoring it to a specific purpose. The ISO/IEC 25010 defines a quality model that comprises six characteristics and 27 sub-characteristics (Table 3.2). The six characteristics are described next:
Functionality: The capability of the software to provide functions which meet stated and implied needs when the software is used under specified conditions;
Reliability: The capability of the software to maintain the level of performance of the system when used under specified conditions;
Usability: The capability of the software to be understood, learned, used and appreciated by the user, when used under specified conditions;
Efficiency: The capability of the software to provide the required performance relative to the amount of resources used, under stated conditions;
Maintainability: The capability of the software to be modified; and
Portability: The capability of software to be transferred from one environment to another.
Table 3.4. Characteristics and Sub-Characteristics in SQuaRE project.
Characteristics
Sub-Characteristics
Functionality SuitabilityAccuracyInteroperabilitySecurityFunctionality Compliance
Reliability MaturityFault ToleranceRecoverabilityReliability Compliance
Usability Understandability
76
LearnabilityOperabilityAttractivenessUsability Compliance
Efficiency Time BehaviorResource UtilizationEfficiency Compliance
Maintainability AnalyzabilityChangeabilityStabilityTestabilityMaintainability Compliance
Portability AdaptabilityInstallabilityReplaceabilityCoexistencePortability Compliance
The usage quality characteristics (i.e. characteristics that can be obtained from the developer or designer feedback of the product) are called Quality in Use characteristics and are modeled with four characteristics: effectiveness, productivity, security and satisfaction.
However, the main drawback of the existing international standards, in this case the ISO/IEC 25010, is that they provide very generic quality models and guidelines, which are very difficult to apply to specific domains such as COTS components and CBSD. Thus, the quality characteristics of this model should be analyzed in order to define and adequate to the component quality characteristics context.
A quality model serves as a basis for determining if a piece of software has a number of quality attributes. In conventional software development, to simply use a quality model is often enough, since the main stakeholders that are interested in software quality are either the developers or the customers that hired these developers. In both cases, the quality attributes may be directly observed and assured by these stakeholders.
3.1.1.2 ISO/IEC 2504n (Quality Evaluation Division)
77
The ISO/IEC 2502n (ISO/IEC 25040) is composed of the ISO/IEC 14598 (ISO/IEC 14598, 2001) standard, which provides a generic model of an evaluation process, supported by the quality measurements from ISO/IEC 9126. This process is specified in four major sets of activities for an evaluation, together with the relevant detailed activities (Figure 3.3). This is an ongoing standard in development.
Figure 3.3. ISO/IEC 25040
The ISO/IEC 2504n is divided in five standards: ISO/IEC 25040 – Evaluation reference model and guide; ISO/IEC 25041 – Evaluation modules; ISO/IEC 25042 – Evaluation process for developers; ISO/IEC 25043 – Evaluation process for acquirers; and ISO/IEC 25044 – Evaluation process for evaluators. Besides providing guidance and requirements for the software product evaluation process (ISO/IEC 25040 and ISO/IEC 25041), it provides three other standards that contain guides for different perspectives of software product evaluation: developers, acquires and evaluators.
3.1.1.3 ISO/IEC 2502n (Quality Measurement Division)
The ISO/IEC 2502n (ISO/IEC 25020, 2007) division tries to improve the quality measurements provided by previous standards like ISO/IEC 9126-2 (external metrics) (ISO/IEC 9126-2, 2003), ISO/IEC 9126-3 (internal metrics) (ISO/IEC 9126-3, 2003) and
78
ISO/IEC 9126-4 (quality in use metrics) (ISO/IEC 9126-4, 2003). However, this standard improves some aspects of quality measurement and the most significantly is the adoption of the Goal-Question- Metrics (GQM) paradigm (Basili et al., 1994), thus, the metrics definition becomes more flexible and adaptable to the software product evaluation context.
The ISO/IEC 2502n is divided in five standards: ISO/IEC 25020 - Measurement reference model and guide; ISO/IEC 25021 – Measurement primitives; ISO/IEC 25022 – Measurement of internal quality; ISO/IEC 25023 – Measurement of external quality; and ISO/IEC 25024 – Measurement of quality in use. These standards contain some examples in how to define metrics for different kinds of perspectives, such as internal, external and quality in use.
1.12 Software Component Certification
After presenting the main standards to reach software product quality, this section will discuss the main concepts involving software components certification, which is an attempt to achieve trust in software components.
According to Stafford et al. (Stafford et al., 2001), certification, in general, is the process of verifying a property value associated with something, and providing a certificate to be used as proof of validity.
A “property” can be understood as a discernable feature of “something”, such as latency and measured test coverage, for example. After verifying these properties, a certificate must be provided in order to assure that this “product” has determined characteristics.
Focusing on a certain type of certification, in this case component certification, Councill (Councill, 2001) has given a
79
satisfactory definition about what software component certification is, definition that was adopted in this thesis:
“Third-party certification is a method to ensure that software components conform to well-defined standards; based on this certification, trusted assemblies of components can be constructed.”
To prove that a component conforms to well-defined standards, the certification process must provide certificate evidence that it fulfills a given set of requirements. Thus, trusted assembly – application development based on third-party composition – may be performed with basis on the previously established quality levels.
Still, third party certification is often viewed as a good way of bringing trust in software components. Trust is a property of an interaction and is achieved to various degrees through a variety of mechanisms. For example, when purchasing a light bulb, one expects that the base of the bulb will screw into the socket in such a way that it will produce the expected amount of light. The size and threading has been standardized and a consumer “trusts” that the manufacturer of any given light-bulb has checked to make certain that each bulb conforms to that standard within some acceptable tolerance of some set of property values. The interaction between the consumer and the bulb manufacturer involves an implicit trust (Stafford et al., 2001).
In the case of the light-bulb there is little fear that significant damage would result if the bulb did not in fact exhibit the expected property values. This is not the case when purchasing a gas connector. In this case, explosion can occur if the connector does not conform to the standard. Gas connectors are certified to meet a standard, and nobody with concern for safety would use a connector that does not have such a certificate attached. Certification is a
80
mechanism by which trust is gained. Associated with certification is a higher requirement for and level of trust than can be assumed when using implicit trust mechanisms (Stafford et al., 2001).
When these notions are applied to CBSD, it makes sense to use different mechanisms to achieve trust, depending upon the level of trust that is required.
In order to achieve trust in components, it is necessary to obtain the components that will be evaluated. According to Frakes et al. (Frakes & Terry, 1996), components can be obtained from existing systems through reengineering, designed and built from scratch, or purchased. After that, the components are certified, in order to achieve some trust level, and stored into a repository system, as shows in Figure 3.4.
A component is certifiable if it has properties that can be demonstrated in an objective way, which means that they should be described in sufficient detail, and with sufficient rigor, to enable their certification (Wallnau, 2003). In order to do that a well-defined component quality model is needed, which incorporates the most common software quality characteristics that are present in the already established models, such as functionally, reliability and performance plus the characteristics that are inherent to CBSE.
Figure 3.4. Process of obtaining, certifying and storing components
81
Regarding the certification process, the CBSE community is still far from reaching a consensus on how it should be carried out, what are its requirements and who should perform it. Further, third party certification can face some difficulties, particularly due to the relative novelty of this area (Goulao et al., 2002a).
1.13 Summary of the Study
Figure 3.5 summarizes the timeline of research in the embedded software component quality and certification area, where the circle represents the proposed standard in this research area, from 1989 to 2004 (Figure 3.5). Besides, there were two projects that failed (represented by an “X”). The arrows indicate that a work was extended by another.
Figure 3.5. Research on embedded software component quality and certification timeline.
The research in the embedded component quality and certification area follows two main directions based on: (i) Formalism: How to develop a formal way to predict component properties? And, How to build components with fully proved correctness properties? And (ii) Component Quality Model: How to establish a well-defined component quality model and what kinds of component properties can be certified?
However, these works still need some effort to conclude the proposed models and to prove their trust, and needs a definition on
82
which requirements are essential to measure quality in components. Even so, a unified and prioritized set of CBSE requirements for reliable components is a challenge in itself (Schmidt, 2003).
1.14 Summary
This Chapter was divided in two parts. The first part of Chapter presented a survey related to the state-of-the-art in the embedded software component quality and certification research. Some approaches found in the literature, including the failure cases, were described. Through this survey, it can be noticed that embedded software components quality and certification is still immature and further research is needed in order to develop processes, methods, techniques, and tools aiming to obtain well-defined standards for component certification.
This second part of Chapter presented the main concepts about software quality and certification, in the context of this thesis, quality related to embedded software components in different domain. It also presented SQuaRE project, a software product quality requirements and evaluation standard that have some ideas regarding component quality assurance. Since trust is a critical issue in CBSE, this Chapter also presented some concepts of component certification, in general as shown, this is a still immature area, and some research is needed in order to acquire the reliability that the market expects from CBSE.
83
4 Embedded software
Component Quality Verification Framework
The evaluation of the quality of embedded software components (i.e. the assessment of their quality attributes) needs to be done by independent parties, at least until software vendors acquire the level of maturity that hardware vendors currently have. They still cannot count on the hardware data sheets and catalogues available for hardware components, which describe all their characteristics.
When a survey of the state-of-the-art in embedded software component certification research was carried out (presented in Chapter 3), it was noted that there is a lack of processes, methods, techniques and tools available in the literature to be used for evaluating component quality in general, and specifically for embedded is much more scarce. This need for processes, methods, techniques and tools to perform the component quality assurance is pointed out by different researchers (Voas, 1998), (Morris et al., 2001), (Wallnau, 2003), (Alvaro et al., 2005), and was evidenced by studies carried out by SEI (Bass et al., 2003), Softex (Softex, 2007) and Lucrédio et al. (Lucrédio et al., 2007). Most researchers agree
84
that component quality is an essential aspect of the CBSE adoption and software reuse success.
In this way, an Embedded Software Component Quality Verification Framework is proposed7. The framework tries to be as complete as possible, in order to provide all insights required for the evaluator to execute the component evaluation. Its idea is to improve the lack of consistency between the available standards for software product quality (ISO/IEC 9126), (ISO/IEC 14598), (ISO/IEC 12119), also including the software component quality context and extend it to the embedded domain. These standards provide a high-level definition of characteristics and metrics for software products but do not provide ways to be used in an effective way, becoming very difficult to apply them without acquiring more knowledge from supplementary sources.
Thus, the main goal of the Embedded Software Component Quality Verification Framework proposed is to provide modules that are consistent enough, each one complementing each other in order to be self-contained (i.e. all information needed to do the component evaluation task are available). In this task, a recent standard will be useful, SQuaRE project8, which has been continually developed and improved. In this way, an overview of the proposed framework is presented in the next session, and the following sessions will present each module with more details.
1.15 Overview of the Framework
Based on the robust framework for software reuse (Almeida et al., 2004) – presented on Chapter 1 – which is being developed by the Reuse in Software Engineering (RiSE) group, there must be a layer that considers the quality aspects of the assets developed during the 7 The framework follows the necessities of the Brazilian software development organizations (Lucrédio et al., 2007), which are looking to increase the productivity and quality of the developed software. Also, it follows the main incentives of the government in this area (Softex, 2007).8 The SQuaRE project presented in Chapter 3 has been developed with this intention but this initiative started in 2005 and until nowadays there are huge efforts around the world in order to finish the first version of the whole standard.
85
software reuse process. This layer is essential as assets reused without quality can decrease the improvements expected with software reuse benefits (Frakes & Fox, 1995) i.e. reusable assets without quality can impact the quality of the whole system and some effort will be needed to correct the errors and faults found. According to Councill (Councill, 2001), it is better to develop components and systems from scratch than reuse an asset without quality or with unknown quality consequently running the risk of negatively impacting the project planning, quality and time-to-market.
The layer that considers the quality aspects of the assets (components) was divided into two groups;
Quality of general propose software component, which is executed in a hardware platform Intel x86 , (server, desktop, notebooks and others) with operational systems: Windows, Linux, Solaris, MACOS and others supported by x86 platform.
Quality of embedded software components, which are executed in different kinds of hardware platforms with different kinds of operational systems.
The process of evaluating embedded software component quality is not a simple one. Firstly, there must be an embedded software component quality model (EQM). Differently from other software product quality models, such as (McCall et al., 1977), (Boehm et al., 1978), (Hyatt et al., 1996), (ISO/IEC 9126, 2001), (Georgiadou, 2003), this model should consider in the embedded domain, the Component-Based Development (CBD) characteristics, sub-characteristics and describe quality attributes that are specific to promote reuse. Moreover, it should be consistent enough to provide characteristics that complement each other with the
86
intention of providing a good quality model in the embedded software component context.
With an embedded software component quality model available, there must be a series of evaluation techniques that allow one to evaluate whether a component conforms to the model. It is very useful to provide the techniques that can be correlated to the quality characteristics proposed in the embedded quality model. Thus, this module is very important as it is impossible to evaluate the quality of the embedded software component without a set of efficient evaluation techniques that cover one or more quality characteristics provided by the embedded software component quality model (first module). Moreover, the techniques provided should consider a set of depth level in such cases where quality attribute analysis must be more or less rigid.
Consistent and good evaluation results can only be achieved by following a high quality and consistent evaluation process (Comella-Dorda et al., 2002). So, the correct usage of these evaluation techniques should follow a well-defined and controllable embedded software component certification process. Through this process the evaluation can be carried out more precisely and effectively, once the evaluator has some guidelines, templates and activities to follow. In addition, the main goal of a well-defined process is that the certification can be repeated and reproduced by different evaluators.
Finally, a set of metrics are needed, in order to track the component’s properties, the completeness of the embedded software component quality model proposed, the efficiency of the evaluation techniques used and the enactment of the certification process. The metrics are important to obtain the feedback of the whole framework and improve the quality of the modules. These four main issues:
(i) an Embedded software component Quality Model,
87
(ii) a Maturity Level Evaluation Techniques,(iii) a Metrics Approach, and(iv) a Component Certification Process,
are the modules of an Embedded Software Component Quality Verification Framework (Figure4.1). Each module will be presented in more detail in the next session.
The framework will allow that the embedded components produced in a Software Reuse Environment are certified before being stored in a Repository System. In this way, software engineers would have a greater degree of trust in the embedded components that are being reused and become more encouraged to reuse assets from the repository system.
According to the three perspectives considered in SQuaRE project, which was presented on Chapter 3, this framework could be used according to two perspectives: acquirers and evaluators. In the first perspective, it should be considered whether the customer has a set of components that contain the same requirements and functionalities, but have different costs, performance, and reliability attributes, among others. In this case, the framework could be applied in order to define which component best fits the customer’s needs and application/domain context. The second perspective should be considered for embedded software component evaluation required by companies in order to achieve trust in its own components, aiming to develop more reliable applications or to sell components with higher quality.
88
Figure 4.1. Embedded Software Component Quality Verification Framework.
On the other hand, this framework does not contemplate the developer’s perspectives, as the tasks, activities the knowledge in certain technologies and domains to evaluate software components makes it very hard for only one developer to execute all activities, independent of his knowledge. This is more clearly visible as the whole framework is presented (i.e. next sessions).
1.16 Embedded software component Quality Model
In the last decade, Component-Based Software Development (CBSD) has generated tremendous interest due to the development of reusable software, which has led to the concept of commercial off-the-shelf (COTS) embedded software components. This approach moves organizations from application development to application assembly. Constructing an application now involves the use of prefabricated pieces, perhaps developed at different times, by different people, and possibly with different uses in mind. The ultimate goal, once again, is to be able to reduce development time, cost, and effort, while improving the flexibility, reliability, and reusability of the final application due to the (re)use of software components already tested and validated.
In this way, assessment and evaluation of software components has become a compulsory and crucial part of any CBSD lifecycle. The
89
risk of selecting a product with unknown quality properties is no longer acceptable and, if this happens, may cause catastrophic results (Jezequel et al., 1997). Thus, the software components quality evaluation, in general, has become an increasingly essential activity in order to bring reliability in (re)using software components.
However, the dream of formal verification is complex and achievable at best only in the case of small source code components. The software factories cannot expect to formally guarantee the correctness of software components markets in order to build large scale software systems. Thus, software factories have to build their software systems with components that may be faulty, i.e. component with unknown quality (Sametinger, 1997). On the other hand, according to Stafford et al. (Stafford et al., 2001), commercial component vendors are not inclined to formally specify their software components too, and it is not certain that it would be cost effective for them to do so.
Unfortunately, it is strictly necessary to evaluate software component quality in order to support the component markets evolution and maturity. And most of the research dedicated to software components is focused on their functional aspects (i.e. component specification, component development, component tests, etc.). In this thesis proposal, the main concern is the evaluation of embedded software component quality, using the Embedded Software Component Quality Verification Framework.
First, it is necessary to define an Embedded software component Quality Model (EQM). However, there are several difficulties in the development of such a model (Goulão et al., 2002a), such as:
(1) which quality characteristics should be considered, (2) how to evaluate them, and
90
(3) who should be responsible for such evaluation.
In general, the main idea behind certification is to bring quality to a certain software product, in this case embedded software components. One of the core goals to achieve quality in components is to acquire reliability on it and, in this way, increase the component market adoption. Normally, the software component evaluation occurs through models that measure its quality. These models describe and organize the component quality characteristics that will be considered during the evaluation. So, to measure the quality of a software component it is necessary to develop a quality model. Thus, this session presents an Embedded software component Quality Model, its characteristics and sub-characteristics, and the quality attributes that compose the model.
1.16.1 The Embedded software component Quality Model (EQM)
In general, there is no consensus on how to define and categorize embedded software product quality characteristics. However, the proposed quality model tries to follow standard terminology as far as possible, in particular the one defined by ISO/IEC 25010.
A quality characteristic is a set of properties of a software product by which its quality can be described and evaluated. A characteristic may be refined into multiple levels of sub-characteristics. An attribute is a quality property to which a metric can be assigned, where a metric is a procedure for examining a component to produce a single datum, either a symbol (e.g. Excellent, Yes, No) or a number. Please note that not all properties are measurable.
A Quality model is the set of characteristics and sub-characteristics, as well as the relationships between them, that
91
provide the basis for specifying quality requirements and for evaluating quality (Bertoa et al., 2002). Of course, the quality model used will depend on the kind of target product to be evaluated. In this sense, the current standards and proposals define “generic” quality models.
The Embedded software component Quality Model (EQM) proposed is based on SQuaRE project (ISO/IEC 25000, 2005), named ISO/IEC 25010 standard, with adaptations for components and in embedded domain. The model is composed of marketing characteristics and some relevant component information that is not supported in other component quality models (Alvaro et al.,2005), (Goulão et al., 2002b), (Bertoa et al., 2002), (Meyer, 2003), (Simão et al., 2003), which will be presented next.
Although recent, some component quality models (Alvaro et al.,2005), (Goulão et al., 2002b), (Bertoa et al., 2002), (Meyer, 2003), (Simão et al., 2003) are described in the literature and analyzed in order to identify directions for proposing a well defined quality model for embedded software component evaluation. The negative and positive points of each model were considered in this study, aiming to identify the characteristics that are really important to such a model.
The first step is to identify several kinds of quality characteristics, classifying them according to different criteria.
First, it is necessary to discriminate between those characteristics that make sense for individual components (that it will be called local characteristics) and those that must be evaluated at the software architecture level (global characteristics).
The moment in which a characteristic can be observed or measured also allows to establish another classification (Preiss et al.,2001):
92
i. characteristics observable at runtime (e.g. Performance) and
ii. characteristics observable during the product cycle-life (e.g. Maintainability).
For embedded components, it is essential to distinguish between internal and external metrics. Internal metrics measure the internal attributes of the product (e.g. specification or source code) during design and coding phases. They are “white-box” metrics. On the other hand, external metrics concentrate on the system behavior during testing and component operation, from an “outsider” view. They are “black-box” metrics.
Finally, it is important to note that there are other kind of marketing characteristics such as price, technical support, license conditions, etc.—not directly related to quality—that may be of great importance when selecting components.
In this way, after analyzing these models and the ISO/IEC 25010, a The EQM was developed. The proposed EQM is composed of seven characteristics, as follows:
Functionality: This characteristic maintains the same meaning for components as for software products. This characteristic expresses the ability of a software component to provide the required services and functions, when used under specified conditions;
Reliability: This characteristic expresses the ability of the software component to maintain a specified level of performance, when used under specified conditions;
Usability: This characteristic expresses the ability of a software component to be understood, learned, used, configured, and executed, when used under specified conditions;
93
Efficiency: This characteristic expresses the ability of a software component to provide appropriate performance, relative to the amount of resources used;
Maintainability: This characteristic describes the ability of a software component to be modified, include corrections, improvements or adaptations to the software, due to changes in the environment, in the requirements, or in the functional specifications;
Portability: This characteristic is defined as the ability of a software component to be transferred from one environment to another; and
Marketability: This characteristic expresses the marketing characteristics of a software component, complementing the quality characteristics of this model.
Table 1 shows the characteristics and sub-characteristics which define the ISO 9126 general software quality model. From this quality model, the main idea is to refine and customize it in order to accommodate to the particular characteristics of embedded components.
Although the model is proposed following the ISO/IEC 25010 standard, some changes were made in order to develop a consistent model to evaluate software components:
The characteristics that were identified as relevant to the component context were maintained;
One characteristic that proved to be not interesting to evaluate components was eliminated;
The name of one of the characteristics was changed in order to adequate it to the component context;
Another level of characteristics was added, containing relevant marketing information for an embedded software component certification process; and
94
Some characteristics that complement the EQM with important component information were established.
The following section presents more details about these changes.
1.16.2 Changes in relation to ISO/IEC 25010
The sub-characteristics present in this model were found in research in the main characteristics in automotive, industrial, medical and others embedded domain, and a consensus of members and a set of software and quality engineers specialist in embedded of a Brazilian software factory.
Table 4.1 summarizes the changes that were performed in relation to ISO/IEC 25010. The characteristics and sub-characteristics that are represented in bold were not present in ISO/IEC 25010. They were added due to the need for evaluating certain CBSD-related properties that were not covered on ISO/IEC 25010. The sub-characteristic that is crossed was present in ISO/IEC 25010, but was removed in the proposed model. Finally, the sub-characteristic in italics had its name changed.
Table 4.1. Changes in the Proposed Embedded software Component Quality
Model, in relation to ISO/IEC 25010.Characteristics
Sub-Characteristics
Functionality Real-timeSuitabilityAccuracyInteroperabilitySecurityComplianceSelf-contained
Reliability MaturityRecoverabilityFault ToleranceSafety
95
Usability UnderstandabilityConfigurabilityLearnabilityOperability
Efficiency Time BehaviorResource behaviorEnergy consumptionMemory utilization
Maintainability AnalyzabilityStabilityChangeabilityTestability
Portability DeployabilityReplaceabilityFlexibilityReusability
Marketability Development timeCompatibles architectures CostTime to marketTargeted marketAffordabilityLicensing
1.16.2.1 Quality sub-characteristics that were created from ISO/IEC 25010
The Real-time sub-characteristic is essential to determine the correctness functionality the embedded component. In embedded domain practically all the systems had real-time requirements.
The Self-contained sub-characteristic is intrinsic to software components and must be analyzed.
The Safety is very important for reliability and dependable embedded systems, so this sub-characteristic is fundamental to compose the quality model.
The Configurability is an essential feature that the developer must analyze in order to determine whether a component can be easily configured. Through this sub-characteristic, the developer is
96
able to preview the complexity of deploying the component into a certain context.
The Energy consumption is crucial for portable, mobile, autonomy system and others. It is essential to determine the project viability, battery size, system autonomy and others.
The Memory utilization sub-characteristics gain emphasis in embedded domain because the resources such as code and data memory are very limited like code and data memory.
Further, the reason why software factories have adopted component-based approaches to software development is the premise of reuse. Thus, the Reusability sub-characteristic is very important to be considered in this model.
A brief description of each new sub-characteristic is presented, as follows:
Real-time : This characteristic indicates the ability to perform a specific task at the specific time, under specified conditions
Self-contained: The function that the component performs must be fully performed within itself;
Safety: Safety is an attribute involving the interaction of a system with the environment and the possible consequences of the system failure
Configurability: The ability of the component to be configurable (e.g. through a XML file or a text file, the number of parameters, etc.);
Energy consumption: This characteristic indicates the quantity of energy required for the component to perform a specific task.
Memory utilization: This characteristic indicates the quantity of data and code memory necessary for the component to perform a specific task.
97
Reusability: The ability of the component to be reused. This characteristic evaluates the reusability level through some points, such as: the abstraction level, if it is platform-specific or not, if the business rules are interlaced with interface code or SQL code, among other points.
Instead of concentrating on quality characteristics alone, another characteristic level was created, called Marketability (last row of Table 4.1). This characteristic presents some sub-characteristics that are important to a software component certification process, such as:
Development time: The time consumed to develop a component;
Compatibles architectures: The list of architectures or platforms that embedded component support.
Cost: The cost of the component; Time to market: The time consumed to make the component
available on the market; Targeted market: The targeted market volume; Affordability: How affordable is the component; and Licensing: The kind of licensing that the software component
is available.
This information is not important to evaluate component quality, but is important to analyze some factors that bring credibility to the component customers (i.e. developers and designers), for example, the time that component was available to the market means that the component is more mature, because bugs are corrected and test cases were applied; the kind of component license is interesting for the customer to analyze the cost/benefit of buying the component; and the target market of a component describes in which domains a certain component can be applied.
98
1.16.2.2 Quality sub-characteristics that were removed from ISO/IEC 25010
Additionally, some sub-characteristics were removed (crossed out) in order to adequate the model to the embedded component context, such as: Suitability, Interoperability, Understandability, Learnability, and Operability. These sub-characteristics are not present in the survey and they were judged to be of minor priority.
In the Maintainability characteristic, the Analyzability sub-characteristic disappeared. Its main concern, according to ISO/IEC 25010, is to assert if there are methods for performing auto-analysis, or identifying parts to be modified. Since a component is developed with some functionality in mind, this kind of auto analysis method is rarely developed. In fact, practical experience has shown that components do not have Analyzability characteristics (Bertoa et al., 2003). For this reason, it was decided, in conjunction with the Reuse in Software Engineering (RiSE) members and a set of embedded software and quality engineers of a Brazilian software factory, that the proposed Embedded software component Quality Model, similarly to (Goulão et al., 2002b), (Bertoa et al., 2002), would not contemplate this characteristic.
1.16.2.3 Quality sub-characteristics that were renamed from ISO/IEC 25010
Concurrently, two sub-characteristics had their names changed, as well as their meaning in this new context.
The Installability, which in the proposed model has the new name of Deployability. After being developed, the components are deployed (not installed) in an execution environment to make their usage possible by other component-based applications that will be further developed. Through this modification, the understandability of this sub-characteristic becomes clearer to the component context.
99
The Time behavior sub-characteristics had its name changed to Real-Time and it is part of functionality characteristics, because the time behavior in embedded systems is related to its correctness.
Another characteristic that changed its meaning was Usability. The reason is that the end-users of components are the application developer and designers that have to build new applications with them, more than the people (end-users) that have to interact with them. Thus, the usability of a component should be interpreted as its ability to be used by the application developer when constructing a software product or a system with it.
Finally, Adaptability was the last characteristic that changed names. Now, it’s called Flexibility. It indicates whether the component can be adapted or flexible to different specified environments.
Basically, the other characteristics of the model maintain the same meaning for embedded software components than for software products, except for little adaptations that are necessary to bring the ISO/IEC 25010 characteristics definition to the component context.
1.16.2.4 Quality characteristics that were extended from ISO/IEC 25010
The previous section presented the major changes, in relation to ISO/IEC 25010, that were introduced in the proposed component quality model. This section presents the quality characteristics from ISO/IEC 25010 that were maintained in the proposed model, with some adaptation to better reflect the CBSD scenario. These are described next:
Functionality: Accuracy: This characteristic evaluates the percentage of
results obtained with the correct precision level demanded;
100
Security: This characteristic indicates how the component is able to control the access to its provided services; and
Compliance: This characteristic indicates whether a component is conforming to any standard (e.g. international standard, certificated in any organization, etc.).
Reliability: Recoverability: This characteristic indicates whether the
component can handle error situations, and the mechanism implemented in that case (e.g. exceptions); and
Fault Tolerance: This characteristic indicates whether the component can maintain a specified level of performance in case of faults.
Efficiency: Resource behavior: This characteristic indicates the amount
of the resources used, under specified conditions.Maintainability:
Stability: This characteristic indicates the stability level of the component in preventing unexpected effect caused by modifications;
Changeability: This characteristic indicates whether specified changes can be accomplished and if the component can easily be extended with new functionalities; and
Testability: This characteristic measures the effort required to test a component in order to ensure that it performs its intended function.
Portability: Replaceability: This characteristic indicates whether the
component is “backward compatible” with its previous versions; and
1.16.3 Summary
101
According to the moment when a characteristic is observed or measured, it can be classified, as shown previously, in two kinds: characteristics that are observable at runtime (that are discernable at component execution time) and characteristics that are observable during the product life-cycle (that are discernable at component development and/or component-based system development). In this moment, the Table 4.2 shows classification for the proposed embedded software component quality model. However, the Marketability characteristic is not classified according to these two classifications.
Table 4.2. The Proposed Component Quality Model, with the sub-characteristics
divided into two kinds: runtime and life-cycle.Characteristic
sSub-Characteristics
Run-timeSub-
CharacteristicsLife cycle
Functionality Real-timeAccuracySecurity
SuitabilityInteroperabilityComplianceSelf-contained
Reliability RecoverabilityFault ToleranceSafety
Maturity
Usability Configurability UnderstandabilityLearnabilityOperability
Efficiency Time BehaviorResource behaviorScalabilityEnergy consumptionMemory utilization
Maintainability AnalyzabilityStability
ChangeabilityTestability
Portability Deployability ReplaceabilityFlexibilityReusability
Once the characteristics and sub-characteristics are defined, there must be a way to determine whether a component fulfills them or not. This is achieved through the use of attributes and metrics.
102
Normally, a quality model (Figure 4.1) consists of four elements:
(i) Characteristics, (ii) Sub-characteristics, (iii) Attributes and (iv) Metrics.
A quality characteristic is a set of properties of a software product through which its quality can be described and evaluated. A characteristic may be refined into multiple levels of sub-characteristic (ISO/IEC 25000, 2005).
Figure 4.2. Relations among the quality model elements.
An attribute is a measurable physical or abstract property of an entity. A metric defines the measurement method and the measurement scale. The measurement process consists of assigning a number or category to an attribute, according to the type of metric that is associated to that attribute (Square project).Next, the quality attributes of the EQM will be presented.
1.16.4 Embedded software Component Quality Attributes
Last section discussed the general points of the proposed embedded software component quality model. This section describes the quality attributes for each sub-characteristic proposed in EQM.
Table 4.2.4 shows the embedded software component quality attributes that are observable at runtime and life-cycle. The quality attributes that are observable during life cycle (Table 4.2.4), could be measured during the development of the component or the component-based system, by collecting relevant information for the model. The table groups the attributes by characteristics and sub-
103
characteristics, and indicates the metrics and kind of metrics used for evaluating each attribute.
Table 4.2.4. Embedded software Component Quality Attributes forSub-Characteristics those are observable at Runtime and Life-cycle.
Characteristics
Sub-Characteristics
(Runtime)
Sub-Characteristic
s(Life-cycle)
Attributes
Functionality
Real-time
1. Response time (Latency)
a. Throughput (“out”)b. Processing Capacity
(“in”)2. Execution time3. Worst case execution
time4. Dead line
Accuracy 5. Correctness
Security6. Data Encryption7. Controllability8. Auditability
Compliance9. Standardization10. Certification
Self-contained 11. Dependability
Reliability
Recoverability 12. Error Handling
Fault Tolerance13. Mechanism
availability14. Mechanism efficiency
Safety15. Environment analyze16. Integrity
Usability Configurability17. Effort to configure18. Understandability
Efficiency
Resource behavior 19. peripheral utilization
Energy consumption20. Mechanism
availability
Data Memory utilization21. Mechanism
availabilityProgram Memory utilization
22. Mechanism availability
Maintainability
Stability 23. Modifiability
Changeability24. Extensibility25. Customizability26. Modularity
Testability 27. Test suite provided28. Extensive component
test cases
10429. Component tests in a
specific environment30. Proofs the
components tests
Portability
Deployability 31. Complexity level
Replaceability32. Backward
Compatibility
Flexibility33. Mobility34. Configuration
capacity
Reusability
35. Domain abstraction level
36. Architecture compatibility
37. Modularity38. Cohesion39. Coupling40. Simplicity
Next, a brief description of each quality attributes is presented:
Functionality Characteristic:Run-time Sub-Characteristics
Real-time 1. Response time (Latency): This attribute measures the
time taken since a request is received until a response has been sent;
a. Throughput (“out”): This attribute measures the output that can be successfully produced over a given period of time;
b. Processing Capacity (“in”): This attribute measures the amount of input information that can be successfully processed by the component over a given period of time;
2. Execution time: This attribute measures the time that the component executes a specified task, under specified conditions.
105
3. Worst case execution time: This attribute measures the time that the component executes a specified task, under any conditions
4. Dead line: This attribute evaluates if the component executes a specified task by the time requirements
Accuracy 5. Correctness: This attribute evaluates if the component
executes as specified by the user requirementsSecurity
6. Data Encryption: This attribute expresses the ability of a component to deal with encryption in order to protect the data it handles;
7. Controllability: This attribute indicates how the component is able to control the access to its provided interfaces;
8. Auditability: This attribute shows if a component implements any auditing mechanism, with capabilities for recording users access to the system and to its data;
Life-cycle Sub-CharacteristicsCompliance
9. Standardization: This attribute indicates if the component conforms to international standards;
10. Certification: This attribute indicates if the component is certified by any internal or external organization;
Self-contained11. Dependability: This attribute indicates if the component
is not self-contained, i.e. if the component depends on other components to provide its specified services;
Reliability Characteristic:Run-time sub Characteristics:
Recoverability
106
12. Error Handling: This attribute indicates whether the component can handle error situations, and the mechanism implemented in that case;
Fault Tolerance 13. Mechanism availability: This attribute indicates the
existence of fault-tolerance mechanisms implemented in the component;
14. Mechanism efficiency: This attribute measures the real efficiency of the fault-tolerance mechanisms that are available in the component;
Safety15. Environment analyze: This attribute involves the
interaction of a system with the environment and the possible consequences of the system failure;
16. Integrity: This attribute is defined as the absence of improper system state alterations;
Usability Characteristic:Run-time Sub-Characteristics:
Configurability 17. Effort to configure: This attribute measures the ability
of the component to be configured;18. Understandability: This attribute measures the capacity
of the component to be understood and its correct use;Efficiency Characteristic:
Run-time Sub-Characteristics:Resource behavior
19. Peripherals utilization: This attribute specifies the Peripherals used by a component;
Energy consumption20. Mechanism availability: This attribute indicates the
existence of basic or complexity energy consumption reduction mechanisms implemented in the component;
Data Memory utilization
107
21. Mechanism availability: This attribute indicates the existence of data/main memory use optimization mechanisms implemented in the component;
Program Memory utilization22. Mechanism availability: This attribute indicates the
existence of program/stored memory use reduction mechanisms implemented in the component;
Maintainability Characteristic:Run-time Sub-Characteristics:
Stability23. Modifiability: This attribute indicates the component
behavior when modifications are introduced; andLife-cycle Sub-Characteristics:
Changeability24. Extensibility: This attribute indicates the capacity to
extend a certain functionality of the component (i.e. which is the percentage of the functionalities that could be extended);
25. Customizability: This attribute measures the number of customizable parameters that the component offers (e.g. number of parameters to configure in each provided interface);
26. Modularity: This attribute indicates the modularity level of the component in order to determine if it is easy or not to modify it, based in its inter-related modules;
Testability27. Test suite provided: This attribute indicates whether
some test suites are provided for checking the functionality of the component and/or for measuring some of its properties (e.g. performance);
28. Extensive component test cases: This attribute indicates if the component was extensively tested before being made available to the market;
108
29. Component tests in a specific environment: This attribute indicates in which environments or platforms a certain component was tested;
30. Proofs the components test: This attribute indicates if the component tests were formally proved;
Portability Characteristic:Run-time Sub-Characteristics:
Deployability31. Complexity level: This attribute indicates the effort
needed to deploy a component in a specified environment.Life-cycle Sub-Characteristics:
Replaceability32. Backward Compatibility: This attribute is used to
indicate whether the component is “backward compatible” with its previous versions or not;
Flexibility33. Mobility: This attribute indicates in which containers
this component was deployed and to which containers this component was transferred;
34. Configuration capacity: This attribute indicates the percentage of the changes needed to transfer a component to other environments;
Reusability35. Domain abstraction level: This attribute measures the
component’s abstraction level, related to its business domain;
36. Architecture compatibility: This attribute indicates the level of dependability of a specified architecture;
37. Modularity: This attribute indicates the modularity level of the component, if it has modules, packages or if all the source files are grouped in a single bunch;
38. Cohesion: This attribute measures the cohesion level between the inter-related parts of the component. A
109
component should have high cohesiveness in order to increase its reusability level;
39. Coupling: This attribute measures the coupling level of the components. A component should have low coupling in order to increase its reusability level; and
40. Simplicity: This attribute indicates if the component modules are well-defined, concise and well-structured.
A brief description of each quality attribute is presented next:
The Marketability characteristics (composed of marketing characteristics) are statically measured through the information that is available in each component (e.g. description, documentation, web-site, etc.). This information is normally made available by the component vendor in the component market web-site.
Besides the market and quality characteristics presented, the model is complemented with other kinds of characteristics. These characteristics bring relevant information for new customers and are composed of Productivity, Satisfaction, Security and Effectiveness. According to ISO/IEC 25010, these characteristics are called Quality in Use (ISO/IEC 25000, 2005). This is the user’s view (i.e. developers or designers) of the component, obtained when they use a certain component in an execution environment and analyze the results according to their expectations. These characteristics show whether the developers or designers can trust a component or not. Thus, Quality in Use characteristics are useful to show the component’s behavior in different environments.
These characteristics are measured through the customer’s feedback. A five-category rating scale is used, ranging from “Very Satisfied” to “Very Dissatisfied”. A “Don’t Know” option is also included. Using this scale, the Satisfaction, Productivity, Security
110
and Effectiveness of the component used by a certain user (developer or designer) can be measured. This user’s feedback is very important to the model in order to describe if a certain component is really good in practice, i.e. in a real environment. Of course, this evaluation is subjective, and therefore it must be analyzed very carefully, possibly confronting this information with other facts, such as the nature of the environment and the user’s characteristics.
As shown in Figure 4.1., a kind of measurement for each quality attribute proposed is necessary. However, the measurement of those quality attributes described during this session will be presented on session 4, which describes the paradigm adopted to define the metrics and give a set of examples to help the evaluation team during the metrics definition, respectively. The idea is to provide a more flexible way of developing the metrics during the evaluation runtime.
These attributes cover most of the important characteristics that help determining if a component has the desired quality level. However, there are other kinds of information that are important in the process of evaluating a component’s quality level, but these were not included in the model because they do not represent quality attributes for embedded software components. Instead, they contain relevant information for a well-defined component certification process. These are presented next.
1.16.5 Other relevant Component Information
In an effective embedded software component certification process, some additional information is needed in order to complement the model. Table 4.2.5 shows the additional characteristics that were identified as being interesting to an embedded software component certification process. These
111
characteristics are called Additional Information and are composed of: Technical Information and Organization Information.
Technical Information is important for developers to analyze the actual state of the component (i.e. if the component has evolved, if any patterns were used in the implementation, which kinds of technical support are available for that product, etc.). Besides, it is interesting to the customer that he knows who is the responsible for that component, i.e. who maintains that component (e.g. a Tata’s9
component is more reliable than a component created by an unknown or a new software factory). Thus, the necessity of the Organization Information was identified.
Table 4.2.5 Additional Information.Additional
InformationTechnical Information
Component Version Programming
Language Patterns Usage Architecture
compatible Program Memory used Technical Support
Organization Information CMMi Level Organization’s
Reputation
The Additional Information sub-characteristics are statically measured through the information that is available in each component (e.g. its description, documentation, web-site, etc.). This information is normally made available by the component vendor in the component market web-site. The characteristics are described by
9 Tata Consultancy Serviceshttp://www.tcs.com
112
a text field containing the information that was found (e.g. price, percentage, program memory used, etc.).
The Additional Information provides relevant component information to the model. The main concern is that these characteristics are the basic information to whatever kind of components available in the market.
1.17 Maturity Level Evaluation Techniques
Not all the selected quality characteristics and sub-characteristics proposed need to be evaluated with the same degree of thoroughness and depth for all types of embedded software components. Nobody would expect the same effort to be allocated to the component evaluation of a railway signal system, and a component from a game system. To ensure this flexibility, the evaluation should be level-oriented.
In this way, different evaluation levels must be used in order to provide specialized service for each kind of software components distributed on different domains and risk-levels, providing confidence in the quality of a software component in these domains, which showed an example in figure 4.3.1.
After the Embedded software component Quality Model (EQM) was defined, it was necessary to establish Maturity Level Evaluation Techniques to evaluate each embedded component quality characteristic and its. The thoroughness of an evaluation is a reflex of the evaluation techniques used. So, an Embedded software component Maturity Model (EMM) was defined. It is based on a model for general propose component created by Alvaro (Alvaro et al., 2007a).
113
Figure 4.3.1 A different evaluation level for each embedded component characteristics.
The figure 4.3.1 shows an example an embedded component whose characteristics were evaluated in different levels.
1.17.1 Embedded software component Maturity Model (EMM)
The model is constituted of quality characteristics levels where the components can be evaluated. There are five levels (they constitute a hierarchy), which identify the depth of the evaluation: evaluation at different levels gives different degrees of confidence in the quality of the embedded software component and the component could increase its level of reliability and quality as it evolves. Thus, each company/customer decides which level is better for evaluating its components, analyzing the cost/benefits of each level. The closer to the last level, the higher is the probability that the component is trustworthy, contains a consistent documentation and can be easily reused.
Thus, there are five levels which form an increasing set of evaluation requirements, where EMM I is the lowest level and EMM V is the highest level. The evaluation level defines the depth or thoroughness of the evaluation. Therefore evaluation at different levels gives different degrees of confidence in the quality of the embedded software component.
114
For instance, the level EMM V contains more rigorous evaluation techniques (requiring a high amount of time and effort to execute the evaluation) which are applied to give more confidence to the software component. On the other hand, as you decrease the EMM levels the techniques used are less rigorous and, consequently, less effort is applied during the evaluation.
The evaluation levels can be chosen independently for each characteristic (i.e. for functionality the component can be evaluated using the techniques from level EMM I, for reliability those techniques from level EMM III, for usability those techniques from level EMM IV and so on). The idea is to provide more flexibility during the selection levels as well, in order to facilitate the model usage and accessibility. Table 4.3.1 gives some indication as to which level a given embedded software component should be evaluated. Each vertical column of Table 4.3.1 represents different layers that the embedded software component should be considered when evaluating its potential damage and related risks. The level of damage in each layer is the first guideline used to decide which EMM level is more interesting for the organization; the important aspects are those related to environment, to safety/security and to economy. However, these are mere guidelines, and should not be considered as a rigid classification scheme. Those few guidelines were based on (Boegh et al., 1993), (Solingen, 2000), (ISO/IEC 25000, 2005) and extended to the component context.
Table 4.3.1. Guidelines for selecting evaluation level.Level Environment Safety/Security Economic Domain
EMM I No damage Few material damage; No specific risk
Negligible economic loss
Entertainment,
EMM II Small/Medium damage properly
Few people disabled
Few economic loss
household
EMM III Damage properly Large number of people disabled
Significant economic loss
Security, Control
115
systemsEMM IV Recoverable
environment damage
Threat to human lives
Large economic
gross
Medical, Financial
EMM V Unrecoverable environmental
damage
Many people killed Financial disaster
Transportation, Nuclear systems
Further, a set of appropriate evaluation techniques were defined. Relevant works from literature that propose evaluation techniques for software product were analyzed (Boegh et al., 1993), (Solingen, 2000), (Tian, 2004), (ISO/IEC 25000, 2005) and the experience of some embedded software quality specialists of Federal University of Pernambuco and a set of embedded software quality/system engineers from a Brazilian software factory helped during this definition. Afterwards, the feedback of relevant researchers on CBD from Mälardalen University10 (Department of Computer and Electrical Engineering) from Sweden and a specialist from ABB Company11 was very important to the model evolution.
Moreover, a set of works from the literature about each single technique were analyzed in order to identify the real necessity of those evaluation techniques. The analyzed works were from diverse areas, such as: software/component Quality Attributes (Larsson et al.,2204), software/component testing (Freedman, 1991), (Councill, 1999), (Gao et al., 2003), (Beydeda and Gruhn, 2003) software/component inspection (Fagan, 1976), (Parnas and Lawford, 2003), software/component documentation (Kotula, 1998), (Lethbridge et al., 2003), (Taulavuori et al., 2004), component interfaces and contracts (pre and post-conditions) (Beugnard et al., 1999), (Reussner, 2003), software/component metrics (Brownsword et al., 2000), (Cho et al., 2001), software/component reliability (Wohlin & Regnell, 1998), (Hamlet et al., 2001), (McGregor et al.,
10 http://www.mdh.se/ide11 http://www.abb.com
116
2003), software component usability (Bertoa et al., 2006), software/component performance (Bertolino & Mirandola, 2003), (Chen et al., 2005), component reusability (Caldiera & Basili, 1991), (Gui & Scott, 2007) and component proofs (Hall, 1990). In this way, those selected techniques bring, each one for each specific aspects, a kind of quality assurance to software components that are essential to integrate them into the evaluation techniques framework.
The establishment of what each level is responsible for is very valuable to the evaluation team during the definition of the evaluation techniques for those levels. In other words, the evaluation team knows what is more important to consider during the component evaluation and try to correlate these necessities with the most appropriate evaluation level. The intention was to build a model where the techniques selected to represent each level should complement each other in order to provide the quality degree needed for each EMM level. The EMM levels and the evaluation techniques are presented on Table 4.3.2. Additionally, in order to understand the meaning of each level, a brief description is presented next:
EMM I: the first level intends to investigate if the component does what is described in its documentation, its reusability level, the effort to use and maintain the component and its correct execution in defined environments. The aim of this level is the compatibility between the documentation and the component’s functionalities;
EMM II: the second step is concerned with the correct execution of the component, applying a set of tests techniques, inspecting the documentation better, if the component uses best practices in the chosen programming language and to evaluate the correct component deployment. The techniques of this level analyze the correct execution of the component;
117
EMM III: the main interests of this level are to evaluate how the component can avoid faults and errors, analyzing the provided and required interfaces looking for correct design and to evaluate a set of programming rules. The aim of this level is to analyze if the component can avoid or tolerate faults and errors during its execution;
EMM IV: in this level the source-code of the component is needed in order to inspect it more precisely. The code is inspected and tested, the provided and required interfaces are revisited and the algorithm complexity is examined in order to prove its performance too. An interesting aspect of this level is the analysis of the Component-Based Development (CBD) process, looking for possibilities of improving the CBD process adopted. The aim of this level is to assure the component’s performance; and
EMM V: the last level is considered the formal proof of the component’s functionalities and reliability. The architecture and the traceability are also examined in this level. Here, the idea is to achieve the highest possible level of trust. As a result, the techniques in this level tend to be the most costly and time consuming. Hence, the ROI (Return On Investment) analysis on this level is very important. The aim of this level is increase the trust in the component as much as possible.
One of the main concerns during EMM definition is that the levels and the evaluation techniques selection must be appropriated to completely evaluate the quality attributes proposed on the EQM, presented in session 4.2. This is achieved through a mapping of the Quality Attributes X Evaluation Technique. For each quality attribute proposed on the EQM, it is interesting that at least one technique is proposed in order to cover it completely, also being capable of measuring it properly. Table 4.3.3 shows this matching between the
118
EQM quality attributes and the proposed EMM evaluation techniques.
Table 4.3.3 shows that the main concern is not to propose a large amount of isolated techniques, but to propose a set of techniques that are essential for measuring each quality attribute, complementing each other and, thus, becoming useful to compose the Maturity Level Evaluation Techniques.
119
Table 4.3.2. Embedded software component Maturity Model.Characteristic
s EMM I EMM II EMM III EMM IV EMM V
Functionality Time constraint analysis Requirements and
Documentation Analysis Accuracy analysis
Evaluation measurement (Time analysis)
Functional Testing (black box), Unit Test, Regression Test (if possible)
System Test Documents Inspection (check
list) Code Inspection
Functional Tests (white-box) with coverage criteria and code inspection
Formal Proof
Reliability Dependability analysis Suitability analysis
Programming Language Facilities (Best Practices)
Error Manipulation analysis Fault tolerance analysis Error Injection analysis
Error recover Reliability growth
model
Formal Proof
Usability Effort to Configure analysis Documentation analysis (Use
Guide, architectural analysis, etc)
Interfaces inspection provided and required
Code and component’s interface inspection correctness and completeness)
Analysis of the pre and post-conditions of the component
User mental model
Efficiency Constraint analyses Accuracy analysis
Evaluation measurement (memory, power and resource)
Memory Analysis Power consumption Analysis Resource Analysis
Tests of performance(memory, power and resource)
Algorithmic complexity
Performance optimization (memory, power and resource)
Performance profiling analysis
Formal Proof
Maintainability
Customizability analysis Extensibility analysis
Inspection of Documents Analysis of the provided test suite (if
exists)
Code metrics and programming rules
Static Analysis
Analysis of the component development process
Traceability evaluation
Component Test Formal Proof
Portability Component execution in specific environment and architectural analysis
Deployment analysis Backward compatibility Mobility analysis
Conformity to programming rules
Environment and architectural constraints evaluation
Analysis of the component’s architecture
120
Cohesion, Coupling, Modularity and Simplicity analyses
Cohesion of the documentation with the source code analysis
Configurable analysis Hardware/Software analysis
Domain abstraction analysis
121
Table 4.3.3. Embedded Quality Attribute X Evaluation Techniques.
Charac-
teristic
Sub-Characteristic
s
QualityAttributes
Evaluation Techniques
Func
tion
alit
yReal-Time Response time (Latency)
a. Throughput (“out”)b. Processing Capacity
(“in”)
• Evaluation measurement (Time analysis)• Time constraint analysis• Formal Proof
Execution time • Evaluation measurementWorst case execution time • Evaluation measurement
• System TestDead line • Evaluation measurement
• System TestAccuracy Correctness • Requirements and
Documentation Analysis• Accuracy analysis• Functional Testing (black box),Unit Test, Regression Test (if possible)• Functional Tests (white-box) with coverage criteria
Security Data Encryption • System Test• Code Inspection
Controllability • System Test• Code Inspection
Auditability • System Test• Code Inspection
Compliance Standardization • Inspection of DocumentsCertification • Inspection of Documents
Self-contained Dependability • Documents Inspection• Code Inspection
Charac-
teristic
Sub-Characteristics
QualityAttributes
Evaluation Techniques
Rel
iabi
lity
Recoverability Error Handling • Programming Language Facilities (Best Practices)• Error Manipulation analysis• Error Injection analysis• Error recover• Reliability growth model• Formal Proof
Fault Tolerance Mechanism available • Suitability analysis• Dependability analysis
122Mechanism efficiency • Error injection analysis
• Programming Language Facilities (Best Practices)• Fault tolerance analysis• Reliability growth model• Formal Proof
Safety Environment analyze • Dependability analysis• Environment analyses• System analyses
Integrity • System analyses
Charac-teristic
Sub-Characteristic
s
QualityAttributes
Evaluation Techniques
Usa
bilit
y
Configurability Effort for configure • Effort to Configure analysis• Interfaces inspection provided and required • Code and component’s interface inspection correctness and completeness)• Analysis of the pre and post-conditions of the component• User mental model
Understandability Documentation analysis (Use Guide, architectural analysis, etc)
Charac-
teristic
Sub-Characteristics
QualityAttributes
Evaluation Techniques
Effic
ienc
y
Resource Behavior
peripheral utilization • Constraint analyses• Evaluation measurement• Resource Analysis• Tests of performance• Performance optimization• Performance profiling analysis• Formal Proof
123Energy consumption
Mechanism available • Constraint analyses• Evaluation measurement• Power consumption analysis• Tests of performance• Performance optimization• Performance profiling analysis• Formal Proof
Data Memory Utilization
Mechanism available • Constraint analyses• Evaluation measurement• Memory analysis• Tests of performance• Performance optimization• Performance profiling analysis• Formal Proof
Program Memory Utilization
Mechanism available • Evaluation measurement• Constraint analyses
Charac-
teristic
Sub-Characteristics
QualityAttributes
Evaluation Techniques
Mai
ntai
nabi
lity
Stability Modifiability • Code metrics and programming rules• Inspection of Documents• Static Analysis
Changeability Extensibility • Effort for operating• Extensibility analysis
Customizability • Customizability analysisModularity • Code metrics and
programming ruleTestability Test suit provided • Analysis of the test-suite
provided (if exists)Extensive component test cases
• Analysis of the component development process
Component tests in a specific environment
• Traceability evaluation
Proofs the components test • Component Test Formal Proof
Charac-
teristic
Sub-Characteristics
QualityAttributes
Evaluation Techniques
124
Port
abili
ty
Deployability Complexity level • Component execution in specific environments and architectural analysis• Deployment analyses• Environment and architectural constraints evaluation
Replaceability Backward Compatibility • Backward compatibility analysis
Flexility Mobility • Mobility analysesConfiguration capacity • Configuration analyses
Reusability Domain abstraction level • Cohesion of the documentation with the source code analysis• Domain abstraction analysis
Architecture compatibility • Conformity to programming rules• Analysis of the component’s architecture• Hardware/Software analysis
Modularity • Modularity analysesCohesion • Cohesion analysesCoupling • Coupling analysesSimplicity • Simplicity analyses
Those evaluation techniques are the start point where the software component evaluator sets up its work. The idea is to incrementally increase the appropriate techniques used during the previous component evaluation and through the evaluation feedback as well. Thus, the EMM will be composed of a set of techniques available to use and the software evaluator will decide which technique is better for each component evaluation, depending on the programming language, component domain, deployment environment, domain risk level, among other factors. It is very interesting that the evaluation team gives its feedback from the EMM techniques in order to increase the amount and quality of those techniques proposed in each level. The same idea is applicable to the Guidelines for selecting evaluation level (Table 4.31), where the guidelines should be improved through evaluations feedback.
125
Also, some initial guidance for estimating the cost of an evaluation can be given. The actual cost of evaluating will depend on the level of evaluation, the size of the component, the amount and quality of the available documentation, special requirements demanded by the customer, laws and regulations, and possibly other factors. No empirical data to prove these factors is available yet, however these factors should be considered as a start point before initiating a software component evaluation.
Based on the guidelines for selecting the evaluation level (Table 4.3.1) and the costs/benefits (some brief guidance cited above), the costumer can choose the level on which the component will be evaluated.
Each technique can be executed using a different kind of process, methods and tools, which depends, basically, on the programming language and deployment environment. The evaluator is responsible for that decision and it becomes very valuable if he/she stores the processes, methods and tools used to execute the techniques selected (this could be stored in a simple table describing each evaluation techniques selected X process/methods/tools defined). Thus, in the future, the evaluation team will have a huge table describing which processes, methods or tools were used for each evaluation technique during previously evaluation and, probably, help him/her in that new selection.
1.18 Metrics Approach
As with any engineering discipline, software development requires a measurement mechanism for feedback and evaluation. Measurement is a mechanism for creating a corporate memory and an aid in answering a variety of questions associated with the enactment of any software process (Basili et al., 1994).
126
Measurement also helps, during the course of a project, to assess its progress, to take corrective action based on this assessment, and to evaluate the impact of such action. Thus, some benefits are listed below:
It helps support project planning (e.g., How much will a new project cost?);
It allows us to determine the strengths and weaknesses of the current processes and products (e.g., How frequent are certain types of errors?);
It provides a rationale for adopting/refining techniques (e.g., What is the impact of the technique XX on the productivity of the projects?);
It allows us to evaluate the quality of specific processes and products (e.g., What is the defect density in a specific system after deployment?).
According to many studies made on the application of metrics and models in industrial environments, measurement, in order to be effective must be:
1. Focused on specific goals;2. Applied to all life-cycle products, processes and resources;3. Interpreted based on characterization and understanding of
the organizational context, environment and goals.
According to Basili et al. (Basili et al., 1994), the measurement must be defined in a top-down fashion. It must be focused, based on goals and models. A bottom-up approach will not work because there are many observable characteristics in software (e.g., time, number of defects, complexity, lines of code, severity of failures, effort, productivity, defect density), but which metrics one uses and how one interprets them it is not clear without the appropriate models and goals to define the context.
127
There are a variety of mechanisms for defining measurable goals that have appeared in the literature: the Quality Function Deployment approach (Kogure & Akao, 1983), the Goal Question Metric approach (Basili et al., 1994), (Basili, 1992), (Basili & Rombach, 1988), (Basili & Selby, 1984), (Basili & Weiss, 1984), and the Software Quality Metrics approach (Boehm et al., 1976), (McCall et al., 1977). However, in this work the Goal-Question-Metric (GQM) approach was adopted, which was the same technique proposed to use in ISO/IEC 25000 looking for track the software product properties.
The Goal Question Metric (GQM) approach is based upon the assumption that for an organization to measure in a purposeful way it must first specify the goals for itself and its projects, then it must trace those goals to the data that are intended to define those goals operationally, and finally provide a framework for interpreting the data with respect to the stated goals. Thus it is important to make clear, at least in general terms, what informational needs the organization has, so that these needs for information can be quantified whenever possible, and the quantified information can be analyzed as to whether or not the goals are achieved.
The result of the application of the Goal Question Metric approach application is the specification of a measurement system targeting a particular set of issues and a set of rules for the interpretation of the measurement data. The resulting measurement model has three levels:
1. Conceptual level - GOAL: A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view, relative to a particular environment. Objects of measurement are:
128
Products: Artifacts, deliverables and documents that are produced during the system life cycle; E.g., specifications, designs, programs, test suites.
Processes: Software related activities normally associated with time; E.g. specifying, designing, testing, interviewing.
Resources: Items used by processes in order to produce their outputs; E.g. personnel, hardware, software, office space.
2. Operational level - QUESTION: A set of questions is used to characterize the way the assessment/achievement of a specific goal is going to be performed based on some characterizing model. Questions try to characterize the object of measurement (product, process, resource) with respect to a selected quality issue and to determine its quality from the selected viewpoint; and
3. Quantitative level - METRIC: A set of data is associated with every question in order to answer it in a quantitative way. The data can be: Objective: If they depend only on the object that is being
measured and not on the viewpoint from which they are taken; E.g. number of versions of a document, staff hours spent on a task, size of a program.
Subjective: If they depend on both the object that is being measured and the viewpoint from which they are taken; E.g. readability of a text, level of user satisfaction.
A GQM model is a hierarchical structure starting with a goal. The goal is refined into several questions that usually break down the issue into its major components. Each question is then refined into metrics, some of them objective, some of them subjective. The same metric can be used in order to answer different questions under the same goal. Several GQM models can also have questions and metrics in common, ensuring that, when the measure is actually
129
taken, the different viewpoints are taken into account correctly (i.e., the metric might have different values when taken from different viewpoints).
The intention is the evaluators consider, during the metrics definition, a set of factors in which the collected results could be improved, such as: Meaning of the Metric, Costs and Complexity to measure, Repeatability, Reproductively, Validity, Objectivity and Impartial. Those factors are essential during the elaboration of the metrics using the GQM technique.
In this way, the GQM will support the evaluation team during the definition of the metrics in order to track the properties of the quality attributes described on EQM, the evaluation techniques presented on EMM as well as the whole certification process (that will be presented on session 4.5). An important aspect is considered the complexity to obtain the data of each metric proposed and if the selected metric can represent completely the information required by the evaluator.
Based on the modules of the Embedded Software Component Quality Verification Framework, some examples will be describe next as example of usage in order to provide insights to the team evaluation during the Define GQM step (vide Session 4.5 and the first activity of the embedded component certification process). The examples will be divided in three steps: (i) the metrics to track the properties of the EQM; (ii) the metrics to track the properties of the certification techniques presented on EMM; and (ii) the metrics to track the properties of the component certification process. However, only few examples of how the GQM is used to measure those properties will be presented in this session.
Thus, the main goal of this session is to provide some metrics definition to guide the team evaluation once there is a complete
130
absence of metrics that could help evaluate embedded software component quality attributes objectively available on literature (Bertoa et al., 2006). However, an important aspect when defining metrics is to be as objective as you can, even though it is frequently very difficult to do so because so many quality attributes are, in this definition, subjective.
Objective measures are those that depend only on the object of study and possibly some physical apparatus. In contrast, subjective measures are the product of some mental activity. While this distinction is clear, an illustration can help explain some less than obvious subsidiary points.
The virtue of objectivity is that the measure is repeatable. But repeatability does not suggest anything about other important qualities of the measure such as reliability. It is not clear whether cyclomatic complexity is more reliable than the proposed subjective measure, and, in fact, there is reason to think just the opposite (Wallnau, 2004).
In this way, the following metrics are defined, as much as possible, in an objective way. However, there are a set of metrics that depends of the evaluation context, evaluation team and the data that could collected during the evaluation, becoming subjective metrics (e.g. one example is the Effort to Configure quality attribute; if one does not have empirical data to classify how much effort is considered low, medium or high to configure certain components, the feeling/experience/knowledge of the evaluation team is necessary within the component’s and environment’s context).
1.18.1 Metrics to track the EMM Properties
Session 4.2 presented the Embedded Component Quality Model (EQM) with its related characteristics, sub-characteristics and quality attributes. Although some basic metrics to measure the
131
Quality in Use and Additional Information characteristics were presented (i.e. these metrics were defined early because the information needed to fulfill these characteristics is very simple and can be statically analyzed), now it is important define metrics to the other quality attributes of the EQM.
A few examples of metrics usage will be presented (in order to show how to define the metrics using GQM). However, it is important to highlight that those metrics must be defined in a component evaluation context. Those ones presented here are only to guide the evaluation team during the definitions of the metrics in the component’s first evaluation process.
For example, for Accuracy Sub-Characteristic the following
metric could be applied:
FunctionalitySub-Characteristic
Accuracy
Quality Attribute CorrectnessGoal Evaluates the percentage of
the results that were obtained with precision
Question Based on the amount of tests executed, how much test results return with precision?
Metric Precision on results / Amount of tests
Interpretation 0 <= x <= 1; which closer to 1 is better
An interesting aspect here is that the evaluation team defines the Interpretation of the metrics definition. Thus, the collection and analysis of those metrics becomes more feasible, repeatable, reproducible and easier to understand the results in a way that the whole team knows how the attribute was collected.
132
A subjective metrics, for example, the Understandability Sub-Characteristic could be measured using the following metric:
UsabilitySub-Characteristic
Configurability
Quality Attribute Document availableGoal Analyses the documentation
availabilityQuestion How much documents are available
to configure the component configurability?
Metric Number of documentsInterpretation As higher and with quality it is
better but it depends of the component complexity, domain, etc.
As shown, it is very complex to measure how configurable a component is. In this way, the metric provided is subjective and it is very dependable of the evaluation team (skills, knowledge, etc.).
1.18.2 Metrics to track the Certification Techniques Properties
With the same objective as the last section, now a few metrics to track the properties of the evaluation techniques of the Embedded software component Maturity Model (EMM), described on session 4.3, will be provided. Different from the last section, each technique proposed here can be measured in different ways and complexity, using different tools, techniques, methods and processes. Thus, the evaluation team should define with a degree of thoroughness which option is
more interesting to measure each evaluation technique proposed. Here, some recognized
tools or methods from literature will be used as basis, considering that the software
components to be evaluated were developed using C/C++ as programming language.
Functionality
133
Quality Attribute
Response Time
EMM level I
Technique Accuracy Analyzes using TPTP tool12
Goal Evaluates the percentage of the time taken between a set of invocations
Question Is the tool is efficient to measure the response time of this kind of component?
Metric Analysis of the results and coverage of the tool
Interpretation Definition the applicability of the tool to measure this quality attributes. If this tool can measure efficient the response time of a set of invocations, it is good. On the other hand, if it is not sufficient to evaluate some functions other tool should be use to complement or to substitute this one. If this tool it is good at evaluating the component, analysis of how many results are generated with the expected accuracy and the formula could be:
Number of results with accuracy /Number of results generated
0 <= x <= 1; which closer to 1 is better
As shown previously, one possible way to measure the quality attribute Response Time using the Accuracy Analysis technique could be through the Test & Performance Tools Platform Project (TPTP) tool. However, the evaluation team must know the tool usage in order to achieve its best utilization and efficiency in measuring the component quality.
1.18.3 Metrics to track the Certification Process Properties
12 Eclipse Test & Performance Tools Platform Project (TPTP) – http://www.eclipse.org/tptp
134
A consistent and good evaluation result can only be achieved by following a high quality and consistent evaluation process (Beus-Dukic et al., 2003). However, to assure the efficiency and efficacy of the process it is important to define some metrics. The idea is to obtain feedback from those metrics in order to improve the activities and steps proposed to evaluate the component quality (that will be presented on session 4.5). Next, two metrics that could be used for this purpose will be presented, as follows.
Component Certification ProcessGoal Adequately evaluate the software
componentQuestion Could the evaluation team evaluate
everything they planned to execute using the documents developed during the process activities?
Metric Total documented functionalities / Total component functionalities (or Total measurement accomplished)
Interpretation
0 <= x <= 1; which closer to 1 is better
Component Certification ProcessGoal Analyze the usability of the templates
providedQuestion Has the template helped during the
certification development?
Metric Evaluation team feedbackInterpretation
If the template helped during the certification development it is good, if not it should be adapted to improve the time of the component certification process.
The metrics presented above are specific to measure certain properties of the certification process however the evaluation team should define metrics as much as they think interesting in order to
135
measure the process capacity to evaluate the embedded software component quality.
1.19 Component Certification Process
The Component Certification Process showed is the default process adopted for component certification in robust framework for software reuse (Almeida et al., 2004) – presented on Chapter 1 – which is being developed by the Reuse in Software Engineering (RiSE) group. It is used for general propose component certification and embedded component certification
The definition of the Embedded Component Quality Model (EQM), the Maturity Level Evaluation Techniques and the Metrics Approach, in previous session, were important to present the quality attributes that must be considered to software component, how to evaluate those quality attributes using some pre-defined and well established techniques, and, how to measure those attributes through the GQM paradigm. However, it is essential to define a set of activities that should be followed in order to guide the evaluation team during the component evaluation. In this way, the component certification could be repeatable and reproducible once each activity contains a well-detailed description, its inputs and outputs, mechanisms to execute and to control.
Consistent and good evaluation results can only be achieved by following a high quality and consistent evaluation process (Comella-Dorda et al., 2003). This does not mean that each evaluation activity requires a highly complex, exquisitely documented process (although sometimes they do), but if you do not follow some kind of consistent process, it is likely that the quality of your results will vary.
In this sense, a Component Certification Process was proposed (Alvaro et al., 2007b) in order to define more precisely the activities necessary to evaluate the component quality.
136
1.19.1 The Component Certification Process
As presented on Chapter 3, the Component Certification Process is based on SQuaRE project, which provides guidance and requirements for the software product evaluation. The idea is to follow this standard as far as possible to develop a new one for software component quality evaluation. Moreover, a set of works from literature which includes processes for software product evaluation and processes for software component assessment aid during the definition of this process (McCall et al., 1977) , (Boegh et al., 1993), (Beus-Dukic et al., 2003), (Comella-Dorda et al., 2002).
In this context, a set of activities to guide the evaluation team during the evaluation were proposed (Figure 4.5.1), which is presented using SADT notation (Ross, 1997).
Figure 4.5.1. Component Certification Process.
Figure 4.5.1 has shown the macro-view of the whole component certification process. Now, each activity of the process will be described in more detail.
137
1.19.1.1 Establish Evaluation Requirements activity
This activity includes specifying the goals and scope of the evaluation, and specifying evaluation requirements. The evaluation requirements specification should identify the quality characteristics (using the Embedded software component Quality Model (EQM)) but also other aspects such as stakeholders to build the evaluation team, the component’s constraints and the component relationships with the software system (at least two systems) or with the target environment. Moreover, it is necessary define the evaluation team where the stakeholders should be carefully selected in order to assure a high-quality component evaluation.
The inputs of this activity are the Component Requirements Document, the Component’s Documentation available and the Embedded Component Quality Model (EQM). Based on these inputs, the evaluation team together with the evaluation customer will Establish the Evaluation Requirements and generate the Evaluation Requirements Specification as output, as shown in Figure 4.5.2.
Figure 4.5.2. Establish Evaluation Requirements activity.
Figure 4.5.2 has shown the steps that should be followed in order to complete execution of the Establish Evaluation Requirements activity and will be presented next.
138
Form an Evaluation Team: The importance of an effective evaluation team should not be underestimated, since without an effective team, an evaluation cannot succeed. People with different skills are essential to the evaluation team, such as: technical experts, domain experts, business analyst, contract personnel, end users, among others. A good balance of power is also important for a good team. The other skills necessary will depend of the evaluation scope and objectives, and can include security professionals, maintenance staff, etc.
According to (Comella-Dorda et al., 2003), to be successful the evaluators need to: (i) understand the impact of software component in the system development process and on the target environment, (ii) determine evaluation requirements for software component; (iii) define software component quality attributes; (iv) select software component evaluation techniques; and (v) employ a software component evaluation process that address the inherent tradeoffs.
While there are no rules for identifying evaluation stakeholders, errors of inclusion (i.e. including additional individuals or groups) are less risky than errors of exclusion (i.e. omitting legitimate stakeholders). Thus, if there is any doubt about including or not one stakeholder it is more interesting to add him/her to the evaluation process. However, the number of stakeholders that will be composed the evaluation team will depend of several aspects, such as: the size of the component, the complexity of the problem solved, the target domain, the algorithm complexity, among other factors. The essential stakeholders in whatever kind of evaluation are the customer and the evaluator responsible (which has large knowledge, at least, in the target domain and in the component technology). More stakeholders could be added to the evaluation team, however, according to (Comella-
139
Dorda et al., 2003), more than 7 stakeholders are not so interesting to the evaluation; Define the Goals and Scope: Some important questions should be answered before the evaluation tasks begins, such as:
o What is the evaluation expected to achieve?
o What are the responsibilities of each member of the team?
o When should the evaluation finish?
o What constraints must the evaluation team adhere to?
o What is the related risk of the component to its target domain?
Often, this basic information is not documented. Moreover, the people tend to be more productive when they have a clear picture of the ultimate goals of the project, in this case, about the component evaluation process. Thus, the evaluation team should answers these questions, focusing on:
o Goals of the evaluation;
o Scope of the evaluation;
o Names of team members and their roles;
o Component and domain risk-level;
o Statement of commitment from both stakeholder(s) and customer(s); and
o Summary of decisions that have already been made.
So, the evaluation team defines the goals, the scope and the related risk-levels of the component domain together with the evaluation customer.
Moreover, it is important to describe, in a general way, the component functionalities and the domain of the component in order to understand them more precisely;
140
Analyze the System and Environment: The evaluation team should analyze the software system (at least two systems that use the software component in evaluation) in order to define the impact of the software component in these system(s), which requirements and functionalities have impact, the architecture constraints, programming language constraints, environment constraints, etc. On the other hand, if the component will be evaluated independent of a software system, the evaluation team should define, as precisely and in as much detail as they can, the environment that the component will be evaluated in (i.e. target environment deploy, target domains of this component, version of the related tools, supplier of those tools, environment constraints, etc.). This step is important to define how large the complexity of the component’s evaluation will be using that environment or that system specified, answering a set of questions:
o How much effort will be spent on providing the whole infrastructure to evaluate the component? What is the complexity and constraints of this environment?
o What is the size of those selected systems? What is the target domain of those systems?
o What is the impact of the software component into selected system?
o What are the component dependencies?
In relation to these last questions, the evaluation team should analyze whether the component has dependencies with other components, modules, systems, etc. So, all dependencies should be described in this step in order to comprehend the behavior of the software components exactly (dependencies, cohesion, coupling, etc.); Define the Quality Characteristics: This step will define a set of quality characteristics that should be considered
141
during the component’s evaluation. Based on the Embedded Component Quality Model (EQM), the evaluation team must define the quality characteristics and the sub-characteristics that will be used to evaluate the software component. It is interesting that the importance related to each quality characteristic should be defined, thus, this identification will aid the next activity where the team evaluator should define the depth of the evaluation. Next, the evaluation team should discuss the selected quality characteristics and its related importance with the customer in order to achieve an agreement between both sides. The levels to considering the importance of the component could be: 1-Not Important; 2-Indiferent; 3-Reasonable; 4-Important; 5-Very Important. A simple table can help in this identification, as show in Table 4.5.1.
Table 4.5.1. Example of Importance’s definition.Characteristic
sSub-
CharacteristicsImportance
Functionality Accuracy 4Functionality Security 3
. . . . . . . . . . . .
Specify the Required Documentation: Based on the quality characteristics selected and its importance level, the evaluation team should define which documentation, assets and information are necessary to execute the evaluation. This information should be shared with the customer in order for him/her provide the information required. After obtaining those documents and attaching them into the evaluation process, the next step could be performed; Define External Quality Characteristics: Besides the quality characteristics presented on the EQM, it is possible that quality characteristics that are not covered on that model are present. In this case, the evaluation team should define the
142
“new” characteristic and reference it based on considerable works in the literature in order to clarify the definition and the importance of the quality characteristic. Further, the evaluation team should define the sub-characteristics and quality attributes of this “new” characteristic defined and attached all information in the Table 4.5.1 presented above, and, at the end of the component evaluation process the team should analyze if it is interesting to put those external quality characteristics in the EQM; Develop the Evaluation Requirements Specification Document: The last step of this activity is the elaboration of a document with all the information collected during the Establish Evaluation Requirements activity and generate the Evaluation Requirements Specification Document which is the main input of the next activity.
1.19.1.2 Specify the Evaluation activity
This activity includes defining the evaluation level, through the Guidelines for Selecting Evaluation Level; the definition of the evaluation techniques used to evaluate each level defined previously, through the Maturity Level Evaluation Techniques; and selecting the metrics that will be used to collect information about all steps of the evaluation process, through the Metrics Approach. The goal here is to detail the specification level as far as possible in order to assure the reproducibility and repeatability of the evaluation. The idea is to assure that other groups of people, which do not participate in the evaluation, can understand and execute the same evaluation again.
The evaluation team should define metrics to track the properties of the quality characteristics and the techniques adopted as well as the whole certification process. An important aspect here is the consideration of the complexity to obtain the data of each
143
metric required and whether the selected metric can represent completely the information required by the evaluation team.
The inputs of this activity are the Evaluation Requirements Specification, the Embedded Component Quality Model (EQM), the Maturity Level Evaluation Techniques and Metrics Approach. The Quality Characteristics Importance, defined in the last activity, acts as control in this activity. Thus, using those assets, the evaluation team will develop the Evaluation Requirements Specification and generate the Specification of the Evaluation as output, according show in Figure 4.5.3.
Figure 4.5.3. Specify the Evaluation activity.
Figure 4.5.3 has shown the steps that should be following to complete executing the Specify the Evaluation activity and will be presented next.
Specify the Quality Attributes: Based on the characteristics and sub-characteristics defined by the previous activity (developed on Define the Quality Characteristics step and Define External Quality Characteristics step), the evaluation team needs to define the quality attributes of each sub-characteristics. For those EQM’s quality characteristics, the evaluation team can use the EQM in order to help them in
144
definition of the quality attributes. And, for those that are external characteristics the evaluation team should define the quality attributes to compose and complement the quality characteristic (if this was not performed yet).
Thus, the evaluation team assures the complete definitions of the quality characteristics necessary to evaluate the software components, as shows on Table 4.5.2. After this step, no more characteristics will be added to the evaluation process and, thus, the evaluation team must guarantee the complete definition of those quality characteristics, as much as possible.
Table 4.5.2. Example of Quality Attributes Definition.Characteristic
sSub-
CharacteristicsQuality
AttributesImportance
Functionality Accuracy Correctness 4Functionality Security Data Encryption 3… … … …
Define the Embedded software component Maturity Model (EMM) level: After specify all quality characteristics and attributes, the evaluation team must define which evaluation technique should be useful to evaluate those attributes. The evaluation team should consider the importance of each quality characteristic (developed on Define the Quality Characteristic step), described in the last activity. Thus, they will define the ECMM level that the software component should attend, using a set of guidelines for selecting evaluation level as basis for this decision (presented in session 4.3 on Table 4.3.1). Thus, the evaluation team selects a mix of levels, where each quality characteristic will be evaluated in a certain EMM level (e.g. functionality will be evaluated on EMM I level, reliability will be evaluated on EMM III level and so on). The report will contain the level achieved for each characteristic separately (e.g. Functionality – EMM II, Reliability – EMM I,
145
Usability – EMM III, etc). The decision of the directions to be followed during the certification should be communicated to the customer until going to the next step. Table 4.5.3 shows the decision of the evaluation techniques for each quality attributes.
Table 4.5.3. Example of Define EMM.Characteristic
sSub-
Characteristics
QualityAttributes
Importance EMM Level/Evaluation Technique
Functionality Accuracy Correctness 4 II. Black-Box Testing
Functionality Security Data Encryption
3 III. Code Inspection
… … … … …
Further, if any external quality characteristic was proposed in the last activity (developed on Define External Quality Characteristics step), the evaluation team should define how this “new” quality characteristic will be evaluated through techniques available in literature or through one of the techniques represented in the Techniques Certification framework (presented on Chapter 4.3); Analysis of the Certification Techniques: The EMM level chooses on last step contains a set of evaluation techniques for each characteristic. The evaluation team, using their expertise, knowledge and know-how in those techniques must analysis such techniques and decide if those techniques are useful to evaluate the target software component or if it is necessary to define other techniques for executing / complement the evaluation. The idea is to define the best technique for each kind of evaluation. It is important to consider here that all of those techniques presented in EMM are recommended techniques to guide the evaluation team during the selection process, however, it is not avoided the proposal of new techniques that is better to evaluate the target
146
software component. Moreover, the evaluation team should justify the adoption of a new technique and if it is reasonable it can be incorporated into the EMM, looking for the incremental improvement of the model; Define Goal-Question-Metric (GQM): The evaluation team will define metrics to track the properties of the quality characteristics selected, the techniques adopted as well as the whole certification process. Through the Metrics Approach (described in session 4.4) the evaluation team will define:
o the metrics to evaluate those quality attributes selected from EQM or from external sources;
o the metrics necessary for each EMM technique used, which requires at least one metric for each evaluation technique that will be used; and
o the metrics to measure the efficiency of the component certification process;
The information collected using these metrics will support the quality assurance of the component evaluation and also provide insights for the next evaluations once they will be available on the Component Evaluation’s Repository; Establish Punctuation Level for Metrics: After defining all the metrics, the evaluation team should consider a punctuation level to facilitate its analysis. For example, if in a certain component evaluation if the metric defined to measure the Security quality attribute achieved less than 40%, it is not accepted; between 41% and 80% it could be considered reasonable; and higher than 81% it is considered acceptable and could receive the certification agreement. This punctuation level will be dependent on the importance of each quality characteristic (Define the Quality Characteristic step) and the evaluation level (Define EMM level step) defined during this activity. However, there is a kind of scale that should be considered as basis in order to start the establishment of the
147
punctuation level. The scale is defined on ISO/IEC 15504-2 (ISO/IEC 15504-2, 2000), as follows:
o N (Not Achieved): 0% to 15% - There is little or no evidence about the presence of the quality attribute on the component;
o P (Partially achieved): 16% to 50% - There is evidence about the presence of the quality attribute on the component. However, the quality attribute aspects have been partially achieved;
o L (Largely achieved): 51% to 85% - There is evidence about the presence of the quality attribute on the component. The component provides the quality attribute aspects implemented but it has not been completely achieved; and
o F (Fully achieved): 86% to 100% - The component provide all necessary fulfillments for the quality attribute in evaluation.
The evaluation team should analyze if the scale proposed on ISO 15504-2 makes sense to the component in evaluation. This is dependent on the reliability level expected by the component, i.e. the EMM level defined to evaluate the component. If necessary, the evaluation team must do some improvements in the scale proposed in order to become more accurate to the component in evaluation.Moreover, the evaluation team should retrieve the previous evaluation in the Component Evaluation’s Repository in order to analyze the punctuation level for the metrics defined in previously evaluations; Develop the Specification of the Evaluation Document: The last step of this activity is elaborating a document with all theinformation collected during the Specify the Evaluation activity and generate the Specification of the
148
Evaluation Document which is the main input of the next activity.
1.19.1.3 Design the Evaluation activity
This activity needs to consider access to component documentation, development tools and personnel, evaluation costs and expertise required, the evaluation schedule and costs, the description of the evaluation environment as detailed as possible, and reporting methods and evaluation tools.
The input of this activity is the Specification of the Evaluation Document. The GQM, defined in the last activity, acts as control in this activity. Based on these inputs, the evaluation team, using a set of tools as support, will develop the Evaluation Plan, which contains detailed and complete information about the evaluation process that should be followed, as show in Figure 4.5.4.
Figure 4.5.4. Evaluation activity design.
Figure 4.5.4 has shown the steps that should be followed to complete the execution of the Design the Evaluation activity and will be presented next.
149
Document the Evaluation Technique/Method: The objective of this step to document all those evaluation techniques or methods proposed to evaluate the component in the last activity (Define the Embedded software component Maturity Model (EMM) level step). The idea is to describe them so that all stakeholders involved in certification process understand the technique/method and can execute the tasks to evaluate the software component.All those techniques proposed on EMM contain its description stored in the Component Evaluation’s Repository and should be reused in order to increase the productivity in this step. However, if a new one technique was adopted in the last activity (Define EMM level step) it is necessary develop this documentation from scratch and, at least, should contain: the name of the technique/method, the reference in literature for this technique, the description and, if necessary, the use guide. The depth of this description will depend of the knowledge on that technique/method of the stakeholders involved in the evaluation team; Select (or Develop) Tools: Based on a set of tools available in literature, on the programming language and on the environment that the component should be deployed (or system that should be evaluated), the evaluation team needs to select the tools. Thus, the evaluation team will select the tools that support the execution of the techniques selected previously (Define the Embedded software component Maturity Model (EMM) level step). If necessary, it is possible develop a specific tool that evaluates a certain (or a set of) technique(s) in the case that the cost/benefit of developing one tool is justified. Further, the evaluation team defined whether any tool is necessary to collect the metrics define in later activity (Define GQM step). Table 4.5.4 shows an example of Table that
150
could be used to store the selected tools for each quality attribute.The description, at least, should consist of: the name of the tool, the version of the tool, the origin of the tool (i.e. its supplier) and the description of the technique (as detailed as the evaluation team consider necessary);
Table 4.5.4. Example of Tools Selection.Characteristic
sSub-
Characteristics
QualityAttributes
EMM Level /Evaluation Technique
Tool used
Functionality Accuracy Correctness II. Black-Box Testing
Junit13,FindBugs14
Functionality Security Data Encryption
III. Code Inspection
PMD15
… … … … …
Define the Environment: Now, it is time to describe the environment that the component will be evaluated in order to establish the context of the evaluation. The main input of this step is the Analyze the System and Environment step developed during the first activity. Thus, the evaluation team will specify the whole environment in as much detail as they can (i.e. software and tools necessary, the versions of the used software/tools, environment installed those software/tools, software/tools and environment constraints, etc.). Based on the environment defined the evaluation team will analyze the component quality. It is important to remember that the final report will contain in which environment(s) the component was certified.The same idea can be applied if the customer would like to certify the software component in a specific software system(s) (at least two);
13 http://www.junit.org/14 http://findbugs.sourceforge.net/15 http://pmd.sourceforge.net/
151
Develop the Evaluation Schedule: When all technological steps have been developed, it is time to analyze all available resources, such as tools, stakeholders, techniques, methods, etc. So, the evaluation team should develop the evaluation schedule with activities and tasks for each stakeholder and the time to execute each task. An agreement with all stakeholders involved in the evaluation process is desirable in order to minimize the risks involved during the component evaluation; Develop the Evaluation Cost: The establishment of the costs of a software project or a certification project has been a hard task. However, in order to define the evaluation costs a few and simple guidelines can be proposed, as following:
o Number of stakeholders involved in the evaluation;
o Skills and experience of those stakeholders;
o Definition of the tasks for each stakeholder;
o Time defined to each stakeholder execute his tasks;
Based on these guidelines, the evaluator responsible can determine the cost of each stakeholder involved in an evaluation process through the stakeholder skills X task to execute X time define to execute each task, and, thus, calculate the total costs of the evaluation process.The idea is to define the costs of the evaluation so that to the customer knows its investment in the component evaluation and approve it. This is not the best one approach to define the costs; however, this is the first step towards to the costs definition. Thus, the intention is to acquire expertise during a set of initial evaluations, store in the Component Evaluation’s Repository and, provide this data to the next component evaluation in order to give some data to be compared and refine the costs of the whole process; Develop the Evaluation Plan Document: The last step of this activity is elaborates a document with all information
152
collected during Design Evaluation activity. The output of this activity is the Evaluation Plan which is the main input of the next activity.
1.19.1.4 Execute the Evaluation activity
This activity includes execution of the evaluation techniques and analysis of the evaluation results. The conclusions should state whether the software component is appropriate for use in the intended environment (and maybe in system(s)) described in later activities.
The input of this activity is the Evaluation Plan Document. The GQM, defined in the second activity, acts as control in this activity. Based on these inputs, the evaluation team, using a set of tools to support them, will develop the Evaluation Report to deliver to the customer and to store in the Component Evaluation’s Repository too, as show in Figure 4.5.5.
Figure 4.5.5. Execute the Evaluation activity.
Figure 4.5.5 has shown the steps that should be followed to complete executing the Execute the Evaluation activity and will be presented next.
Configure the environment: Based on Define the Environment step, defined in last activity, the stakeholder(s)
153
responsible for this activity will configure the whole environment in order to start the evaluation of the component; Execute the Evaluation: Based on the Evaluation Plan, the stakeholder(s) responsible for this activity will execute the techniques/methods, using the defined tools and the metrics adopted to collect information about the component evaluation; Collect Data: Collecting data provides a basis for analysis. Good data collection is simple, repeatable, measures what is intended to measure, and captures information in a form suitable for analysis. Accurate data collection is one of the keys to successful software component evaluation, yet the act of collecting data is full of surprises – a few good ones and more than a few bad ones (Comella-Dorda et al., 2003).
It is important to remember that the quality of the evaluation is only as good as the data collected. Confidence in the final results can be improved by ensuring that the data collection is as accurate as possible.
Thus, the collection process should be carefully executed in order to provide good data for the next step. All data obtained from the tools used must be stored in a table like Table 4.5.5, based on the metrics and interpretation defined in GQM Define step for each quality attribute. So, the data analysis will be executed based on this table provided by this step;
Table 4.5.5. Example of Data Collection.Characteristi
csSub-
Characteristics
Quality Attribute
s
SCMM Level /
Evaluation Technique
Tool used
Results
Functionality Accuracy Correctness
II. Black-Box Testing
Junit, FindBugs
0.7
154
Functionality Security Data Encryption
III. Code Inspection
PMD 0.8
… … … … … …
Analyze the Results: After finishing the execution, the evaluation team will pick up all data stored in the previous table to analyze and develop a complete report about the evaluation. Those data should be carefully analyzed and correlated to each other in order to analyze the component quality in a complete view of its functionalities. It means that one data result may affect/interfere in other data result and vice versa, necessitating correlation of the results. If possible, it is desirable to compare this evaluation with other similar ones stored in Component Evaluation’s Repository in order to provide insights to the evaluation team during the data analysis. Subsequently, this report will be stored in a Component Evaluation’s Repository for sharing the knowledge acquired during this evaluation to the next component evaluations; Develop the Evaluation Report: The final result is an Evaluation Report that describes all quality attributes chosen, all techniques defined, the methods, process and tools used, the metrics defined for this evaluation, the data collected in the last step and the evaluation results, i.e. if the component was approved or not in the required evaluation level.
If possible, the evaluation team can describe a set of suggestions in order to improve the quality of the component. The goal of the recommendation is to provide some information to the customer improving the component quality. The evaluators learn many lessons about the use of the component during its evaluation (i.e. component architecture, deployment constraints, tailoring, wrapping, and testing and maintenance activities) and can contribute with important recommendations to the customer. This
155
information is important in the case that the component is approved and, much more interesting if the component is rejected.
1.19.1.5 Process Summary
Some individuals believe that following any documented process is a waste, particularly when the end goal is to save time and money. According to (Comella-Dorda et al., 2003), the highly informal COTS evaluation processes share the blame for the failure due to its lack of activities to follow. The process described here is a means of performing a component evaluation and not an end in itself. Expect to tailor this process for a specific purpose, and do not let it get in the way of getting good data and making an informed recommendation. At least, it is important to consider that the evaluation team is the main responsible for executing this process and should be carefully defined in order to assure that the certification will be efficiently developed.
156
5 Conclusion and
future worksAs like writed at begin, The demands that companies in these
electronic products must satisfy include low production costs, short time to market and high quality. The cost and time to market issue is addressed by means of the rapidly emerging Component-based Development (CBD) approach. In CBSE, the proper search, selection and evaluation process of components is considered the corner stone for the development of any effective component-based system. So, assessment and evaluation of software components has become a compulsory and crucial part of any CBSD lifecycle. Many organizations failure in their attempts to select an appropriate approach for use in Component-Based Software Development (CBSD), which is being used in a wide variety of application areas, mostly in embedded domain where the correct operations of the components are often critical for business success and, in some cases, human safety. Thus, the software components quality evaluation has become an essential activity in order to bring reliability in (re)using software components.
In this way, in order to properly enable the evaluation of embedded software components, supplying the real necessities of the embedded design of building system fast, cheap and high quality systems, an Embedded software Component Quality Verification Framework is mandatory. Thus, this thesis presented the whole framework and its related-modules: the Embedded software
157
component Quality Model (EQM), the Maturity Level Evaluation Techniques, the Metrics Approach and the Component Certification Process.
1.20 Contributions
The main goal of this research is to demonstrate that component certification is not only possible and practically viable, but also directly applicable in the embedded system industry. In this way, some evaluations have been envisioned in conjunction with the industry for acquiring trust and maturation to the proposed embedded software component quality verification framework.
This work could be split on four main aspects:
the implementation of research related to requirements and needs of embedded systems design;
to show the challenge of the use component-based approach for embedded systems and it specific requirements in different domain;
the realization of a survey related to the state-of-the-art in embedded software component quality and certification research; and
the proposition of an Embedded software Component Quality Verification Framework to evaluate the embedded software component quality.
The Chapters 1 contributes with a research about the main problems, needs and challenges of embedded systems design. After that, a proposed solution was present for these explicit problems.
The Chapter 2 details the Embedded System Design, and has showed the component-based approach for small and large systems. The second part, documents the survey of specifics requirements for
158
embedded systems in different domains. Finally, the needs and priorities in research are discussed.
A Survey on Embedded software Component Quality and Certification was showed in Chapter 3. The main research contributions found in the literature, from the 90’s until today, were analyzed in order to understand how the software component certification area has evolved during this timeline. Through this study, it became possible to elaborate a well-defined embedded software component quality verification framework, consisting of four modules;
In Chapter 4 The Embedded software Component Quality Verification Framework was proposed. The survey showed that embedded software component quality is important to the embedded systems design. In order to supply this necessity, an Embedded software Component Quality Verification Framework was defined, which contains a set of modules that complement each other trying to supply all information required to evaluate the quality of an embedded software component. Each module was defined and discussed with some quality experts around the world in industries and the academic environment, in order to improve them as much as possible. Moreover, the modules are still undergoing evolution through its usage in the industry (i.e. RiSE projects).
Other works found in the literature were not evaluated in either academic or industrial scenarios, making the real efficiency to evaluate software components unknown (Meyer, 2003), (Bertoa et al., 2002), (Goulão et al., 2002b), (Simão et al., 2003), (Beus-Dukic et al., 2003), (Comella-Dorda et al., 2002) and (Karlsson, 2006). Still on, the works considering only specific aspects of software component quality (i.e. some researchers works with component quality model, other works with specific kind of software component metrics, other works with component evaluation process, and so on) and do not
159
provide detailed steps that should be carefully followed to accomplish the component evaluation. The works presented a high-level proposal and, in this way, it is very difficult to apply some of them because they do not provide detailed description about that in order to facilitate its applicability.
Compared to the works above, the embedded software component quality verification framework proposed in this thesis will be applied, evaluated and tested in a university laboratory and, if it is possible, applied in a Brazilian Software Company, called C.E.S.A.R. Thus, the framework can become more efficient to solve the necessities of the embedded design and component market (Heineman et al., 2001). Moreover, the framework is compound of four modules that complement each other in an effort to evaluate the component quality level. The steps that should be followed are carefully described in order to facilitate the execution of the framework by the evaluation team.
1.21 Future Work
During the research it was perceived that there are needs for a number of improvements. Some of them are the following:
Component Certification Center. The long term plan could be to achieve a degree of maturity that could be used as an embedded component certification standard for embedded industry, making it possible to create a Component Certification Center. Through the Brazilian projects that the RiSE group is involved in, this “dream” will become reality through the maturation of the process and the reliability of the embedded software component reuse on that process;
Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc. The analysis of many global properties from
160
component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.
Tool Support. In any engineering discipline a tool support is necessary in order to aid the usage of the proposed processes, methods, techniques, etc. In that way, it is vitally important to develop a tool that supports the whole embedded software component quality verification framework activities since there is a lot of information produced during the evaluation process that could be missed without a tool support; and
Risk and Cost/Benefit Management Model. An interesting aspect to the customer is the cost/benefit and risk management that the component quality assurance could bring to its business in order to analyze if the costs relative to the component quality assurance are acceptable or not (Keil & Tiwana, 2005). In this way, a Risk and Cost/Benefit Management Model is very interesting to complement the embedded software component quality verification framework and should be carefully designed to provide the real risks, costs and possible benefits to the component customer.
1.22 Chronogram
The research work to perform up until the defense of the thesis can be divided into five distinct activity:
Table 5.3. Chronogram.Activity Deadline
Deliverable
Suggest and review metrics for remaining quality attributes Jan
16 - Jan
Define a document/form model for register embedded component evaluation
30 - Jan
Finalization of case of study for proposed framework evaluation
13 - Feb
Completion of doctoral thesis 1 - MarDoctoral thesis defense 27 - Mar
161
162
6 Bibliographic
References
(Åkerholm et al.,2004) M. Åkerholm; J. Fredriksson; K. Sandström; I. Crnkovic. Quality Attribute Support in a Component Technology for Vehicular Software. Fourth Conference on Software Engineering Research and Practice in Sweden, Linköping, Sweden, 2004.
(Almeida et al., 2004) Almeida, E. S.; Alvaro, A.; Lucrédio, D.; Garcia, V. C.; Meira, S. R. L. RiSE Project: Towards a Robust Framework for Software Reuse, In: The IEEE International Conference on Information Reuse and Integration (IRI), Las Vegas, USA, pp. 48-53, 2004.
(Alvaro et al., 2005) Alvaro, A.; Almeida, E. S.; Meira, S. R. L. A Software Component Certification: A Survey, In: The 31st IEEE EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), Component-Based Software Engineering (CBSE) Track, Porto, Portugal, 2005.
(Alvaro et al., 2006) Alvaro, A.; Almeida, E.S.; Meira, S. R. L. A Software Component Quality Model: A Preliminary Evaluation, In: The 32st IEEE EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), Component-Based Software Engineering (CBSE) Track, Cavtat/Dubrovnik, Croatia, 2006.
163
(Alvaro et al., 2007a) Alvaro, A.; Almeida, E.S.; Meira, S. L. A Software Component Maturity Model (SCMM), In: The 33st IEEE EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), Component-Based Software Engineering (CBSE) Track, Lübeck, Germany, 2007.
(Alvaro et al., 2007b) Alvaro, A.; Almeida, E. S.; Meira, S.R.L. Component Quality Assurance: Towards a Software Component Certification Process. In: The IEEE International Conference on Information Reuse and Integration (IRI), Las Vegas, USA. IEEE Press. 2007.
(Atkinson et al.,2005) C. Atkinson1, C. Bunse, C. Peper, H. Gross. Component-based software development for embedded systems: An overview of current research trends. State-of-the-Art Survey. Berlin: Springer, 2005. (Lecture Notes in Computer Science 3778), pp. 1-7
(Avižienis et al. , 2001) Avižienis A., Laprie J-C., Randell B., Fundamental Concepts of Computer System Dependability, IARP/IEEE-RAS Workshop on Robot Dependability: Technological Challenge of Dependable, Robots in Human Environments, 2001
(Barbacci et al, 1995) Barbacci M., Klein M., Longstaff T., and Weinstock C. B., Quality Attributes, report CMU/SEI-95-TR-021, CMU/SEI, 1995.
(Basili & Rombach, 1988) Basili, V.R.; Rombach, H.D. The TAME Project: Towards Improvement-Oriented Software Environments, In: IEEE Transactions on Software Engineering, Vol. 14, No.6, pp.758-773, 1988.
(Basili & Selby, 1984) Basili, V.R.; Selby, R.W. Data Collection and Analysis in Software Research and Management, In: Proceedings of the American Statistical Association and Biomeasure Society, Joint Statistical Meetings, Philadelphia, 1984.
164
(Basili & Weiss, 1984) Basili, V.R.; Weiss, D.M. A Methodology for Collecting Valid Software Engineering Data, In: IEEE Transactions on Software Engineering, Vol. 10, No. 06, pp. 728-738, 1984.
(Basili et al., 1994) Basili, V.R.; Caldiera, G.; Rombach, H.D. The Goal Question Metric Approach, In: Encyclopedia of Software Engineering, Vol. II, September, pp. 528-532, 1994.
(Basili, 1992) Basili, V.R. Software Modeling and Measurement: The Goal Question Metric Paradigm, In: Computer Science Technical Report Series, CS-TR-2956 (UMIACS-TR-92-96), University of Maryland, College Park, MD, September 1992.
(Bass et al., 2000) Bass, L.; Buhman, C.; Dorda, S.; Long, F.; Robert, J.; Seacord, R.; Wallnau, K. C. Market Assessment of Component-Based Software Engineering, In: Software Engineering Institute (SEI), Technical Report, Vol. I, May, 2000.
(Bertoa et al., 2002) Bertoa, M.; Vallecillo, A. Quality Attributes for COTS Components, In: The 6th IEEE International ECOOP Workshop on Quantitative Approaches in Object-Oriented Software Engineering (QAOOSE), Spain, Vol. 01, No. 02, pp. 128-144, 2002.
(Bertoa et al., 2006) Bertoa, M.F.; Troya, J.M.; Vallecillo, A. Measuring the Usability of Software Components. In: Journal of Systems and Software, Vol. 79, No. 03, pp. 427-439, 2006.
(Bertolino & Mirandola, 2003) Bertolino, A.; Mirandola, R. Towards Component-Based Software Performance Engineering, In: Proceedings of 6th ICSE Workshop on Component-Based Software Engineering, USA, 2003.
(Beugnard et al., 1999) Beugnard, A.; Jezequel, J.; Plouzeau, N.; Watkins, D. Making component contract aware, In: IEEE Computer, Vol. 32, No. 07, pp. 38-45, 1999.
(Beus-Dukic et al., 2003) Beus-Dukic, L.; Boegh, J. COTS Software Quality Evaluation, In: The 2nd International Conference on
165
COTS-Based Software System (ICCBSS), Lecture Notes in Computer Science (LNCS), Springer-Verlag, Canada, 2003.
(Beydeda & Gruhn, 2003) Beydeda, S.; Gruhn, V. State of the art in testing components, In: The 3th IEEE International Conference on Quality Software (ICQS), USA, 2003.
(Boegh et al., 1993) Boegh, J.; Hausen, H-L.; Welzel, D. A Practioners Guide to Evaluation of Software, In: The IEEE Software Engineering Standards Symposium, pp. 282-288, 1993.
(Boehm et al., 1976) Boehm, W.; Brown, J.R.; Lipow, M. Quantitative Evaluation of Software Quality, In: The Proceedings of the Second International Conference on Software Engineering, pp.592-605, 1976.
(Boehm et al., 1978) Boehm, B.; Brown, J.R.; Lipow, H.; MacLeod, G. J.; Merrit, M. J. Characteristics of Software Quality, Elsevier North Holland, 1978.
(Brinksma et al., 2001) E. Brinksma et al., ROADMAP - Componentbased Design and Integration Platforms, W1.A2.N1.Y1, Project IST-2001-34820, ARTIST - Advanced Real-Time Systems
(Brown, 2000) Brown A. W., Large-Scale Component-Based Development, Prentice Hall, 2000.
(Brownsword et al., 2000) Brownsword, L.; Oberndorf, T.; Sledge, C. A.: Developing New Processes for COTS-Based Systems. In: IEEE Software, July/August, pp. 48-55, 2000.
(Caldiera & Basili, 1991) Caldiera, G.; Basili, V. Identifying and Qualifying Reusable Software Components, In: IEEE Computer, Vol. 24, No. 02, pp. 61–71, 1991.
(Camposano & Wilberg, 1996)R. Camposano, J. Wilberg, “Embedded System Design”, in Design Automation for Embedded Systems, vol. 1, pp. 5-50, Jan 1996.
(Carvalho & Junior et al., 2004a) Carvalho, Fernando F.; Júnior, M.N. O.; Maciel, P. R. M.; Barreto, R. S. Towards a Software Power Cost Analysis Framework Using Colored Petri Net. In: 14th
166
International Workshop PATMOS 2004, Volume 3254, Pages: 362-371, Springer-Verlag, August, 2004.
(Carvalho & Junior et al., 2004b) Carvalho, Fernando F.; Júnior, M.N. O.; Maciel, P. R. M.; Barreto, R. S. A Software Power Cost Analysis based on Colored Petri Net. In: 25th International Conference on Applications and Theory of Petri Nets (ICATPN'04) in conjunction Workshop on Token Based Computing (ToBaCo), July, 2004.
(Chen et al., 2005) Chen, S.; Liu, Y.; Gorton, I.; Liu, A. Performance Predication of Component-based Applications. In: Journal of Systems and Software, Vol. 01, No. 07, pp. 35-46, 2005.
(Cho et al., 2001) Cho, E. S.; Kim, M. S.; Kim, S. D. Component Metrics to Measure Component Quality, In: The 8th IEEE Asia-Pacific Software Engineering Conference (APSEC), pp. 419-426, 2001.
(CMMI, 2000) CMMI Product Development Team. CMMI for Systems Engineering/Software Engineering/Integrated Product and Process Development/Supplier Sourcing, Version 1.1 Continuous Representation, In: CMU/SEI-2002-TR-011, ESC-TR- 2002-011. Carnegie Mellon University/Software Engineering Institute (CMU/SEI), November, 2000.
(Comella-Dorda et al., 2002) Comella-Dorda, S.; Dean, J.; Morris, E.; Oberndorf, P. A Process for COTS Software Product Evaluation, In: The 1st International Conference on COTS-Based Software System (ICCBSS), Lecture Notes in Computer Science (LNCS), Springer-Verlag, USA, 2002.
(Comella-Dorda et al., 2003) Comella-Dorda, S.; Dean, J.; Lewis, G.; Morris, E.; Oberndorf, P.; Harper, E. A Process for COTS Software Product Evaluation, In: Technical Report, CMU/SEI-2003-TR-017, 2003.
(Councill, 1999) Councill, W. T. Third-Party Testing and the Quality of Software Components, In: IEEE Computer, Vol. 16, No. 04, pp. 55-57, 1999.
167
(Councill, 2001) Councill, B. Third-Party Certification and Its Required Elements, In: The 4th Workshop on Component-Based Software Engineering (CBSE), Lecture Notes in Computer Science (LNCS), Springer-Verlag, Canada, May, 2001.
(Crnkovic & Larsson, 2002) I. Crnkovic and M. Larsson. Building Reliable Component-Based Software Systems. ArtechHouse, 2002.
(Crnkovic, 2005) Crnkovic, I. Component-based software engineering for embedded systems I - Proceedings of the 27th international conference on Software Engineering ICSE’05, May , Missouri, USA,2005
(Drouin, 1995) Drouin, J-N. The SPICE Project: An Overview. In: The Software Process Newsletter, IEEE TCSE, No. 02, pp. 08-09, 1995.
(Druckner, 2003) L. Druckner, “SystemC Verification Library speeds transaction-based verification”, in EETimes Online, http://www.eetimes.com/, 24 Feb. 2003
(Fagan, 1976) Fagan, M. Design and Code Inspections to Reduce Errors in Program Development. In: IBM Systems Journal, Vol. 15, No. 03, pp. 182-211, 1976.
(Frakes & Fox, 1995) Frakes, W. B.; Fox, S. Sixteen Questions about Software Reuse, In: Communications of the ACM, Vol. 38, No. 06, pp. 75-87, 1995.
(Frakes & Terry, 1996) Frakes, W.; Terry, C. Software Reuse: Metrics and Models, In: ACM Computing Survey, Vol. 28, No. 02, pp. 415-435, 1996.
(Freedman, 1991) Freedman, R.S. Testability of Software Components, In: IEEE Transactions on Software Engineering, Vol. 17, No. 06, June 1991.
(Gao et al., 2003) Gao, J.Z.; Jacob, H.S.J.; Wu, Y. Testing and Quality Assurance for Component Based Software, Artech House, 2003.
168
(Georgiadou, 2003) Georgiadou, E. GEQUAMO-A Generic, Multilayered, Customisable, Software Quality Model. In: Software Quality Journal, Vol. 11, No. 04, pp. 313-323, 2003.
(Goulao et al., 2002a) Goulao, M.; Brito e Abreu, F. The Quest for Software Components Quality, In: The 26th IEEE Annual International Computer Software and Applications Conference (COMPSAC), England, pp. 313-318, 2002.
(Goulão et al., 2002b) Goulão, M.; Abreu, F. B. Towards a Component Quality Model, In: The 28th IEEE EUROMICRO Conference, Work in Progress Session, Dortmund, Germany, 2002.
(Gui & Scott, 2007) Gui, G.; Scott, P.D. Ranking reusability of software components using coupling metrics. In: Journal of Systems and Software, Vol. 80, No. 09, pp. 1450-1459, 2007.
(Hall, 1990) Hall, A., Seven Myths of Formal Methods, In: IEEE Software, pp. 11-20, 1990.
(Hamlet et al., 2001) Hamlet, D.; Mason, D.; Woit. D. Theory of Software Component Reliability, In: 23rd International Conference on Software Engineering (ICSE), 2001.
(Hissam et al., 2003) Hissam, S. A.; Moreno, G. A.; Stafford, J.; Wallnau, K. C. Enabling Predictable Assembly, In: Journal of Systems and Software, Vol. 65, No. 03, pp. 185-198, 2003.
(Hyatt et al., 1996) Hyatt, L.; Rosenberg, L.; A Software Quality Model and Metrics for Risk Assessment, In: NASA Software Technology Assurance Center (SATC), 1996.
(ISO/IEC 1131-3,1995) IEC, Application and Implementation of IEC 1131-3, IEC Geneva, 1995.
(ISO/IEC 12119, 1994) ISO 12119, Software Packages – Quality Requirements and Testing, International Standard ISO/IEC 12119, International Standard Organization (ISO), 1998.
(ISO/IEC 14598, 1998) ISO 14598, Information Technology – Software product evaluation -- Part 1: General Guide, International Standard ISO/IEC 14598, International Standard Organization (ISO), 1998.
169
(ISO/IEC 15504-2, 2003) ISO/IEC 15504-2, Information technology. Software process assessment. Part 2 : a reference model for processes and process capability, International Standard ISO/IEC 15504-2, International Standard Organization (ISO), 2003.
(ISO/IEC 25000, 2005) ISO/IEC 25000, Software product quality requirements and evaluation (SQuaRE), Guide to SQuaRE, International Standard Organization, July, 2005.
(ISO/IEC 61131-3,1995) IEC. Application and implementation of IEC 61131-3. Technical report, IEC, Geneva, 1995.
(ISO/IEC 9126, 2001) ISO 9126, Information Technology – Product Quality – Part1: Quality Model, International Standard ISO/IEC 9126, International Standard Organization (ISO), 2001.
(Jahnke, Niere and Wadsack, 2000)(Jezequel et al., 1997) Jezequel, J. M.; Meyer, B. Design by Contract:
The Lessons of Ariane, In: IEEE Computer, Vol. 30, No. 02, pp. 129-130, 1997.
(Jezequel et al., 1997) Jezequel, J. M.; Meyer, B. Design by Contract: The Lessons of Ariane, In: IEEE Computer, Vol. 30, No. 02, pp. 129-130, 1997.
(Kalaoja et al., 1997)J. Kalaoja, E. Niemelä, H. Perunka, “Feature Modelling of Component-Based Embedded Software," step,pp.444, 8th International Workshop on Software Technology and Engineering Practice (STEP '97) (including CASE '97), 1997.
(Karlson, 2006) D. Karlson, “Verification of Component-based Embedded Systems Designs, Dissertation, Linkoping University, 2006.
(Kocher et al., 2004) Kocher, P.; Ruby Lee; McGraw, G.; Raghunathan, A.; Ravi, S. Security as a new dimension in embedded system design, DAC 2004, June 7–11, 2004, San Diego, California, USA.
170
(Kogure & Akao, 1983) Kogure, M.; Akao, Y. Quality Function Deployment and CWQC in Japan, In: Quality Progress, pp.25-29, 1983.
(Kopetz & Suri et al., 2003) H. Kopetz and N. Suri. Compositional design of RT systems: A conceptual basis for specification of linking interfaces. In Proc. 6th IEEE International Symposium on Object-oriented Real-Time Distributed Computing (ISORC), Hokkaido, Japan, May 2003.
(Kotula, 1998) Kotula, J. Using Patterns To Create Component Documentation, In: IEEE Software, Vol. 15, No. 02, March/April, pp. 84-92, 1998.
(Krueger, 1992) Krueger, C. W. Software Reuse, In: ACM Computing Surveys, Vol. 24, No. 02, pp. 131-183, 1992.
(Larsson, 2000) Larsson M., Applying Configuration Management Techniques to Component-Based Systems, Licentiate Thesis, Dissertation 2000-007, Department of Information Technology Uppsala University., 2000.
(Larsson, 2004) M. Larsson, Predicting Quality Attributes in Component-based Software Systems, Phd Thesis, Mälardalen University Press, March 2004.
(Lethbridge et al., 2003) Lethbridge, T.; Singer, J.; Forward, A. How Software Engineers use Documentation: The State of the Practice, In: IEEE Software, Vol. 20, No. 06, pp. 35-39, 2003.
(Lucrédio et al., 2007) Lucrédio, D.; Brito, K.S.; Alvaro, A.; Garcia, V.C.; Almeida, E.S.; Fortes, R.P.M.; Meira, S.R.L. Software Reuse:Brazilian Industry Scenario, In: Journal of Systems and Software, Elsevier, 2007.
(Lüders et al., 2002) F. Lüders, I. Crnkovic, and A. Sjögren. Case study: Componentization of an industrial control system. In Proc. 26th Annual International Computer Software and Applications Conference - COMPSAC 2002, Oxford, UK, Aug. 2002. IEEE Computer Society Press.
171
(McCall et al., 1977) McCall, J.A; Richards, P.K.; Walters, G.F. Factors in Software Quality, Griffiths Air Base, Nova York, Rome Air Development Center (RADC) System Command, TR-77-369, Vol. I, II and III, 1977.
(McGregor et al., 2003) McGregor, J. D.; Stafford, J. A.; Cho, I. H. Measuring Component Reliability, In: The 6th Workshop on Component- Based Software Engineering (CBSE), Lecture Notes in Computer Science (LNCS) Springer-Verlag, USA, pp. 13-24, 2003.
(Merrit, 1994) Merrit, S. Reuse Library, In: Encyclopedia of Software Engineering, J.J. Marciniak (editor), John Willey & Sons, pp. 1069-1071, 1994.
(Meyer, 2003) Meyer, B. The Grand Challenge of Trusted Components, In: The 25th IEEE International Conference on Software Engineering (ICSE), USA, pp. 660–667, 2003.
(Möller et al., 2004)(Morris et al., 2001) Morris, J.; Lee, G.; Parker, K.; Bundell, G. A.;
Lam, C. P.; Software Component Certification. In: IEEE Computer, Vol. 34, No. 09, pp. 30-36, 2001.
(Müller et al.,2002)P. O. Müller, C.M. Stich, and C. Zeidler. Component-Based Embedded Systems
(Ommering et al.,2000) R. van Ommering, F. van der Linden, and J. Kramer. The Koala component model for consumer electronics software. IEEE Computer, 33(3):78–85, March 2000.
(Ommering, 2002) 14. R. van Ommering. Building product populations with software components. In Proceedings of the 24th international conference on Software engineering,. ACM Press, 2002.
(Parnas & Lawford, 2003) Parnas, D.; Lawford, M. The Role of Inspection in Software Quality Assurance. In: IEEE Transactions on Software Engineering, Vol. 29, No. 08, pp. 674-676, 2003.
172
(Paulk et al., 1993) Paulk, M.; Curtis, B.; Chrissis, M.; Weber, C. Capability Maturity Model for Software, Version 1.1, In: Software Engineering Institute, Carnegie Mellon University, CMU/SEI-93-TR-24, DTIC Number ADA263403, February, 1993.
(Poore et al., 1993) Poore, J.; Mills, H.; Mutchler, D. Planning and Certifying Software System Reliability, In: IEEE Computer, Vol. 10, No. 01, pp. 88-99, 1993.
(Preiss et al.,2001) Otto Preiss, Alain Wegmann, and Jason Wong. On quality attribute based software engineering. In Proc. of the 27th Euromicro Conference, Warsaw, Poland, September 2001. IEEE CS Press.
(Pressman, 2005) Pressman, R. Software Engineering: A Practitioner’s Approach. McGraw-Hill. 6th Edition. 2005.
(Reussner, 2003) Reussner, R. H. Contracts and quality attributes of software components, In: The 8th International Workshop on Component-Oriented Programming (WCOP) in conjunction with the 17th ACM European Conference on Object Oriented Programming (ECCOP), 2003.
(Rohde et al., 1996) Rohde, S. L.; Dyson, K. A.; Geriner, P. T.; Cerino, D. A. Certification of Reusable Software Components: Summary of Work in Progress, In: The 2nd IEEE International Conference on Engineering of Complex Computer Systems (ICECCS), Canada, pp. 120-123, 1996.
(Ross, 1997) Ross, D.T. Structured Analysis (SA): A Language for Communicating Ideas, In: IEEE Transaction on Software Engineering, Vol. 03, No. 01, pp. 16-34, 1997.
(Sametinger, 1997) Sametinger, J. Software Engineering with Reusable Components. Springer Verlag, USA, 1997.
(Savage et al.,2000)W. Savage, J. Chilton and R. Camposano, “IP Reuse in the System on a Chip Era”, in Proc. ISSS, pp. 2-7, 2000
173
(Simão et al., 2003) Simão, R. P. S.; Belchior, A. Quality Characteristics for Software Components: Hierarchy and Quality Guides, Component-Based Software Quality: Methods and Techniques, In: Lecture Notes in Computer Science (LNCS) Springer-Verlag, Vol. 2693, pp. 188-211, 2003.
(Softex, 2007) Softex, Perspectivas de desenvolvimento e uso de componentes na indústria brasileira de software e serviços, (in portuguese) Campinas: SOFTEX, 2007. Available at: http://golden.softex.br/portal/softexweb/uploadDocuments/Componentes(2).pdf
(Solingen, 2000) Solingen, R.v. Product focused software process improvement: SPI in the embedded software domain, PhD Thesis, Technische Universiteit Eindhoven, 2000.
(Stafford et al., 2001) Stafford, J.; Wallnau, K. C. Is Third Party Certification Necessary?, In: The 4th Workshop on Component-Based Software Engineering (CBSE), Lecture Notes in Computer Science (LNCS) Springer-Verlag, Canada, 2001.
(Szyperski et al. 1998) C. Szyperski. Component Software: Beyond Object-Oriented Programming. ACM, Press and Addison-Wesley, New York, N.Y., 1998.
(Taulavuori et al., 2004) Taulavuori, A.; Niemela, E.; Kallio, P. Component documentation — a key issue in software product lines, In: Journal Information and Software Technology, Vol. 46, No. 08, June, pp. 535–546, 2004.
(Tian, 2004) Tian, J. Quality-Evaluation Models and Measurements, In: IEEE Software, Vol. 21, No.03, pp.84-91, May/June, 2004.
(Urting et al., 2001) Urting, D.; Van Baelen, S; Holvoet, T.; Berbers, Y. Embedded software development: components and contracts. Proceedings of the IASTED International Conference on Parallel and Distributed Computing and Systems, ACTA Press, 2001, pp. 685-690.
174
(Van Ommering, 2002) Van Ommering R., "The Koala Component Model", in Crnkovic I. and Larsson M. (editors): Building Reliable Component-Based Software Systems, ISBN 1-58053-327-2, Artech House, 2002.
(Voas, 2000) Jeffrey Voas. Usage-based software certification process, IEEE Computer Society Press, Volume 33 , Pages: 32 – 37, 2000.
(Voas et al., 2000) Voas, J. M.; Payne, J. Dependability Certification of Software Components, In: Journal of Systems and Software, Vol.52, No. 2-3 , pp. 165-172, 2000.
(Voas, 1998) Voas, J. M. Certifying Off-the-Shelf Software Components, In: IEEE Computer, Vol. 31, No. 06, pp. 53-59, 1998.
(Wallin, 2002) Wallin, C. Verification and Validation of Software Components and Component Based Software Systems, In: Building Reliable Component-Based Systems, I. Crnkovic, M. Larsson (editors), Artech House Publishers, July, pp. 29-37, 2002.
(Wallnau, 2003) Wallnau, K. C. Volume III: A Technology for Predictable Assembly from Certifiable Components. In: Software Engineering Institute (SEI), Technical Report, Vol. III, April, 2003.
(Wallnau, 2004) Wallnau, K., Software Component Certification: 10 Useful Distinctions, In: Technical Note CMU/SEI-2004-TN-031, 2004. Available at: http://www.sei.cmu.edu/publications/documents/04.reports/04tn031.html.
(Wijnstra, 2001) Wijnstra, Jan Gerben. Quality Attributes and Aspects of a Medical Product Family. Proceedings of the 34th Hawaii International Conference on System Sciences - 2001
(Wohlin & Regnell, 1998) Wohlin, C.; Regnell, B. Reliability Certification of Software Components, In: The 5th IEEE
175
International Conference on Software Reuse (ICSR), Canada, pp. 56-65, 1998.
(Wohlin et al., 1994) Wohlin, C.; Runeson, P. Certification of Software Components, In: IEEE Transactions on Software Engineering, Vol. 20, No. 06, pp 494-499, 1994.
176
Appendix A. Metrics Example
Chapter 4 presented some brief examples of metrics definition using the GQM approach. In order to help the evaluation team during the metrics definition, now some other related metrics that could be considered in a component evaluation will be described. However, the more complex the sub-characteristics and its related quality attribute are, the more difficult it is to provide metrics examples without a well-defined context.
Example of Metrics to track Functionality Characteristics
For Functionality characteristics there are five sub-characteristics that could be evaluated: Accuracy, Security, Suitability, Interoperability, Compliance and Self-Contained. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows.
For the Accuracy Sub-Characteristic the following metrics could be applied:
FunctionalitySub-Characteristic AccuracyQuality Attribute CorrectnessGoal Evaluates the percentage of the
results that were obtained with precision
Question Based on the amount of tests executed, how much test results return with precision?
Metric Precision on results / Amount of tests
Interpretation 0 <= x <= 1; closer to 1 being better
177
For the Security Sub-Characteristic, the following metrics could be applied: Functionality
Sub-Characteristic SecurityQuality Attribute Data EncryptionGoal the encryption of the input and
output data of the component.Question How complete is the data
encryption implementation?Metric Number of services that must
have data encryption / Number of services that have encryption
Interpretation 0 <= x <= 1; closer to 1 being better
FunctionalitySub-Characteristic SecurityQuality Attribute ControllabilityGoal if the component provide any
control mechanism.Question How controllable is the
component access?Metric Number of provided interfaces
that control the access / Number of provided interfaces
Interpretation 0 <= x <= 1; closer to 1 being better
FunctionalitySub-Characteristic SecurityQuality Attribute AuditabilityGoal Evaluate if the component
provide any audit mechanism.
Question How controllable is the component audit mechanism?
Metric Number of provided interfaces that log-in the access (or any kind of data) / Number of provided interfaces
Interpretation 0 <= x <= 1; closer to 1 being better
• Example of Metrics to track Reliability Characteristics
178
For Reliability characteristic there are three sub-characteristics that could be evaluated: Recoverability, Fault Tolerance and Safety. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows. Besides those that are presented here, some reliability metrics for software components found on literature can be considered (McGregor et al., 2003), (Shukla et al., 2004).
For the Recoverability Sub-Characteristic the following metrics could be
applied:
ReliabilitySub-Characteristic RecoverabilityQuality Attribute Error HandlingGoal Evaluates the ability of the
component to avoid error situations.
Question How many functions provide any mechanism that can avoid error situations?
Metric Number of error handlings provided / Number of functions.
Interpretation 0 <= x <= 1; closer to 1 being better
For the Fault Tolerance Sub-Characteristic, the following metrics could be applied;Reliability
Sub-Characteristic Fault ToleranceQuality Attribute Mechanism availableGoal Evaluates the functions that
contain fault tolerance mechanism
Question How many functions provide the fault tolerance mechanism?
Metric Number of functions that contain any kind of fault tolerance mechanism / Number of functions
Interpretation 0 <= x <= 1; closer to 1 being better
Reliability
179
Sub-Characteristic Fault ToleranceQuality Attribute Mechanism EfficiencyGoal Evaluates the efficiency of the
fault tolerance mechanismQuestion - How is the efficiency of the
functions that provide any kind of fault tolerance mechanism?- What it is the range of the data lost?
Metric - Number of functions that contain any kind of fault tolerance mechanism / Number of mechanisms that are considered efficient - Total number of interfaces that exchanged data to outside / Number of interface that lost data
Interpretation 0 <= x <= 1; closer to 1 being better
ReliabilitySub-Characteristic SafetyQuality Attribute Environment analyzeGoalQuestionMetricInterpretation
ReliabilitySub-Characteristic IntegrityQuality Attribute Environment analyzeGoalQuestionMetricInterpretation
• Example of Metrics to track Usability Characteristics
For the Usability characteristic, there are one sub-characteristic that could be evaluated: Configurability. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows. Besides the ones presented here, some usability metrics for software components found on
180
literature can be considered (Bertoa & Vallecillo, 2004), (Bertoa et al., 2006).
For the Configurability Sub-Characteristic the following metrics could be
applied:
UsabilitySub-Characteristic ConfigurabilityQuality Attribute Effort to configureGoal Evaluates the time necessary to
configure the component.Question How much time is needed to
configure the component in order to work correctly in a system?
Metric Time spent to configure correctly.
Interpretation The faster it is to configure the component the better, but it depends of the component and environment complexity.
UsabilitySub-Characteristic ConfigurabilityQuality Attribute Understandability Goal Analyses the documentation
availability. Question How many documents are
available to understand the component functions?
Metric Number of documents.Interpretation The higher number of documents
available is better, but it depends of the component complexity, domain, etc.
• Example of Metrics to track Efficiency Characteristics
For the Efficiency characteristic there are four sub-characteristics that could be evaluated: Resource Behavior, Energy Consumption, Data Memory Utilization and Program Memory Utilization. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows.
181
For the Resource Behavior Sub-Characteristic the following metrics could be applied:
EfficiencySub-Characteristic Resource BehaviorQuality Attribute Peripheral utilizationGoal Analyzes the amount of
Peripheral required to its correct work.
Question How much Peripheral is enough for the component to work correctly?
Metric Amount of Peripheral necessary for the component to work correctly/amount of Peripheral available on the execution environment.
Interpretation 0 <= x <= 1; which closer to 0 being better
EfficiencySub-Characteristic Energy ConsumptionQuality Attribute Mechanism availableGoal Evaluates the functions that
contain energy consumption optimization mechanism
Question How many functions provide the energy consumption optimization mechanism?
Metric Number of functions that contain any kind of energy consumption optimization mechanism / Number of functions
Interpretation 0 <= x <= 1; closer to 1 being better
EfficiencySub-Characteristic Data Memory UtilizationQuality Attribute Mechanism availableGoal Evaluates the functions that
contain Data Memory optimization mechanism
Question How many functions provide the Data Memory optimization mechanism?
Metric Number of functions that contain
182
any kind of Data Memory optimization mechanism / Number of functions
Interpretation 0 <= x <= 1; closer to 1 being better
EfficiencySub-Characteristic Program Memory UtilizationQuality Attribute Mechanism availableGoal Evaluates the functions that
contain Program Memory optimization mechanism
Question How many functions provide the Program Memory optimization mechanism?
Metric Number of functions that contain any kind of Program Memory optimization mechanism / Number of functions
Interpretation 0 <= x <= 1; closer to 1 being better
• Example of Metrics to track Maintainability Characteristics
For the Maintainability characteristic there are three sub-characteristics that could be evaluated: Stability, Changeability and Testability. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows.
For the Stability Sub-Characteristic the following metrics could be applied:
MaintainabilitySub-Characteristic StabilityQuality Attribute ModifiabilityGoal Evaluates the flexibility to
change the component source code in order to improve its functions.
Question How modifiable is the component?
Metric Execute a set of modifications and analyze the component behavior.
Interpretation Analyze the amount of
183
modifications done and the amount of modifications that works well
MaintainabilitySub-Characteristic ChangeabilityQuality Attribute ExtensibilityGoal Evaluates the flexibility to extend
the component functions
Question How extensible is the component?
Metric Execute a set of extensions and analyze the new component behavior.
Interpretation Analyze the amount of extensions done and the amount of extensions that work well
MaintainabilitySub-Characteristic ChangeabilityQuality Attribute CustomizabilityGoal Analyzes the customizable
parameters that the component offers
Question How much parameters are provided to customize each function of the component?
Metric Number of provided interfaces / Number of parameters to configure the provided interface
Interpretation 0 <= x <= 1; which closer to 1 is better
MaintainabilitySub-Characteristic ChangeabilityQuality Attribute ModularityGoal Analyzes the internal
organization of the component.Question How logically separated are the
component concerns?Metric Packaging analysisInterpretation If the component contains some
packages that isolate each logical concern it probably has good modularity. On the other hand, if the component doesn’t contain a well defined internal
184
structure the modularity level is slower.
MaintainabilitySub-Characteristic TestabilityQuality Attribute Test suite providedGoal Analyzes the ability of the
component to provide some test suite for checking its functions.
Question - Is there any test suite?- How is the coverage of this test suite based on the whole component functions?
Metric - Analysis of the test suites provided - Number of test suites provided / Number of functions.
Interpretation 0 <= x <= 1; closer to 1 being better
MaintainabilitySub-Characteristic TestabilityQuality Attribute Extensive component test casesGoal Analyzes if the component was
extensively tested before being made available to the market.
Question - How many tests cases are executed? - What is the coverage of these test cases?
Metric Number of functions / Number of test cases
*Still on it is interesting to analyze the number of bugs that were corrected during the test case.
Interpretation The test cases coverage is very important to be analyzed and the number of bugs discovered during the execution of the tests.
MaintainabilitySub-Characteristic TestabilityQuality Attribute Component test in a specific
environmentGoal Analyzes the environments
where the component can work
185
wellQuestion In which environment this
component can be executed without errors?
Metric Number of environments that work well /Number of environments defined on specification.
Interpretation 0 <= x <= 1; closer to 1 being better
MaintainabilitySub-Characteristic TestabilityQuality Attribute Proofs the component testGoal Analyzes if the tests are formally
provedQuestion How is the coverage of the proof
in the test cases?Metric Proofs AnalysisInterpretation It is interesting to note if the
amount of formal proof covers the whole test cases provided by the component. As higher it is better.
• Example of Metrics to track Portability Characteristics
For the Portability there are four sub-characteristics that could be evaluated: Deployability, Replaceability, Adaptability and Reusability. These have a set of quality attributes, for which will be presented at least one metric as example of usage, as follows.
For the Deployability Sub-Characteristic the following metrics could be applied:
PortabilitySub-Characteristic DeployabilityQuality Attribute Complexity levelGoal Analyzes how complex it is to
deploy a component in its specific environment(s)
Question How much time does it take to deploy a component in its environment?
Metric Time taken for deploying a
186
component in its environmentInterpretation Estimate the time first and then
compare with the actual time taken to deploy the component
PortabilitySub-Characteristic ReplaceabilityQuality Attribute Backward CompatibilityGoal Analyzes the compatibility with
previous versionsQuestion What is the compatibility with
previous versions?Metric Correct results / Set of same
invocations in different component versions
Interpretation 0 <= x <= 1; closer to 1 being better
PortabilitySub-Characteristic FlexibilityQuality Attribute MobilityGoal Analyzes the ability of the
component to be transferred from one environment to another
Question Can the component be transferred to other environment without any changes?
Metric - Analyze the component constraints and environment constraints - Analyze the component specification - Deploy the component in environment specified on documentation
* Possible metric: Number of environments where the component works correctly / Number of environments described in its specification
Interpretation 0 <= x <= 1; closer to 1 being better
PortabilitySub-Characteristic FlexibilityQuality Attribute Configuration CapacityGoal Analyzes the ability of the
component to be transferred from one environment to another, considering the related
187
changesQuestion How much effort is needed to
adapt the component to a new one environment?
Metric - Analyze the component constraints and environment constraints - Analyze the component specification - Deploy the component in environment specified on documentation - Time taken to adapt the component in its specified environments
Interpretation Analyze the time taken to deploy the component in each environment defined
PortabilitySub-Characteristic ReusabilityQuality Attribute Domain abstract levelGoal Analyzes the correct separation
of concerns in the componentQuestion Can the component be reused in
other domain applications?- Does the component have inter-related business code?
Metric Analyzes the source code and tries to reuse the component in other domains
Interpretation If the component does not contain business code related to specific domain and can be reused around a set of domains, it is good candidate to be reused. On the other hand, if it does have code related to a specific domain and it becomes difficult to reuse it around some domain, the component is not good candidate to be reusable and should be revised.
PortabilitySub-Characteristic ReusabilityQuality Attribute Architecture compatibilityGoal Analyzes the level of
dependability of a specified
188
architectureQuestion Was the component correctly
designed based on the architecture constraints defined?
Metric Analysis of the component design based on some documentation and source code
Interpretation Understand the architecture constraints and analyze if the component follows that one specification during its development and implementation. Based on this, the component can be considered good to be reused or not.
PortabilitySub-Characteristic ReusabilityQuality Attribute ModularityGoal Analyzes the internal
organization of the componentQuestion How logically separated are the
component concerns?Metric Packaging analysisInterpretation If the component contains some
packages that isolate each logical concern, it should have good modularity and become more reusable and extensible. On the other hand, if the component doesn’t contain a well define internal structure, the modularity level is slower and, consecutively, the reusability level decreases.
PortabilitySub-Characteristic ReusabilityQuality Attribute CohesionGoal Analyzes the cohesion between
the internal modules/ packages/ functionalities of the component
Question How is the cohesion level of the component?
Metric Analysis of the inter-related partsInterpretation A component should have high
cohesiveness in order to increase its reusability level
189
PortabilitySub-Characteristic ReusabilityQuality Attribute CouplingGoal Analyzes the coupling between
the internal modules/ packages/ functionalities of the component
Question How is the coupling level of the component?
Metric Analysis of the inter-related parts (Called modules, methods, etc)
Interpretation A component should have low coupling in order to increase its reusability level
PortabilitySub-Characteristic ReusabilityQuality Attribute SimplicityGoal Analyzes the way the component
is organizedQuestion How simple is the component’s
organization?Metric Number of modules, average
module size and cyclomatic complexity.
Interpretation The component’s organization should be easily understandable and (re)usable. The simpler the better.