1

Pós-Graduação em Ciência da Computação

“An Embedded Software Component Quality Verification Framework”

Fernando Ferreira de Carvalho

PHD THESIS PROPOSE

Universidade Federal de Pernambuco

posgraduacao@cin.ufpe.br

www.cin.ufpe.br/~posgraduacao

RECIFE, 11/2008

UNIVERSIDADE FEDERAL DE PERNAMBUCO

CENTRO DE INFORMÁTICA

PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO

Fernando Ferreira de Carvalho

“An Embedded Software Component Quality Verification Framework"

ESTE TRABALHO FOI APRESENTADO À PÓS-GRADUAÇÃO EM CIÊNCIA DA COMPUTAÇÃO DO CENTRO DE INFORMÁTICA DA UNIVERSIDADE FEDERAL DE PERNAMBUCO COMO REQUISITO PARCIAL PARA OBTENÇÃO DO GRAU DE DOUTOR EM CIÊNCIA DA COMPUTAÇÃO.

A PHD. THESIS PRESENTED TO THE FEDERAL UNIVERSITY OF PERNAMBUCO IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF PHD. IN COMPUTER SCIENCE.

ADVISOR: Silvio Lemos Meira

RECIFE, NOVEMBER/2008

R ESUMO

Um dos maiores desafios para indústria hoje é fornecer produtos com alto nível de qualidade e funcionalidade com um baixo custo e curto tempo de desenvolvimento. Os requisitos de custo e tempo de desenvolvimento têm sido abordados com bastante êxito pela engenharia de software baseada em componentes (CBSE). A tecnologia baseada em componentes é amplamente utilizada nas indústrias mecânica e de componentes eletrônicos, entre outras áreas, mas na indústria de software embarcados ainda não alcançou o nível desejado. É de conhecimento geral que a força de um corrente está intimamente ligada com a força de cada elemento que a compõe. Da mesma forma, a qualidade dos atributos de um sistema embarcado, como seu desempenho, sua confiabilidade e segurança, dependem do nível de qualidade de cada componente que integra o sistema. Por este motivo, a abordagem baseada em componentes, embora atraente para muitas razões, é difícil utilizar em domínios nos quais atributos de qualidade são de vital importância. Em CBSE, utilização de mecanismos apropriados de pesquisa, seleção e avaliação do processo de componentes, é considerada o ponto chave para o desenvolvimento de qualquer sistema baseado em componentes. Por este e outros motivos, esta tese propõe um Framework para Verificação da Qualidade de Componentes de Software Embarcados, que prove mecanismos de avaliação dos atributos de qualidade do componente de software embarcado, baseando-se módulos bem definidos que se complementam uns aos outros em busca da qualidade dos componentes embarcados, de forma consistente. O sonho do desenvolvedor de sistemas embarcados é que a avaliação dos atributos de qualidade de seus componentes alcance níveis semelhantes ao dos componentes utilizados nos sistemas mecânicos. Pela introdução de verificação e certificação da qualidade de componentes de software no projeto de sistemas embarcados, esta tese é um pequeno passo para alcançar este sonho.

A bstract

One of the major challenges to industry today is to provide products with high degrees of quality and functionality at low cost and short time to market. The cost and time to market requirements have quite successfully been addressed by the component-based software engineering (CBSE). Unfortunately, satisfactory solutions for handling quality are not yet available. Component technologies are widely used in the electronics and mechanics industries, but software industry has not reached the desired level. It is well known that the strength of a chain is proportional to the strength of its interconnected elements. Similar to this concept, the quality attributes of the final embedded system, such as its performance, scalability or reliability, depends on the quality level of the components that integrate the system. For these reasons, the component-based approach, although attractive for many reasons, is difficult to apply in domains in which quality attributes are of primary importance. In CBSE, the proper search, selection and evaluation process of components is considered the corner stone for the development of any effective component-based system. For these and other reasons, this thesis proposes an Embedded Software Component Quality Verification Framework that provides mechanisms for assessing quality attributes of embedded software components, based upon well-defined modules that complement each other looking for the embedded component quality in a consistent way. Having the means to reason about the qualities of an embedded software design in the same way as one can reason about the qualities of a mechanical design is a dream of embedded design. By introducing component quality verification and certification in embedded systems design, this thesis is a small step towards fulfilling this dream.

L ist of Figures

18Figure 1.1. The Framework for Software Reuse.

19Figure 1.2. The Component Certification Process was divided into two parts:

20Figure 1.3. Embedded Software Component Quality Verification Framework.

31Figure 2.1. An embedded system encompasses the CPU as well as many other resources.

33Figure 2.2. Component technology for small embedded Systems

41Figure 2.3. The results. Y-axis: priority of quality attributes in a scale 1 (highest),

54Figure 3.1. Structure of Prediction-Enabled Component Technology (Wallnau, 2003).

63Figure 3.2. SQuaRE Architecture

68Figure 3.3. ISO/IEC 25040

71Figure 3.4. Process of obtaining, certifying and storing components

72Figure 3.5. Research on embedded software component quality and certification timeline.

78Figure 4.1. Embedded Software Component Quality Verification Framework.

90Figure 4.2. Relations among the quality model elements.

100Figure 4.3.1 A different evaluation level for each embedded component characteristics.

119Figure 4.5.1. Component Certification Process.

120Figure 4.5.2. Establish Evaluation Requirements activity.

125Figure 4.5.3. Specify the Evaluation activity.

129Figure 4.5.4. Design the Evaluation activity.

133Figure 4.5.5. Execute the Evaluation activity.

L ist of Tables

37Table 2.2. Summary of relevant characteristics in Crnkovic’s research.

40Table 2.2.2. Priority classes used to classify the importance of the different quality attributes.

58Table 3.1. Software quality standards.

59Table 3.2. IEC61131

61Table 3.3. DO-178B

67Table 3.4. Characteristics and Sub-Characteristics in SQuaRE project.

83Table 4.1. Changes in the Proposed Embedded software Component Quality

83Model, in relation to ISO/IEC 25010.

89Table 4.2. The Proposed Component Quality Model, with the sub-characteristics

89divided into two kinds: runtime and life-cycle.

91Table 4.2.4. Embedded software Component Quality Attributes for

91Sub-Characteristics those are observable at Runtime and Life-cycle.

98Table 4.2.5 Additional Information.

101Table 4.3.1. Guidelines for selecting evaluation level.

105Table 4.3.2. Embedded software component Maturity Model.

106Table 4.3.3. Embedded Quality Attribute X Evaluation Techniques.

123Table 4.5.1. Example of Importance’s definition.

125Table 4.5.2. Example of Quality Attributes Definition.

126Table 4.5.3. Example of Define EMM.

130Table 4.5.4. Example of Tools Selection.

134Table 4.5.5. Example of Data Collection.

140Table 5.3. Cronogram.

A cronyms

Terms - Descriptions

B2B

- Business to Business

CBD

- Component-Based Development

CBSE

- Component-Based Software Engineering

CMU/SEI - Carnegie Mellon University’s Software Engineering Institute

COTS

- Commercial Off-The-Self

CBSD

- Component-Based Software Development

COM

- Component Object Model

CCM

- CORBA Component Model

CMM

- Capability Maturity Model

CMMI - Capability Maturity Model Integrated

EQM

- Embedded software component Quality Model

EMM

- Embedded software component Maturity Model

GQM

- Goal Question Metric Paradigm

ISO

- International Organization for Standardization

IEC

- International Electro-technical Commission

PECT

- Prediction-Enabled Component Technology

PACC

- Predictable Assembly from Certifiable Components

RiSE

- Reuse in Software Engineering group

SPICE - Software Process Improvement and Capability dEtermination

OMG

- Object Management Group

XML

- eXtensible Markup Language

C ontents

111Introduction

141.1Motivation

201.1.1The needs and priorities in research

241.2Problem formulation

251.3Proposed solution

261.4Out of Scope

281.5Statement of Contributions

281.6Organization of the Thesis

302Embedded systems design

312.1Basic concepts for Component-Based embedded systems

322.1.1Component-based approach for small embedded systems

332.1.2Component-based approach for large embedded systems

352.2Specific requirements for embedded systems

372.2.1Industrial Automation

382.2.2Automotive

412.2.3Medical

422.2.4Consumer Electronics

432.2.5Other domains

432.3The needs and priorities in research

463Embedded Software Component Quality and Certification: A Survey

563.1Failures in Software Component Certification

573.1.1Failure in National Information Assurance Partnership (NIAP).

573.1.2Failure in IEEE.

573.2Standardization: Software Quality and Certification

593.2.1ISO/IEC 61131

603.2.2RTCA DO 178B

623.2.3ISO/IEC 61508

623.2.4ISO/IEC 25000 (SQuaRE project)

693.3Software Component Certification

723.4Summary of the Study

733.5Summary

744Embedded software Component Quality Verification Framework

754.1Overview of the Framework

784.2Embedded software component Quality Model

804.2.1The Embedded software component Quality Model (EQM)

834.2.2Changes in relation to ISO/IEC 25010

844.2.2.1Quality sub-characteristics that were created from ISO/IEC 25010

864.2.2.2Quality sub-characteristics that were removed from ISO/IEC 25010

874.2.2.3Quality sub-characteristics that were renamed from ISO/IEC 25010

884.2.2.4Quality characteristics that were extended from ISO/IEC 25010

894.2.3Summary

904.2.4Embedded software Component Quality Attributes

984.2.5Other relevant Component Information

994.3Maturity Level Evaluation Techniques

1004.3.1Embedded software component Maturity Model (EMM)

1094.4Metrics Approach

1144.4.1Metrics to track the EMM Properties

1154.4.2Metrics to track the Certification Techniques Properties

1164.4.3Metrics to track the Certification Process Properties

1174.5Component Certification Process

1184.5.1The Component Certification Process

1194.5.1.1Establish Evaluation Requirements activity

1244.5.1.2Specify the Evaluation activity

1294.5.1.3Design the Evaluation activity

1324.5.1.4Execute the Evaluation activity

1354.5.1.5Process Summary

1365Conclusion and future works

1375.1Contributions

1395.2Future Work

1405.3Chronogram

1416Bibliographic References

155Appendix A. Metrics Example

1.1.1 Introduction

Embedded system is at the heart of many systems in use today. Added value to products is to a large extent provided by the software. Furthermore, production cost reduction is imperative and is often achieved by introducing software that permits the use of less complex hardware. Domains in which the use of software is now essential include, among others, the automotive, medical systems, process control, and manufacturing industries. Embedded systems are often systems consisting of software and hardware, the software part incorporates many software components that must cooperate without fail. It is platform-dependent, consequently difficult to port, upgrade, and customize, and offers limited opportunities for reuse, even within a single application domain.

The demands that companies in these electronic products must satisfy include low production costs, short time to market and high quality. The cost and time to market issue is addressed by means of the rapidly emerging Component-based Development (CBD) approach. Adding new functionalities to existing products must be done at low cost and high quality. In CBD, software systems are built as an assembly of components already developed and prepared for integration.

The potential benefits of component-based development are as attractive in the domain of embedded systems as they are in other areas of the software industry. These include reduction of costs, improved quality, specialization of expertise, effective management of complexity, reduced time to market, increased productivity, a greater degree of consistency, and a wider range of usability (Brown, 2000). Additionally, CBD offers an opportunity to increase productivity by providing natural units for reuse (components and architectures), by raising the level of abstraction for system construction, and by separating services from their configurations to facilitate evolution (Müller et al., 2002).

However, these benefits are much more difficult to realize in embedded systems development than in other areas of software engineering because the problems of composing extra-functional requirements such as quality, reliability and performance are much more acute. When building new applications from existing components it is not only necessary to ensure that they behave as expected, but also that these extra-functional properties are composed correctly. Until recently it was difficult to do this in an economically viable way, but recent trends have combined to make component-based software engineering as imperative for embedded systems development as it is in other domains. According to Atkinson (Atkinson at al., 2005), there are three important trends with embedded domain:

· The shear growth in the number and ubiquity of embedded systems in the world around us;

· The growth in the size and complexity of software in embedded systems.

· The creation of larger product families in order to tailor products variants to the needs of specific customers and/or market segments.

New functionality is frequently added to products to meet market requirements. The new functionality is often introduced by adding components to the system. This is the reason why the component-based development approach is attractive to industry.

Different classes of components are used in different industrial settings, and different component technologies target different application domains. Industrial component technologies such as IEC 61131-3 (ISO/IEC 1131-3, 1995) and Koala (Van Ommering, 2002) are used to build applications. Other component classes are used to provide the infrastructure in systems for examples, databases, real-time operating systems, user interfaces and communication components. This diversity means that it is possible to use components at different levels in a system.

The quality aspects of software products are not, however, addressed adequately by component-based development. Component-based applications are sensitive to evolution of the system. As components and applications have separate lifecycles and different kinds of requirements, there is some risk that a component will not completely satisfy the particular requirements of certain applications or that it may have characteristics unknown to the application developer. When introducing changes on the application level (changes such as updating the operating system, updating other components, changes in the application), there is a risk that the change introduced will reduce the reliability of the system due to different unforeseen mismatches (Larsson, 2000).

Components are in general considered as black boxes with little or no information easily accessible. The information needed from the software components relates to their quality attributes. If the components’ attributes are not known, the attributes of the system of which they are part of is even more difficult to reason about. Industrial products are required to meet high functional and quality standards. Customers often pay for the functionality they require and take the quality of the product for granted. This business model influences the way software is developed. Unfortunately, as software developers tend to focus on delivering the functionality without giving the quality requirements the same priority it is possible to ensure the quality characteristics and the functionality of a product by means of extensive testing, but this is generally an expensive and inefficient approach. A much more effective approach would be to ensure the quality attributes of the product on the basis of the quality of the components of which it is constructed.

A software program computes functions with certain functional attributes. There are other attributes that we care for more than the functional ones, these attributes are expressed in terms of quality attributes. A quality attribute is originally the characteristic of a system but today we also recognize quality attributes of software. Security, safety, performance and dependability are examples of system characteristics. Quality attributes are primarily attributes of systems which more and more are achieved by software.

Quality attributes are often referred to as properties, non-functional or extra-functional attributes because they relate to the quality of the component and not explicitly to the component functionality (Barbacci et al, 1995). The quality attributes describe the characteristics of a component. The properties of a component are the concrete accessible values that represent the characteristics. A component can express its quality attributes in terms of its properties.

1.1.1 Motivation

This creates a big challenge for embedded-software development. In the years to come, the key to success will be the ability to successfully develop high-quality embedded systems and software on time. As the complexity, number, and diversity of applications increase, more and more companies are having trouble achieving sufficient product quality and timely delivery. To optimize the timeliness, productivity, and quality of embedded software development, companies must apply software engineering technologies that are appropriate for specific situations.

Unfortunately, the many available software development technologies do not take into account the specific needs of embedded-systems development. This development is fundamentally different from that of non embedded systems. Technologies for the development of embedded systems should address specific constraints such as severe timing constraints, limited memory and power use, predefined hardware platform technology, and hardware costs. Existing development technologies do not address their specific impact on, or necessary customization for, the embedded domain. Nor do these technologies give developers any indication of how to apply them to specific areas in this domain—for example, automotive systems, telecommunications, or consumer electronics. Consequently, tailoring a technology for a specific use is difficult. Furthermore, the embedded domain is driven by reliability factors, cost factors, and time to market. So, this embedded domain needs specifically targeted development technologies. In industry, the general feeling is that the current practice of embedded software development is unsatisfactory. However, changes to the development process must be gradual; a direction must be supplied.

Embedded systems comprise a scale from ultra small devices with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume products in which the system dependability is the most important feature. All these different requirements have impact on feasibility, on use, and on approach in component-based development.

There is no universal definition of an embedded system. However, there is a certain consensus that the following features are common to most embedded systems (Camposano & Wilberg, 1996):

· They are part of a larger system (host system), hence the term embedded, with which they continuously or frequently interact. Usually, the embedded system serves as a control unit inside the host system.

· They have a dedicated functionality and are not intended to be reprogrammable by the end-users. Once an embedded system is built, its functionality does not change throughout its lifetime. For example, a device controlling the engine of a car will probably never be reprogrammed to decode Mp3s. A desktop computer, on the other hand, has a wide range of functionality, including web browsing, word processing, gaming, advanced scientific calculator, etc.

· They have real-time behavior. The systems must, in general, respond to their environment in a timely manner.

· They consist of both hardware and software components. In order to cope with the wide and unpredictable range of applications, the hardware of a general purpose computer has to be meticulously designed with the risk of wasting resources. However, since the set of applications to be run on an embedded system is known at design-time, including their performance requirements, the hardware can be tuned at design-time for best performance at minimal cost. Similarly, software must also be optimized to build a globally efficient HW/SW system.

There are many domains in which embedded systems are used extensively. Some of them are: Telecommunication, avionics and aerospace, transportation, computer games, home electronics, navigation systems, etc. While there are many similarities between these domains there are also very different requirements for their functional and extra-functional properties. The consequences are that the requirements for component-based technologies are different, and consequently it expects to have one component model that summarizes all quality attributes in common in different domains. The expectations are that only one component models will exist, some common characteristics, such as basic principles of component specifications through interfaces, basic composition and run-time services, certain patterns, and similar.

All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes (Crnkovic, 2005). These properties can be classified in run-time and life cycle extra-functional properties.

A common characteristic of all systems is the increasing importance of software. For example, software development costs for industrial robots today constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten to fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies

One of the most compelling reasons for adopting component-based approaches in embedded software design is the premise of reuse. The idea is to build software from existing components primarily by assembling and replacing interoperable parts. The implications for reduced development time and improved product quality make this approach very attractive (Krueger, 1992).

Software reuse is not a new idea. Since McElroy’s pioneer work, “Mass Produced Software Components” (McElroy, 1968), the idea of reusing software components on a large scale is being pursued by developers and research groups. This effort is reflected in the literature, which is very rich in this particular area of software engineering.

McIlroy’s work does not consider an essential requirement for these systems: the assets certification. In a real environment, a developer that retrieves a faulty component from the repository would certainly lose his trust in the system, becoming discouraged to make new queries. Thus, it is extremely important to assert the quality of the assets that are stored into the repository before making them available for reuse. Despite this importance, the software engineering community had not explored these issues until recently. In this way, a new research area arose: components certification and quality assurance (Wohlin et al., 1994), (Mingins et al., 1998), (Morris et al., 2001), (Schmidt, 2003), (Wallnau, 2003). However, several questions still remain unanswered, such as:

(i) How certification should be carried out?

(ii) What are the requirements for a certification process? and,

(iii) Who should perform it? (Goulão et al., 2002a).

This is the reason why there is still no well-defined standard to perform component certification (Voas et al., 2000), (Morris et al., 2001).

This thesis addresses embedded software component quality issues with a particular emphasis on component quality certification for reuse proposes. It also covers issues related to design of component-based embedded systems. This thesis is investigating effective ways to demonstrate that embedded component certification is not only possible and practically viable, but also directly applicable in the embedded software design. Through certification, some benefits can be achieved, such as: higher quality levels, reduced maintenance time, investment return, reduced time-to-market, among others. According to Weber et al. (Weber et al., 2002), the need for quality assurance in software development has exponentially increased in the past few years.

Moreover, this thesis is a part of a big project that aims to develop a robust framework for software reuse (Almeida et al., 2004), in order to establish a standard for component development; to develop a repository system; and to develop a general software component certification process. This project has been developed in collaboration between the industry and academia (the RiSE group1 and three other universities), in order to generate a well-defined model for developing, evaluating quality, storing and, subsequently, making it possible for software factories to reuse software components.

The framework (Figure 1.1) that is being developed has two layers. The first layer (on the left) is composed of best practices related to software reuse. Non-technical factors, such as education, training, incentives, and organizational management are considered. This layer constitutes a fundamental step prior to the introduction of the framework in the organizations. The second layer (on the right) is composed of important technical aspects related to software reuse, such as processes, environments, tools, and a certification process, which is the focus of this thesis.

Figure 1.1. The Framework for Software Reuse.

However, to better support the embedded domain and our specific quality characteristics, the component certification process was divided into two parts. The first of these, focuses on general propose software component certification, in general, the software component used in IBM-PC compatible. The other, is a component certification processes that aligns to specific requirements and constraints to develop quality characteristics for embedded domain.

Figure 1.2. The Component Certification Process was divided into two parts:

i – General Propose Certification and ii - Embedded Component Certification.

The process of certifying components proposed is not a simple one. First, there should be an Embedded software component Quality Model (EQM). Differently from other software product quality models, such as (McCall et al., 1977), (Boehm et al., 1978), (Hyatt et al., 1996), (ISO/IEC 9126, 2001), (Georgiadou, 2003), this model should consider Component-Based Development (CBD) characteristics applied to embedded domain, and describe attributes that are specific to the promotion of reuse. With the availability of an embedded software component quality model, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different level of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics is required, in order to track the components properties and the enactment of the certification process. These four main issues:

(i) Embedded software component Quality Model,

(ii) Maturity Level Evaluation Techniques,

(iii) Metrics Approach, and

(iv) Certification Process,

Are the modules of an Embedded Software Component Quality Verification Framework (Figure 1.3) that is being investigated as a part of the RiSE project.

Figure 1.3. Embedded Software Component Quality Verification Framework.

The framework will allow that the components produced in a Software Reuse Environment are certified before being stored on a Repository System. In this way, software engineers would have a greater degree of trust in the components that are being reused.

1.1.1 The needs and priorities in research

The component-based approach on system level, where hardware components are designed with embedded software, has been successfully used for many years. Also large-grain generic components like protocol stacks, RTOSs, etc. have been used for a long time. In addition to this, technology supporting a component-based approach has been developed; either in the form of proprietary component models, or by using reduced versions of some widely used component models.

Still, there are needs for a number of improvements (Crnkovic, 2005). Some of them are the following (differently important for different domains):

· There is a lack of widely adopted component technology standards which are suitable for embedded systems. For smaller-size embedded systems, it is important that a system composed of components can be optimized for speed and memory consumption, which is still missing in most of the component technologies available today.

· There is also a need for interoperability between different component technologies. Specification technology is not sufficiently developed to guarantee a priori component interoperability.

· Component certification: In order to transfer components across organizations, techniques and procedures should be developed for ensuring the trustworthiness of components.

· Most current component technologies do not support important extra-functional properties.

· There is a need for generic platform services, for, e.g., security and availability.

· Tools that support component based development are still lacking.

· There is a lack of efficient implementations of component frameworks (i.e., middleware), which have low requirements on memory and processing power. Major needs for the further development of component technology for embedded systems are the following (Brinksma et al., 2001).

· Need for adopted component models and frameworks for embedded systems. A problem is that many application domains have application-dependent requirements on such a technology.

· Need for light-weight implementations of component frameworks. In order to support more advanced features in component-based systems, the run-time platform must provide certain services, which however must use only limited resources.

· Obtaining extra-functional properties of components: Timing and performance properties are usually obtained from components by measurement, usually by means of simulation. Problems with this approach are that the results depend crucially on the environment (model) used for the measurements may not be valid in other environments, and that the results may depend on factors which cannot easily be controlled. Techniques should be developed for overcoming these problems, thereby obtaining more reliable specifications of component properties.

· Platform and vendor independence: Many current component technologies are rather tightly bound to a particular platform (either run-time platform or design platform). This means that components only make sense in the context of a particular platform.

· Efforts to predict system properties: The analysis of many global properties from component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.

· Component noninterference: Particularly in safety-critical applications, there is a need to ensure separation and protection between component implementations, in terms of memory protection, resource usage, etc.

· Tool support: The adoption of component technology depends on the development of tool support. The clearly identified priorities of CBSE for embedded systems are:

· Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc.

· Development of widely adopted component models for real-time systems. Such a model should be supported by technology for generating necessary runtime infrastructure (which must be light-weight), generation of monitors to check conformance with contracts, etc.

Further, according to Voas (Voas, 1998), to foster an emerging software component marketplace, it must be clear for buyers whether a component’s impact is positive or negative. Ideally, buyers should have this information before buying a component. Component buyers could then choose an appropriate component according to its certification level. With this information, system builders could make better design decisions and be less fearful of liability concerns, and component vendors could expect a growing marketplace for their products.

The Carnegie Mellon University’s Software Engineering Institute (CMU/SEI) (Bass et al., 2000) studied industry trends in the use of software components from technical and business perspectives. A distinct set of inhibitors to adopting software component technology emerged from the conducted surveys and interviews of earlier adopters of software component technology.

From this data and from the interviews, the study concludes that the market perceives the following key inhibitors for component adoption, presented here in decreasing order of importance:

· Lack of available components;

· Lack of stable standards for component technology;

· Lack of certified components; and

· Lack of an engineering method to consistently produce quality systems from components.

The software engineering community is already attempting to reduce the gaps related to the two first inhibitors.

However, in relation to the third inhibitor, the community is still a fledgling. Further research is required in order to assure the production of certified components, especially when combined with the lack of component-based software engineering techniques that deliver predictable properties (the last inhibitor).

Important researches on software components, such as Heineman (Heineman et al., 2001), Councill in (Councill, 2001), Crnkovic (Crnkovic, 2005), Wallnau (Wallnau, 2003), Wallnau (Wallnau, 2004), Schneider & Han (Schneider & Han, 2004) and Andreou & Tziakouris (Andreou & Tziakouris, 2007) indicates that the future of software components is certification. These authors state that certification is a necessary precondition for CBSE to be successfully adopted and to achieve the associated economic and social benefits that CBSE could yield. With the success of CBSE, software developers will have more time to develop, instead of spending their time addressing problems associated with understanding and fixing someone else’s code. Certified components used during development will have predetermined and well established criteria, thus reducing the risk of system failure and increasing the likelihood that the system will comply with design standards.

1.2.1 Problem formulation

The previous section stated that designers increasingly often build systems using reusable components due to the complexity of their designs. Therefore, there is an increasing need to efficiently and effectively qualify such systems. Quality assurance, in particular certification, which can effectively cope with this situation and take advantage of the component-based structure, need to be developed.

The evaluation of the quality of components (i.e. the assessment of their quality attributes) needs to be done by independent parties, at least until software vendors acquire the level of maturity that hardware vendors currently have. The software industry still far from counting with the hardware data sheets and catalogues available for hardware components that describe all their characteristics. However, it is necessary to have them for software components too if it wants to talk about a “real” Component-based Software Engineering. Many organizations struggle in their attempts to select and evaluate an appropriate component in their system. For this reason, a well-defined and consistent software component quality assurance is essential for the component market adoption (i.e. without an efficient mechanism to select/evaluate the component quality.

According to Morris et al. (Morris et al., 2001), there is a lack of an effective assessment of software components in general. Besides, the international standards that address the software products’ quality issues (in particular, those from ISO and IEEE) have shown to be too general for dealing with the specific characteristics of embedded components. While some of their characteristics are appropriate to the evaluation of components in embedded domain, others are not well suited for that task.

However, the main drawback of the existing international standards, in this case the ISO/IEC 25010, is that they provide very generic quality models and guidelines, which are very difficult to apply to specific domains such as embedded components and CBSE. Thus, the quality characteristics of this model should be analyzed in order to define and adequate to the component quality characteristics context.

A unified and prioritized set of CBSE requirements for trustworthy components is a challenge in itself (Schmidt, 2003). This fact may be due also to the relatively novelty of this area (Goulão et al., 2002a). Moreover, there is a lack of processes, methods, techniques and tools that support the component certification activity (Alvaro et al., 2005). The existent processes and/or methods dealing only with specific aspects of software component (Alvaro et al., 2005) were not evaluated into industrial scenarios. In fact, they are based on the researchers’ experience, and the real efficiency of evaluating software components using these process/methods remains unknown.

One of the keys to success in industry in the future will be the ability to develop high-quality embedded systems and their software on time in order cost reduced to remain competitive. Organizations whose aim is to construct software by integrating components – rather than developing software from scratch – will not be able to meet their objectives if they cannot find sufficient number of components and component versions that satisfy certain functional and quality requirements. Without a quality level, the component usage may have catastrophic results (Jezequel et al., 1997). However, the common belief is that the market components are not reliable and this prevents the emergence of a mature software component market (Trass et al., 2000). Thus, the components market quality problems must be resolved in order to increase the reliability, and third-party certification programs would help to acquire trust in the market components (Heineman et al., 2001).

1.3.1 Proposed solution

This work defines an Embedded Software Component Quality Verification Framework, which includes the steps of definition of embedded software component quality, maturity level evaluation techniques, metrics and component certification, based on a set of activities, metrics and guidelines. Moreover, the process is based on the state-of-the-art in the area and its foundations and elements are discussed in details

In this way, the main goals of this thesis were to define Embedded Software Component Quality Verification Framework that is composed of four inter-related modules in order to assure the component quality degree. This framework was proposed with basis in the standards ISO/IEC 25010, ISO/IEC 9126, ISO/IEC 14598 adapted for component context and embedded domain. Different from other approaches in the literature (Goulão et al., 2002b), (Beus-Dukic et al., 2003), (Cho et al., 2001), (Gui and Scott, 2007) that provide only isolated aspects to assure the component quality, this thesis tries to investigate and make available a complete framework with all necessary mechanisms to execute the component evaluation activities.

Through these evaluations it is expected a continuous evolution of the whole framework in a way that the software industry can start to trust in it and evaluate its own software components.

In order to achieve the main objective, the process is based on the following foundations:

First, there should be an embedded software component quality model. With an embedded software component quality model available, there must be a series of techniques that allow one to evaluate whether a component conforms to the model in different levels of maturity. The correct usage of these techniques should follow a well-defined and controllable certification process. Finally, a set of metrics are needed, in order to track the components properties and the enactment of the certification process. These four main modules:

· Embedded software component Quality Model,

· Maturity Level Evaluation Techniques,

· Metrics Approach, and

· Certification Process.

1.4.1 Out of Scope

As the proposed reuse process is part of a broader context, a set of related aspects will be left out of its scope. Nevertheless, as these aspects were envisioned since the initial definitions of the process, they can be added in the future with some adjustments. Thus, the following issues are not directly addressed by this work:

· Cost Model: Cost estimation is a key requirement for CBSD before the actual development activities can proceed. Cost is a function of the enterprise itself, its particular development process, the selected solution, and the management and availability of the resource during the development project (Cechich et al., 2003), (Mahmooda et al, 2005). A cost model is useful to help the software engineer during the analysis of the software component available to purchase (or to select or to evaluate). However, it makes sense when, first, you have defined processes, methods, techniques and tools to execute the selection and/or the evaluation task in order to identify the cost/benefit to purchase or to evaluate a component.

· Formal Proof: Meyer (Meyer, 2003) and Karlsson (Karlsson, 2006) proposes a formal approach in order to acquire trust in software components. His idea is to build or to use software components with fully proved properties or characteristics. The intention is to develop software components that could be reliable to the software market.

This thesis does not consider cost model because the whole framework that is the basis for embedded software component evaluation will be considered in this first moment and, after that a cost model to help organizations to define if it is viable evaluate certain kinds of components (or not) will be useful. The formal quality assurance is not considered mainly because the software component market is not inclined to formally specify their software components. This kind of approach is highly expensive, in terms of development time and level of expertise that is needed, and component developers still do not know if it is cost effective to spend effort in this direction without having specific requirements such as strict time constraints or high reliability. However, the Embedded software component Maturity Model (EMM) provides formal proof evaluation techniques that could be useful in some scenarios, depending of the customer’s necessities and the cost/benefit to do so;

1.5.1 Statement of Contributions

As a result of the work presented in this thesis, the following contributions can be enumerated:

· An extensive study of the key developments in the field of quality and certification of embedded software component, in an attempt to analyze this research area and identify trends to follow;

· A survey of the state-of-the-art of quality and certification of general propose software component in order to understand and identify the weak and strong points of existing processes;

· A development of an Embedded Software Component Quality Verification Framework aiming to provide a complementary mechanism to standards ISO/IEC to evaluate the component quality;

· Definition of the Embedded software component Quality Model (EQM), based on the new standard, the Software Product Quality Requirements and Evaluation (SQuaRE) project (ISO/IEC 25000, 2005) and Component Quality Model (CQM) (Alvaro et al., 2006);

· A development of the Embedded software component Maturity Model (EMM) in order to provide different thoroughness levels of evaluation techniques and a set of guidelines for selecting those evaluation levels;

· A development of the Component Certification Process in order to provide a high quality and consistent evaluation process; and

· A development of the Metrics Approach that is composed of a set of valuable measures to be considered as starting point during the component evaluations;

1.6.1 Organization of the Thesis

The remainder of this thesis is organized as follows.

Chapter 2 presents a brief overview of embedded system design, component-based development areas and requirements for embedded in different domain. The main concepts of these topics are considered.

Chapter 3 presents, in the first part, the survey of the state-of-the-art of the embedded software component quality and certification area that was performed. The failure cases that can be found in the literature are also described in this chapter. Subsequently, in the final part, this chapter describes the aspects related to the software quality and certification concepts and standardization. The intention is to show that software reuse depends on quality.

Chapter 4 presents the Embedded Software Component Quality Verification Framework proposed and its related modules. Session 4.2 describes the proposed Embedded software component Quality Model (EQM), showing its characteristics, its sub-characteristics, the quality attributes and the metrics that are related to the model. Session 4.3 presents the Embedded software component Maturity Model (EMM) that is composed of a set of evaluation levels in order to provide flexibility to the component evaluation. Further, a set of guidance is delineated to direct the evaluation team during the selection of levels. Session 4.4 presents the Metrics Approach and the paradigm adopted to define the metrics is also presented. Some few examples of metrics usage are presented. Session 4.5 presents the Software Component Certification Process, describing all activities and steps that should be followed to execute the component evaluation activity in a more controllable way.

Chapter 5 summarizes the main contributions of this work, presents the related works, the concluding remarks and the future work that will be accomplished during next months.

2.1.1 Embedded systems design

Embedded systems comprise a scale from ultra small devices with simple functionality, through small systems with sophisticated functions, to large, possibly distributed systems, where the management of the complexity is the main challenge. An embedded system may be represented by a dedicated processor surrounded by dedicated hardware systems, performing very specific tasks (Carvalho & Junior et al., 2004). Further, it can distinguish between systems produced in large quantities, in which the low production costs are very important and low-volume products in which the system dependability is the most important feature. All these different requirements have an impact on feasibility, on use, and on approach in component-based development. A common characteristic of all systems is increasing importance of software. For example, software development costs for industrial robots currently constitute about 75% of total costs, while in car industry it is currently about 30%. Some ten, fifteen years ago this number was about 25% for robots and negligible for cars (Crnkovic, 2005). A second common characteristic is increasing interoperability. While previously the embedded systems were mainly used for controlling different processes today they are integrated with information systems of infotainment technologies. In this chapter, a short overview of embedded systems design will be shown.

Figure 2.1. An embedded system encompasses the CPU as well as many other resources.

2.1.1 Basic concepts for Component-Based embedded systems

In classic engineering disciplines a component is a self contained part or subsystem that can be used as a building block in the design of a system. In software engineering, there are many different suggestions for precise definitions of components in component based software development. The best accepted understanding of component in the software industry world is based on Szyperski’s definition (Szyperski et al. 1998). From this definition it can be assumed that a component is an executable unit, and that deployment and composition can be performed at run-time.

In the domains of embedded systems this definition is largely followed, in particular the separation between component implementation and component interface. However the demands on the binary or executable from is not directly followed. A component can be delivered in a form of a source code written in a high-level language, and allows build-time (or design-time) composition. This more liberal view is partly motivated by the embedded systems context, as will be discussed below.

Many important properties of components in embedded systems, such as timing and performance, depend on the characteristics of the underlying hardware platform. Kopetz and Suri (Kopetz and Suri et al., 2003) propose to distinguish between software components and system components. Extra-functional properties, such as performance, cannot be specified for a software component in isolation. Such properties must either be specified with respect to a given hardware platform, or be parameterized on (characteristics of) the underlying platform. A system component, on the other hand, is defined as a self-contained hardware and software subsystem, and can satisfy both functional and extra functional properties.

2.1.1 Component-based approach for small embedded systems

The specific characteristics of embedded systems lead to specific requirements of component technologies. In particular the approaches in development process and component specifications using interfaces are different form those implemented in the component technologies widely used in other domains.

The component interface summarizes the properties of the component that are externally visible to the other parts of the system. As for embedded systems non-functional properties are as important as functional there is a tendency to include specification of extra-functional properties in the component interface (for example timing properties). This allows more system properties to be determined when the system is designed, i.e. such interface enables verification of system requirements and prediction of system properties from properties of components.

In general-purpose component technologies, the interfaces are usually implemented as object interfaces supporting polymorphism by late binding. While late binding allows connecting of components that are completely unaware of each other beside the connecting interface, this flexibility comes with a performance penalty and increased risk for system failure. Therefore the dynamic component deployment is not feasible for small embedded systems.

Taking into account all the constraints for real-time and embedded systems, we can conclude that there are several reasons to perform component deployment and composition at design time rather than run-time (Crnkovic & Larsson et al., 2002): This allows composition tools to generate a monolithic firmware for the device from the component-based design and in this way achieve better performance and better predictability of the system behavior. This also enables global optimizations: e.g., in a static component composition known at design time, connections between components could be translated into direct function calls instead of using dynamic event notifications. Finally, verification and prediction of system requirements can be done statically from the given component properties.

This implies that the component-based characteristic is mostly visible at design time. To achieve an efficient development process tools should exist which will provide support for component composition, component adaptation and static verification and prediction of system requirements and properties from the given component properties.

There may also be a need for a run-time environment, which supports the component framework by a set of services. The framework enables component intercommunication (those aspects which are not performed at design time), and (where relevant) control of the behavior of the components. Figure 2.1 shows different environments in a component life cycle. The figure is adopted from (Crnkovic & Larsson et al., 2002).

Figure 2.2. Component technology for small embedded Systems

2.1.2 Component-based approach for large embedded systems

For large embedded systems the resource constraints are not the primary concerns. Complexity and interoperability play a much more important role. Also due to complexity, the development of such systems is very expensive and cutting the development costs is highly prioritized. For this reason general-purpose component technologies are of more interest than in the case for small systems.

The technology mostly used in large systems is Microsoft COM and recently .NET, and to a smaller extent different implementations of CORBA, although none of these technologies provides support for real-time. The systems using these technologies belong to the category of soft-real time systems. Often a component technology is used as a basis for additional abstraction level support, which is specified either as standards or proprietary solutions. The main reason for wide use of component-based technology is the possibility of reusing solutions in different ranges of products, efficient development tools, standardized specifications and interoperation, and integration between different products.

One successful example of the adoption of a component-based technology is the initiative OPC Foundation (OLE process control Foundation, www.opcfoundation.org), an organization that consists of more than 300 member companies worldwide, it is responsible for a specification that defines a set of standard interfaces based upon Microsoft’s OLE/COM and recently .NET technology. OPC consists of a standard set of interfaces, properties, and methods for use in process-control and manufacturing-automation applications. OPC provides a common interface for communicating with diverse process-control devices, regardless of the controlling software or devices in the process. The application of the OPC standard interface makes possible interoperability between automation/control applications, field systems/devices and business/office applications.

Another example of a component-based approach is development and use of the standard IEC 61131 (ISO/IEC 61131-3, 1995). IEC 61131 defines a family of languages that includes instruction lists, assembly languages, structured text, a high level language similar to Pascal, ladder diagrams, or function block diagrams (FBD). Function blocks can be viewed as components and interfaces between blocks are released by connecting in-ports and out-ports. Function lock execution may be periodic or event-driven. IEC 61131 has been successfully used in development of industrial process automation systems for many years.

Large embedded systems that must fulfill real-time requirements usually do not use general-purpose component-based technologies. However in some cases, such as for ABB controllers, a reduced version of COM has been used on top of a real-time operating system (Lüders et al., 2002). The reused version includes facilities for component specification using the interface description language (IDL), and some basic services at run-time such as component deployment has been used. These services have been implemented internally. Different communication protocols and I/O drivers have been identified as components.

2.2.1 Specific requirements for embedded systems

Embedded systems vary from very small systems to very large systems. For small systems there are strong constrains related to different recourses such as power or memory consumption and others. In most of the cases, embedded systems are real-time systems. For these as well as for large embedded systems the demands on reliability, functionality, efficiency and other characteristics that depends on domain or application. Finally, in many domains, the product life cycle is very long – in can stretch to several decades.

All these characteristics have strong implications on requirements. Most of the requirements of embedded systems are related to non-functional characteristics, generally designated as extra-functional properties or quality attributes. Unfortunately, the priority quality attributes vary according to domain application. These properties can be classified in run-time and life cycle extra-functional properties. In this way, research developed by Crnkovic (Crnkovic, 2005) lists four main characteristics that must be considered to embedded systems, they are listed below:

(i) Real-time properties: a violation of time requirements even of a proper functional response violates the system functionality. The real-time properties:

a - Response time or latency,

b - Execution time,

c - Worst case execution time,

d - Deadline.

(ii) Dependability is defined as an ability of a system to deliver a service that can justifiably be trusted and an ability of a system to avoid failures that are more severe and frequent than is acceptable to the users (Crnkovic, 2005). The main means to attain dependability are related to avoidance of faults: fault prevention, fault tolerance, fault removal and fault forecasting and it is characterized by several attributes (Avižienis et al., 2001):

a - Reliability,

b - Availability,

c- Integrity,

d- Safety,

e -Confidentiality

f - Maintainability.

(iii) Resource consumption. Many embedded systems have strong requirements for low and controlled consumption of different resources. The reasons may be the size of the systems and/or the demands on lower production costs. Examples of such restrictions and constraints are:

a - Power consumption,

b - Memory consumption,

c - Execution (CPU) time,

d - Computation (CPU) power.

(iv) Life cycle properties. In many domains the embedded systems have very long life time running round the clock year after year. During the lifetime of a system several generations of hardware and software technologies can be used. The long life systems must be able to cope with these changes introduced either into the surrounding environment or into the systems themselves.

In this way, the research concludes that many of the most important requirements of the embedded systems are related to extra-functional properties. This implies that development and maintenance of such systems are very costly. In particular activities related to verification and guaranteed behavior (formal verification, modeling, tests, etc.) and maintenance (adaptive maintenance, debugging, regressive testing, etc.) require a lot of effort. For these reasons the technologies and processes that lead to lower costs for these activities are very attractive and desirable.

Table 2.2. Summary of relevant characteristics in Crnkovic’s research.

Characteristics

Sub-characteristics

Real-time properties

Response time or latency

execution time

worst case execution time

deadline

Dependability

Reliability

Availability

integrity

confidentiality

safety

Resource consumption

Power consumption

computation (CPU) power

memory Consumption

execution (CPU) time,

Life cycle properties

maintainability

2.2.1 Industrial Automation

Typical application domains of industrial automation are in control of industrial processes, power supply, and industrial robots. Industrial automation domain comprises a large area of control, monitoring and optimization systems. They typically include large pieces of software that have been developed over many years (often several decades). Most control systems are manufactured in rather large volumes, and must to a large extent be configurable to suit a variety of customer contexts. They can be classified according to different levels of control (Crnkovic & M. Larsson, 2002):

(i) Process level (for example, a valve in a water pipeline, a boiler, etc.),

(ii) Field level that concerns sensors, actuators, drivers, etc.

(iii) Group control level that concerns controller devices and applications which control a group of related process level devices in a closed-loop fashion,

(iv) Process control level i.e. operator stations and processing systems with their applications for plant-wide remote supervision and control and overview the entire process to be controlled,

(v) Production or manufacturing management level that includes systems and applications for production planning.

Notice that, even if the higher levels are not embedded, they are of uttermost importance as they need to be interoperable with the lower level which greatly influences the possible choices of the component model and in fine the design choices. The integration requirements have in many cases led to a decision to use component technologies which are not appropriate for embedded systems but provide better integration possibilities. Depending on the level, the nature of the requirements and the implementation will be quite different. In general, the lower the level, the stronger are the real-time requirements (including timing predictability) and the resource limitations. Also, the component based approach will include different concepts at different levels. The most important quality attributes in industrial automation, following the researchers, is:

(i) lowest levels:

a - Availability,

b - Timeliness,

c - Reliability

(ii) higher levels:

a - Performance,

b - Usability, and

c - Integrability.

2.2.2 Automotive

To provide a domain specific classification of the importance of quality attributes for software in vehicles, and discusses how the attributes could be facilitated by a component technology. Åkerholm (Åkerholm et al., 2005) executed a research in main vehicle industries. The research method is divided into three ordered steps:

1 - During the first step a list of relevant quality attributes was gathered;

2 - In the next step technical representatives from a number of vehicular companies placed priorities on each of the attributes in the list reflecting their companies view respectively;

3 - Finally a synthesis step was performed, resulting in a description of the desired quality attribute support in a component technology for vehicular systems.

The list of quality attributes have been collected from different literature trying to cover qualities of software that interest vehicular manufactures. In order to reduce a rather long list, attributes with clear similarities in their definitions have been grouped in more generic types of properties, e.g., portability and scalability are considered covered by maintainability. Although such grouping could fade the specific characteristics of a particular attribute, it put focus on the main concerns. In the ISO/IEC 9126 standard (ISO/IEC 9126, 2001), 6 quality attributes (functionality, reliability, usability, efficiency, maintainability, and portability) are defined for evaluation of software quality. However, the standard has not been adopted fully in this research; it is considered too brief and does not cover attributes important for embedded systems (e.g., safety, and predictability). Furthermore, concepts that sometimes are mixed with quality attributes (for example fault tolerance) are not classified as quality attributes, rather as methods to achieve qualities (as for example safety). Finally, functionality is of course one of the most important quality attributes of a product, indicating how well it satisfies stated or implied needs. However, it focuses on quality attributes beyond functionality often called extra-functional or non-functional properties. The resulting list of quality attributes is presented below. Having been presented with the basic characteristics of quality attributes related to component technologies tailored for vehicular systems below:

1. Safety

2. Reliability

3. Predictability

4. Usability

5. Extendibility

6. Maintainability

7. Efficiency

8. Testability

9. Security

10. Flexibility

Representatives from the technical staff of several companies have been requested to prioritize a list of quality attributes, reflecting each of the respective companies’ view. The attributes have been grouped by the company representatives in four priority classes as shown in Table 2.1. The nature of the quality attributes imply that no quality attributes can be neglected. It is essential to notice that placing an attribute in the lowest priority class (4) does not mean that the company could avoid that quality in their software, rather that the company does not spend extra efforts in reaching it. The following companies have been involved in the classification process:

· Volvo Construction Equipment

· Volvo Cars

· Bombardier Transportation

· Scania

· ABB Robotics

Table 2.2.2. Priority classes used to classify the importance of the different quality attributes.

Priority

Description

1

very important, must be considered

2

important, something that one should try to consider

3

less important, considered if it can be achieved with a small effort

4

Unimportant, do not spend extra effort on this

As the last step the authors provide a discussion where we have combined the collected data from the companies with the classification of how to support different quality attributes in (Larsson, 2004). The combination gives an abstract description of where, which, and how different quality attributes should be supported by a component technology tailored for usage in the vehicular industry.

Figure 2.3. The results. Y-axis: priority of quality attributes in a scale 1 (highest), to 4 (lowest). X-axis: the attributes, with the highest prioritized attribute as the leftmost, and lowest as rightmost. Each of the companies has one bar for each attribute, textured as indicated below the X-axis.

2.2.3 Medical

Wijnstra (Wijnstra et al., 2001) described their experience with quality attributes and aspects in the development of a medical imaging product family. To meet such requirements for a range of medical imaging products, a product family has been defined. Wijnstra distinguished between quality attributes and aspects as views. A quality attribute of an artifact is an observable property of the artifact. A quality attribute can be observable at development-time or at run-time.

Following Wijnstra, quality attributes and aspects are used to add structure to the various phases of the development process. They form a supporting means for achieving completeness, i.e. have all relevant concerns been taken into account? In a product family context where the family members are constructed from a component-based platform, it is especially useful to achieve aspect-completeness of components, allowing system composition without worrying about individual aspects.

Software architecture is defined as “the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships among them”. The architecture must accommodate the quality attributes as imposed on the system, which can be handled via structures in the high-level architecture, aspects, or rules & guidelines.

The most characteristic and important quality attributes found in Wijnstra’s research for the medical imaging product family is given below:

1. Reliability

2. Safety

3. Functionality

4. Portability

5. Modifiability

a. Configurability

b. Extensibility and Evolvability

6. Testability

7. Serviceability

2.2.4 Consumer Electronics

Consumer electronics products, such as TV, VCR, and DVD, are developed and delivered in form of product families which are characterized by many similarities and few differences and in form of product populations which are sets of products with many similarities but also many differences. Production is organized into product lines -this allows many variations on a central product definition.

A product line is a top-down, planned, proactive approach to achieve reuse of software within a family or population of products. It is based on use of a common architecture and core functions included into the product platform and basic components. The diversity of products is achieved by inclusion of different components. Because of the requirements for low hardware and production costs, general-purpose component technologies have not been used, but rather more dedicated and simpler propriety models have been developed. An example of such a component model is the Koala component model used at Philips (Ommering et al., 2000), (Ommering, 2002). Koala is a component model and an architectural description language to build a large diversity of products from a repository of components. Koala is designed to build consumer products such as televisions, video recorders, CD and DVD players and recorders, and combinations of them.

2.2.5 Other domains

There are many domains in which embedded systems are used extensively. Some of them are: Telecommunication, avionics and aerospace, transportation, computer games, home electronics, navigation systems, etc. While there are many similarities between these domains there are also very different requirements for their functional and extra-functional properties. The consequences are that the requirements for component-based technologies are different, and consequently it expects to have one component model that summarizes all quality attributes in common in different domains. The expectations are that only one embedded quality component models will exist, some common characteristics, such as basic principles of component specifications through interfaces, basic composition and run-time services, certain patterns, and similar.

2.3.1 The needs and priorities in research

The component-based approach on system level, where hardware components are designed with embedded software, has been successfully used for many years. Also large-grain generic components like protocol stacks, RTOSs, etc. have been used for a long time. In addition to this, technology supporting a component-based approach has been developed; either in the form of proprietary component models, or by using reduced versions of some widely used component models. Still, there are needs for a number of improvements. Some of them are the following (differently important for different domains):

- There is a lack of widely adopted component technology standards which are suitable for embedded systems. For smaller-size embedded systems, it is important that a system composed of components can be optimized for speed and memory consumption, which is still missing in most of the component technologies available today.

- There is also a need for interoperability between different component technologies. Specification technology is not sufficiently developed to guarantee a priori component interoperability.

- Most current component technologies do not support important extra-functional properties

- There is a need for generic platform services, for, e.g., security and availability.

- Tools that support component based development are still lacking

- There is a lack of efficient implementations of component frameworks (i.e., middleware), which have low requirements on memory and processing power.

Major needs for the further development of component technology for embedded systems are the following (Brinksma et al., 2001).

- Need for adopted component models and frameworks for embedded systems. A problem is that many application domains have application-dependent requirements on such a technology.

- Need for light-weight implementations of component frameworks. In order to support more advanced features in component-based systems, the run-time platform must provide certain services, which however must use only limited resources.

- Obtaining extra-functional properties of components: Timing and performance properties are usually obtained from components by measurement, usually by means of simulation. Problems with this approach are that the results depend crucially on the environment (model) used for the measurements may not be valid in other environments, and that the results may depend on factors which cannot easily be controlled. Techniques should be developed for overcoming these problems, thereby obtaining more reliable specifications of component properties.

- Platform and vendor independence: Many current component technologies are rather tightly bound to a particular platform (either run-time platform or design platform). This means that components only make sense in the context of a particular platform.

- Efforts to predict system properties: The analysis of many global properties from component properties is hindered by inherent complexity issues. Efforts should be directed to finding techniques for coping with this complexity.

- Component certification: In order to transfer components across organizations, techniques and procedures should be developed for ensuring the trustworthiness of components.

- Component noninterference: Particularly in safety critical applications, there is a need to ensure separation and protection between component implementations, in terms of memory protection, resource usage, etc.

- Tool support: The adoption of component technology depends on the development of tool support. The clearly identified priorities of CBSE for embedded systems are:

- Predicting system properties. A research challenge today is to predict system properties from the component properties. This is interesting for system integration, to achieve predictability, etc.

- Development of widely adopted component models for real-time systems. Such a model should be supported by technology for generating necessary runtime infrastructure (which must be light-weight), generation of monitors to check conformance with contracts, etc.

3.1.1 Embedded Software Component Quality and Certification: A Survey

This Chapter presents a survey of cutting edge embedded software component quality and certification research, in an attempt to analyze the trends in CBSE/CBD applied embedded systems projects and to probe some of the component quality and certification research directions.

Some relevant research works explore the theory of component certification in academic scenarios, but the literature is not so rich in reports related to practical embedded software component certification experience. In this way, this chapter presents a survey of embedded software component quality and certification research, from the early 90s until today. Another relevant aspect observed is that in the pioneering works the focus was mainly on mathematical and test-based models while more recently researchers have focused on techniques and models based on predicting quality requirements.

Works related to certification process in order to evaluate embedded software component quality are surveyed, however the literature contains several works related to software component quality achievement, such as: component and contracts (Urting et al., 2001), Feature Modeling of CBD embedded (Kalaoja, 1997), Security as a new dimension in embedded (Kocher, 2004), Quality attributes of a Medical Family (Wijnstra, 2001), Usage-based software certification process (Voas, 2000), component testing (Councill, 1999), (Beydeda & Gruhn, 2003), component verification (Wallin, 2002), (Reussner, 2003). Since the focus of this survey is on processes for assuring component quality, it does not cover these works, which deal only with isolated aspects of component quality.

The first research published about embedded software certification system was focused on mathematical and test based models. In 1993, Poore (Poore et al., 1993) developed an approach based on the usage of three mathematical models (sampling, component and certification models), using test cases to report the failures of a system later analyzed in order to achieve a reliability index. Poore et al. were concerned with estimating the reliability of a complete system, and not just the reliability of individual software units, although, they did consider how each component affected the system’s reliability.

One year later, in 1994, Wohlin et al. (Wohlin et al., 1994) presented the first method of component certification using modeling techniques, making it possible not only to certify components but to certify the system containing the components as well. The method is composed of the usage model and the usage profile. The usage model is a structural model of the external view of the components, complemented with a usage profile, which describes the actual probabilities of different events that are added to the model. The failure statistics from the usage test form the input of a certification model, which makes it possible to certify a specific reliability level with a given degree of confidence. An interesting point of this approach is that the usage and profile models can be reused in subsequent certifications, with some adjustments that may be needed according to each new situation. However, even reusing those models, the considerable amount of effort and time that is needed makes the certification process a hard task.

One different work that can be cited was published in 1994. Merrit (Merrit, 1994) presented an interesting suggestion: the use of components certification levels. These levels depend on the nature, frequency, reuse and importance of the component in a particular context, as follows:

• Level 1: A component is described with keywords and a summary and is stored for automatic search. No tests are performed; the degree of completeness is unknown;

• Level 2: A source code component must be compiled and metrics are determined;

• Level 3: Testing, test data, and test results are added; and

• Level 4: A reuse manual is added.

Although simple, these levels represent an initial component maturity model. To reach the next level, the component efficiency and documentation should be improved. The closer to level four, the higher is the probability that the component is trustworthy and may be easily reused. Moreover, Merrit begins to consider other important characteristics related to component certification, such as attaching some additional information to components, in order to facilitate their recovery, defining metrics to assure the quality of the components, and providing a component reutilization manual in order to help its reuse in other contexts. However, this is just a suggestion of certification levels and no practical work was actually done to evaluate it.

Subsequently, in 1996, Rohde et al. (Rohde et al., 1996) provided a synopsis of in-progress research and development in reuse and certification of embedded software components at Rome Laboratory of the US Air Force, where a Certification Framework (CF) for software components was being developed. The purpose of the CF was: to define the elements of the reuse context that are important to certification; to define the underlying models and methods of certification; and, finally, to define a decision-support technique to construct a context-sensitive process for selecting and applying the techniques and tools to certify components. Additionally, Rohde et al. developed a Cost/Benefit plan that describes a systematic approach of evaluating the costs and benefits of applying certification technology within a reuse program.

After analyzing this certification process, Rohde et al. found some points that require improved formulation in order to increase the certification quality, such as the techniques to find errors (i.e. the major errors are more likely to be semantic, not locally visible, rather than syntactic, which this process was looking for) and thus the automatic tools that implements such techniques. In summary, Rohde et al. considered only the test techniques to obtain the defects result in order to certify software components. This is only one of the important techniques that should be applied to component certification.

Two years later, Voas (Voas, 1998) defined a certification methodology using automated technologies, such as black-box testing and fault injection to determine if a component fits into a specific scenario.

This methodology uses three quality assessment techniques to determine the suitability of a candidate COTS component;

(i) Black-box component testing is used to determine whether the component quality is high enough;

(ii) System-level fault injection is used to determine how well a system will tolerate a faulty component;

(iii) Operational system testing is used to determine how well the system will tolerate a properly functioning component, since even these components can create system wide problems.ww.trusted-components.org

The methodology can help developers to decide whether a component is right for their system or not, showing how much of someone else’s mistakes the components can tolerate.

According to Voas, this approach is not foolproof and perhaps not well suited to all situations. For example, the methodology does not certify that a component can be used in all systems. In other words, Voas focused his approach in certifying a certain component within a specific system and environment, performing several types of tests according to the three techniques that were cited above.

Another work involving component test may be seen in (Wohlin and Regnell, 1998), where Wohlin and Regnell extended their previous research (Wohlin et al., 1994), now, focusing on techniques for certifying both components and systems. Thus, the certification process includes two major activities:

(i) usage specification (consisting of a usage model and profiles), and

(ii) certification procedure, using a reliability model.

The main contribution of that work is the division of components into classes for certification and the identification of three different ways of certifying software systems:

(i) Certification process, in which the functional requirements implemented in the component are validated during usage-based testing in the same way as in any other testing technique;

(ii) Reliability certification of component and systems, in which the component models that were built are revised and integrated to certify the system that they form; and,

(iii) Certify or derive system reliability, where the focus is on reusing the models that were built to certify new components or systems.

In this way, Wohlin and Regnell provided some methods and guidelines for suitable directions to support software component certification. However, the proposed methods are theoretical without experimental study. According to Wohlin et al., “both experiments in a laboratory environment and industrial case studies are needed to facilitate the understanding of component reliability, its relationship to system reliability and to validate the methods that were used only in laboratory case studies” (pp. 09). Until now, no progress in those directions was achieved.

Another relevant work was presented in 2000 by Jahnke, Niere and Wadsack (Jahnke, Niere and Wadsack, 2000). They developed a methodology for semi-automatic analysis of embedded software component quality. This approach evaluates data memory (RAM) utilization by the component. This approach is applied in Java technology development components. Java is a multiplatform language, so components developed from a project in a determinate platform could be easily ported to another platform. This technology employs intensive use of RAM, additionally a virtual machine to interpret the code is necessary. However, embedded systems usually have a small RAM capacity.

The work is restricted because it verifies the component quality from only one point of view, use of data memory, among many other characteristics to be considered. Furthermore, the language most commonly used for embedded development is C and C + +, Java is widely used for the development of desktops systems.

In 2001, Stafford et al. (Stafford et al., 2001) developed a model for the component marketplaces that supports prediction of system properties prior to component selection. The model is concerned with the question of verifying functional and quality-related values associated with a component. This work introduced notable changes in this area, since it presents a CBD process with support for component certification according to the credentials, provided by the component developer. Such credentials are associated to arbitrary properties and property values with components, using a specific notation such as . Through credentials, the developer chooses the best components to use in the application development based on the “credibility” level.

Stafford et al. also introduced the notion of active component dossier, in which the component developer packs a component along with everything needed for the component to be used in an assembly. A dossier is an abstract component that defines certain credentials, and provides benchmarking mechanisms that, given a component, will fill in the values of these credentials. Stafford et al. finalized their work with some open questions, such as: how to certify measurement techniques? What level of trust is required under different circumstances? Are there other mechanisms that might be used to support trust? If so, are there different levels of trust associated with them and can knowledge of these differences be used to direct the usage of different mechanisms under different conditions?

Besides these questions, there are others that must be answered before a component certification process is achieved, some of these apparently as simple