um sistema fuzzy de suporte à decisão para meta-avaliação ... · estatística da fundação...

Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v. 15, n. 56, p. 447-462, jul./set. 2007

Um sistema fuzzy de suporte à decisão

para meta-avaliação uma nova

abordagem e um estudo de caso

desenvolvidos no Brasil*

Informes e Participações

Ana Carolina Letichevsky

Dra. em Engenharia Elétrica,PUC-Rio

Profª. do Departamento deInformática, PUC-Rio

Estatística da Fundação Cesgranrio

[email protected]

Marley Maria Bernardes

Rebuzzi Vellasco

Ph.D, University of London,Grã-Bretanha

Profª do Departamento de

Engenharia Elétrica, PUC-Rio

[email protected]

Ricardo Tanscheit

Ph.D., University of London,Grã-Bretanha.

Prof. do Departamento de

Engenharia Elétrica, [email protected]

ResumoEste artigo apresenta uma nova metodologiapara meta-avaliação que utiliza conjuntos fuzzye lógica fuzzy. Esta metodologia é composta porum instrumento de coleta de dados e por umsistema hierárquico de inferência fuzzy. As van-tagens da metodologia proposta são: (i) o ins-trumento de coleta de dados,que permite respostas interme-diárias; (ii) a capacidade dosistema de inferência de seadaptar a necessidades espe-cíficas; e (iii) transparência,através da utilização de regraslingüísticas que facilitam tan-to o entendimento como adiscussão de todo o proces-so. As regras foram construí-das com base nas diretrizesestabelecidas pelo Joint Com-mittee on Standards for Edu-cational Evaluation(1994) epor informações fornecidaspor especialistas. O sistemapode auxiliar avaliadores queainda não têm experiência emmeta-avaliação. Um estudode caso é apresentado comovalidação da metodologiaproposta.Palavras-chave: Meta-avaliação. Lógicafuzzy. Avaliação de programas educa-cionais. Padrões de avaliação.

Abstract

A fuzzy decision support

system for meta-evaluation

a new approach and a

case study performed in

BrazilThis paper presents a newmethodology for meta-evaluation that makes useof fuzzy sets and fuzzylogic. It is composed of adata collection instrumentand of a hierarchical fuzzyinference system. Theadvantages of theproposed methodologyare: (i) the instrument,which allows intermediateanswers; (ii) the inferenceprocess ability to adapt tospecific needs; and (iii)transparency, through theuse of linguistic rules thatfacilitate both theunderstanding and thediscussion of the wholeprocess. The rules are

based on guidelines established by theJoint Committee on Standards forEducational Evaluation (1994) and also

*A revised and improved version of the paper presented at the 2006 American Evaluation Association Conference “TheConsequences of Evaluation”, Portland, Oregon, United States

448 Ana Carolina Letichevsky, Marley Maria Bernardes Rebuzzi Vellasco e Ricardo Tanscheit


represent the view of experts. The systemcan provide support to evaluators thatmay lack experience in meta-evaluation.A case study is presented as a validationof the proposed methodology.Keywords: Meta-evaluation. Fuzzy logic.Evaluation of educational programs.Evaluation standards.

ResumenUn sistema de suporte ala decisión para meta-evaluación y susconsecuencias: nuevaabordaje y estudio decaso realizado en BrasilEste artículo presenta una nuevametodología para meta-evaluaciónque utiliza conjuntos fuzzy e lógicafuzzy. Esta metodología es compuestapor un instrumento de coleta de dadosy por uno sistema de inferencia fuzzy.Las ventajas de la metodologíapropuesta son: (i) el instrumento decoleta de dados, que permiterespuestas intermediarias; (ii) lacapacidad del sistema de inferenciade adaptarse a necesidadesespecíficas; (iii) transparencia, através de la utilización de reglaslingüísticas que facilitan tanto elentendimiento como la discusión detodo el proceso. Las reglas fueronconstruidas con base en las directricesestablecidas por el Joint Committee onStandards for EducationalEvaluation(1994) y por informacionesfornecidas por especialistas. El sistemapuede auxiliar evaluadores que aúnno tienen experiencia en meta-

evaluación. Un estudio de caso espresentado comovalidación de laMetodología propuesta.Palabras clave: Meta-evaluación.Lógica fuzzy. Evaluación de programaseducacionales. Padrones de evaluación.

IntroductionAssuring the quality of an evaluation is agreat challenge to evaluators. Meta-evaluation (SCRIVEN, 1991) is themechanism used nowadays to face thischallenge. The main focus of thediscussions about meta-evaluation is theexcellence criterion for an evaluation. Thisis only the starting point to obtain aquality meta-evaluation; it is necessary togo beyond and find new methodologiesthat confer more flexibility to the executionof evaluative processes and that supply aprecise and timely answer.

Meta-evaluation can be carried out indifferent ways, but frequently checklists(STUFFLEBEAM, 2001) are used as aninstrument to collect data. Checklists areinstruments with items or assertions abouta specific focus with options of closedanswers. In case of a meta-evaluationprocess, those assertions must investigatethe presence of each one of thestandards of a true evaluation in thetarget evaluation process. Traditionally,data are collected and treated based onclassic logic, where the frontier between apoint of the scale and another one isalways clear. However, it is not easy for ahuman being to precisely define thisfrontier, since it may be of a fuzzy nature.In Fuzzy Set Theory, one given elementcan belong to more than one set withdifferent grades of membership(TANSCHEIT, 2004).

Um sistema fuzzy de suporte à decisão para meta-avaliação uma

nova abordagem e um estudo de caso desenvolvidos no brasil 449


The same difficulty that exists in the stage ofdata collection is also faced in thetreatment of information, since, in order tomake an evaluation, excellence criteriamust be established. These serve toelaborate a value judgment and canconstitute rule bases, generally supplied byexperts, that are used to verify whether ornot the result meets a certain criterion. Thisstudy presents a methodology developed inBrazil for meta-evaluation that makes useof the concepts of fuzzy sets and fuzzylogic. This allows for the use ofintermediate answers in the process of datacollection. In other words, instead ofdealing with crisp answers(“accomplished” or “not accomplished”,for example), it is possible to indicate thatan excellence criterion was partiallyaccomplished in different levels. Theanswers of this instrument are treatedthrough the use of a Mamdani-typeinference system (MAMDANI; ASSILIAN,1975), so that the result of the meta-evaluation is eventually obtained.

Meta-evaluationThe ConceptWhen an evaluative process is designed,several aspects, such as evaluativequestions, methods and techniques ofdata collection, identification of therespondents, should, from the beginning,be negotiated with whomever is in chargeof the evaluation and with therepresentatives of those being evaluated.Evaluators and clients must be aware ofthe bias in the evaluative process, andseek to minimize it whenever possible;when this is not possible, they must reportit. After all, meta-evaluations are carriedout so that the bias is minimized and thequality of an evaluative process in all its

stages is ensured. This includes decisionsconcerning the execution of theevaluation, the definition of its purpose,design, information collection andanalysis, elaboration of budget andcontract, management, setting up of theteam, among others.

In the same fashion that it isrecommended that evaluations be carriedout in the formative and summativeperspectives (SCRIVEN, 1967), meta-evaluations should also be carried outhaving in view those two perspectives,which up to a certain point complementone another. The formative meta-evaluation is conducted along theevaluative process to improve on theevaluation. Ideally, its starting pointshould coincide with that of theevaluation. The main objective is toprovide the team responsible for carryingout the evaluative process with usefulinformation, in order to improve theprocess while it is still in progress. Thesummative meta-evaluation is carried outat the end of the evaluation, in search ofconclusive answers about its merit andrelevance to those who ordered it, as wellas to users and others interested in theprocess. The aim is to give credibility tothe evaluation and to the final resultsgenerated by it. In other words, while therole of the formative meta-evaluation is toimprove the evaluative process throughoutits development, the role of the summativeevaluation is to give an account to thoseinvolved and to the community at large,and to contribute to the improvement offuture processes.

The concern with the creation ofstandards (principles obtained through



consensus among the people involved inthe practice of evaluation, which, ifachieved, will guarantee the quality of anevaluation) that the evaluative processshould follow is an old one and,perhaps, as old as the concern withevaluation itself. This is a hard task, notonly on account of the technical difficultyinherent to it, but also, above all, as aresult of the difficulty to sensitize,mobilize, and reach consensus amongdifferent pertinent people, so as toproduce a technically good work,accepted by those who carry out theevaluation, those who are evaluated,and those who make use of it(LETICHEVSKY et al., 2007).

To discuss procedures of meta-evaluationis to discuss the quality of the evaluativeprocess. Therefore, it is fundamental toconsider also the standards that anevaluator should follow, in the light ofpre-established criteria of an evaluation ofquality.

Practical aspectsIt is necessary to ensure that instrumentsfor data collection are adequate forobtaining the data one really intends tocollect. The validation of the instrumentsfor data collection can be made indifferent ways. In the case of instrumentsthat collect qualitative information, suchvalidation can be made with theassistance of specialists in the area orthrough a comparison among differentevaluators, techniques, and instruments.In the case of quantitative information,the use of a Confirmatory Factor Analysisis recommended (BOLLEN, 1998). This isa technique of reduction of datadimension just like Exploratory Factor

Analysis, which is better known and morefrequently used. The fundamentaldifference is that Confirmatory FactorAnalysis is carried out from theapplication of a model of structuralequation and, therefore, a theoreticalmodel is assumed beforehand relatinglatent variables (not observable) to theobservable variables. In the ExploratoryFactor Analysis, on the other hand, eachand every latent variable may have aninfluence on the observable variables,since the number and nature of the latentfactors before processing the analysis isunknown. It is precisely this differencethat makes the first one confirmatory andthe second one eminently exploratory.Another important difference is that in theConfirmatory Factor Analysis the errorsare also modelled and may (or may not)be correlated, whereas in FactorExploratory Analysis it is assumed that theerrors may not be correlated (which is notalways true). In the Confirmatory FactorAnalysis, the model previouslyestablished is adjusted for the purpose ofminimizing calculated residues throughthe difference between the variance andcovariance matrixes observed andcalculated. The Factor ExploratoryAnalysis is vastly employed, without anytype of test for checking whether theerrors are not, in fact, correlated. Ideally,different types of instruments for datacollection should be used, consideringboth quantitative and qualitativeinformation.

As to the quality of information, there aretwo aspects that must be observed: (i) thequality and adequacy of the sources ofinformation and (ii) the adequatetreatment of databases. When choosing




the sources of information, it is importantto ensure (whenever applicable) that alldifferent groups of possible informantsabout the evaluative focus are considered.On the other hand, after collecting dataand before calculating the indicators, it isfundamental to remove from thedatabases information that does not reflectthe latent trace one wants to measure.Thus, in the case of instruments thatcollect quantitative data, those thatpresent responses with always the samepattern, no responses, or indication ofobjection, must be excluded, as well asany other instruments that when filled indo not reflect a useful informationregarding to what one wants to measure(BRYK; RAUDENBUSH, 1992). When theinformation is a qualitative one, thisproblem may be avoided through datatriangulation (FRANSES; GELUK;HOMELEN, 1999).

When choosing the most adequatetechnique for modelling and dataanalysis, it is important to be clear aboutwhich evaluative question or questionsone intends to answer, since an adequatetechnique to search for the answer to agiven question may not be adequate foranother one (LETICHEVSKY, 2004). Forexample, if an evaluation of studentperformance is carried out with the aimof determining in which schools the beststudents are, it is possible to work directlywith the students’ scores (results). On theother hand, if one’s intention is toidentify which are the most efficientschools, it is necessary to consider thatstudents present some differences andbring diversified life experiences, bothwith regard to formal education and togeneral knowledge (GOLDSTEIN, 1995).

Socioeconomic levels of families vary,and so does the previous knowledge ofstudents (FLETCHER, 1997). Thus,students with a higher socioeconomiclevel, or students with a broader scope ofpre-existing knowledge, would tend todisplay a better performance. Interactionsbetween the student and the schoolenvironment also interfere with hisperformance and should also beincorporated into the models (YANG etal., 1999a). It is in this context that thereappears the need to isolate the effectsthat do not depend on school, that is,those that are not, and will never be,under the control of administrators,teachers, pedagogical and support team.In this sense, one generally intends toisolate in particular the effects of thesocioeconomic level of students and theschools which, somehow, have impacton their performance. Thus, what onewants to measure is the value added bythe school (YANG et al., 1999b).Traditional methods of study of causeand effect relationship involve models ofregression made at a single level, whereone dependent variable is explained by aset of independent variables plus oneerror. In this specific case, the dependentvariable is the student proficiency(estimated by his performance in contenttests). However, is it possible to acceptthe validity of these models ignoring therelations between different hierarchicallevels and the way these relations impacton the results of the study? The intuitiveanswer is no, and statistical studiesconfirm this. When one analysesmultilevel questions through models of asingle level, errors may possibly becommitted, and, therefore, multilevelmodels must be used (RASBASH, 1999).



Similarly to what happens with schools,the same care must be taken in theevaluation of the performance of sectionsor boards of directors within a businesscompany or the impact achieved by thebeneficiaries of social programs withsimilar objectives (LETICHEVSKY, 2004).

A meta-evaluation can be carried out inseveral ways, through the use ofdifferent instruments, but the use ofchecklists has been the procedure mostadopted by many evaluators andevaluation centers, generatingsatisfactory results (PENNA FIRME;LETICHEVSKY, 2002). Checklists are“lists of things to be checked or done”.In practice, when a checklist istransformed into an instrument for datacollection, a set of instruments is createdwhere each item is a statement and therespondent must simply indicate whetherthe statement is true or not. In thespecific case of meta-evaluation, suchstatement tries to investigate thepresence of the patterns adopted in theevaluative process that is the focus ofmeta-evaluation. Checklists represent anefficient instrument, in a friendly format,for sharing lessons learned in practice(STUFFLEBEAM, 1994).

Fuzzy Logic: basic conceptsConcepts of Fuzzy Set Theory and ofFuzzy Logic can be used to translate, inmathematical terms, the impreciseinformation expressed by a set oflinguistic IF-THEN rules. If a humanbeing is capable of articulating itsreasoning as a set of IF-THEN rules, it ispossible to create an inference system,the algorithm which may beimplemented through a computer

program, where Fuzzy Set Theory andFuzzy Logic provide the mathematicaltools for dealing with such linguisticrules (TANSCHEIT, 2004).

Fuzzy Logic studies the formal principles ofapproximate reasoning and is based onFuzzy Set Theory. Fuzzy Logic deals withintrinsic imprecision, associated with thedescription of the properties of aphenomenon, and not with theimprecision associated with themeasurement of the phenomenon itself.

In ordinary sets theory, the concept ofmembership of an element to a set is veryprecise: either the element belongs or doesnot belong to a given set. Given a set A ina universe X, membership of an element xof X to the set A is expressed by thecharacteristic function ƒ

A:

1 if x∈AƒA(x)=

0 if x∉A

L.A. Zadeh generalized the characteristicfunction so that it could assume an infinitenumber of values in the interval [0, 1].Given a fuzzy set A, in a universe X,membership is expressed by the functionμ

A(x): X→[0,1]. A is now represented by a

set of ordered pairs, A={μA(x)/x} x∈X,

where μA(x) indicates to what extent x is

compatible with the set A.

The support set of A is the set of elementsin the universe X for which μ

A(x)>0. Thus,

a fuzzy set may be seen as the mapping ofthe support set in the interval [0, 1].

A linguistic variable is a variablewhose values are names of fuzzy sets.For example, the answer to an item ofa certain instrument of data collection




may be a linguistic variable assumingthe values “poor”, “good”, and“excellent”. These values are describedby means of fuzzy sets, defined bymembership functions.

Consider the linguistic variable scale (inan instrument for data collection), withvalues poor, good and excellent, definedby the membership functions shown inFigure 1. Answers up to 2.5 have amembership grade equal to 1 in the poorset; the membership grade in this setdecreases as the response increases. Ananswer of 5 is considered “totallycompatible” with the good set, whereasresponses above 5 present a membershipgrade different from zero in excellent.

Figure 1 - Example of membership functions.

Membership functions may havedifferent shapes, depending on theconcept that one wishes to representand the context in which they will beused. Context is extremely important,since the concepts of poor, good andexcellent, for example, are extremelysubjective. Membership functions maybe defined by the user, but they areusually of a standard form (triangular orgaussian, for example). In practiceshapes can be adjusted, in accordancewith the results, in a trial-and-errorprocedure.

In fuzzy logic, a conditional statement (IFx is A THEN y is B) is expressedmathematically by a membership function,which denotes the degree of truth of theimplication A→B.As to inference, the modus ponens ofpropositional logic (premise 1: x is A;premise 2: IF x is A THEN y is B;consequence: y is B) is extended tothe generalized modus ponens,described as:Premise 1: x is A*.Premise 2: IF x is A THEN y is B.Consequence: y is B*

While in classical logic the rule generatesa consequence only if premise 1 is theexact antecedent of the rule (and the resultis exactly the consequent of that rule), infuzzy logic a rule is activated if there is adegree of similarity different from zerobetween premise 1 and the antecedent ofthe rule. The result will be a consequentwith a degree of similarity to theconsequent of the rule.

A Fuzzy Inference System, shown in Figure2, does the mapping from precise (crisp)inputs to a crisp output. The crisp inputsmay be measurement or observation data,which is the case of the large majority ofpractical applications. These inputs arefuzzified (mapped to fuzzy sets), which canbe viewed as the activation of relevantrules for a given situation. Once theoutput fuzzy set is computed through theprocess of inference, a defuzzification isperformed, since in practicalapplications crisp outputs are generallyrequired. The rules are linguistic IF-THEN statements sentences andconstitute a key aspect in theperformance of a fuzzy inference system.



Figure 2 – Fuzzy Inference System

The methodologyThe methodology adopted in this work(LETICHEVSKY et al., 2007) is composedof an instrument for data collection(Checklist for the Meta-evaluation ofPrograms/Projects) and of a fuzzyinference system to treat data related to

the meta-evaluation of projects andprograms (Figure 3). The instrument fordata collection was constructed from theadaptation of the checklist for meta-evaluation developed by the EvaluationCenter of the Western Michigan University,whereas the fuzzy inference system wasbased on the thirty standards developedby the Joint Committee on Standards forEducational Evaluation.

Figure 3 –Methodology for Meta-Evaluation.

Instrumentfor Data

Collection

FuzzyInferenceSystem

Meta -Evaluation

Results




Due to the complexity of the problem, thefuzzy inference system was subdivided intothirty-six rule bases, organized into ahierarchical structure composed of threelevels (Figure 4):• Level 1: Standards rule bases

• Level 2: Categories rule bases• Level 3: Meta-evaluation rule basesThe hierarchical inference system, with theproposed three levels, is shown in Figure4. The whole system was implemented byusing the MatLab© Fuzzy Toolbox.

Figure 4 – Inference System Rule bases.



First, the standards rule bases (level 1) werebuilt. Since each standard is evaluated onthe basis of six criteria, the rules at this levelhave at most six antecedents. Each criterionhas three linguistic values (insufficient,satisfactory and excellent) associated with it.The membership functions of the 3 fuzzy setsare shown in Figure 5.

Figure 5 – Linguistic values for level 1input variables.

To the output variables of this level thevalues insufficient, satisfactory and excellentare also associated, as shown in Figure 6.Thirty standards rules were developed.

Figure 6 – Linguistic values for level 1output variables.

As an example of linguistic rules of thislevel, consider the U2 rule base, whichrefers to the credibility of the evaluatorand aims to measure to what extentpeople conducting the evaluation areboth trustworthy and competent to performit. The following linguistic variables wereconsidered in the rules antecedents:- U21: competent evaluators- U22: trustworthy evaluators- U23: evaluators address stakeholders’concerns;

- U24: evaluators are responsive to issuesof differences;- U25: evaluators help stakeholdersunderstand and assess the evaluationplan and process;- U26: evaluators pay attention tostakeholders’ criticisms and suggestions.Some of the linguistic rules generated forthis rule base are:If U22 is insufficient then U2 is insufficientIf U21 is satisfactory and U22 issatisfactory and U23 is excellent and U24is insufficient and U25 is satisfactory andU26 is insufficient then U2 is insufficient.

In the rule bases of level 2, the numberof antecedents varies in accordance withthe number of standards present in eachcategory, that is: category Utility hasseven; category Feasibility has three;category Propriety has eight; andcategory Accuracy has twelve. In the caseof the category Accuracy, the largenumber of input variables wouldjeopardize the development andunderstanding of linguistic rules.Therefore, the solution was to create tworules bases for accuracy: one with thestandards that are directly related toinformation and its quality (Accuracy Irule bases) and another with standardsthat refer to the analysis and disclosureof information (Accuracy II rule bases).

The inputs to the inference systems oflevel 2 are the outputs from level 1(linguistic variables with three valueseach). In the rule bases of level 2, theoutputs are the linguistic variables thatrepresent the category and have fiveassociated values (insufficient, regular,satisfactory, good and excellent),specified by fuzzy sets defined by the




membership functions shown in Figure7. The category Feasibility, for example,considers the following standards:F1: practical proceduresF2: political viabilityF3: cost effectiveness.Examples of rules are:If F1 is insufficient and F2 is insufficientand F3 is insufficient then F is insufficientIf F1 is excellent and F2 is sufficient andF3 is excellent then F is good

Figure 7 - Linguistic values for level 2output variables.

Results by category are obtained at level 2and not only facilitate the elaboration ofrecommendations and adjustments butalso enable some categories to gothrough the process of meta-evaluation atdifferent moments, when the instrument isused in a formative character.

The meta-evaluation rule base isresponsible for the generation of the finalresult. Rules at this level have fiveantecedents (outputs from level 2). Theconsequent is a linguistic variable withfive values, as shown in Figure 8.Examples of rules at this level are:

If utility is excellent and feasibility is goodand propriety is excellent and accuracy Iis excellent and accuracy II is excellentthen meta-evaluation is excellent

If utility is insufficient than meta-evaluation is insufficient

Figure 8 – Linguistic values for level 3output variables.

When the instrument is totally filled out,the inference system computes thirty sixresults. Thus, besides calculating a resultfor each standard, the system alsocalculates one for utility, one for feasibility,one for propriety, two for accuracy, andthe general result of evaluation. Each oneof the standards results may beinsufficient, satisfactory or excellent andthe others: insufficient, regular,satisfactory, good or excellent.

A case studyA case study was developed, based on theevaluation of the initial stage of aneducational program aimed at a low-income population. The length of theprogram was of about six months, and itwas applied in ten towns in the North,Northeast, and Southeast of Brazil, allwith low HDI (HumanDevelopment Index). Steps taken for datacollection and validation of the proposedmethodology are shown in Figure 9.

The first step for data collection was theselection of respondents (meta-evaluators) (1). Five types of respondentswere selected:EPP: Evaluator who Participated in theProcess (1a), integrated by thoseevaluators who took part in the evaluativeprocess focus of this case study.ENPP Evaluator who did Not Participatein the Process (1b), consisting of



evaluators in general who did not knowthe evaluative process focus of the meta-evaluation and whose first contact with itwas through the document ‘Description ofthe Evaluative Process - Focus of theCase Study’.PPIP: Professionals who Participated inthe Implementation of the EvaluatedPrograms (1c), including those who areinterested in and are users of theevaluative process focus.META: Meta-evaluator who carriedout the meta-evaluation external to theevaluation focus of the case study (1d).STU: STUdent of evaluation who did notparticipate in the process (1e), integratedby students who did not know theevaluative process focus of the meta-evaluation and who had their first contactwith it through the same documenthanded to the ENPP evaluators.

The next step was to contact selectedrespondents (2) that could ask foradditional explanation (3) on theevaluative process focus. After the datacollection instrument was given back (4),the fuzzy inference system (5) was fed sothat the initial processing (6) of datacould be carried out. Based on the fuzzyinference system outputs and on theanswers of the other instruments,preliminary results were discussed (7), theresults being considered in accordancewith the category of the respondent. Afterthat, adjustments were done to fuzzyinference system (8), especially incorrection and enlargement of rule bases.Afterwards, a new processing (9) wascarried out, and with the new results it waspossible to conclude a validation of theproposed methodology (10). Eventually,final results were discussed (11).

The analysis of results was based on thecomparison between the results generated bythe fuzzy inference system (starting with inputsfrom different evaluators) and the grades given.

Table 1 presents a summary of the resultsof the inference system and the gradesgiven to the meta-evaluation and to utility,feasibility, propriety, and accuracy,according to the type of respondent.

Figure 9 – Procedure to validate theproposed methodology.




Results provided by the inference systemand the grades given by the EPP, ENPPand the meta-evaluator are verycoherent. This coherence is justified bythe fact that the three groups arecomposed, in general, by evaluatorswho have a wide experience, and, inmany cases, with specific education inthe area. All human beings are able toevaluate, and generally they do it manytimes a day, but the actual evaluator isone who is capable of judging thevalue on the basis of excellence criteriaand previously established values. Thisis a competence achieved through thestudy of evaluation and thedevelopment of an evaluative culture.

In the case of PPIPs, the results are alsocoherent. The exception is the discrepancy, inpropriety, between the fuzzy inference system’sresult of 5.34 and the grade good (G).

In the STU group, results were incoherentwhen compared to the grades given.Students probably do not have muchexperience in the area, and are stillacquiring theoretical knowledge about theevaluation field. It is natural for them tohave some difficulties to perform theevaluation and to separate their personalopinions, based only on their ownstandards and values, from a valuejudgment done in the light of previouslyexpressed excellence criteria.

Table 1- Inference System Results and Grades* according to the type of respondent.



*I - insufficient; A - average; S - satisfactory; G - good; E – excellent.

(Continuação)

The harmony between the fuzzy inferencesystem’s results and the grades given by EPP,ENPP and meta-evaluators validate themethodology proposed here. It is important toemphasize that the coherence between resultswas confirmed by the feedback given by EPP,ENPP and META. Of 20 respondents, 16have given feedback. In general, theseevaluators presented their feedback accordingto the four categories of the Joint Committeeon Standards for Educational Evaluation.

ConclusionThis work presented a methodology formeta-evaluation based on fuzzy setsconcepts. This new methodology makesuse of a fuzzy inference system and

consists of 36 rule bases organized inthree levels: Standards (level 1), Category(level 2), and Meta-evaluation (level 3).The hierarchical structure and the rulebases were built in accordance with thestandards of a true evaluation proposedby the Joint Committee on Standards forEducational Evaluation.

In this methodology the inference systememploys linguistic rules provided by experts;this favors understanding and the update ofrules. It may incorporate contradictory rules,which is not possible when traditional logic isused, and can deal with intrinsic imprecisionthat exists in complex problems, as is the caseof meta-evaluation. The system was built on




the basis of evaluation standards of the JointCommittee on Standards for EducationalEvaluation; thus, it is able to reach a broadrange of users. It is expected that this work mayhelp evaluators, those who order evaluations,and those who make use of results.

As for future work, the methodology shallbe applied to performance, business and

institutional evaluations. It may also beadapted to other patterns, as, forexample, those suggested by theEuropean Society and by the JointCommittee on Standards for EducationalEvaluation. The use of a model ofStructural Equations and FactorialAnalysis may be used to aggregatedifferent meta-evaluators results.

ReferencesBOLLEN, K. A. Structural equations with latent variable. New York: John Wiley & Sons, 1998.

BRYK, A.; RAUDENBUSH, S. Hierarchical Linear Models. Newbury Park, CA: SagePublications, 1992.

FLETCHER, P. À procura do ensino eficaz: relatório de pesquisa. Brasília, DF: PNUD/MEC/SAEB, 1997.

FRANSES, P. H.; GELUK, I.; HOMELEN, V. P. Modeling item nonresponse inquestionnaires. Quality & Quantity, Netherlands, n. 33, p. 203-213, 1999.

GOLDSTEIN, H. Multilevel statistical models. London: Edward Arnold, 1995.

JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION. The ProgramEvaluation Standards. 2nd ed. Newbury Park, CA: Sage Publications, 1994.

LETICHEVSKY, A. C. La categoria precisión en la evaluación y en la meta evaluación:aspectos prácticos y teóricos. In: CONFERENCIA DE RELAC, 1., 2004, Lima. Trabajopresentado… Peru, Lima, 2004.

LETICHEVSKY, A. C. et al. A new approach to meta-evaluation Using Fuzzy Logic. In:NEDJAH, N. et al. (Ed.). Intelligent educational machines: methodologies andexperiences: systems engineering. [S.l.: s. n.]. 2007. Books in Series. In press.

MAMDANI, E. H.; ASSILIAN, S. An experiment in linguistic synthesis with a Fuzzy LogicController. International Journal of Man-Machine Studies, v. 7, n. 1, p. 1-13, 1975.

MENDEL, J. M. Fuzzy Logic Systems for Engineering: a tutorial. Proceedings of the IEEE,Raleigh, NC, v. 83, n. 3, p. 345-377, 1995.

MORTIMORE, P. The nature and findings of school effectiveness research in primarysector. In: RIDDELL, S.; BROWN, S. (Ed.). School effectiveness research: its messages forschool improvement. London: HMSO, 1991.



PENNA FIRME, T.; LETICHEVSKY, A. C. O desenvolvimento da capacidade deavaliação no século XXI: enfrentando o desafio através da meta-avaliação. Ensaio:avaliação e políticas públicas em educação, Rio de Janeiro, v. 10, n. 36, p. 289-300,jul./set. 2002.

PFEFFERMANN, D. et al. Weighting for unequal selection probabilities in multilevelmodels. Journal of the Royal Statistical Society, London, Serie B, n. 1, p. 23-40, 1998.

RASBASH, J. et al. MlwiN Beta version: Multilevel Models Project. London: Institute ofEducation, University of London, 1999.

RAUDENBUSH, S.; Bryk, A. Hierachical Linear Models: applications and Data Analysis.2nd ed. Newbury Park, CA: Sage, 2002.

SCRIVEN, M. The methodology of evaluation. In: AMERICAN EDUCATIONALRESEARCH ASSOCIATION. Perspectives of curriculum evaluation. Chicago, IL: RandMcNally, 1967. (Monograph Series on Curriculum Evaluation; No. 1).

______. Evaluation Thesaurus. 4th ed. Newbury Park, CA: Sage, 1991.

SHADISH, W. R. et al. Guiding principles for evaluators. San Fracisco: Jossey-Bass,1995. (New Directions for Program Evaluation; No. 66).

STUFFLEBEAM, D. Empowerment evaluation, objectivist evaluation, and evaluationstandards: where the future of evaluation should not go and where it needs to go.Evaluation Practice, Beverly Hills, CA, v. 15, n. 3, p. 321-38, 1994 .

______. The methodology of metaevaluation as reflected in metaevaluations by thewestern Michigan University. Journal of Personal Evaluation, v. 14, n. 1, p. 95-125,2001.

TANSCHEIT, R. Sistemas Fuzzy. Rio de Janeiro: Departamento de Engenharia Elétrica,PUC- Rio, 2004.

YANG, M. et al. MlwiN macros for advanced multilevel modelling: version 2.0:Multilevel Models Project. London: Institute of Education, University of London, 1999a.

______. The use of assessment data for school improvement purposes. Oxford Review ofEducation, Oxford, n. 25, p. 469-483, 1999b.

Recebido em: 30/07/2007Aceito para publicação em: 09/08/2007

um sistema fuzzy de suporte à decisão para meta-avaliação ... · estatística da fundação...

Documents