More Menu

Methodological constraints and challenges in Program evaluation.

byPosted at

Contents

  1. The shoestring approach
  2. Budget constraints
  3. Time constraints
  4. Data constraints 
  5. Five-tiered approach
  6. Methodological challenges presented by language and culture

The shoestring approach

The "shoestring evaluation approach" is intended to help evaluators working with a limited budget, data access or availability, and turnaround time conduct effective, methodologically rigorous evaluations (Bamberger, Rugh, Church & Fort, 2004). This method has been developed in response to the growing demand for more rapid and cost-effective evaluation processes in the face of budget constraints, time constraints, and data scarcity. However, it is not always possible to design an assessment to meet the highest possible standards. Many programmes don't include an evaluation procedure in their planning or budgeting. As a result, many evaluation processes do not begin until after the programme has already begun, which can result in time, budget, or data constraints for the evaluators, affecting the evaluation's reliability, validity, or sensitivity. > The shoestring approach helps to ensure that the maximum possible methodological rigour is achieved within these constraints.

Budget constraints 

Most original projects do not include a budget for conducting an evaluation, so programmes frequently face budget constraints (Bamberger et al., 2004). As a result, evaluations are automatically given smaller budgets, which are insufficient for a thorough evaluation. Budget constraints may make it difficult to apply the most appropriate methodological instruments effectively. As a result of these constraints, the amount of time available to conduct the evaluation may be limited (Bamberger et al., 2004). Budget constraints can be overcome by simplifying the evaluation design, reducing the sample size, experimenting with cost-effective data collection methods (such as using volunteers to collect data, shortening surveys, or relying on focus groups and key informants), or seeking out reliable secondary data (Bamberger et al., 2004).

Time constraints

When an evaluator is summoned to conduct an evaluation while a project is already underway, if they are given limited time to do the evaluation compared to the study's life, or if they are not given enough time to plan adequately, the most time constraint that an evaluator can face. When the evaluator is unfamiliar with the area or country where the programme is located, time constraints are especially problematic (Bamberger et al., 2004). Time constraints can be overcome using the methods listed above under budget constraints, as well as careful planning to ensure effective data collection and analysis within the time constraints.

Data constraints 

Multiple methods, such as combining qualitative and quantitative data, can improve validity and save time and money through triangulation. These constraints can also be overcome with careful planning and consultation with programme stakeholders. The evaluative process can be streamlined and reduced while maintaining credibility by clearly identifying and understanding client needs prior to the evaluation. Multiple methods, such as combining qualitative and quantitative data, can improve validity and save time and money through triangulation. These constraints can also be overcome with careful planning and consultation with programme stakeholders. The evaluative process can be streamlined and reduced while maintaining credibility by clearly identifying and understanding client needs prior to the evaluation.

Overall, time, money, and data constraints can have a negative impact on the evaluation's validity, reliability, and transferability. The shoestring approach was developed to help evaluators overcome the limitations mentioned above by identifying ways to save money and time, reconstruct baseline data, and ensure maximum quality while working within existing constraints (Bamberger et al., 2004)

Five-tiered approach

The five-tiered approach to evaluation expands on the strategies that underpin the shoestring approach to evaluation. Jacobs (1988) developed it as a different way to evaluate community-based programmes, and it was used to evaluate a statewide child and family programme in Massachusetts, USA. The five-tiered approach is proposed as a conceptual framework for better matching evaluations to programme characteristics as well as the unique resources and constraints that each evaluation context entails. In other words, the five-tiered approach aims to tailor the evaluation to each evaluation context's specific needs.

The earlier tiers (1-3) generate descriptive and process-oriented information while the later tiers (4-5) determine both the short-term and the long-term effects of the program. The five levels are organized as follows:
  • Tier 1: needs assessment (sometimes referred to as pre-implementation) 
  • Tier 2: monitoring and accountability
  • Tier 3: quality review and program clarification (sometimes referred to as understanding and refining) 
  • Tier 4: achieving outcomes 
  • Tier 5: establishing impact 
Purpose(s) are identified for each tier, as well as tasks that enable the tier's identified purpose to be achieved. For example, the first tier, Needs Assessment, would be used to document a community's need for a programme. Working with all relevant stakeholders, that tier's task would be to assess the community's needs and assets.

Family support programmes that emphasise community and participant empowerment are said to benefit from the five-tiered approach. This is because it promotes a participatory approach involving all stakeholders, and empowerment is achieved through this process of reflection.

Methodological challenges presented by language and culture

The goal of this section is to highlight some of the methodological challenges and dilemmas that evaluators may face when conducting programme evaluations in developing countries. Donor agencies from the developed world are major sponsors of evaluation in many developing countries, and these agencies require regular evaluation reports to maintain accountability and control of resources, as well as generate evidence for the program's success or failure. However, evaluators face numerous obstacles and challenges when attempting to implement an evaluation programme that employs techniques and systems that were not developed in the context to which they are applied. Differences in culture, attitudes, language, and political process are among the issues.

Ebbutt defines culture as a "constellation of both written and unwritten expectations, values, norms, rules, laws, artefacts, rituals, and behaviours that pervade a society and shape how people behave socially." Many aspects of the evaluation process, including data collection, evaluation programme implementation, and analysis and understanding of the evaluation results, are influenced by culture. Instruments that have traditionally been used to collect data, such as questionnaires and semistructured interviews, must be sensitive to cultural differences if they were developed in a different cultural context. The evaluator's understanding and meaning of the constructs he or she is attempting to measure may differ from that of the sample population, so concept transference is critical, as it affects the quality of the data collection done by evaluators, as well as the analysis and results generated by the data.

Language, which is closely linked to culture, plays an important role in the evaluation process. Language can be a significant barrier to communicating concepts to the evaluator, and translation is frequently required. There are numerous issues with translation, including loss of meaning as well as translator exaggeration or enhancement of meaning. For example, terms that are contextually specific may not have the same weight or meaning in another language. Data collection instruments, in particular, must take meaning into account, as the subject matter may not be considered sensitive in one context but may be sensitive in another. When administering data collection tools, evaluators must consider two important concepts: lexical equivalence and conceptual equivalence. The question of lexical equivalence is how to phrase a question in two languages using the same words. This is a difficult task, and while techniques like back-translation can help the evaluator, they may not result in perfect meaning transfer. The following point is conceptual equivalence. It is unusual for concepts to transfer without ambiguity from one culture to another.

Instruments that have not been adequately tested and piloted may produce results that are not useful because the concepts measured by the instrument may have taken on a different meaning, rendering the instrument unreliable and invalid.

When attempting to conduct a programme evaluation in a developing country, evaluators must take into account the methodological challenges posed by differences in culture and language.

Post a Comment
Tag,

Comments

Thank You