top of page
Kākā 2.jpg

RA5 - Te Hoa Arotake Critical Friend: Evaluating and Measuring Success

At the heart of Oranga was a commitment to meaningful, impactful results, and the integration of mātauranga Māori with scientific methodologies. Te Hoa Arotake: Critical Friend plays a pivotal role in ensuring the success of the Oranga research programme by supporting each research area, facilitating self-reflection, and strengthening inter-project collaboration, and then reporting on that. 
 
Through a values-driven, principles-focused evaluation framework, Oranga's critical friends ensured that each research area operated with clarity, alignment, and effectiveness. Their work fostered an environment of intellectual rigour, cultural integrity, and practical application, ensuring that Oranga’s findings informed meaningful conservation outcomes.

Background | Whakapapa

Te Hoa Arotake: Critical Friend, was developed after conversations with subcontractors who wanted to ensure Oranga remained a safe space for them to work at the nexus of mātauranga and science, with both traditional scientists and traditional knowledge holders. Further, the funders knew that Oranga would be subject to increased scrutiny because it sat firmly at the top end of the Vision Mātauranga Policy scale, and addressed issues of importance to Māori communities and citizens.   Thus, the critical friend was seen as pivotal in providing not only support for the leaders of the four research areas, but also reassurance for funders, agencies and others that the delivery and thinking behind each project was tested by international experts - ensuring it upheld best practice research standards. 

 

The team of three 'friends' brought expertise in evaluation, measurement, scientific research, and international collaboration to Oranga,  creating and maintaining a safe space for:

✅ Intellectual debate: Encouraging rigorous analysis and critical questioning.

✅ Reflection: Ensuring research objectives remained aligned with values and goals. 

✅ Pastoral care: Providing emotional and professional support to researchers.

 

By embedding relational accountability within the evaluation process, we ensured that the Oranga research programme upheld its core principles while delivering tangible benefits to communities, and for ecosystems. 

Developing the Research | Whakawhanake Rangahau

Early on in the development of the Oranga programme there was an elevated interest in the proposed research, not only from Māori but also from non-Māori (nationally and internationally). This led the team to adopt the view that the impacts from this programme would, and should, have wider importance than just the immediate aims of each project: the whole should be greater than the sum of its parts. 

This focus on system "impact" vs short-term outputs was not only what the researchers wanted, but what Māori communities, government agencies, funders and wider communities of interest were asking for. Accordingly, the critical friends sought early on to define and refine measures of progress and impact from across Oranga, seeking early input from each project team. This process also contributed to building greater awareness within the teams and across the programme about how Oranga was progressing towards its goals. While balancing 'impact' with 'outputs' can be tricky, the collective approach and shared values that underpinned the work in Oranga enabled the team to realise the programmes goals. 

Te Hoa Arotake: Critical Friend, received just over 9% of the Oranga budget (1% of the Ngā Rākau Taketake budget) and was led collectively by the three critical friends. They were contracted via their home institutions:  Forest Research UK, the University of Saskatchewan, and Two Bridges Consulting Canada. 

Research Methodology | Tikanga Rangahau

Te Hoa Arotake: Critical Friends developed a measurement framework that was both principles-focused and values-based (Patton, 2018). This approach allowed each research area to retain its own identity and leadership while ensuring coherence across the wider Oranga programme. Key aspects included: 

  • Collaborative Design: Each research area was involved in defining their own success measures. 

  • Kanohi ki te kanohi (Face-to-Face) Engagement: We prioritised in person interactions, supported by virtual meetings during COVID-19 and the extreme weather events, to foster strong relationships.

  • Regular Checkpoints: Ongoing assessments were conducted to identify strengths, challenges, and areas for refinement. 

  • Interdisciplinary Communication: Creating pathways for shared learning between research teams.  

Our evaluation strategy ensured that the programme's research objectives were not only met but also enriched by reflection, adaptation, and continuous learning. 

Core Values | Ngā Uara

At the foundation of evaluation work were key values that drove our evaluation and guidance, forming our core theory of change: 

  • Knowledge:  about deepening understanding of people, place, and lore, including kaitiakitanga and ecosystem dynamics. 

  • Recognition: which acknowledges how kauri ora and mātauranga Māori are essential for novel conservation solutions. 

  • Relations: strengthens collaborations to ensure sustainable, community-led conservation partnerships that promote coexistence with each other and our ecosystems (kauri ora)

  • Responsiveness: to the needs of the community.

  • Mentorship: building capacity, supporting training, knowledge exchange and mobilisation.

  • Renew: support for the value of Māori processes and mātauranga.

  • Transform: colonial systems by navigating shifts in those systems and ways of doing and being.

  • Sustainability and Preservation: of knowledge, including conservation.

  • Sovereignty: of knowledge, data and processes is upheld.

  • Equity and Respect: means honouring and following protocols, accepting responsibility, and committing to anti-racist practices, and values like kaitiakitanga.

We also crafted indicators to track progress so that we always knew what we were doing, how we were doing it, and - most importantly - why we were doing it!

Click here to understand more about our framework and here to watch a video explanation of it.

First values-based diagram used to create the measurement framework adopted by the RAs.

​Measuring Success​​​​

We gathered evidence through interviews with project leaders and key participants either kanohi ki te kanohi, face-to-face, or via zoom (especially during Covid and weather-related events). We connected at regular intervals and especially at key points in their timelines - RA leads and their key researchers were interviewed over 20 times during this process. This was done strategically to gather data as the projects operationalised their research, so we could facilitate reflections on the projects impacts. Importantly, these narratives also helped us to understand how, where and why impacts were occurring. 

We also scanned the quarterly reports for each RA (from 2020 to the end of 2024) and used that as a basis to track connections and outputs. Together, the interviews and information from the quarterly reports formed the basis for our analysis and reporting.

The critical friend role contributed to Oranga's success by: 

✔️ Providing strategic guidance that strengthened each research area's approach and outcomes. 

✔️ Ensuring alignment between mātauranga Māori and scientific methodologies, fostering a unique and effective research model. 

✔️ Enhancing collaboration between research areas, ensuring knowledge was shared and amplified. 

✔️ Supporting researchers through structured reflection, adaptive strategies, and pastoral care. 

✔️ Developing a replicable framework for future Indigenous-led conservation projects. 

These successes demonstrated that the Critical Friend role was not merely an external evaluator but an integral part of ensuring that Oranga's research was rigorous, ethical, and impactful. 

Tar_Simon.png

Assoc. Prof

Simon Lambert

University of Saskatchwen

(now part-time with TTW)

mariella.jpg

Dr Mariella Marzano

Forest Research, UK

7f90dd_d83d1d4846374ea7b9d9e4410c197976~mv2.webp

Micheal Heimlick

Two Bridges Consulting  (now TTW)

Outcomes and Outputs of Oranga | Ngā Hua me Ngā Putanga 

 

From the outset, all Oranga researchers accepted the need, and recognised the value, in publishing scientific reports and academic articles. As Māori, we are proud of our knowledge, our mātauranga, and our lived experiences, and are happy to share our data and evidence when appropriate protocols are adhered to, and we are in control of or fully included in the processes. The list of publications from the five RAs is testimony to the intellectual rigour, hard work, and support Oranga received from communities and science peers.

A particularly satisfying outcome was to hear of and observe the development of skills and knowledge across the rangatahi who engaged in different projects with their communities. Given the growing need for science and research skills in the New Zealand workforce, we see Oranga as providing not just “new recruits” to the system but recruits specifically trained in kaupapa Māori research approaches.

 

Perhaps the biggest impact the Oranga programme had was how it highlighted the changes happening across Aotearoa New Zealand’s science and innovation ecosystem. It highlighted the commitment most scientists and research organisations have to including Māori communities, people, and businesses, and their knowledge in both applied and blue-sky research. Over the course of the programme, we saw the inclusion of Māori principles and approaches as key to transforming private and public services across Aotearoa, and to the building of new institutional arrangements and systems for effective biosecurity. The Oranga team started to be seen as leaders within the Biosecurity space, and because of this, were approached more to be on research committees, to help and support their programmes.

In addition, and contributing to this primary impact, the programme is actively dispelling myths about Kauri Dieback & Myrtle Rust investment and research with Māori and highlighting Mātauranga Māori-led Innovation in the biosecurity space.

2.png

Second, and complimentary, values-based diagram used to create the measurement framework adopted by the RAs.

As of late 2024, the Oranga Programme has:

  • Drafted and submitted 27 publications, all of them peer reviewed (6 peer-reviewed articles or chapters published,  6 peer-reviewed articles in press,  1 Masters thesis completed, 2 programme reports, and 12 reports, publications, guidelines).

  • Given 77 presentations, including in community and academic settings.

  • Posted 14 video outputs on building capacity and showing results.

  • Created art installations, documentaries, waiata, seed banking maps, and opinion pieces published internationally (the Guardian, Samara KEW) and locally.

  • Attracted the attention of mainstream media 32 times, including National Geographic and the New York Times. 

An example of the style of measurement adopted for Oranga and of an output by RA5.

Below is a summary of impacts created by the Oranga Programme, grouped by the 2024 goals:

  • By 2024 - The Māori environmental worldview is understood to be an intrinsic part of restoring the ngahere from kauri dieback and myrtle rust 

    • Project leads saw an increased acceptance of Māori approaches and paradigms across plant, forest and biosecurity research systems. This was seen in acknowledgments of Māori research, researchers and knowledge holders in spaces where it had previously been dismissed in the past. Concretely this was demonstrated through the ever-increasing amount of external interest in the work of the research teams, largely from non-Māori media and organisations. While it was noted that project leads felt the need to protect the research from potential misuse and ridicule, many also saw the increased interest, particularly from non-Māori, as a sign of positive change, and acceptance that this approach to research on kauri ora, was the correct and valid approach. 

    • Oranga programme funding helped create the structure, capacity and spaces (physical and intellectual) for those with pre-existing capabilities and knowledge to meaningfully work within kauri ora spaces. This extended to both researchers and kaumātua. In other words, Oranga mobilised existing capacity and promoted the knowledge necessary to work with kauri ora. Many pointed to the tuakana/teina relationship as another facilitating factor for this outcome: participants were both learners and teachers throughout the research. The benefit of this approach was that it created space for co-learning between researchers, Iwi and hapū, and kaumātua and rangatahi.

  • By 2024 - Māori are leading positive system change in forest biosecurity leading to increased resilience to tree pests and diseases, especially as related to kauri dieback and myrtle rust.

    • The extensive relational work done by the project teams has paid off in the sense of increased individual, Iwi, and hapū capacities. Capacity increases have taken many different forms but include: increases in kauri ora and kaitiakitanga knowledge, professional development for Māori and non-Māori researchers, changes in how systems relate to kauri ora and Māori research, and increases in the availability of much needed equipment for kaitiaki to carry out work on the ground.

    • There has been an expansion of internal and external connections, and a broadening and deepening of relationships and partnerships. We have been able to describe this networking through the Critical Friend evaluation process. Instead of simply counting the number of partnerships, we focused on how the relational approach the project teams took to carry out their work led to a foundational set of relationships that propelled the work forward, supported by Kaupapa Māori principles. This happened despite restrictions presented by both COVID-19 and an extreme weather event. Project leaders believe that the connections and relationships they formed will extend beyond the scope of Oranga, as evidenced by the increasing number of community referrals they are receiving - perhaps the ultimate indicator of success for a Māori researcher.

    • Funding through Oranga provided the project teams with the opportunity, as well as a safe space, to create relationships. Evidence suggests that is having an ongoing positive impact on forest health Māori-led research.

  • By 2024 - Affected Māori communities are empowered to protect and restore their ngahere from kauri dieback and myrtle rust.

    • Evidence suggests that the research has helped Iwi, hapū, individuals, organisations, and government agencies to better understand the deep relationship between kauri dieback, myrtle rust, and the surrounding ecosystem (including non-Māori individuals and organisations). This saw kaumātua shift how this relationship was described by focusing on positivity and life as opposed to death and dying (Oranga led the move to using the term Kauri Ora instead of Kauri Dieback). This fundamental shift in understanding how work should be designed and carried out, facilitated a Māori-based understanding of how to approach the research. It provided real motivation and hope to the project leaders and those they were working with, which inevitably created buy-in for the research and facilitated other impacts.

    • Project leaders and key participants have identified personal and professional growth, facilitated by the Māori-led approach the research took. Project teams have developed key skills in research administration, navigation of systems, and confidence using Māori approaches against a backdrop of largely scientific (colonial) paradigms. The teams felt that the true impact of this approach to research will continue to be felt well into the future. Increased capacity will not only contribute to the health of the kauri in the short-term, but in the long-term will help Māori researchers and knowledge holders make a significant difference in future research projects (environmental or otherwise).

Key Outputs of RA5

Peer-reviewed Publication

  • Lambert, S., Heimlick, M., Marzano, M., Mark-Shadbolt, M., & Smith, V. 2024. Theory of Change Visuals: Using Diagrams, Metaphors, and Symbols to Communicate Complex Ideas and Get Buy-in. Canadian Journal of Program Evaluation. 40(1), https://doi.org/10.3138/cjpe-2024-0028

Conference Presentation

  • Theory of Change Visuals: Using Visuals and Metaphors in Measurement (above)

Other Outputs

  • Marae Framework video (below) & report

  • Oranga Programme Measurement Framework

  • Oranga Programme Mid term Report (Reflections on outcomes and impacts)

The Marae Framework Video created by members of Oranga.

Addressing Potential Criticisms

​Q) Was this just about oversight and compliance? 

A) No. The Critical Friend was not about checking boxes or enforcing compliance, it was a relational, values-driven approach that fostered growth, reflection, and alignment. Our role was to enhance, not dictate. 

Q) Did this approach prioritise science over mātauranga Māori? 

A) No. This evaluation framework ensured balance, actively uplifting mātauranga Māori alongside scientific inquiry, ensuring both knowledge systems shaped and informed research approaches and conservation strategies. In addition, two of the critical friends were not New Zealanders, which meant mātauranga had to be clearly explained and rationalised . 

​​

Q) How did this contribute to real-world conservation outcomes? 

A) By integrating reflective practice, relationship-building, and iterative learning, this approach ensured that research findings translated into action, benefiting forests, conservation efforts, and communities in tangible ways. 

Q) Why does this research matter for Aotearoa New Zealand? 

A) The Critical Friend approach in Oranga ensured that conservation research in Aotearoa: 

  • Centred Indigenous knowledge and community leadership.

  • Produced rigorous, meaningful research with real-world impact. 

  • Strengthened collaboration across disciplines and research teams. 

  • Created an adaptable, reflective research model for future use.

By upholding these principles, Te Hoa Arotake (Critical Friend) ensured that Oranga's work was not only scientifically robust but also ethically grounded, culturally informed, and environmentally impactful. 

bottom of page