During the first stage of the project, we ran an iterative process that led to selection of a working set of indicators and measures.
As a starting point, we took the Institute for Healthcare Improvement (IHI) Whole System Measures  that were developed as a balanced set of metrics to measure the overall quality of a health system. We felt the IHI measures strongly aligned with our objectives. We reviewed the measures as a ‘straw man’ to test coverage and relevance in a New Zealand context. This resulted in several IHI measures being adopted and adapted for the first indicator set. We also undertook extensive research to identify and assess existing indicators from both New Zealand and overseas quality indicator frameworks, to supplement the set in key areas of interest and to help address identified gaps in coverage.
Through a workshop and survey process, including assessment of each potential indicator against seven strategic criteria, we reduced the long list of potential indicators down to a narrower working set of 17, including four system-level indictors and 13 contributory measures. We then undertook detailed work to complete definition standards, undertake feasibility testing and present preliminary results for this set.
In July 2012, the Commission published a report presenting our draft indicator framework and preliminary results for the working set of indicators, and targeted a range of areas for stakeholder feedback.
Subsequently, we launched an extensive stakeholder engagement process that ran from August to October 2012. The process included several streams of activity, including: an electronic survey; a series of five seminars; commentary on specific indicators collected via the Health Quality Measures New Zealand website, some discussion sessions around specific measures with sector experts and targeted feedback from the Commission’s consumer network.
Early in November 2013, we published a summary of the feedback we received during the stakeholder engagement process.
Broadly, there was strong support for the overall framework, the way we approached this work and how the work is positioned within the context of the Commission’s measurement and evaluation work.
We collected useful and detailed commentary on specific indicators and their construct, which has led to the revision of our working list into the final set of nine indicators. We also gathered a huge amount of feedback on ways in which the process could be developed and improved, and several suggestions for future indicators that would be useful and interesting to investigate further.
 Martin LA, Nelson EC, Lloyd RC, Nolan TW. 2007. Whole System Measures. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement. (Available at www.IHI.org ).