Community Services Outcomes Measurement Framework User Guide

Purpose of this guide

This guide demonstrates how to align your government agency commissioning plans and service agreements to the Community Services Outcomes Measurement Framework (Framework). The Framework will enable a shared understanding of outcomes across government agencies and the community services sector and to better harness data to demonstrate the effectiveness of community services.

Purpose of the Framework

The Framework sets out the community outcomes that the Western Australian Government seeks to achieve in delivering community services. It will help to better meet the needs of the people of Western Australia by:

• building a shared understanding of outcomes across government agencies and the community services sector;
• supporting agencies and the community services sector to adopt an outcomes-focused approach in community services planning and commissioning;
• promoting consistency in the measurement and evaluation of outcomes and activities;
• enabling innovation, learning, and improvement; and
• encouraging agencies and the community services sector to work together to achieve outcomes.

Where the Framework applies

The Framework applies across all commissioning plans, programs and service agreements under the Delivering Community Services in Partnership (DCSP) Policy. Government agencies undertaking community services commissioning activities are encouraged to use the Framework to support the design, delivery, and evaluation of the community services they deliver. The Framework applies to all new commissioning plans and service agreements and the relevant reporting requirements described in each commissioning plan.

Ownership of the Framework

The Framework is an initiative of the Supporting Communities Forum. It was initially developed through Department of Finance (Finance) funding to the Western Australian Council of Social Service, under the guidance and oversight of the Supporting Communities Forum.

Finance will undertake ongoing implementation of the Framework in consultation with government agencies and the community services sector, with oversight from the Supporting Communities Forum.

Government agencies are responsible for integrating the Framework into their community services commissioning and service delivery management and may contact Finance for guidance in doing so.

The Community Services Outcomes Measurement Framework

The Framework consists of domains, community outcomes, indicators and measures.

Domains

Domains are the generalised set of wellness categories for whole populations of the Western Australian community. The Framework consists of seven domains:

• Safe – we are free and safe from harm
• Stable – we are financially secure and have suitable, stable housing
• Healthy – we are healthy and well
• Equipped – we choose how to live our lives
• Empowered – we have the skills, experiences, and resources to contribute to our community and economy
• Connected – we connect with culture, our communities, our environment and to each other
• Sustainable – our built, natural, and service environments are clean, liveable, and sustainable

For each domain, the Framework articulates both the community outcomes and possible indicators to measure achievement.

Community outcomes and sub-outcomes

The community outcomes and sub-outcomes identify the critical wellbeing outcomes for each domain. There are several community outcomes for each domain (numbered one through 31) and several sub-outcomes (numbered 1.1 through 31.2), which can be viewed in the Framework. Community outcomes are long-term outcomes (usually longer than the service agreement term), often expressed on a population level, can be affected by events outside of the services, and may or may not be achieved by the services even if all outputs (deliverables) are provided to a high standard.

Service level outcomes

Community services use service level outcomes as well as community outcomes. Service level outcomes are outcomes that can reasonably be expected to be achieved for service recipients during the term of the service agreement if services are delivered to the expected standard. Service level outcomes are not included in the Framework but will still need to be determined and measured for each community service.

Indicators and measures

The indicators identify elements of change under each community outcome that quantify how well a service is working. Indicators at this level demonstrate progress towards the strategic community outcomes and domains. They provide measurable information to determine if a service is implementing activities as expected and achieving its outcomes. Not only can indicators help you understand what happened or changed, but they can also help you ask further questions about how these changes happened.

You should define several measures for each indicator, through a collaborative approach with the service providers delivering the program. When defining these measures, agencies should focus upon data that reflect the best evidence-based practice and data that is readily available and easy to collect. Once the indicator and measure are defined, record them in your agency’s data library.

The Framework shows the suggested indicators for each community outcome.

The data library

Government agencies are required to use a data library to service as a reference point for all outcomes, indicators, measurements, and collection methodologies that a government agency uses. The data library is a tool with which government agencies can record and establish their processes to capture and report against each service agreement.

Government agencies should record their program and service agreement outcomes and data measures. Over time the data library will form a map of what measurement and data collection processes best represent each outcome. The data library will also identify the data collected by each program and service agreement. Over time the data library will allow agencies to reference a standardised set of measurements that best reflect each indicator.

Finance has a data library template which government agencies may use and adapt to their needs. Please contact Finance if you would like to access the template. Alternatively, government agencies are free to develop their own data library templates.

How the Framework can benefit the delivery of community services for Western Australia

The Framework will benefit government agencies and the community services sector in several ways.

• It will align programs and service agreements across the community services sector – the Framework can be used to map all community services programs and service agreements to a consistent set of outcomes, ensuring that programs focus on consistent goals.
• It will identify cross-agency activities that are contributing to the same domain – the Framework can be used to identify all programs that contribute to an outcome across the whole sector, which can unlock opportunities for learning, collaboration, and the ability to target outcomes from a multi-program approach.
• It strengthens engagement with individuals by articulating government activities under outcome domains seen as relevant and meaningful – individuals can understand what the program intends to help them achieve and where it fits into the bigger picture.
• It can be used to identify any gaps in services and strategically develop or commission appropriate initiatives to address these gaps – the government has transparency around where community needs are being under-served and can use this to create precisely targeted intervention programs.
• It will provide insight into the total value of investment into achieving specific outcomes across the sector – the government can understand the cost of delivering outcomes for the community and understand where best to direct funding.
• It will deliver better results for community services recipients in Western Australia – the Framework will provide a cohesive view of the community services sector, with more significant opportunities to understand what is working and what we can do better.

Planning for outcomes

Planning the outcomes for a community services program or service agreement requires several steps. Agencies will lead the co-design of outcomes using these steps:

1. Defining activities through a program logic mapping process;
2. Selecting community outcomes and service level outcomes;
3. Selecting indicators and measurement methodology;
4. Collecting data; and
5. Feedback and iteration.

Participants in the co-design of outcomes should include relevant people within your government agency, as well as service providers, service recipients and other community members. 

1. Define your activities through program logic mapping 

Defining your program outcomes

Your community service needs to have a clear intention for the change being sought. A program logic model is one tool that can help to articulate how your activity will achieve a specific outcome.

Program logic mapping process and template

A program logic model provides a structured approach to identify the cause-and-effect nature of how a service will create positive benefits. The program logic model follows the steps below:

1. Situation – what problem is the service attempting to solve?
2. Inputs – what resources are needed to conduct the activities?
3. Activities – what actions and processes will be undertaken to deliver the outputs?
4. Outputs – what products or services will be delivered or produced by the service provider and to whom?
5. Outcomes – what is the impact at the individual, and population level?
6. Assumptions – list the assumptions you have made regarding links between the inputs and activities. Does evidence support these assumptions?
7. External Factors – list the external factors that could impact service delivery and intended outcomes.

A community services program logic template and a program logic model guideline are available on wa.gov.au.

2. Select the community outcomes and service level outcomes

Using the Framework and in consultation with relevant stakeholders including potential service providers, choose at least two relevant community outcomes that your program or service agreement aims to achieve. Consider ways that activities may contribute to changes across all areas of the Framework and try to identify as many impacts that arise from the program as possible.

A program logic modelling workshop is a useful way to identify, group and order your outcomes, indicators, and measures. This guide used in conjunction with the program logic model guideline will provide a starting point for the workshop process.

Using the Framework does not prevent you from using other community outcomes. However, you should nominate at least two community outcomes from the Framework.

Service level outcomes are not included in the Framework as they are more specific to the services being commissioned; however, you are required to develop service level outcomes and measures in partnership with the community services sector. Service level outcomes, along with community outcomes, are included in the request template.

3. Select indicators and measurement methodology for community outcomes

The outcomes are the changes your program hopes to achieve. To know whether those changes have happened, you need to use indicators. Indicators are the signals that suggest something has happened. When you have chosen your community outcomes, select the most relevant indicator linked to the community outcome from the Framework, with input from stakeholders including potential service providers.

The Framework does have some suggested indicators; if the existing indicators are not appropriate for a particular service you can define and recommend a relevant indicator through your co-design process. There are several considerations when selecting an indicator:

• The chosen indicators should have a clear linkage to the desired outcomes, be validated and practical;
• The indicator should be common in community services so that comparison of indicators is possible;
• The indicator should be evidence-based and inform service delivery;
• Generally, indicators will begin with a statement;
• Statements such as "The number of." or "The percentage of . . ."; and
• Begin by measuring the priority outcomes and draw on data you are already collecting.

Next, each indicator should have a definition for measurement methodology. Some essential questions to consider during this process include:

• What will we measure to demonstrate the services are delivered? (linked to outputs)
• What will we measure to demonstrate that the change has occurred? (linked to the community outcomes)
• Do we have existing data? (secondary data sources)
• Where can the data be sourced?
• Do we need to collect new data? (primary data sources)
• Who will be responsible for collecting the data?
• When do we need the data?
• Are there any issues around data access or disclosure (e.g. legislative or other restrictions)?
• Is the data source accurate?

Once you have determined your indicators and measures, record them in your government agency’s data library. For each selected population outcome, the data library should contain fields to identify the indicator, the data, and the collection methodology.

4. Collection of data

Once outcomes and indicators are defined, you can use many different forms to establish a data collection process (e.g. surveys and questionnaires, structured and semi-structured interviews, and tailored client assessment tools).

The best way to develop a complete assessment of outcomes is to use a mix of approaches, often to draw on multiple data sources.

Data collection methods cannot be determined without input from service providers, as they are frequently responsible for collecting the data. We recommend that you select methodologies that have the following characteristics:

• a standardised collection methodology within the sector
• a currently used collection methodology by your agency or service provider. Seek insight from your service provider and align to what they are already collecting where possible
• the collection methodology will not induce a bias in the answer, and
• the collection methodology is not overly intrusive or burdensome on the service provider or the service recipient.

Data should be kept in one central digital database within each agency to be analysed and used. All new data collection systems should be pilot tested. Treat any data collected in the first round of data collection with caution while testing your collection methods. Also, remember that you will learn what works best for data collection over time.

5. Feedback and iteration

Using feedback to monitor and achieve better outcomes

Regularly review the results of your outcomes measurement to ensure you achieve your intended impact. Is the data confirming that services are achieving the intended outcomes, or does it show mixed results? Is the data showing that services realise benefits you did not expect? Can the service provider share insights into the service based on the data?

A valuable way to review the data is to examine it in line with your program logic model. By checking each area, you can uncover a range of information to make changes to your program to deliver improved outcomes.

• Situation: Have you understood the situation correctly?
• Inputs: Are you using the appropriate resources?
• Activities: Are you delivering activities appropriately?
• Outputs: Are you delivering the right outputs?
• Outcomes: Are you achieving the results you intended?
• Assumptions: Are your assumptions about linkages correct?

An extended list of program logic model feedback and iteration questions is available in the package of materials for the Framework.

6. A working example of using the Framework

Below we will demonstrate how to use the Framework through an example.

You are commissioning a service designed to improve rates of school attendance for kindy and pre-primary aged children. You are planning the service and will need to undertake the process below to identify information for the Request document.

STEP 1: Define your service activities using program logic mapping

Using the program logic model, work with stakeholders to co-design all the elements across the template. The elements include the situation, inputs, activities and outputs.

STEP 2: Select community outcomes from the Framework

Go to the Framework and identify the community outcomes and sub-outcomes area. Look through the list under the heading community outcomes and select a minimum of two of these outcomes that are consistent with what you have identified in your program logic mapping.

For our example, we identify two outcomes that are most relevant to the services, both within the ‘equipped’ domain:

22. Our children start school ready to learn; and

23. We successfully transition between schooling levels and employment.

We then identify the most relevant sub-outcomes within those community outcomes:

22.1 Early childhood education; and

23. 2 Children and young people are engaged and supported with learning.

STEP 3: Select indicators and select measures

For each of the sub-outcomes, identify the indicators. Within the Framework, we have some suggested indicator areas. To use this, we can look across the Framework for each of the outcomes to identify potential indicators.

In our example, we have identified the following indicators:

For each of these indicators, identify measures that are most relevant to the service through consultation and collaboration with key stakeholders such as service providers, peak bodies and consumer representatives.

Use measures relevant to the indicator, already collected and commonly used across the sector by your agency and service provider. The fields in your agency’s data library will give you the specific elements needed to identify a measure and the collection methodology.

For our example, the team running the service agreement looked towards data they already collected and consulted with service providers to identify the best measures for each indicator. They selected the following:

STEP 4: Plan and collect data

For each indicator and measure, use your agency’s data library to record the data collection methodology by completing all fields. When collecting data, the data itself will need to be recorded digitally (ideally in a database format such as an excel spreadsheet).

For our example, the agency in partnership with service providers determines that the best way to collect data will be through a combination of departmental data and a before/after survey of service recipients.

STEP 5: Add this information into your planning document

The information identified through the program logic process should inform the relevant areas within your planning documents. This information includes the outcomes, proposed indicators and measures, and data collection methodology. As data becomes available throughout service delivery, these metrics can ensure the service delivers the intended outcomes.

More information is available regarding evaluating your service agreement in the program logic model guideline.

Outcomes are determined during service design and planning, but this does not mean they are unable to be changed during service delivery. It may be necessary for you to negotiate changes to outcomes, indicators and measures during the term of a service agreement. Service agreements are flexible contracts that allow for continuous improvement via service agreement variations when appropriate and agreed by both parties.

Collecting and reporting data comes at a cost to service providers, and ultimately must be funded through the price of service delivery. Government agencies need to carefully consider the use of all data they request, and to accept feedback from the community services sector on the suitability and or administrative burden of requested data.