Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out by the Cultural & Indigenous Research Centre Australia (CIRCA). A Monitoring and Evaluation Framework and a set of National Indicators have been developed by CIRCA to guide the TIS program evaluation. It is a requirement within TIS funding agreements that some of the data collected locally, related to local project outcomes, will feed into the broader National Evaluation, by reporting against the five National TIS Performance Indicators.
The Department of Health (DoH) has described five TIS program objectives and associated national indicators that will be used to assess TIS program progress. Each of these indicators is described below, along with examples of possible data sources and data collection methods.
Outcomes related to this indicator include the following:
Examples of possible data sources provided by CIRCA include:
Numerical data on the number and type of groups/organisations or individuals involved in your project, or the number of people attending your activities/smoke-free events is a simple way of addressing this indicator. This indicator is concerned with increasing the reach of your activity, so if your project is working, the number of organisations or people involved in your activities should increase over time. If they don’t then you will need to think about why this is happening. It might be, for example, that there are no more local organisations to involve in your activities. In this case, maintaining different organisations’ involvement will become a key task for you.
Another way of measuring reach is to find out what people think about your TIS activities and if they have learnt anything new about smoking, such as ways to quit, or support for quitting that is available in your community. This is because reach is not just about the number of people who attend your activities, but about how many listen and understand the message. This kind of data can be either quantitative or qualitative, and can be collected through:
You could also provide a detailed description of what you did to engage the community, reflecting on what worked and what didn’t work and why.
The outcome related to this indicator is:
There is simple data you can collect around the number and type of organisations or services you have partnered with as part of your project activities. You can also collect more detailed data which describes the quality and extent of your collaboration with different partners. You can also show how your project partnerships have improved access to culturally appropriate support to quit, for example through case studies of the partnership journey.
You might also want to think about the quality of your relationships with different organisations you have entered into partnership with. Different ways of doing this include tools that involve the grant recipient organisation and their partner/s individually assessing the relationship and then coming together to discuss and move forward. Examples of these tools include:
From the above examples, you can see there is simple data you can collect around the number and/or percentage of staff in your own organisation or others, who have undertaken training related to TIS. This might be a formal training course on, for example, brief intervention. Or it might be more informal training about locally available quit support services (for example having a yarn with the local Quitline team).
You can also collect more detailed data which describes the impact of training on staff development (e.g. knowledge about smoking matters and quit services) and their confidence in supporting community members asking for advice. You can collect this data through:
The specific numerical data that you might be able to provide for this indicator will depend on the records that your organisation keeps, or what information other services in the region (e.g. Quitline, clinical services) are able to share with you. If you are not able to access this kind of data, or your organisation does not keep detailed or accurate records, other ways to find out how many people in the community are accessing quit services include:
You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:
From the above examples you can see there is simple data you can collect around the number of events locally that are smoke-free, how many organisations you have worked with to develop/improve smoke-free policies or to increase worker compliance with policies. You might also be able to count how many homes are smoke-free, or what people think about being smoke-free. Good ways of doing this include:
You can also collect more in depth data on understanding about what being smoke-free means and how important it is to people to try to be smoke-free. You can ask the smokers if they go without smokes at home, in the car, near their children, at work, or on other occasions, and how they handle smoke-free times. Interviews and focus groups are good ways to get this information.
Finally you might want to describe how you have supported the communities in your region to become more smoke-free.
The Australian Indigenous HealthInfoNet acknowledges the Traditional Owners of the lands and waters of Australia and the Torres Strait.
We respect all Aboriginal and Torres Strait Islander people—their customs and their beliefs. We also pay our respects to Elders past and present, with particular acknowledgement to the Whadjuk people of the Nyoongar nation, the traditional owners of the lands where our offices are located.