TIS evaluation

Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out. There are two evaluators of the National TIS program. Part A of the evaluation will be conducted by the Cultural & Indigenous Research Centre Australia (CIRCA). Part B of the evaluation will be conducted by the Australian National University (ANU).

National TIS Performance Indicators

Six National Indicators will be used to assess TIS program progress. Each of these indicators is described below, along with the data sources described by CIRCA and examples of data collection methods. Click on any of the data collection methods listed under each indicator to learn more about that specific data collection technique. You can also find more monitoring and evaluation resources to help you here: Resources to monitor and evaluate your program

NBPU TIS strongly recommends that you collect both quantitative and qualitative data. This will allow you to address the National Indicators in full and tell the story of your successes. It will also ensure strong data for your own internal monitoring and evaluation of your activities as part of your continual quality improvement (CQI) process.

National Indicator 1: Implementation of evidence-based population health promotion activities aimed at preventing the uptake of smoking and supporting the promotion of cessation

The outcomes related to this indicator are:

  • Increased community involvement in and support for initiatives to reduce the uptake of smoking and increase sustained cessation
  • Increased leadership and advocacy role of community leaders in tobacco control
  • Increased understanding by the community of the health impacts of smoking
  • Population health promotion activities are locally relevant and have community support.

Data sources described by CIRCA are:

  • Number of community members participating in population health promotion activities and events
  • Number of community leaders participating in population health promotion activities and events
  • Number and type of evidence-based population health promotion activities including social marketing, community education and community engagement
  • Number and reach as evidenced by social media analytics, other media activities, and production/distribution of health promotion materials.

This indicator focuses on two aspects of your activity:

  • the extent of community reach and engagement of your activities (please note however that any activities targeted on either priority groups, people who do not attend ACCHS, or to increase geographical reach are not to be reported here, as these are the focus of Indicators 5 and 6)
  • the type of evidence-based population health promotion activities you do.

Community reach and engagement

Numerical data (counts) of the number of community members and community leaders involved in or attending your activities/smoke-free events is a simple way of addressing this indicator. Social media analytics are also important. This indicator is concerned with increasing the reach of your activity, so if your project is working, the number of community members and leaders involved in your activities should increase over time. If they don’t then you will need to think about why this is happening. Do you need to find new ways to engage community interest? You might also want to think about how representative of community the people you do see at your activities are. For example, are there certain groups in your community you are just not reaching, such as full-time workers? How could you make sure your activities are accessible to everyone? Or it might be that all community leaders are now involved in your activities. In this case, maintaining their involvement will become a key task for you.

You also need to find out what people think about your TIS activities and if they have learnt anything new about smoking, such as ways to quit, or support for quitting that is available in your community. This is because reach is not just about the number of people who attend your activities, but about how many listen to and understand the message (the third outcome under this indicator). This kind of data can be either quantitative or qualitative, and can be collected through:

As this data is not directly captured by the six monthly progress report, it will be important to provide this information as part of the story around your activity (case study or success story). You should also give a description of what you did to engage the community, reflecting on what worked and what didn’t work and why. What are the strengths of your community and its leaders that have made this activity a success? You will also need to consider what the risks and challenges have been for your team, and how you have worked to overcome them.

Evidence-based population health promotion activities

Simple numerical data (counts) for each type of evidence-based population health promotion activity you do will address this part of the indicator. You will also need to record the location of your activities (e.g. neighbourhood, town or region). Activities of specific interest under this indicator are:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

To avoid repetition, partnership working and collaborations developed do not need to be reported here. This information will be captured under Indicator 2. It will also be important to demonstrate (through your Action Work Plan) that these activities are evidence-based.

National Indicator 2: Partnerships and collaborations facilitate support for tobacco control

The outcome related to this indicator is:

  • Collaborations and partnerships built between TIS organisations and external support for tobacco control initiatives.

Data sources described by CIRCA are:

  • Number and type of organisations involved in planning/implementing TIS activities
  • Number and type of collaborative projects/partnership activities
  • Number and type of partnerships with local service providers to enable increased geographical reach
  • Number and type of partnerships with local service providers to enable increased reach to priority groups.

There is simple data you can collect around the number and type of organisations, services or individuals you have partnered with as part of your population health promotion activities. Of specific interest are partnerships with:

  • mainstream services
  • ACCHS
  • schools
  • community organisations/sporting clubs
  • local, state or federal government organisations
  • community leaders or community champions
  • networks/interagency groups.

As you can see from the data sources described by CIRCA, it will be useful to think about the purpose of these partnerships, for example if they increase your reach into areas within the region you are contracted to service, or priority populations including pregnant women, or people who do not typically attend ACCHS. You should also collect more detailed data which describes the quality and extent of your collaboration with different partners and how these partnerships have improved geographical or priority group reach, for example through case studies of the partnership journey.

Think about the quality of your relationships with the different organisations you have entered into partnership with as well. Different ways of doing this include tools that involve the grant recipient organisation and their partner/s individually assessing the relationship and then coming together to discuss and move forward. Examples of these tools include:

This information will be an important part of providing the story around your activity (case study or success story). What did you do to engage the organisations, what worked, what didn’t work and why? What are the strengths of your organisation and those you have partnered with that have made this activity a success? What were the risks and challenges and how have you worked to overcome them?

National Indicator 3: Increased access to Quit support through capacity building

The outcomes related to this indicator are:

  • Improved access to culturally appropriate support to quit
  • Increase in awareness of Quitline among community members and local health services
  • Increases in skills among those professionals in contact with Aboriginal and Torres Strait Islander peoples
  • Increases in Quitline referrals made throughout the TIS program.

Data sources described by CIRCA are:

  • Number of Quitline referrals
  • Number of referrals to other services for Quit support, e.g Quit support groups
  • Number of FTE positions with a focus on tobacco control
  • Number of FTE positions with a focus on tobacco control that are currently filled
  • Number and type of assistance provided to organisations to establish, maintain or improve brief interventions
  • Number and percentage of staff with a major focus on tobacco control/TIS staff who have completed formal training
  • Number and percentage and role of staff who do not have a major focus on tobacco control (e.g clinicians) who have completed formal training in brief advice, smoking cessation or tobacco control.

This indicator is concerned with community access to quit services. Access to healthcare services is only possible if:

  • appropriate and affordable services are available and have capacity
  • people are aware of and trust those services and feel the service provided meets their needs.

If both of these are in place, we should see good uptake of services.

Availability and capacity of appropriate services

From the data sources described by CIRCA, you can see there is simple data you can collect around the number and percentage of staff in your own organisation or others, who have:

  • a focus on tobacco control
  • undertaken training related to TIS.

This will provide information about the availability and capacity of appropriate services. CIRCA are interested in formal training courses which staff have done – specifically Quitskills.

You also need to report the number of organisations you have supported to develop or maintain their capacity to carry out brief interventions with clients. This could be providing or facilitating a training day for new staff, or a refresher course for staff in another organisation.

You should also collect more detailed data which describes the impact of training on staff knowledge and skills, and their confidence in supporting community members asking for advice (the third outcome under this indicator). You can collect this data through:

You should report this information as part of your case study or success story as it is not captured directly on the six monthly report. You can also report any relevant training carried out at your own or another organisation that was not Quitskills as part of this story.

Awareness and uptake of services

You will need to collect simple data (counts) of:

  • the number of written referrals made to Quitline
  • the number of referrals made to other services for Quit support.

A good way of finding out whether awareness of services has increased in (a) the community and (b) healthcare practitioners (the second outcome under this indicator) is through:

  • surveys with community members about smoking status, knowledge of quit services, and use of quit services
  • surveys of service providers about number of clients seen from the community and knowledge of available quit services.

This will provide good data to support your case study or success story. Examples of surveys you can use or adapt for your own region are available from the NBPU Survey Question Bank. You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:

You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 4: Reduced exposure to second hand smoke

The outcomes related to this indicator are:

  • Increase in smoke-free homes, workplaces and public spaces
  • Increase in activities aimed at minimising exposure to passive smoking.

Data sources described by CIRCA are:

  • Number and type of smoke-free space or workplace policies adopted and/or reviewed by relevant organisations
  • Number of local events organised to be smoke-free
  • Number and type of assistance provided to organisations to establish, maintain or improve a smoke-free policy
  • Number of smoke-free homes and/or pledges to keep homes smoke-free.

From these data sources you can see there is simple data you should collect around the number of events locally that you have supported to be smoke-free, how many organisations you have worked with to develop/improve smoke-free policies or to increase worker compliance with policies. You can also count how many homes and cars are smoke-free or, how many people have pledged to be smoke free. Good ways of doing this include:

  • surveys with local people or organisations, using a short questionnaire like this one: Smoke free homes questionnaire
  • pledges taken at events or through social media
  • observation of smoking behaviours in the community or in workplaces, for example using an environmental scan like the one provided by AH&MRC in their Aboriginal tobacco resistance toolkit.

You will also need to report on your own organisation’s smoke-free policy, including whether or not staff and board members comply with smoke-free policies:

  • indoors
  • outdoors, except within any designated areas
  • in work vehicles
  • in uniform
  • in work time.

You should also collect more in-depth data on understanding about what being smoke-free means and how important it is to people to try to be smoke-free. You can ask the smokers if they go without smokes at home, in the car, near their children, at work, or on other occasions, and how they handle smoke-free times. Good ways to get this information are:

  • interviews
  • focus groups.

Finally, you should describe how you have supported the communities in your region to become more smoke-free and how people have responded to these activities. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 5: Increased focus on priority groups, e.g pregnant women

The outcomes related to this indicator are:

  • Evidence based approaches are being used to reach priority groups
  • Increase in population health promotion activities targeting priority groups, particularly pregnant women.

Data sources described by CIRCA are:

  • Number of people in priority groups participating in/reached by population health promotion activities
  • Number and type of population health promotion activities that have a specific focus on pregnant women and other identified priority groups.

This indicator is concerned with whether your activities are reaching and engaging priority groups in your service area. Some of these groups have been described nationally (e.g. pregnant women), however there may also be other groups of people which are a priority for your community (e.g. older men).

A simple count of the priority groups you have identified for your service delivery area, and the number and type of targeted activities you do for each of these groups will also be important (e.g. targeted health education sessions, targeted social marketing or targeted events). You will also need to report the number of people participating in these activities. Activities of interest include:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

You should also collect more in-depth data to tell the story of the difference your activities are making for priority groups. This could include questions about new knowledge about the benefits of being smoke free, what being smoke-free means to them, and how important it is to your priority groups. You might ask mums-to-be what they are doing to keep their home and car smoke-free, if they are trying or intending to quit, and if so has anyone else in the family joined them on their smoke-free journey. Good ways to get this information include:

This will provide good data to support your case study or success story.

You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 6: Increased reach into communities

The outcomes related to this indicator are:

  • Increase in reach (including geographical reach) of population health promotion activities
  • Increase in reach to community members, including those who do not attend Aboriginal Community Controlled Health Services.

Data sources described by CIRCA are:

  • Number and location of activities conducted that extend geographical reach of activities
  • Number and type of population health promotion activities and partnerships that have a specific focus on people who do not attend Aboriginal Community Controlled Health Services.

This indicator focuses on geographical reach. You need to show that the activities you do extend across your contracted service area and include community members who do not routinely use ACCHO services. Simple numerical data (counts) will address this indicator, and include the number, type and location of your activities. You will also need to report the number of people who attended. Specific activities of interest include:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

You should also provide a detailed description of what you did to engage the communities across your service area, reflecting on what worked and what didn’t work and why. It will be useful to report this as part of your case study or success story. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.