TIS evaluation

Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out by the Cultural & Indigenous Research Centre Australia (CIRCA). A Monitoring and Evaluation Framework and a set of National Indicators have been developed by CIRCA to guide the TIS program evaluation. It is a requirement within TIS funding agreements that some of the data collected locally, related to local project outcomes, will feed into the broader National Evaluation, by reporting against the five National TIS Performance Indicators.

National TIS Performance Indicators

The Department of Health (DoH) has described five TIS program objectives and associated national indicators that will be used to assess TIS program progress. Each of these indicators is described below, along with examples of possible data sources and data collection methods.

National Indicator 1: Quality and reach of community engagement

Outcomes related to this indicator include the following:

  • increased community/regional involvement and support
  • increased leadership and advocacy role of community leaders in tobacco cessation
  • increased focus on priority groups
  • increased understanding by the community of the health impacts of smoking
  • increased understanding by the community of quitting pathways.

Examples of possible data sources provided by CIRCA include:

  • data on community groups/organisations involved in planning and implementing TIS activities
  • number and type of social marketing and community education activities undertaken (type of activities, target audience, and number reached)
  • feedback collected during social marketing and community education activities on message recall (qualitative)
  • number of community leaders, community groups and other sectors engaged in tobacco reduction work in the grant recipient region
  • information on how your TIS activities have engaged the community to support tobacco reduction (qualitative).

Numerical data on the number and type of groups/organisations or individuals involved in your project, or the number of people attending your activities/smoke-free events is a simple way of addressing this indicator. This indicator is concerned with increasing the reach of your activity, so if your project is working, the number of organisations or people involved in your activities should increase over time. If they don’t then you will need to think about why this is happening. It might be, for example, that there are no more local organisations to involve in your activities. In this case, maintaining different organisations’ involvement will become a key task for you.

Another way of measuring reach is to find out what people think about your TIS activities and if they have learnt anything new about smoking, such as ways to quit, or support for quitting that is available in your community. This is because reach is not just about the number of people who attend your activities, but about how many listen and understand the message. This kind of data can be either quantitative or qualitative, and can be collected through:

  • questionnaire surveys
  • interviews
  • focus/yarning groups.

You could also provide a detailed description of what you did to engage the community, reflecting on what worked and what didn’t work and why.

National Indicator 2: Organisations involved in tobacco reduction in this region

The outcome related to this indicator is:

  • collaborations and partnerships built between TIS operations and external support for quitting.

Examples of possible data sources provided by CIRCA include:

  • number and type of services where links have been made by TIS grant recipients
  • case studies on collaboration to show the impact of linking with local organisations to deliver integrated services, such as health checks (qualitative).

There is simple data you can collect around the number and type of organisations or services you have partnered with as part of your project activities. You can also collect more detailed data which describes the quality and extent of your collaboration with different partners. You can also show how your project partnerships have improved access to culturally appropriate support to quit, for example through case studies of the partnership journey.

You might also want to think about the quality of your relationships with different organisations you have entered into partnership with. Different ways of doing this include tools that involve the grant recipient organisation and their partner/s individually assessing the relationship and then coming together to discuss and move forward. Examples of these tools include:

National Indicator 3: Building capacity to support quitting

The outcome related to this indicator is:

  • increase in skills among those professionals in contact with Aboriginal and Torres Strait Islander people.

Examples of possible data sources provided by CIRCA include:

  • data on participation in the training of grant recipient staff and staff from other organisations
    • number and percent of staff in grant recipient organisation with major focus on tobacco control who have undertaken formal training
    • number, percent and roles of staff in grant recipient organisation who do not have main focus on tobacco control who have undertaken training
    • number, percent and roles of staff in other organisations who have undertaken formal training
    • number of community leaders who have undertaken formal training
  • feedback from training surveys
    • impact of TIS activities on understanding and confidence of community leaders
    • percentage of those participating in training who report increased understanding of health impacts and quitting pathways, increased confidence in ability to provide advice/support to clients and ability to facilitate/refer clients to appropriate quit pathways and deliver brief interventions.

From the above examples, you can see there is simple data you can collect around the number and/or percentage of staff in your own organisation or others, who have undertaken training related to TIS. This might be a formal training course on, for example, brief intervention. Or it might be more informal training about locally available quit support services (for example having a yarn with the local Quitline team).

You can also collect more detailed data which describes the impact of training on staff development (e.g. knowledge about smoking matters and quit services) and their confidence in supporting community members asking for advice. You can collect this data through:

  • questionnaire surveys
  • interviews
  • focus/yarning groups.

National Indicator 4: Referrals to appropriate quitting support

The outcome related to this indicator is:

  • improved access to culturally appropriate support to quit.

Examples of possible data sources provided by CIRCA include:

  • number of clients who are current smokers, ex-smokers or have never smoked
  • number of clients from TIS supported health services in the region whose clinical records include a record of advice or assistance to stop smoking being provided
  • number of clients from TIS supported health services in the region attending quit support groups (if relevant)
  • number of clients referred to Quitline services
  • number of pharmaceutical referrals for NRT products
  • information on how your TIS activity has supported access to quitting support (qualitative).

The specific numerical data that you might be able to provide for this indicator will depend on the records that your organisation keeps, or what information other services in the region (e.g. Quitline, clinical services) are able to share with you. If you are not able to access this kind of data, or your organisation does not keep detailed or accurate records, other ways to find out how many people in the community are accessing quit services include:

  • surveys with community members about smoking status, knowledge of quit services, use of quit services
  • surveys of service providers about number of clients seen from the community.

You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:

  • interviews with community members
  • interviews with service providers
  • focus/yarning groups with community members.

National Indicator 5: Supporting smoke-free environments

The outcome related to this indicator is:

  • increase in smoke-free homes, workplaces and public spaces.

Examples of possible data sources provided by CIRCA include:

  • number of smoke-free public areas and events established in your TIS region
  • number and type of workplaces with smoke-free policies implemented in your TIS region
  • number and type of smoke-free policies in your TIS region established, maintained or improved
    • in workplaces/organisations
    • at community events
    • in public areas
  • information on the promotion of smoke-free homes, workplaces and public spaces
    • cultural appropriateness
    • community awareness
    • community knowledge/involvement
  • information on how your TIS activities have supported these changes (qualitative).

From the above examples you can see there is simple data you can collect around the number of events locally that are smoke-free, how many organisations you have worked with to develop/improve smoke-free policies or to increase worker compliance with policies. You might also be able to count how many homes are smoke-free, or what people think about being smoke-free. Good ways of doing this include:

  • questionnaire surveys with local people or organisations
  • observation of smoking behaviours in the community or in workplaces.

You can also collect more in depth data on understanding about what being smoke-free means and how important it is to people to try to be smoke-free. You can ask the smokers if they go without smokes at home, in the car, near their children, at work, or on other occasions, and how they handle smoke-free times. Interviews and focus groups are good ways to get this information.

Finally you might want to describe how you have supported the communities in your region to become more smoke-free.