Reporting on your TIS activities

You will need to provide a formal written report to the TIS program funder, the Australian Government Department of Health (DoH). The Department has provided a new reporting template which is also available for download from this page. Teams should note that contrary to the current Standard Grant Agreement, which has six and 12 month progress reporting, each progress report for TIS funded teams, will now cover a six month period.

TIS RTCG six month performance report template

Tips for performance reporting

The performance report is your opportunity to show the difference your program is making in the communities across your region. As there is a limited number of words, you need to make them count. The NBPU has developed a number of tips for presenting the progress you are making, which should be read along with the guidance document and good practice example:

  • Make sure the report includes the full range of information about what is changing in your region. A good way to do this is to make reporting a team effort. For example, if the organisation manager or TIS Coordinator is responsible for writing the report, they might want to sit with the team and a whiteboard and go through the key points before they start writing.
  • Please ensure you list all of your activities under the appropriate TIS Performance Indicator. You need to show how each of your activities is making a difference in the community. Think about what has changed since the last report (outcomes).
  • You should also add attachments to your report, e.g. examples or links to resources you have produced such as videos or posters.
  • Rather than just saying that something has changed, show how you know you have made a difference (present the evidence and remember to provide some context around these changes).
  • If you are reporting numbers (quantitative data) include the total number of people who you had contact with as well as the number who made a change (e.g. 14 of the 20 people attending your quit support group during this reporting period were able to cut down their smoking).
  • When reporting the percentage of people making a change, remember that it is best practice to include the actual number as well (e.g. 70% of people attending your quit support group during this period were able to cut down their smoking, which is 14/20 people).
  • When reporting stories (qualitative data) make sure you focus on the part of the story that shows how your activities are making a difference to smoking. Whilst it’s great that the netball team you sponsor has won all of their games recently, what matters for the performance report are things such as whether all the games have been smoke-free. If your players are ambassadors explain how they helped support your activities, messaging and so on.
  • It’s really important that your report brings out the voices of the community and provides narratives where possible, so the report talks about real people and what the program is doing for them (so use short quotes or images to illustrate your report and provide ‘good news’ stories).
  • Some TIS teams have been asking for advice about when someone’s status changes from a smoker to an ex-smoker (for reporting on activity outcomes). This will be discussed with the program evaluators once they come on board to ensure everyone is collecting appropriate data. However, in the meantime our stance on defining ex-smoker status is as follows:
    • the NBPU would recommend using the international convention which is four weeks (28 days). Whilst this may not seem like a long time, since those first few weeks are the worst and most of the cravings should be done by that point, anyone who lasts that long is well on the road to giving up the smokes for good. It is also useful to make a distinction between people who have been ex-smokers for <12 months (short term) and >12 months (long term quit). We suggest these two categories because we know that if you are able to abstain for 12 months, the chances of relapse are statistically much reduced and most people at that point will remain smoke-free. However you decide to categorise people, remember that it is important to define clearly the criteria you are using (four weeks, 12 months etc).

Monitoring and evaluation methods

There are two types of data that you can collect, quantitative and qualitative:

Quantitative data is numerical and includes audits or counts. For example you could count the number of attendees at an activity, the number of activities carried out, the number of referrals to a quit support group.

Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. You might gather this through an interview when you ask people to tell you in their own words what they thought about an activity, or what the benefits of going to a quit support group have been. However qualitative data can take many forms – people might also be asked to express their feelings through photographs, paintings, drama, or other imagery.

The type of data you collect will depend partly on the question you want answered but also on the sort of data collection methods that will work best in your community.

Data collection methods

Counts

Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, smokers, non-smokers, male / female), and the outcome of participation (e.g. N who were referred to Quitline or quit support services). It is important to make sure you collect good quality data by accurate and consistent counting (making sure you count the same thing in the same way each time).

Box 1: Example of a case study

Newest family member makes grandma think about quitting again

Quitting smoking is a tough job, and not every attempt to do so will be a successful one. The arrival of Karen West’s newest grandson, Tarnah, has made this grandmother rethink her smoking habit. Karen is from the Kaladoon mob near Mt Isa, and, although not a heavy smoker, she has smoked about 15 cigarettes a day for nearly 40 years. Karen has tried once before to quit and managed to cut her smoking down to two cigarettes a day using the support of Zyban. ‘It was really easy, and I didn’t feel like I needed any more than those two a day’.

While Zyban helped Karen with her cravings, her first attempt at quitting only lasted a short while. She can’t recall what it was exactly that led her to smoking again, but remembers feeling that once she had started she might as well keep going. Apart from Zyban, she didn’t use any other strategies to support her quitting attempt and doesn’t know if this would have made any difference.

Asked what impact smoking has had on her health, Karen feels that her fitness has been affected by breathlessness and she suffers from high blood pressure. She realises that if she can give up smoking, her fitness and blood pressure would improve.

It is the impact her smoking has on those around her that concerns her most. While she doesn’t smoke around her grandchildren, she would like to be a non-smoking role model for them as well. Her daughter would be Karen’s biggest support should she make another quitting attempt, and has strictly enforced the no-smoking rule around baby Tarnah.

Although Karen does not have a particular quitting strategy in mind as yet, she is beginning to think about how quitting would improve her life and the importance of having a good reason to motivate her in this difficult challenge. She doesn’t think that she will qualify to get Zyban on the PBS again this year, since she has already used it, and will have to think about other ways she can cope with the cravings she might feel both physically and behaviourally.

We thank Karen for sharing her story and wish her luck and encouragement in making the next step along the road to quitting.

Material adapted from:

Case study

Case studies are a useful way of understanding how different parts of a project have worked together to produce a particular outcome. Case studies focus on an individual person, workplace, process or type of activity. For example you might focus on how one person in the community has managed to successfully quit:

  • What helped them decide to quit?
  • What supports did they use to quit?
  • How did your project help in this journey?

Or you could reflect on how you were able to increase smoke-free environments in your region:

  • Where have smoke-free environments increased? (e.g. homes, workplaces, community events or venues)
  • Which of your activities helped this change happen? (e.g. running smoke-free events, supporting workplaces to put smoke-free policies in place, working with families to go smoke-free)
  • What were the challenges you faced and how did you overcome them? (e.g. where workers were reluctant to go smoke-free, gaining support from CEOs and providing education on benefits of being smoke-free and providing tips for staying smoke-free were most effective).

An example of a case study is provided in Box 1. You can find out more information in this 

resource.

Focus group/yarning group

A focus group is a planned discussion with selected individuals, which you use to gather information, or opinions – for example on your activities – or to measure local knowledge of the benefits of quitting, what quit support is available and so on.

The group should be held in a familiar and comfortable environment so participants feel at ease. It is good practice to provide snack and drinks so that participants feel welcome and valued. This also sets a friendly and informal tone for the session. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).

It is important to know what you want to ask before going into the focus group – this Focus groups resource provides a guide to writing focus group questions. You can also find an example of questions for focus groups on TIS topics such as the second module in the Aboriginal tobacco resistance toolkit on workplace smoke-free policies. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.

A more detailed guide on running a focus group can be found here.

Interviews

Interviews are one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it.

As with a focus group you need to choose a setting for the interview where you and your interviewee feel comfortable, and where there will be no interruptions. You also need to have planned your questions, and thought about how to record what the person you are interviewing says. Will you take notes, or make an audio recording? Again you must get the person’s agreement if you want to record them.

This Semi-structured interviews resource can be a helpful guide to developing and writing interview questions for evaluation. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found on the Resources to monitor and evaluate your program page. More general information can be found in this Interviews factsheet.

Observation

There are two types of observation:

  • direct observation – this is where the observer is an unidentified ‘fly on the wall’ (you will need permission if you want to observe people, whether in a workplace or school or other venue/location)
  • participant observation – where the observer takes part in an activity with the participants and asks questions.

Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as the example provided by the Aboriginal Health & Medical Research Council (AH&MRC) in its Aboriginal tobacco resistance toolkit is a good way of recording your observations:

Questionnaire survey

This is a good method of gathering information from a large number of people. This information is collected by asking everyone the same questions. The kind of data you get will depend on the questions you ask. For example questions which give people a fixed set of answers (closed questions) will give you quantitative data:

Question: Did you find the quit support group helpful?
Answer: Yes/No

You can then count how many people say ‘yes’ and how many say ‘no’. Or you could ask them to rate how satisfied they are with the support group on a scale of 1-5:

Question: How helpful did you find the support group?
Answer: 1 2 3 4 5
Very Helpful Somewhat Helpful  Neither helpful nor unhelpful Somewhat unhelpful Very unhelpful

You can then count how many found the group very helpful, somewhat helpful and so on. However questions with yes/no answers or simple rating scales like this might not tell you everything you want to know. You might want to ask a question that allows people to give a free written response;

Question: What did you think of the quit support group?
Answer: ____________________________________

People can write what they like and this will give qualitative data. An advantage of this is that you will get much more detailed information about what people think. The disadvantage is that you might have to spend a lot of time organising answers so as to make sense of what is being said. The type of question you ask and the response choices you give people are therefore very important.

A helpful guide to developing and writing survey questions for evaluation can be found here. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found in the Survey question bank on the Resources to monitor and evaluate your program page along with a template for a post workshop feedback questionnaire that you could use to find out about participant satisfaction with activities. An example of using a questionnaire for evaluation can be seen in Box 2.

You might also want to find out more about what members of your project team and experts in the community think about the importance of your organisation’s activities for the community as a whole. To do this you can carry out something known as a Constituent survey of outcomes.

Box 2: Using a questionnaire to evaluate an activity

During the delivery of ‘Operation Smoke Signals’, an evaluation was done to see whether it was meeting the project’s objectives. Using a questionnaire, participants were asked for their feedback about the quitting course to measure their satisfaction levels. They were asked questions, such as:

  • Was the venue suitable, the session times convenient and the facilities adequate?
  • Did they find the content useful?
  • What did they like about the course?
  • What didn’t they like about the course?
  • How would they improve the course?

The results of the questionnaire showed that participants felt comfortable taking part in the course, that they found the staff friendly and the topics covered helpful. However, some had to drop out because they had problems with transportation and child care. To improve the running of the course, the local community bus is now picking up those participants who need a lift, and child care is being offered by the health service.

Material adapted from:

Story telling

Stories are a good way of capturing individual experiences. They are usually verbal, but can use a range of other methods such as writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.

A guide to using verbal story telling for evaluation, and how to use templates to collect significant change stories can be found here. You can also find a significant change story template here or on the Resources to monitor and evaluate your program page. Section four of the Talkin’ up good air: Australian Indigenous tobacco control resource kit also has some useful information and advice on gathering stories.

Examples of other creative approaches to gathering data including playing games, creating a storyboard or using photography can be found in this Creative strategies resource.

TIS evaluation

Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out. There are two evaluators of the National TIS program. Part A of the evaluation will be conducted by the Cultural & Indigenous Research Centre Australia (CIRCA). Part B of the evaluation will be conducted by the Australian National University (ANU).

National TIS Performance Indicators

Six National Indicators will be used to assess TIS program progress. Each of these indicators is described below, along with the data sources described by CIRCA and examples of data collection methods. Click on any of the data collection methods listed under each indicator to learn more about that specific data collection technique. You can also find more monitoring and evaluation resources to help you here: Resources to monitor and evaluate your program

NBPU TIS strongly recommends that you collect both quantitative and qualitative data. This will allow you to address the National Indicators in full and tell the story of your successes. It will also ensure strong data for your own internal monitoring and evaluation of your activities as part of your continual quality improvement (CQI) process.

National Indicator 1: Implementation of evidence-based population health promotion activities aimed at preventing the uptake of smoking and supporting the promotion of cessation

The outcomes related to this indicator are:

  • Increased community involvement in and support for initiatives to reduce the uptake of smoking and increase sustained cessation
  • Increased leadership and advocacy role of community leaders in tobacco control
  • Increased understanding by the community of the health impacts of smoking
  • Population health promotion activities are locally relevant and have community support.

Data sources described by CIRCA are:

  • Number of community members participating in population health promotion activities and events
  • Number of community leaders participating in population health promotion activities and events
  • Number and type of evidence-based population health promotion activities including social marketing, community education and community engagement
  • Number and reach as evidenced by social media analytics, other media activities, and production/distribution of health promotion materials.

This indicator focuses on two aspects of your activity:

  • the extent of community reach and engagement of your activities (please note however that any activities targeted on either priority groups, people who do not attend ACCHS, or to increase geographical reach are not to be reported here, as these are the focus of Indicators 5 and 6)
  • the type of evidence-based population health promotion activities you do.

Community reach and engagement

Numerical data (counts) of the number of community members and community leaders involved in or attending your activities/smoke-free events is a simple way of addressing this indicator. Social media analytics are also important. This indicator is concerned with increasing the reach of your activity, so if your project is working, the number of community members and leaders involved in your activities should increase over time. If they don’t then you will need to think about why this is happening. Do you need to find new ways to engage community interest? You might also want to think about how representative of community the people you do see at your activities are. For example, are there certain groups in your community you are just not reaching, such as full-time workers? How could you make sure your activities are accessible to everyone? Or it might be that all community leaders are now involved in your activities. In this case, maintaining their involvement will become a key task for you.

You also need to find out what people think about your TIS activities and if they have learnt anything new about smoking, such as ways to quit, or support for quitting that is available in your community. This is because reach is not just about the number of people who attend your activities, but about how many listen to and understand the message (the third outcome under this indicator). This kind of data can be either quantitative or qualitative, and can be collected through:

As this data is not directly captured by the six monthly progress report, it will be important to provide this information as part of the story around your activity (case study or success story). You should also give a description of what you did to engage the community, reflecting on what worked and what didn’t work and why. What are the strengths of your community and its leaders that have made this activity a success? You will also need to consider what the risks and challenges have been for your team, and how you have worked to overcome them.

Evidence-based population health promotion activities

Simple numerical data (counts) for each type of evidence-based population health promotion activity you do will address this part of the indicator. You will also need to record the location of your activities (e.g. neighbourhood, town or region). Activities of specific interest under this indicator are:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

To avoid repetition, partnership working and collaborations developed do not need to be reported here. This information will be captured under Indicator 2. It will also be important to demonstrate (through your Action Work Plan) that these activities are evidence-based.

National Indicator 2: Partnerships and collaborations facilitate support for tobacco control

The outcome related to this indicator is:

  • Collaborations and partnerships built between TIS organisations and external support for tobacco control initiatives.

Data sources described by CIRCA are:

  • Number and type of organisations involved in planning/implementing TIS activities
  • Number and type of collaborative projects/partnership activities
  • Number and type of partnerships with local service providers to enable increased geographical reach
  • Number and type of partnerships with local service providers to enable increased reach to priority groups.

There is simple data you can collect around the number and type of organisations, services or individuals you have partnered with as part of your population health promotion activities. Of specific interest are partnerships with:

  • mainstream services
  • ACCHS
  • schools
  • community organisations/sporting clubs
  • local, state or federal government organisations
  • community leaders or community champions
  • networks/interagency groups.

As you can see from the data sources described by CIRCA, it will be useful to think about the purpose of these partnerships, for example if they increase your reach into areas within the region you are contracted to service, or priority populations including pregnant women, or people who do not typically attend ACCHS. You should also collect more detailed data which describes the quality and extent of your collaboration with different partners and how these partnerships have improved geographical or priority group reach, for example through case studies of the partnership journey.

Think about the quality of your relationships with the different organisations you have entered into partnership with as well. Different ways of doing this include tools that involve the grant recipient organisation and their partner/s individually assessing the relationship and then coming together to discuss and move forward. Examples of these tools include:

This information will be an important part of providing the story around your activity (case study or success story). What did you do to engage the organisations, what worked, what didn’t work and why? What are the strengths of your organisation and those you have partnered with that have made this activity a success? What were the risks and challenges and how have you worked to overcome them?

National Indicator 3: Increased access to Quit support through capacity building

The outcomes related to this indicator are:

  • Improved access to culturally appropriate support to quit
  • Increase in awareness of Quitline among community members and local health services
  • Increases in skills among those professionals in contact with Aboriginal and Torres Strait Islander peoples
  • Increases in Quitline referrals made throughout the TIS program.

Data sources described by CIRCA are:

  • Number of Quitline referrals
  • Number of referrals to other services for Quit support, e.g Quit support groups
  • Number of FTE positions with a focus on tobacco control
  • Number of FTE positions with a focus on tobacco control that are currently filled
  • Number and type of assistance provided to organisations to establish, maintain or improve brief interventions
  • Number and percentage of staff with a major focus on tobacco control/TIS staff who have completed formal training
  • Number and percentage and role of staff who do not have a major focus on tobacco control (e.g clinicians) who have completed formal training in brief advice, smoking cessation or tobacco control.

This indicator is concerned with community access to quit services. Access to healthcare services is only possible if:

  • appropriate and affordable services are available and have capacity
  • people are aware of and trust those services and feel the service provided meets their needs.

If both of these are in place, we should see good uptake of services.

Availability and capacity of appropriate services

From the data sources described by CIRCA, you can see there is simple data you can collect around the number and percentage of staff in your own organisation or others, who have:

  • a focus on tobacco control
  • undertaken training related to TIS.

This will provide information about the availability and capacity of appropriate services. CIRCA are interested in formal training courses which staff have done – specifically Quitskills.

You also need to report the number of organisations you have supported to develop or maintain their capacity to carry out brief interventions with clients. This could be providing or facilitating a training day for new staff, or a refresher course for staff in another organisation.

You should also collect more detailed data which describes the impact of training on staff knowledge and skills, and their confidence in supporting community members asking for advice (the third outcome under this indicator). You can collect this data through:

You should report this information as part of your case study or success story as it is not captured directly on the six monthly report. You can also report any relevant training carried out at your own or another organisation that was not Quitskills as part of this story.

Awareness and uptake of services

You will need to collect simple data (counts) of:

  • the number of written referrals made to Quitline
  • the number of referrals made to other services for Quit support.

A good way of finding out whether awareness of services has increased in (a) the community and (b) healthcare practitioners (the second outcome under this indicator) is through:

  • surveys with community members about smoking status, knowledge of quit services, and use of quit services
  • surveys of service providers about number of clients seen from the community and knowledge of available quit services.

This will provide good data to support your case study or success story. Examples of surveys you can use or adapt for your own region are available from the NBPU Survey Question Bank. You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:

You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 4: Reduced exposure to second hand smoke

The outcomes related to this indicator are:

  • Increase in smoke-free homes, workplaces and public spaces
  • Increase in activities aimed at minimising exposure to passive smoking.

Data sources described by CIRCA are:

  • Number and type of smoke-free space or workplace policies adopted and/or reviewed by relevant organisations
  • Number of local events organised to be smoke-free
  • Number and type of assistance provided to organisations to establish, maintain or improve a smoke-free policy
  • Number of smoke-free homes and/or pledges to keep homes smoke-free.

From these data sources you can see there is simple data you should collect around the number of events locally that you have supported to be smoke-free, how many organisations you have worked with to develop/improve smoke-free policies or to increase worker compliance with policies. You can also count how many homes and cars are smoke-free or, how many people have pledged to be smoke free. Good ways of doing this include:

  • surveys with local people or organisations, using a short questionnaire like this one: Smoke free homes questionnaire
  • pledges taken at events or through social media
  • observation of smoking behaviours in the community or in workplaces, for example using an environmental scan like the one provided by AH&MRC in their Aboriginal tobacco resistance toolkit.

You will also need to report on your own organisation’s smoke-free policy, including whether or not staff and board members comply with smoke-free policies:

  • indoors
  • outdoors, except within any designated areas
  • in work vehicles
  • in uniform
  • in work time.

You should also collect more in-depth data on understanding about what being smoke-free means and how important it is to people to try to be smoke-free. You can ask the smokers if they go without smokes at home, in the car, near their children, at work, or on other occasions, and how they handle smoke-free times. Good ways to get this information are:

  • interviews
  • focus groups.

Finally, you should describe how you have supported the communities in your region to become more smoke-free and how people have responded to these activities. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 5: Increased focus on priority groups, e.g pregnant women

The outcomes related to this indicator are:

  • Evidence based approaches are being used to reach priority groups
  • Increase in population health promotion activities targeting priority groups, particularly pregnant women.

Data sources described by CIRCA are:

  • Number of people in priority groups participating in/reached by population health promotion activities
  • Number and type of population health promotion activities that have a specific focus on pregnant women and other identified priority groups.

This indicator is concerned with whether your activities are reaching and engaging priority groups in your service area. Some of these groups have been described nationally (e.g. pregnant women), however there may also be other groups of people which are a priority for your community (e.g. older men).

A simple count of the priority groups you have identified for your service delivery area, and the number and type of targeted activities you do for each of these groups will also be important (e.g. targeted health education sessions, targeted social marketing or targeted events). You will also need to report the number of people participating in these activities. Activities of interest include:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

You should also collect more in-depth data to tell the story of the difference your activities are making for priority groups. This could include questions about new knowledge about the benefits of being smoke free, what being smoke-free means to them, and how important it is to your priority groups. You might ask mums-to-be what they are doing to keep their home and car smoke-free, if they are trying or intending to quit, and if so has anyone else in the family joined them on their smoke-free journey. Good ways to get this information include:

This will provide good data to support your case study or success story.

You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 6: Increased reach into communities

The outcomes related to this indicator are:

  • Increase in reach (including geographical reach) of population health promotion activities
  • Increase in reach to community members, including those who do not attend Aboriginal Community Controlled Health Services.

Data sources described by CIRCA are:

  • Number and location of activities conducted that extend geographical reach of activities
  • Number and type of population health promotion activities and partnerships that have a specific focus on people who do not attend Aboriginal Community Controlled Health Services.

This indicator focuses on geographical reach. You need to show that the activities you do extend across your contracted service area and include community members who do not routinely use ACCHO services. Simple numerical data (counts) will address this indicator, and include the number, type and location of your activities. You will also need to report the number of people who attended. Specific activities of interest include:

  • social marketing campaigns
  • social media activities
  • development and distribution of resources
  • community education
  • community engagement (including event attendance/support).

You should also provide a detailed description of what you did to engage the communities across your service area, reflecting on what worked and what didn’t work and why. It will be useful to report this as part of your case study or success story. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

Evaluation documents

Evaluation 2018 – 19 to 2021 – 22

A national evaluation of the TIS program will take place. The evaluation of the TIS program for the next phase of the program will be split into a two-part process. The Cultural and Indigenous Research Centre Australia (CIRCA) will conduct part A of the TIS evaluation. The organisation conducting part B is the Australian National University (ANU).

The Monitoring and Evaluation Framework for the TIS program 2018-19 to 2021-22 is available here. This document provides a framework for part A of the evaluation of the TIS program, and includes information on:

  • the TIS program
  • the purpose of the evaluation
  • the questions the evaluation aims to answer
  • how these questions will be answered.

Evaluation 2015 – 2018

preliminary evaluation report of the TIS program was produced by CIRCA. The report provides an evaluation of the first year of the TIS program (Jan – Dec 2016), with a focus on the regional tobacco control grants delivering localised Aboriginal and Torres Strait Islander tobacco interventions. A number of recommendations are made to further enhance outcomes from the program, which are directed at the Government, the Department of Health, and key program stakeholders.

The final evaluation of the TIS program from 2015-2018 was produced by CIRCA, and released in March 2019. The evaluation looked at: how effective the TIS program is; how well it meets the needs of Aboriginal and Torres Strait Islander communities; and whether it’s on target to achieve its long term goals. The report found that the program is on track to achieve long-term objectives to reduce tobacco use among Aboriginal and Torres Strait Islander people.

Monitoring and evaluation

How will you determine if your program is working? This section will provide you with the tools to monitor and evaluate your programs.

Monitoring and evaluating your project

The processes of monitoring and evaluation use carefully planned and well-thought-out methods to measure the success of a project (or program) in meeting its goals. They are an important part of the project management process, because they provide:

  • evidence of what is working
  • guidance on what could be done better, which can be used to improve your project’s performance (progress towards and achievement of results)
  • a check on whether you are meeting your project aims
  • feedback to everyone involved in the project, including community members and partner organisations
  • compliance with funding body reporting requirements.

Key terms used when talking about monitoring and evaluation are shown in Box 1.

Box 1: Monitoring and evaluation key terms

  • performance: what the project is achieving (observable results)
  • measurement: how we determine the impact of a project or program on intended outcomes (e.g. using a questionnaire to find out how many people have smoke-free homes or conducting interviews to find out how people keep their homes smoke-free)
  • indicator: measures that show the extent of progress toward outcomes, especially differences in the lives of the people the project is working for
  • data collection: process used to gather evidence (e.g. giving smoke-free event participants a questionnaire survey)
  • output: what the project is producing with its resources (e.g. a specific activity, product or service)
  • outcome: results and impacts of the project (e.g. a percentage reduction in smoking, a change in behaviour).

Monitoring and evaluation are related processes, but each has a different focus:

  • monitoring provides the organisation and key stakeholders with early indicators of progress, and usually focuses on project outputs (the activities that a project has delivered)
  • evaluation systematically assesses progress towards achieving outcomes.

Table 1 provides examples of different outputs and outcomes for the TIS program. Monitoring outputs relies on describing and counting project activities and the number of people who come to events. In the past TIS reporting focused only on what is included in the outputs column – ‘what we do and who we reach.’ Now, however, you are being asked to think more about ‘what difference do our activities make?’ This is a question about your project outcomes.

Table 1: Examples of outputs and outcomes for TIS

Outputs Outcomes

Smoke-free workshops delivered to 100 workers in 10 community organisations

85 workers have increased knowledge of benefits of smoke-free workplace and increased commitment to being smoke-free at work

 8 Organisations are smoke-free

Brief intervention training provided to all staff (N=20) in TIS-funded organisation

20 staff have increased skills to support TIS activities

Most staff describe increased confidence when working with community members

You can find out more about the ‘how, who and what’ of TIS project evaluation in Box 2.

Box 2: The how, who and what of evaluation

How to plan an evaluation?

It’s a good idea to work out an evaluation plan for your project, so you can keep track of the evaluation. In your plan consider the following points:

  • What are the reasons for evaluating your project?
  • Who will read or listen to your evaluation?
  • What sort of resources will you need for your evaluation?
  • What are your evaluation questions?

Who will do the evaluation?

Work out who will be responsible for organising and writing up the evaluation.

Who should participate in an evaluation?

Good evaluations involve those who are interested and affected by your project. Involving people from the earliest stages of the project’s development to the final evaluation can encourage local communities to set up, control and own a project. Putting together a working group (a group appointed to study or report on a particular question) of community members also helps the evaluation process. A working group brings the values and shared interests of the community into the process. Often they are also the ones who are best placed to talk about the needs of their community.

What should you evaluate?

What aspect of the project do you want to evaluate? For example, do you want  to know if the project’s objectives were met? Or do you want to know what people liked about the project? There are lots of evaluation questions you could use. Examples of the sorts of questions you might use to answer different questions are provided in the 

.

What sort of resources will you need for your evaluation?

Be realistic about what resources are available to undertake the evaluation. This includes funding, time, staff, salaries, material, equipment and operating costs.

Source: material adapted from Kruger K, McMillan N, Russ P and Smallwood H (2007) Talkin’ up good air: Australian Indigenous tobacco control resource kit. Melbourne: Centre for Excellence in Indigenous Tobacco Control

[14810]

Or for a more in-depth general discussion of behaviour change evaluation and definitions of some other terms used in monitoring and evaluation, you might find the Evaluation toolbox useful.

When should I think about monitoring and evaluation?

Monitoring and evaluation should be built into your project from the start. It is also a requirement of TIS program funding to measure your outcomes every six months for Indigenous Australians’ Health Programme (IAHP) grant agreement progress reporting. The

 developed by HealthInfoNet on behalf of the Australian Government (2012), describes a simple six step process to building evaluation into a project plan:

    1. Plan the project – what do you want to achieve, who do you want to involve, what will you do and how will you do it?
    2. Plan an evaluation – decide and consult about what you will measure, identify key questions, identify local sensitivities, identify good processes, and allocate necessary resources.
    3. Design the evaluation – decide on the methods you will use to collect the information you want, such as counting how many attend a smoke-free community event (output), giving out a short questionnaire asking what people thought of the event, and whether those who attended learnt anything about smoking harms, benefits of quitting, or quit support available (outcomes).
    4. Collect and record your information – do this systemically to get a true picture of what your project is achieving.
    5. Analyse your information – see if the program is achieving what you intended, or whether are are any unexpected outcomes, identify the lessons to learn. Decide if you need to make any changes to your project to keep on track.
    6. Provide feedback on your findings – let the people involved in the project such as the community, your organisation and the participants know about what was achieved.

Examples of ways to present your data using charts and dashboards for good visual impacts can be found in this monitoring and evaluation resource or on the Resources to monitor and evaluate your program page here.

Figure 1: Keeping your project on track

 

 

Figure 1 shows how monitoring and evaluation forms a central part of your project. You can use the information that you collect to make sure activities are on track and that you are achieving the outcomes you intended, adapting and improving your project as you go. This process of using monitoring and evaluation data to improve your project or activities as you go is known as continual quality improvement (CQI). If you are working in a clinical setting you might be interested in the 

developed by Menzies School of Health Research in association with One21Seventy. More tips for how to use monitoring and evaluation data are provided in Box 3.

 

 

 

 

Box 3: Using monitoring and evaluation data

Projects do not always run as planned or as expected, and are a learning experience for all those involved. Monitoring your activities and evaluating outcomes will help you keep track of progress, identify important changes and make it easier to see what you have achieved.  Answering the following questions may help keep your project on track:

  • What do the early results mean in terms of what you are trying to achieve?
  • Should you be changing the way you are doing things?
  • How do you continue the positive changes you are achieving?
  • Are you doing things in the best possible way?
  • Are you talking to the right people about the progress of your activities?
  • Are you making progress, making a difference?

Source: adapted from Australian Indigenous HealthInfoNet (2012) [2]


[1] This material is used with permission from the Commonwealth of Australia. Talkin’ up good air contains materials that were contributed by Quit Victoria (Anti-Cancer Council of Victoria), Apunipima Cape York Health Council, National Heart Foundation of Australia (NSW Division), Council of Social Services of New South Wales and Queensland Health and which remain their property.

[2] Australian Indigenous HealthInfoNet (2012) Healthy, Deadly and Strong: Healthy Lifestyle Worker Toolkit. Perth, WA: Australian Indigenous HealthInfoNet

References