You will need to provide a formal written report to the TIS program funder, the Australian Government Department of Health (DoH). The Department has provided a new reporting template which is also available for download from this page. Teams should note that contrary to the current Standard Grant Agreement, which has six and 12 month progress reporting, each progress report for TIS funded teams, will now cover a six month period.

TIS RTCG six month performance report template

In addition to the general tips for performance reporting below, please listen to this short presentation (20 minutes) which concerns completion of the performance report due September 2020

Performance reporting presentation with Penney Upton – 2020

*Click on ‘play slide show’ to get the full narrated version which only takes 20 mins

Notes from Performance Reporting workshops – 2020

This document provides advice about the indicators for performance reporting and answers to some general questions.

Tips for performance reporting

The performance report is your opportunity to show the difference your program is making in the communities across your region. As there is a limited number of words, you need to make them count. The NBPU has developed a number of tips for presenting the progress you are making, which should be read along with the guidance document and good practice example:

• Make sure the report includes the full range of information about what is changing in your region. A good way to do this is to make reporting a team effort. For example, if the organisation manager or TIS Coordinator is responsible for writing the report, they might want to sit with the team and a whiteboard and go through the key points before they start writing.
• Please ensure you list all of your activities under the appropriate TIS Performance Indicator. You need to show how each of your activities is making a difference in the community. Think about what has changed since the last report (outcomes).
• You should also add attachments to your report, e.g. examples or links to resources you have produced such as videos or posters.
• Rather than just saying that something has changed, show how you know you have made a difference (present the evidence and remember to provide some context around these changes).
• If you are reporting numbers (quantitative data) include the total number of people who you had contact with as well as the number who made a change (e.g. 14 of the 20 people attending your quit support group during this reporting period were able to cut down their smoking).
• When reporting the percentage of people making a change, remember that it is best practice to include the actual number as well (e.g. 70% of people attending your quit support group during this period were able to cut down their smoking, which is 14/20 people).
• When reporting stories (qualitative data) make sure you focus on the part of the story that shows how your activities are making a difference to smoking. Whilst it’s great that the netball team you sponsor has won all of their games recently, what matters for the performance report are things such as whether all the games have been smoke-free. If your players are ambassadors explain how they helped support your activities, messaging and so on.
• It’s really important that your report brings out the voices of the community and provides narratives where possible, so the report talks about real people and what the program is doing for them (so use short quotes or images to illustrate your report and provide ‘good news’ stories).
• Some TIS teams have been asking for advice about when someone’s status changes from a smoker to an ex-smoker (for reporting on activity outcomes). This will be discussed with the program evaluators once they come on board to ensure everyone is collecting appropriate data. However, in the meantime our stance on defining ex-smoker status is as follows:
  • the NBPU would recommend using the international convention which is four weeks (28 days). Whilst this may not seem like a long time, since those first few weeks are the worst and most of the cravings should be done by that point, anyone who lasts that long is well on the road to giving up the smokes for good. It is also useful to make a distinction between people who have been ex-smokers for <12 months (short term) and >12 months (long term quit). We suggest these two categories because we know that if you are able to abstain for 12 months, the chances of relapse are statistically much reduced and most people at that point will remain smoke-free. However you decide to categorise people, remember that it is important to define clearly the criteria you are using (four weeks, 12 months etc).

There are two types of data that you can collect, quantitative and qualitative:

Quantitative data is numerical and includes audits or counts. For example you could count the number of attendees at an activity, the number of activities carried out, the number of referrals to a quit support group.

Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. You might gather this through an interview when you ask people to tell you in their own words what they thought about an activity, or what the benefits of going to a quit support group have been. However qualitative data can take many forms – people might also be asked to express their feelings through photographs, paintings, drama, or other imagery.

The type of data you collect will depend partly on the question you want answered but also on the sort of data collection methods that will work best in your community.

Data collection methods

Counts

Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, smokers, non-smokers, male / female), and the outcome of participation (e.g. N who were referred to Quitline or quit support services). It is important to make sure you collect good quality data by accurate and consistent counting (making sure you count the same thing in the same way each time).

Case studies and success stories

Case studies and success stories are both ways of showing the impact of your project. This story of impact is really important and the reason your M&E has to go beyond just numerical counts. CIRCA have provided a space for you to tell these stories on the six-monthly performance report.

You should provide a story that shows the impact your activities have against each of the program indicators. The focus of this story might be:

• change for individuals or families
• a successful event
• training that you have facilitated
• organisational change.

Case studies and success stories are very similar. The main difference is the point of view from which the story is written:

• a case study is written from the participant point of view. So they might describe a family’s experience of going smoke free in the home and car. Or the thoughts and feelings of a group of workers after a brief intervention training session.  Many case studies focus on individual stories like the example provided here.
• a success story is written from the point of view of the organisation. An example can be found here.

Case studies and success stories should be specific and include relevant information (data) as evidence of the difference you have made. They can be illustrated with quotes and photos – but remember to get people’s permission to share their information. You can find out more about how to gather data for an individual case study here. You can also download templates for writing your case studies and success stories.

Focus group/yarning group

A focus group is a planned discussion with selected individuals, which you use to gather information, or opinions – for example on your activities – or to measure local knowledge of the benefits of quitting, what quit support is available and so on.

The group should be held in a familiar and comfortable environment so participants feel at ease. It is good practice to provide snack and drinks so that participants feel welcome and valued. This also sets a friendly and informal tone for the session. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).

It is important to know what you want to ask before going into the focus group – this Focus groups resource provides a guide to writing focus group questions. You can also find an example of questions for focus groups on TIS topics such as the second module in the Aboriginal tobacco resistance toolkit on workplace smoke-free policies. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.

A more detailed guide on running a focus group can be found here.

Interviews

Interviews are one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it.

As with a focus group you need to choose a setting for the interview where you and your interviewee feel comfortable, and where there will be no interruptions. You also need to have planned your questions, and thought about how to record what the person you are interviewing says. Will you take notes, or make an audio recording? Again you must get the person’s agreement if you want to record them.

This Semi-structured interviews resource can be a helpful guide to developing and writing interview questions for evaluation. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found on the Resources to monitor and evaluate your program page. More general information can be found in this Interviews factsheet.

Observation

There are two types of observation:

• direct observation – this is where the observer is an unidentified ‘fly on the wall’ (you will need permission if you want to observe people, whether in a workplace or school or other venue/location)
• participant observation – where the observer takes part in an activity with the participants and asks questions.

Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as the example provided by the Aboriginal Health & Medical Research Council (AH&MRC) in its Aboriginal tobacco resistance toolkit is a good way of recording your observations:

• Environmental scan key
• Environmental template key.

Questionnaire survey

This is a good method of gathering information from a large number of people. This information is collected by asking everyone the same questions. The kind of data you get will depend on the questions you ask. For example questions which give people a fixed set of answers (closed questions) will give you quantitative data:

You can then count how many people say ‘yes’ and how many say ‘no’. Or you could ask them to rate how satisfied they are with the support group on a scale of 1-5:

You can then count how many found the group very helpful, somewhat helpful and so on. However questions with yes/no answers or simple rating scales like this might not tell you everything you want to know. You might want to ask a question that allows people to give a free written response;

People can write what they like and this will give qualitative data. An advantage of this is that you will get much more detailed information about what people think. The disadvantage is that you might have to spend a lot of time organising answers so as to make sense of what is being said. The type of question you ask and the response choices you give people are therefore very important.

A helpful guide to developing and writing survey questions for evaluation can be found here. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found in the Survey question bank on the Resources to monitor and evaluate your program page along with a template for a post workshop feedback questionnaire that you could use to find out about participant satisfaction with activities. An example of using a questionnaire for evaluation can be seen in Box 1.

You might also want to find out more about what members of your project team and experts in the community think about the importance of your organisation’s activities for the community as a whole. To do this you can carry out something known as a Constituent survey of outcomes.

Story telling

Stories are a good way of capturing individual experiences. They are usually verbal, but can use a range of other methods such as writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.

A guide to using verbal story telling for evaluation, and how to use templates to collect significant change stories can be found here. You can also find a significant change story template here or on the Resources to monitor and evaluate your program page. Section four of the Talkin’ up good air: Australian Indigenous tobacco control resource kit also has some useful information and advice on gathering stories.

Examples of other creative approaches to gathering data including playing games, creating a storyboard or using photography can be found in this Creative strategies resource.

Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out. There are two evaluators of the National TIS program. Part A of the evaluation will be conducted by the Cultural & Indigenous Research Centre Australia (CIRCA). Part B of the evaluation will be conducted by the Australian National University (ANU).

National TIS Performance Indicators

Six National Indicators will be used to assess TIS program progress. Each of these indicators is described below, along with the data sources described by CIRCA and examples of data collection methods. Click on any of the data collection methods listed under each indicator to learn more about that specific data collection technique. You can also find more monitoring and evaluation resources to help you here: Resources to monitor and evaluate your program

NBPU TIS strongly recommends that you collect both quantitative and qualitative data. This will allow you to address the National Indicators in full and tell the story of your successes. It will also ensure strong data for your own internal monitoring and evaluation of your activities as part of your continual quality improvement (CQI) process.

National Indicator 1: Implementation of evidence-based population health promotion activities aimed at preventing the uptake of smoking and supporting the promotion of cessation

National Indicator 2: Partnerships and collaborations facilitate support for tobacco control

National Indicator 3: Increased access to Quit support through capacity building

This will provide good data to support your case study or success story. Examples of surveys you can use or adapt for your own region are available from the NBPU Survey Question Bank. You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:

• interviews with community members
• interviews with service providers
• focus/yarning groups with community members.

You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.

National Indicator 4: Reduced exposure to second hand smoke

National Indicator 5: Increased focus on priority groups, e.g pregnant women

Evaluation 2018-19 to 2021-22

A national evaluation of the TIS program will take place. The evaluation of the TIS program for the next phase of the program will be split into a two-part process. The Cultural and Indigenous Research Centre Australia (CIRCA) will conduct part A of the TIS evaluation. The organisation conducting part B is the Australian National University (ANU).

The Monitoring and Evaluation Framework for the TIS program 2018-19 to 2021-22 is available here. This document provides a framework for part A of the evaluation of the TIS program, and includes information on:

• the TIS program
• the purpose of the evaluation
• the questions the evaluation aims to answer
• how these questions will be answered.

Evaluation 2015 – 2018

A preliminary evaluation report of the TIS program was produced by CIRCA. The report provides an evaluation of the first year of the TIS program (Jan – Dec 2016), with a focus on the regional tobacco control grants delivering localised Aboriginal and Torres Strait Islander tobacco interventions. A number of recommendations are made to further enhance outcomes from the program, which are directed at the Government, the Department of Health, and key program stakeholders.

The final evaluation of the TIS program from 2015-2018 was produced by CIRCA, and released in March 2019. The evaluation looked at: how effective the TIS program is; how well it meets the needs of Aboriginal and Torres Strait Islander communities; and whether it’s on target to achieve its long term goals. The report found that the program is on track to achieve long-term objectives to reduce tobacco use among Aboriginal and Torres Strait Islander people.