You will need to provide a formal written report to the TIS program funder, the Australian Government Department of Health (DoH). The Department has provided a new reporting template which is also available for download from this page. Teams should note that contrary to the current Standard Grant Agreement, which has six and 12 month progress reporting, each progress report for TIS funded teams, will now cover a six month period.
TIS RTCG six month performance report template
Please download this template to use for your performance reporting.
Performance reporting review process – June 2022
This PDF outlines the performance reporting review process.
Monitoring and reporting on social media activities with Penney Upton (narrated slides)
Please listen to this short presentation (20 minutes) which discusses the monitoring and reporting of social media activities, including reach, engagement and impact. Please note: The presentation will start playing automatically when opened.
Tips for completing your performance report – July 2022
This document provides general advice and answers frequently asked questions about the Indicators for performance reporting.
There are two types of data that you can collect, quantitative and qualitative:
Quantitative data is numerical and includes audits or counts. For example you could count the number of attendees at an activity, the number of activities carried out, the number of referrals to a quit support group.
Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. You might gather this through an interview when you ask people to tell you in their own words what they thought about an activity, or what the benefits of going to a quit support group have been. However qualitative data can take many forms – people might also be asked to express their feelings through photographs, paintings, drama, or other imagery.
The type of data you collect will depend partly on the question you want answered but also on the sort of data collection methods that will work best in your community.
Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, people who smoke, people who do not smoke, male / female), and the outcome of participation (e.g. N who were referred to Quitline or quit support services). It is important to make sure you collect good quality data by accurate and consistent counting (making sure you count the same thing in the same way each time).
Case studies and success stories are both ways of showing the impact of your project. This story of impact is really important and the reason your M&E has to go beyond just numerical counts. CIRCA have provided a space for you to tell these stories on the six-monthly performance report.
You should provide a story that shows the impact your activities have against each of the program indicators. The focus of this story might be:
Case studies and success stories are very similar. The main difference is the point of view from which the story is written:
Case studies and success stories should be specific and include relevant information (data) as evidence of the difference you have made. They can be illustrated with quotes and photos – but remember to get people’s permission to share their information. You can find out more about how to gather data for an individual case study here. You can also download templates for writing your case studies and success stories.
A focus group is a planned discussion with selected individuals, which you use to gather information, or opinions – for example on your activities – or to measure local knowledge of the benefits of quitting, what quit support is available and so on.
The group should be held in a familiar and comfortable environment so participants feel at ease. It is good practice to provide snack and drinks so that participants feel welcome and valued. This also sets a friendly and informal tone for the session. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).
It is important to know what you want to ask before going into the focus group – this Focus groups resource provides a guide to writing focus group questions. You can also find an example of questions for focus groups on TIS topics such as the second module in the Aboriginal tobacco resistance toolkit on workplace smoke-free policies. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.
A more detailed guide on running a focus group can be found here.
Interviews are one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it.
As with a focus group you need to choose a setting for the interview where you and your interviewee feel comfortable, and where there will be no interruptions. You also need to have planned your questions, and thought about how to record what the person you are interviewing says. Will you take notes, or make an audio recording? Again you must get the person’s agreement if you want to record them.
This Semi-structured interviews resource can be a helpful guide to developing and writing interview questions for evaluation. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found on the Resources to monitor and evaluate your program page. More general information can be found in this Interviews factsheet.
There are two types of observation:
Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as the example provided by the Aboriginal Health & Medical Research Council (AH&MRC) in its Aboriginal tobacco resistance toolkit is a good way of recording your observations:
This is a good method of gathering information from a large number of people. This information is collected by asking everyone the same questions. The kind of data you get will depend on the questions you ask. For example questions which give people a fixed set of answers (closed questions) will give you quantitative data:
You can then count how many people say ‘yes’ and how many say ‘no’. Or you could ask them to rate how satisfied they are with the support group on a scale of 1-5:
You can then count how many found the group very helpful, somewhat helpful and so on. However questions with yes/no answers or simple rating scales like this might not tell you everything you want to know. You might want to ask a question that allows people to give a free written response;
People can write what they like and this will give qualitative data. An advantage of this is that you will get much more detailed information about what people think. The disadvantage is that you might have to spend a lot of time organising answers so as to make sense of what is being said. The type of question you ask and the response choices you give people are therefore very important.
A helpful guide to developing and writing survey questions for evaluation can be found here. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found in the Survey question bank on the Resources to monitor and evaluate your program page along with a template for a post workshop feedback questionnaire that you could use to find out about participant satisfaction with activities. An example of using a questionnaire for evaluation can be seen in Box 1.
Box 1: Using a questionnaire to evaluate an activity
During the delivery of ‘Operation Smoke Signals’, an evaluation was done to see whether it was meeting the project’s objectives. Using a questionnaire, participants were asked for their feedback about the quitting course to measure their satisfaction levels. They were asked questions, such as:
The results of the questionnaire showed that participants felt comfortable taking part in the course, that they found the staff friendly and the topics covered helpful. However, some had to drop out because they had problems with transportation and child care. To improve the running of the course, the local community bus is now picking up those participants who need a lift, and child care is being offered by the health service.
Material adapted from:
Stories are a good way of capturing individual experiences. They are usually verbal, but can use a range of other methods such as writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.
A guide to using verbal story telling for evaluation, and how to use templates to collect significant change stories can be found here. You can also find a significant change story template here or on the Resources to monitor and evaluate your program page. Section four of the Talkin’ up good air: Australian Indigenous tobacco control resource kit also has some useful information and advice on gathering stories.
Examples of other creative approaches to gathering data including playing games, creating a storyboard or using photography can be found in this Creative strategies resource.
Monitoring and evaluation will take place at two levels for the TIS program. It is expected that all TIS funded organisations will monitor their activities and evaluate project outcomes at a local level. In addition, a national evaluation of all the components of the TIS program will be carried out. There are two evaluators of the National TIS program. Part A of the evaluation will be conducted by the Cultural & Indigenous Research Centre Australia (CIRCA). Part B of the evaluation will be conducted by the Australian National University (ANU).
Six National Indicators will be used to assess TIS program progress. Each of these indicators is described below, along with the data sources described by CIRCA and examples of data collection methods. Click on any of the data collection methods listed under each indicator to learn more about that specific data collection technique. You can also find more monitoring and evaluation resources to help you here: Resources to monitor and evaluate your program
NBPU TIS strongly recommends that you collect both quantitative and qualitative data. This will allow you to address the National Indicators in full and tell the story of your successes. It will also ensure strong data for your own internal monitoring and evaluation of your activities as part of your continual quality improvement (CQI) process.
The outcomes related to this indicator are:
Data sources described by CIRCA are:
This indicator focuses on two aspects of your activity:
Numerical data (counts) of the number of community members and community leaders involved in or attending your activities/smoke-free events is a simple way of addressing this indicator. Social media analytics are also important. This indicator is concerned with increasing the reach of your activity, so if your project is working, the number of community members and leaders involved in your activities should increase over time. If they don’t then you will need to think about why this is happening. Do you need to find new ways to engage community interest? You might also want to think about how representative of community the people you do see at your activities are. For example, are there certain groups in your community you are just not reaching, such as full-time workers? How could you make sure your activities are accessible to everyone? Or it might be that all community leaders are now involved in your activities. In this case, maintaining their involvement will become a key task for you.
You also need to find out what people think about your TIS activities and if they have learnt anything new about smoking, such as ways to quit, or support for quitting that is available in your community. This is because reach is not just about the number of people who attend your activities, but about how many listen to and understand the message (the third outcome under this indicator). This kind of data can be either quantitative or qualitative, and can be collected through:
As this data is not directly captured by the six monthly progress report, it will be important to provide this information as part of the story around your activity (case study or success story). You should also give a description of what you did to engage the community, reflecting on what worked and what didn’t work and why. What are the strengths of your community and its leaders that have made this activity a success? You will also need to consider what the risks and challenges have been for your team, and how you have worked to overcome them.
Simple numerical data (counts) for each type of evidence-based population health promotion activity you do will address this part of the indicator. You will also need to record the location of your activities (e.g. neighbourhood, town or region). Activities of specific interest under this indicator are:
To avoid repetition, partnership working and collaborations developed do not need to be reported here. This information will be captured under Indicator 2. It will also be important to demonstrate (through your Action Work Plan) that these activities are evidence-based.
The outcome related to this indicator is:
There is simple data you can collect around the number and type of organisations, services or individuals you have partnered with as part of your population health promotion activities. Of specific interest are partnerships with:
As you can see from the data sources described by CIRCA, it will be useful to think about the purpose of these partnerships, for example if they increase your reach into areas within the region you are contracted to service, or priority populations including pregnant women, or people who do not typically attend ACCHS. You should also collect more detailed data which describes the quality and extent of your collaboration with different partners and how these partnerships have improved geographical or priority group reach, for example through case studies of the partnership journey.
Think about the quality of your relationships with the different organisations you have entered into partnership with as well. Different ways of doing this include tools that involve the grant recipient organisation and their partner/s individually assessing the relationship and then coming together to discuss and move forward. Examples of these tools include:
This information will be an important part of providing the story around your activity (case study or success story). What did you do to engage the organisations, what worked, what didn’t work and why? What are the strengths of your organisation and those you have partnered with that have made this activity a success? What were the risks and challenges and how have you worked to overcome them?
This indicator is concerned with community access to quit services. Access to healthcare services is only possible if:
If both of these are in place, we should see good uptake of services.
From the data sources described by CIRCA, you can see there is simple data you can collect around the number and percentage of staff in your own organisation or others, who have:
This will provide information about the availability and capacity of appropriate services. CIRCA are interested in formal training courses which staff have done – specifically Quitskills.
You also need to report the number of organisations you have supported to develop or maintain their capacity to carry out brief interventions with clients. This could be providing or facilitating a training day for new staff, or a refresher course for staff in another organisation.
You should also collect more detailed data which describes the impact of training on staff knowledge and skills, and their confidence in supporting community members asking for advice (the third outcome under this indicator). You can collect this data through:
You should report this information as part of your case study or success story as it is not captured directly on the six monthly report. You can also report any relevant training carried out at your own or another organisation that was not Quitskills as part of this story.
You will need to collect simple data (counts) of:
A good way of finding out whether awareness of services has increased in (a) the community and (b) healthcare practitioners (the second outcome under this indicator) is through:
This will provide good data to support your case study or success story. Examples of surveys you can use or adapt for your own region are available from the NBPU Survey Question Bank. You can also provide more qualitative data around how your project has supported local communities in the region to get better access to quit support. This could be based on:
You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.
From these data sources you can see there is simple data you should collect around the number of events locally that you have supported to be smoke-free, how many organisations you have worked with to develop/improve smoke-free policies or to increase worker compliance with policies. You can also count how many homes and cars are smoke-free or, how many people have pledged to be smoke free. Good ways of doing this include:
You will also need to report on your own organisation’s smoke-free policy, including whether or not staff and board members comply with smoke-free policies:
You should also collect more in-depth data on understanding about what being smoke-free means and how important it is to people to try to be smoke-free. You can ask the people who smoke if they go without smokes at home, in the car, near their children, at work, or on other occasions, and how they handle smoke-free times. Good ways to get this information are:
Finally, you should describe how you have supported the communities in your region to become more smoke-free and how people have responded to these activities. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.
This indicator is concerned with whether your activities are reaching and engaging priority groups in your service area. Some of these groups have been described nationally (e.g. pregnant women), however there may also be other groups of people which are a priority for your community (e.g. older men).
A simple count of the priority groups you have identified for your service delivery area, and the number and type of targeted activities you do for each of these groups will also be important (e.g. targeted health education sessions, targeted social marketing or targeted events). You will also need to report the number of people participating in these activities. Activities of interest include:
You should also collect more in-depth data to tell the story of the difference your activities are making for priority groups. This could include questions about new knowledge about the benefits of being smoke free, what being smoke-free means to them, and how important it is to your priority groups. You might ask mums-to-be what they are doing to keep their home and car smoke-free, if they are trying or intending to quit, and if so has anyone else in the family joined them on their smoke-free journey. Good ways to get this information include:
This will provide good data to support your case study or success story.
This indicator focuses on geographical reach. You need to show that the activities you do extend across your contracted service area and include community members who do not routinely use ACCHO services. Simple numerical data (counts) will address this indicator, and include the number, type and location of your activities. You will also need to report the number of people who attended. Specific activities of interest include:
You should also provide a detailed description of what you did to engage the communities across your service area, reflecting on what worked and what didn’t work and why. It will be useful to report this as part of your case study or success story. You will also need to report any risks and challenges for your team in delivering against this indicator, and how you have worked to overcome these challenges.
A national evaluation of the TIS program will take place. The evaluation of the TIS program for the next phase of the program will be split into a two-part process. The Cultural and Indigenous Research Centre Australia (CIRCA) will conduct part A of the TIS evaluation (implementation and process evaluation). The Australian National University (ANU) will conduct part B (impact evaluation).
The Monitoring and Evaluation Framework for the TIS program 2022/23 – 2025/26 is in development. This document will provide a framework for part A of the evaluation of the TIS program.
The implementation and process evaluation of the TIS program from 2018/19-2021/22 was conducted by CIRCA, and a summary of key findings was released in September 2022. The evaluation looked at:
The report shows that the program is implementing evidence-based approaches to promote population health as intended, and is achieving successful outcomes.
Please find archived TIS evaluation documents here.
How will you determine if your program is working? This section will provide you with the tools to monitor and evaluate your activities.
The processes of monitoring and evaluation use carefully planned and well-thought-out methods to measure the success of an activity in meeting its goals. They are an important part of the project management process, because they provide:
Key terms used when talking about monitoring and evaluation are shown in Box 1.
Box 1: Monitoring and evaluation key terms
Monitoring and evaluation are related processes, but each has a different focus:
Table 1 provides examples of different outputs and outcomes for the TIS program. Monitoring outputs relies on describing and counting project activities and the number of people who come to events. In the past TIS reporting focused only on what is included in the outputs column – ‘what we do and who we reach.’ Now, however, you are being asked to think more about ‘what difference do our activities make?’ This is a question about your activity outcomes.
Table 1: Examples of outputs and outcomes for TIS
Smoke-free workshops delivered to 100 workers in 10 community organisations
85 workers have increased knowledge of benefits of smoke-free workplace and increased commitment to being smoke-free at work
8 Organisations are smoke-free
Social media campaign (call to action – take the ‘we can be smokefree’ pledge).
45 families took the pledge
Follow up survey showed that this has potential to reach over 500 individuals living in these homes
You can find out more about the ‘how, who and what’ of TIS activities evaluation in Box 2.
Box 2: The how, who and what of evaluation
How to plan an evaluation?
It’s a good idea to work out an evaluation plan for your activities, so you can keep track of the evaluation. In your plan consider the following points:
Who will do the evaluation?
Work out who will be responsible for organising and writing up the evaluation.
Who should participate in an evaluation?
Good evaluations involve those who are interested and affected by your activities. Involving people from the earliest stages of the activity’s development to the final evaluation can encourage local communities to set up, control and own an activity. Putting together a working group (a group appointed to study or report on a particular question) of community members also helps the evaluation process. A working group brings the values and shared interests of the community into the process. Often they are also the ones who are best placed to talk about the needs of their community.
What should you evaluate?
What aspect of the activities do you want to evaluate? For example, do you want to know if the activities’ objectives were met? Or do you want to know what people liked about the activities? There are lots of evaluation questions you could use. Examples of the sorts of questions you might use to answer different questions are provided in the
.
What sort of resources will you need for your evaluation?
Be realistic about what resources are available to undertake the evaluation. This includes funding, time, staff, salaries, material, equipment and operating costs.
Source: material adapted from Kruger K, McMillan N, Russ P and Smallwood H (2007) Talkin’ up good air: Australian Indigenous tobacco control resource kit. Melbourne: Centre for Excellence in Indigenous Tobacco Control
[14810]
Or for a more in-depth general discussion of behaviour change evaluation and definitions of some other terms used in monitoring and evaluation, you might find the Evaluation toolbox useful.
Monitoring and evaluation should be built into your activities from the start. It is also a requirement of TIS program funding to measure your outcomes every six months for Indigenous Australians’ Health Programme (IAHP) grant agreement progress reporting. The
developed by HealthInfoNet on behalf of the Australian Government (2012), describes a simple six step process to building evaluation into an activities plan:
Examples of ways to present your data using charts and dashboards for good visual impacts can be found in this monitoring and evaluation resource or on the Resources to monitor and evaluate your program page here.
Figure 1: Keeping your activities on track
Figure 1 shows how monitoring and evaluation forms a central part of your activities. You can use the information that you collect to make sure activities are on track and that you are achieving the outcomes you intended, adapting and improving your activities as you go. This process of using monitoring and evaluation data to improve your activities as you go is known as continual quality improvement (CQI). For support in undertaking CQI, you can download Reach, implementation, satisfaction and effectiveness (RISE): a tool for supporting CQI for TIS. More tips for how to use monitoring and evaluation data are provided in Box 3.
Box 3: Using monitoring and evaluation data
Activities do not always run as planned or as expected, and are a learning experience for all those involved. Monitoring your activities and evaluating outcomes will help you keep track of progress, identify important changes and make it easier to see what you have achieved. Answering the following questions may help keep your activities on track:
Source: adapted from Australian Indigenous HealthInfoNet (2012) [2]
[1] This material is used with permission from the Commonwealth of Australia. Talkin’ up good air contains materials that were contributed by Quit Victoria (Anti-Cancer Council of Victoria), Apunipima Cape York Health Council, National Heart Foundation of Australia (NSW Division), Council of Social Services of New South Wales and Queensland Health and which remain their property.
[2] Australian Indigenous HealthInfoNet (2012) Healthy, Deadly and Strong: Healthy Lifestyle Worker Toolkit. Perth, WA: Australian Indigenous HealthInfoNet
