TIS six monthly performance reporting (1 July 2023 – 30 June 2027)
From 1 January 2024, TIS six monthly performance reporting will be via the Australian Government’s online system – the Health Data Portal (HDP). All lead organisations will need to complete an online form provided through this portal. User help for the portal is available here or by contacting the HDP team: indigenousreporting@health.gov.au.
Submission dates for 1 July 2023 to 30 June 2027 are as follows:
NBPU TIS will provide feedback within two weeks for AWP Word documents received by the due date indicated above.
The Performance Report Guidance below includes an example of how to complete the different sections of the online template.
Please contact your NBPU TIS Project Officer if you need any further information on the online reporting process.
There are two types of data that you can collect, quantitative and qualitative:
Quantitative data is numerical and includes audits or counts described below.
Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. Qualitative data can take many forms. You might gather people’s feedback through interviews or you could ask them to share how they feel through photographs, paintings, drama, or other imagery.
The type of data you collect will depend partly on the question you want to answer but also on the data collection methods that work best in your community.
Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, gender, if they smoke, where are they from), and the outcome of participating in TIS activities (e.g. N referred to Quitline, N taking a smokefree pledge). It is important to collect good quality data by making sure you count the same thing in the same way each time you do an activity.
Case studies and success stories are both ways of showing the impact of your project. This story of impact is really important and the reason your monitoring and evaluation has to go beyond just numerical counts.
CIRCA have provided a space for you to tell these stories on the six-monthly performance report.
You should provide a story that shows the impact your activities have against each of the program indicators. The focus of this story might be:
Success Story template
Case studies and success stories are very similar. The main difference is the point of view (PoV) from which the story is written:
Case studies and success stories should be specific and include relevant information (data) as evidence of the difference you have made. They can be illustrated with quotes and photos – but remember to get people’s permission to share their information. You can download templates for writing your case studies and success stories.
You can use a range of methods for collecting data for success stories including writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.
Examples of other creative approaches to gathering data for stories includes playing games, creating a storyboard or using photography. You can find in this Creative strategies resource.
A focus group is a planned discussion with selected individuals to gather information, or people’s opinions. For example you might want to know what people think about your activities, or to find out what they have learnt from a social marketing campaign. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).
It is important to know what you want to ask before going into the focus group – this Focus groups resource provides a guide to writing focus group questions.. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.
In this short video Ninti One’s Aboriginal Community Researchers show how to run a focus group here.
Interviews are typically one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it. In this short video Sunil George from the NBPU explains in more detail how to plan and carry out interviews to collect monitoring and evaluation data.
More information can also be found in this Interviews factsheet.
In this short video Ninti One’s Aboriginal Community Researchers show you how to write open ended questions for interviews and focus groups.
Preparing open-ended questions
Yarning is an important part of Aboriginal and Torres Strait Islander cultures. If you want to use yarning as a way to collect monitoring and evaluation data, you must follow local cultural protocols and practices. Seeking guidance from local Elders should be your starting point.
Yarning for gathering data is recognised as a meaningful and rigorous form of knowledge production (AIATSIS 2020). You might yarn with one person or with two or three together. Or you could yarn with a larger group as part of a yarning circle. Yarning is less formal than western approaches such as interviews or focus groups. Yarning allows information to unfold through story telling in a relaxed and flexible process that is culturally safe for Indigenous people. You can read more about the principles of yarning as a data collection method here.
Yarning must be used appropriately. We recommend you:
If you are interested in learning more about yarning as a research method, a narrative review of yarning was published in 2022 by a group of Indigenous academics including Michelle Kennedy, Raglan Maddox and Dawn Bessarab. The review paper is available here.
Figure 1 based on Bessarab, D., & Ng’Andu, B. (2010)
There are two types of observation:
Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as this Environmental scan template created by the NBPU TIS is a good way of recording your observations.
This is a good method of gathering information from a large number of people. Surveys collect information by asking everyone the same questions. In this short video Penney Upton from the NBPU explains how surveys can be useful to collect monitoring and evaluation data.
Survey Template
This template will help you structure your survey and ask the right questions. Examples of the kinds of questions you might want to ask can be found in the Survey question bank. There is also a template for a post workshop feedback questionnaire that you could modify to find out about participant satisfaction with educational activities.
Monitoring and evaluation is taking place at two levels for the TIS program. All TIS funded organisations are expected to monitor their activities and evaluate project outcomes at a local level. A national evaluation of all the components of the TIS program is also being carried out:
National Indicator 1: Eligible, evidence-based population health promotion activities are being implemented, monitored and improved upon (if necessary).
National Indicator 2: Activities are reaching intended audiences.
National Indicator 3: Eligible TIS activities and messaging are co-designed with local Aboriginal and Torres Strait Islander peoples to ensure they are culturally safe for the local community.
National Indicator 4: Formal and informal partnerships are engaged and facilitate support for eligible TIS activities
National Indicator 5: Intended audiences have increased awareness about how to access quit support.
National Indicator 6: Intended audiences have increased intentions to remain smoke-free, quit and/or stay quit.
National Indicator 7: Intended audiences have increased intentions to make and/or keep environments smoke-free.
A national evaluation of the TIS program will take place. The evaluation of the TIS program for the next phase of the program will be split into a two-part process. The Culturally Inclusive Research Centre Australia (CIRCA) will conduct part A of the TIS evaluation (implementation and process evaluation). The Australian National University (ANU) will conduct part B (impact evaluation).
The TIS Program Logic 2023-26 is an overview of what the TIS program is trying to achieve over the next implementation period. It has been developed by CIRCA for Part A of the TIS evaluation. It maps out the inputs, activities, and outputs, as well as the desired short-, medium-, and long-term outcomes for the program. Short-term outcomes are focused on changes in people’s understanding, medium-term outcomes are focused on changes in people’s intentions or motivations, and long-term outcomes are about behavioural changes. The diagram includes small icons so you can easily see which stakeholders the different outcomes refer to.
The implementation and process evaluation of the TIS program from 2018/19 – 2021/22 was conducted by CIRCA. The evaluation looked at:
You can read CIRCA’s Tackling Indigenous Smoking Program final evaluation report (2018/19 – 2021/22) here.
The report shows that the program is implementing evidence-based approaches to promote population health as intended, and is achieving successful outcomes.
Please find archived TIS evaluation documents here.
How will you determine if your program is working? This section will provide you with the tools to monitor and evaluate your activities.
The processes of monitoring and evaluation use carefully planned and well-thought-out methods to measure the success of an activity in meeting its goals. They are an important part of the project management process, because they provide:
Key terms used when talking about monitoring and evaluation are shown in Box 1.
Box 1: Monitoring and evaluation key terms
Monitoring and evaluation are related processes, but each has a different focus:
Table 1 provides examples of different outputs and outcomes for the TIS program. Monitoring outputs relies on describing and counting project activities and the number of people who come to events. In the past TIS reporting focused only on what is included in the outputs column – ‘what we do and who we reach.’ Now, however, you are being asked to think more about ‘what difference do our activities make?’ This is a question about your activity outcomes.
Table 1: Examples of outputs and outcomes for TIS
Smoke-free workshops delivered to 100 workers in 10 community organisations
85 workers have increased knowledge of benefits of smoke-free workplace and increased commitment to being smoke-free at work
8 Organisations are smoke-free
Social media campaign (call to action – take the ‘we can be smokefree’ pledge).
45 families took the pledge
Follow up survey showed that this has potential to reach over 500 individuals living in these homes
You can find out more about the ‘how, who and what’ of TIS activities evaluation in Box 2.
Box 2: The how, who and what of evaluation
How to plan an evaluation?
It’s a good idea to work out an evaluation plan for your activities, so you can keep track of the evaluation. In your plan consider the following points:
Who will do the evaluation?
Work out who will be responsible for organising and writing up the evaluation.
Who should participate in an evaluation?
Good evaluations involve those who are interested and affected by your activities. Involving people from the earliest stages of the activity’s development to the final evaluation can encourage local communities to set up, control and own an activity. Putting together a working group (a group appointed to study or report on a particular question) of community members also helps the evaluation process. A working group brings the values and shared interests of the community into the process. Often they are also the ones who are best placed to talk about the needs of their community.
What should you evaluate?
What aspect of the activities do you want to evaluate? For example, do you want to know if the activities’ objectives were met? Or do you want to know what people liked about the activities? There are lots of evaluation questions you could use. Examples of the sorts of questions you might use to answer different questions are provided in the
.
What sort of resources will you need for your evaluation?
Be realistic about what resources are available to undertake the evaluation. This includes funding, time, staff, salaries, material, equipment and operating costs.
Source: material adapted from Kruger K, McMillan N, Russ P and Smallwood H (2007) Talkin’ up good air: Australian Indigenous tobacco control resource kit. Melbourne: Centre for Excellence in Indigenous Tobacco Control
[14810]
Or for a more in-depth general discussion of behaviour change evaluation and definitions of some other terms used in monitoring and evaluation, you might find the Evaluation toolbox useful.
Monitoring and evaluation should be built into your activities from the start. It is also a requirement of TIS program funding to measure your outcomes every six months for Indigenous Australians’ Health Programme (IAHP) grant agreement progress reporting. The
developed by HealthInfoNet on behalf of the Australian Government (2012), describes a simple six step process to building evaluation into an activities plan:
Examples of ways to present your data using charts and dashboards for good visual impacts can be found in this monitoring and evaluation resource or on the Resources to monitor and evaluate your program page here.
Figure 1: Keeping your activities on track
Figure 1 shows how monitoring and evaluation forms a central part of your activities. You can use the information that you collect to make sure activities are on track and that you are achieving the outcomes you intended, adapting and improving your activities as you go. This process of using monitoring and evaluation data to improve your activities as you go is known as continual quality improvement (CQI). For support in undertaking CQI, you can download Reach, implementation, satisfaction and effectiveness (RISE): a tool for supporting CQI for TIS. More tips for how to use monitoring and evaluation data are provided in Box 3.
Box 3: Using monitoring and evaluation data
Activities do not always run as planned or as expected, and are a learning experience for all those involved. Monitoring your activities and evaluating outcomes will help you keep track of progress, identify important changes and make it easier to see what you have achieved. Answering the following questions may help keep your activities on track:
Source: adapted from Australian Indigenous HealthInfoNet (2012) [2]
[1] This material is used with permission from the Commonwealth of Australia. Talkin’ up good air contains materials that were contributed by Quit Victoria (Anti-Cancer Council of Victoria), Apunipima Cape York Health Council, National Heart Foundation of Australia (NSW Division), Council of Social Services of New South Wales and Queensland Health and which remain their property.
[2] Australian Indigenous HealthInfoNet (2012) Healthy, Deadly and Strong: Healthy Lifestyle Worker Toolkit. Perth, WA: Australian Indigenous HealthInfoNet
