Reporting on your TIS activities

TIS six monthly performance report template (1 July 2023 – 30 June 2026)

From 1 January 2024, TIS six monthly performance reporting will be via the Australian Government’s online system – the Health Data Portal (HDP). All lead organisations will need to complete an online form provided through this portal.

The document below has been created to give you an idea of the content that will be needed for the report (click to download).

If required you can also use this document to gather the information you need to complete your online report (for example, from partner organisations where you are delivering the program as a consortium).

Please note: you should not use this document to submit your report. From 1 January 2024 performance report data will only be accepted via the online form provided in the portal. Word document templates or PDFs will no longer be accepted.

The Performance Report Guidance below includes an example of how to complete the different sections of the online template.

Please contact your NBPU TIS Project Officer if you need any further information on the online reporting process.

Monitoring and Evaluation Methods

There are two types of data that you can collect, quantitative and qualitative:

Quantitative data is numerical and includes audits or counts described below.

Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. Qualitative data can take many forms. You might gather people’s feedback through interviews or you could ask them to share how they feel through photographs, paintings, drama, or other imagery.

The type of data you collect will depend partly on the question you want to answer but also on the data collection methods that work best in your community.

Data collection methods

Counts

Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, gender, if they smoke, where are they from), and the outcome of participating in TIS activities (e.g. N referred to Quitline, N taking a smokefree pledge). It is important to collect good quality data by making sure you count the same thing in the same way each time you do an activity.

Case studies and success stories

Case studies and success stories are both ways of showing the impact of your project. This story of impact is really important and the reason your monitoring and evaluation has to go beyond just numerical counts.

CIRCA have provided a space for you to tell these stories on the six-monthly performance report.

You should provide a story that shows the impact your activities have against each of the program indicators. The focus of this story might be:

  • an effective social marketing campaign
  • community change such as more smokefree homes

    Success Story template

  • a successful smokefree event
  • organisational change.

Case studies and success stories are very similar. The main difference is the point of view (PoV) from which the story is written:

  • a case study is written from the participant PoV. So they might describe a family’s experience of going smoke free in the home and car. Many case studies focus on individual stories like the example provided here. You will see the story explains how TIS population health promotion activities encouraged this person to make a positive change.
  • a success story is written from the PoV of the organisation. An example can be found here. You can read other examples on the TIS Success Stories page.

Case studies and success stories should be specific and include relevant information (data) as evidence of the difference you have made. They can be illustrated with quotes and photos – but remember to get people’s permission to share their information. You can download templates for writing your case studies and success stories.

You can use a range of methods for collecting data for success stories including writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.

Examples of other creative approaches to gathering data for stories includes playing games, creating a storyboard or using photography. You can find in this Creative strategies resource.

Focus group

A focus group is a planned discussion with selected individuals to gather information, or people’s opinions. For example you might want to know what people think about your activities, or to find out what they have learnt from a social marketing campaign. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).

It is important to know what you want to ask before going into the focus group – this Focus groups resource provides a guide to writing focus group questions.. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.

In this short video Ninti One’s Aboriginal Community Researchers show how to run a focus group here.

Interviews

Interviews are typically one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it. In this short video Sunil George from the NBPU explains in more detail how to plan and carry out interviews to collect monitoring and evaluation data.

More information can also be found in this Interviews factsheet.

In this short video Ninti One’s Aboriginal Community Researchers show you how to write open ended questions for interviews and focus groups.

Preparing open-ended questions

Yarning and Yarning Circles

Yarning is an important part of Aboriginal and Torres Strait Islander cultures. If you want to use yarning as a way to collect monitoring and evaluation data, you must follow local cultural protocols and practices. Seeking guidance from local Elders should be your starting point.

Yarning for gathering data is recognised as a meaningful and rigorous form of knowledge production (AIATSIS 2020). You might yarn with one person or with two or three together. Or you could yarn with a larger group as part of a yarning circle. Yarning is less formal than western approaches such as interviews or focus groups. Yarning allows information to unfold through story telling in a relaxed and flexible process that is culturally safe for Indigenous people. You can read more about the principles of yarning as a data collection method here.

Yarning must be used appropriately. We recommend you:

  • Take advice from local Elder’s about whether or not yarning would be an appropriate method for you to use;
  • Work closely with local cultural advisors (Elders or others nominated by them) to develop your yarning guide;
  • Ensure you are comfortable with, and know how to apply the different types of yarning (social, research topic, collaborative and therapeutic) described by Bessarab and Ng’andu 2010;
  • Follow local cultural advice about the principles and process of collecting your data (e.g. who, where, and when).

If you are interested in learning more about yarning as a research method, a narrative review of yarning was published in 2022 by a group of Indigenous academics including Michelle Kennedy, Raglan Maddox and Dawn Bessarab. The review paper is available here.

Figure 1 based on Bessarab, D., & Ng’Andu, B. (2010)

Observation

There are two types of observation:

  • direct observation – this is where the observer is an unidentified ‘fly on the wall’ (you will need permission if you want to observe people, whether in a workplace or school or other venue/location)
  • participant observation – where the observer takes part in an activity with the participants and asks questions.

Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as this Environmental scan template created by the NBPU TIS is a good way of recording your observations.

Questionnaire survey

This is a good method of gathering information from a large number of people. Surveys collect information by asking everyone the same questions. In this short video Penney Upton from the NBPU explains how surveys can be useful to collect monitoring and evaluation data.

Survey Template

This template will help you structure your survey and ask the right questions. Examples of the kinds of questions you might want to ask can be found in the Survey question bank. There is also a template for a post workshop feedback questionnaire that you could modify to find out about participant satisfaction with educational activities.

 

National TIS Performance Indicators

Monitoring and evaluation is taking place at two levels for the TIS program. All TIS funded organisations are expected to monitor their activities and evaluate project outcomes at a local level. A national evaluation of all the components of the TIS program is also being carried out:

  • A process and implementation evaluation led by Cultural & Indigenous Research Centre Australia (CIRCA);
  • An impact and outcomes evaluation led by the Australian National University (ANU).

National TIS Performance Indicators (1 January 2023 – 30 June 2023)

Teams completing performance reports for 1 January – 30 June 2023 should refer to the PDF below which outlines the National TIS Performance Indicators for this funding period.

National TIS Performance Indicators (From 1 July 2023)

National Indicator 1: Eligible, evidence-based population health promotion activities are being implemented, monitored and improved upon (if necessary).

National Indicator 2: Activities are reaching intended audiences.

National Indicator 3: Eligible TIS activities and messaging are co-designed with local Aboriginal and Torres Strait Islander peoples to ensure they are culturally safe for the local community.

National Indicator 4: Formal and informal partnerships are engaged and facilitate support for eligible TIS activities

National Indicator 5: Intended audiences have increased awareness about how to access quit support.

National Indicator 6: Intended audiences have increased intentions to remain smoke-free, quit and/or stay quit.

National Indicator 7: Intended audiences have increased intentions to make and/or keep environments smoke-free.

Evaluation documents

Evaluation 2022/23 – 2025/26

A national evaluation of the TIS program will take place. The evaluation of the TIS program for the next phase of the program will be split into a two-part process. The Cultural and Indigenous Research Centre Australia (CIRCA) will conduct part A of the TIS evaluation (implementation and process evaluation). The Australian National University (ANU) will conduct part B (impact evaluation).

The TIS Program Logic 2023-26 is an overview of what the TIS program is trying to achieve over the next implementation period. It has been developed by CIRCA for Part A of the TIS evaluation. It maps out the inputs, activities, and outputs, as well as the desired short-, medium-, and long-term outcomes for the program. Short-term outcomes are focused on changes in people’s understanding, medium-term outcomes are focused on changes in people’s intentions or motivations, and long-term outcomes are about behavioural changes. The diagram includes small icons so you can easily see which stakeholders the different outcomes refer to.

Evaluation 2018/19 – 2021/22

The implementation and process evaluation of the TIS program from 2018/19 – 2021/22 was conducted by CIRCA. The evaluation looked at:

  • program appropriateness, evidence base and national supports;
  • whether it’s on target to achieve its short and medium term goals;
  • program enablers and barriers.

You can read CIRCA’s Tackling Indigenous Smoking Program final evaluation report (2018/19 – 2021/22) here.

The report shows that the program is implementing evidence-based approaches to promote population health as intended, and is achieving successful outcomes.

Please find archived TIS evaluation documents here.

Very brief advice

Very brief advice (VBA) is an evidence-based approach to increasing quit attempts. The purpose of VBA is to engage with people who smoke and get them to think about quitting. The aim is not to tell people how they should behave (quit smoking), but to guide them to the quit support that is available. VBA focuses on offering help by providing:

  • information about where to go for help to quit
  • encouragement and support for change
  • a referral to quit support.

VBA is an opportunistic, non-intrusive and respectful approach which can be used by anyone, in any setting, including community settings. VBA is not smoking cessation therapy and does not require formal counselling skills or knowledge of the stages of behaviour change because:

  • assessing clinical factors such as how much the person smokes or their level of addiction is not important because there is no safe level of smoking
  • evaluating stage of behaviour change is not important because most people who smoke know it is bad for them, want to stop, and have probably tried to quit already.

In simple terms VBA is a short conversation lasting from 30 seconds to a couple of minutes during which you:

  • Ask the person if they smoke
  • Advise them about the best way to quit (behavioural support (e.g. Quitline) combined with stop smoking medication like NRT)
  • Act by directing them to locally available support (provide information or referral).

Because of the opportunistic non-clinical nature of VBA, anyone who has contact with people who smoke from Aboriginal and Torres Strait Islander communities should be able to provide this including:

  • staff from health organisations, who are the first point of contact for clients (e.g., receptionists and drivers)
  • staff from non-health organisations that have regular contact with Aboriginal and Torres Strait Islander clients
  • smoke-free ambassadors or other volunteers who work on community events/outreach activities.

According to a review of the evidence, brief simple advice about quitting smoking increases the likelihood that someone who smokes will successfully quit and remain smoke-free 12 months later.

VBA is a modification of the 5As (Ask, Advise, Assess, Assist, Arrange) approach to brief intervention which is often recommended for use in a clinical setting. You can read more about this clinical approach on the fact sheet: Key facts about behavioural support for smoking cessation.

Social media and social networking

Social media and social networking tools such as Facebook, Instagram, TikTok, YouTube, Snapchat and Twitter are increasingly being used to help tackle smoking, particularly with young adults. One of the advantages of these platforms is that they are accessible, low cost and familiar to young people. About 20.5 million Australians are active users of social media – around 80% of the total population. Research by the McNair Ingenuity Research Institute in 2014 found that Facebook is a popular means of communication among Aboriginal and Torres Strait Islander people.

The use of Facebook and Twitter as a way of communicating is a popular approach for many healthcare services. However the value of these tools seems to lie more in their networking functions. Social media is interactive and user-driven, meaning it has the potential to provide real-time peer to peer support and discussion around tobacco use.

There is currently a lack of evidence of the effectiveness of using social media in tobacco control. Studies that do exist tend to be descriptive, with a focus on the acceptability of the medium to support quitting, or an analysis of posts. A study on using Facebook to reduce smoking among Aboriginal and Torres Strait Islander people found that there was potential for health services to incorporate a strategy of using paid local social media ‘champions’ or ‘ambassadors’ to disseminate tobacco control messages on Facebook through community networks. It also found that:

  • posts were more likely to be shared if they:
    • were child-focused
    • featured Aboriginal and Torres Strait Islander content
    • were perceived as practical, relevant and credible, with a direct and unambiguous message.
  • posts were less likely to be shared if they:
    • included disgusting imagery about health impacts
    • were focused on the environment
    • were ambiguous or sarcastic.

The NBPU TIS have produced a Key facts about social media factsheet and infographic that provide key information for TIS workers about using social media to communicate messages about smoking.

Social media factsheet

Social media infographic

 

Claudine Thornton’s social media training

A bespoke online social media training course has been developed for the TIS workforce by Claudine Thornton Creative. You will learn about marketing terminology and consumer behaviour tactics. Understanding how marketers use emotion over logic in promoting cigarettes, means you can use the same tactics to persuade people to be smoke-free. The course focuses on how to use emotional availability to reverse engineer tobacco marketing. The course lasts around one and a half hours and comprises seven modules each split into 5-10 minute segments.So you can complete it in one go, or in short pieces. The course is free to access, simply open this link and then click the ‘Enroll for free’ button.

Menzies School of Health Research has produced a tips and tricks resource for people working in health promotion and tobacco control, Social media in health promotion and tobacco control: tips and tricks. An accompanying PowerPoint presentation, Can Facebook help Aboriginal and Torres Strait Islander people to quit smoking? is also available.

Monitoring and reporting social media activities presentation

Monitoring and reporting on social media activities

This presentation from A/Prof Penney Upton discusses the monitoring and reporting of social media activities, including:

  • reach
  • engagement
  • impact.

Please note: The presentation will start playing automatically when opened.

Mass media and social marketing

Mass media and social marketing campaigns aim to reduce the number of people who smoke by changing attitudes, beliefs and intentions surrounding tobacco use. Mass media campaigns take a traditional marketing approach to this aim, treating the desire to be smoke-free as a product to be sold. In contrast, social marketing uses knowledge of specific community barriers to develop more targeted marketing approaches.

Both approaches use education about the negative consequences of smoking and the benefits of not smoking for two purposes:

  • to prevent the uptake of smoking (particularly in young people)
  • to promote quit attempts in people who currently smoke.

It is thought these campaigns help to prevent smoking by changing people’s expectations toward smoking, so that tobacco use is no longer accepted as the ‘usual, cool or necessary thing to do.’

A 2021 publication from the Mayi Kuwayu study showed that education provided by the TIS program has changed awareness and understanding of the impact of smoking on long term health in Aboriginal and Torres Strait Islander adults. We know that negative attitudes towards smoking are an important precursor to smoking behaviour change. Anti-smoking attitudes are more likely to lead to quit attempts, and to successful long term smoke free living.

There is evidence that both mass media and social marketing campaigns can help prevent smoking from starting, encourage people to stop smoking, and prevent relapse among recent quitters by reminding them about why they chose to stop smoking. However one of the biggest effects is in relation to promoting access to cessation support services such as Quitline, counselling and other health professionals. Developing capacity to support people who are ready to quit – by taking the systems approach described under Planning, is therefore essential if programs are to be sustainable.

A campaign’s impact is influenced by:

  • repetition – the more often the messages are heard, the more likely they are to sink in
  • reach – how far the messages are spread, how large the audience is
  • intensity – how regularly the messages are heard
  • how long the campaign is – it needs to be long enough for all target audiences to have heard the message but not so long that they get sick of it and tune out.

Don’t Make Smokes Your Story campaign ad

The relevance of the message has been found to be important for audience engagement. The context, characters and role models used in advertising or community activities must seem believable, if people who smoke are to connect to them. When a campaign does not relate to how people see themselves, they find it hard to become interested in the content. While there is some evidence that mass media campaigns do influence attitudes and beliefs of Aboriginal people and Torres Strait Islanders in regard to smoking, more specific local messages tailored for Aboriginal and Torres Strait Islander people seem to be most effective. Evidence from the Talking About The Smokes project also supports the importance of using targeted advertising.

The Australian Government’s Don’t Make Smokes Your Story campaign is a good example of how advertising and community-based activities can work together to encourage behaviour change among Aboriginal and Torres Strait Islander people who smoke. You can adapt and use these materials and resources for your own activities.

Smoke-free

Reducing second-hand and third-hand smoke

Reducing second-hand and third-hand smoke is an important aim. This is because second-hand and third-hand smoke can be very harmful. The evidence also shows that if smoking is seen as ‘normal’ at a community level, young people are more likely to start smoking, and people who currently smoke will find it harder to quit. Increasing the extent to which a community is smoke-free is associated with less smoking and more success in quitting. A 2021 publication from the Mayi Kuwayu study showed that areas of Australia where the TIS program is present, compared to non-TIS areas, have a significantly lower prevalence of smoking inside households.

Relevant activities include the following:

Developing smoke-free policies in the workplace

To be successful, smoke-free policies need community participation – not just consultation – in their development. Policies that have local ownership and commitment are more likely to be followed. There is also evidence that introducing smoke-free policies in the workplace can lead to increased support for smoke-free spaces in other areas such as smoke-free homes and cars. Successful smoke-free workplace policies also result in more workers wanting to quit.

‘Keep our place a smoke-free space’ is a resource package developed to support the implementation of smoke-free workplace policies. ‘Keep our place a smoke-free space’ was a collaboration between NBPU TIS, the Office of the Registrar of Indigenous Corporations (ORIC) and TIS teams from WA, Qld, SA and NT. You can view the poster presentation about the project that was presented at the Lowitja Institute International Indigenous Health and Wellbeing Conference 2023 here.

Resources include:

Smoke-free workplace wallchart

Workshop resources

An information leaflet promoting the initiative to Aboriginal and Torres Strait Islander organisations is available here. TIS teams can download this leaflet and personalise with their contact details before sharing with local businesses.

Combining locally owned smoke-free policies with access to quit support services increases the success of these policies. Working in an environment with a smoke free policy can also encourage individuals to quit.

 

Supporting smoke-free homes and cars

Second-hand and third-hand smoke is a health risk factor, particularly for children. Children are at greater risk for a number of reasons, including their size, faster breathing rates and less developed respiratory and immune systems. Second-hand smoke is associated with a number of childhood illnesses including:

  • asthma
  • croup
  • bronchitis
  • bronchiolitis
  • pneumonia
  • ear, nose and throat infections.

Second-hand smoke is also believed to contribute to the risk of sudden infant death syndrome (SIDS).

Less well known, but probably just as harmful is third-hand smoke. Third-hand smoke is the tobacco smoke toxins from second-hand smoke that get into people who smoke’s hair and clothes and build up on surfaces and dust in areas where people smoke. Evidence shows that these toxins stay in homes and cars for a long time after the cigarette has been extinguished, even several months later. They may even become more toxic over time.

Bullinah Aboriginal Health Service’s Solid Mob ‘Our home is a smoke free zone’ campaign

Third-hand smoke is an emerging area of research and we don’t yet understand all the health hazards. However, children (especially infants), are more vulnerable to third-hand smoke toxins. This is because infants crawl over contaminated floors and mouth contaminated surfaces like furniture and toys. We also know that infants consume up to a quarter of a gram of dust every day. That’s twice as much as adults.

There is good evidence that having a smoke-free home and car improves children’s health. There is also some evidence that keeping the home smoke-free helps to prevent uptake of smoking by young people. There are therefore many good reasons to support people to have smoke-free homes and cars.

2018 review found that intensive counselling methods or motivational interviewing with parents to be most effective for reducing children’s tobacco smoke exposure in the home. There is some indication that school-based education, intensive home visits, brief education provided to parents in clinics (including scheduled children’s health checks) and culturally sensitive health promotion brochures may also help reduce second-hand smoke in the home and car.

Menzies School of Health Research has developed Healthy starts: reducing the health effects of smoking around Indigenous babies and childrena resource to support health providers when discussing second-hand tobacco smoke with families, and to encourage families to have smoke-free homes.

Running smoke-free events

Griffith AMS hosting a smoke-free community event

Any opportunity to reduce second-hand smoke is a good investment because there is no safe level of exposure to second-hand smoke, and smoke-free events both denormalise smoking and encourage people to think about quitting. The Talking About The Smokes (TATS) study found that support for smoke-free festivals and events is strong in Aboriginal and Torres Strait Islander communities, particularly among those who don’t smoke:

  • for people who have never smoked, support was 71%
  • for ex-smokers 65%
  • for people who do not smoke daily 70%
  • half of all people who smoke daily were in favour of smoke-free events (51%).

Smoke-free pledges

Pledging to be smoke-free demonstrates an active commitment and motivation to change. Evidence shows that when someone makes a public declaration (pledge) they are more likely to follow through with that promise, both for themselves, but also because of what others might think of them if they don’t maintain their promise. We also know that people are more likely to stick with a commitment that has real value, purpose and meaning to them.

Smoke-free pledges are a good population health promotion tool because:

  • group pledges (e.g. family, organisation or community) seem to be more effective than individual ones – this is probably because of the social support that a group pledge provides
  • pledges can be linked to any environment – e.g. smoke-free workplaces, smoke-free homes
  • pledges can be used as a part of different activities – e.g. at community events and workplace education sessions
  • as well as supporting behaviour change, pledges can be used to monitor an activity’s reach and impact.

The Wellington Aboriginal Corporation Health Service Quit B Fit Team encouraging people to take a pledge to not smoke in their homes, cars and workplaces

You can find further information on smoke-free environments in the video and factsheet produced by NBPU TIS which can be downloaded here.

Monitoring and Evaluation Guidance

How will you determine if your program is working? This section will provide you with the tools to monitor and evaluate your activities.

Monitoring and evaluating your activities

The processes of monitoring and evaluation use carefully planned and well-thought-out methods to measure the success of an activity in meeting its goals. They are an important part of the project management process, because they provide:

  • evidence of what is working
  • guidance on what could be done better, which can be used to improve your activity’s performance (progress towards and achievement of results)
  • a check on whether you are meeting your activity’s aims
  • feedback to everyone involved in the activity, including community members and partner organisations
  • compliance with funding body reporting requirements.

Key terms used when talking about monitoring and evaluation are shown in Box 1.

Box 1: Monitoring and evaluation key terms

  • performance: what the activity is achieving (observable results)
  • measurement: how we determine the impact of a activity or program on intended outcomes (e.g. using a questionnaire to find out how many people have smoke-free homes or conducting interviews to find out how people keep their homes smoke-free)
  • indicator: measures that show the extent of progress toward outcomes, especially differences in the lives of the people the activity is working for
  • data collection: process used to gather evidence (e.g. giving smoke-free event participants a questionnaire survey)
  • output: what the activity is producing with its resources (e.g. a specific product or service)
  • outcome: results and impacts of the activity (e.g. a percentage reduction in smoking, a change in behaviour).

Monitoring and evaluation are related processes, but each has a different focus:

  • monitoring provides the organisation and key stakeholders with early indicators of progress, and usually focuses on project outputs (the activities that a project has delivered)
  • evaluation systematically assesses progress towards achieving outcomes.

Table 1 provides examples of different outputs and outcomes for the TIS program. Monitoring outputs relies on describing and counting project activities and the number of people who come to events. In the past TIS reporting focused only on what is included in the outputs column – ‘what we do and who we reach.’ Now, however, you are being asked to think more about ‘what difference do our activities make?’ This is a question about your activity outcomes.

Table 1: Examples of outputs and outcomes for TIS

Outputs Outcomes

Smoke-free workshops delivered to 100 workers in 10 community organisations

85 workers have increased knowledge of benefits of smoke-free workplace and increased commitment to being smoke-free at work

 8 Organisations are smoke-free

Social media campaign (call to action – take the ‘we can be smokefree’ pledge).

45 families took the pledge

Follow up survey showed that this has potential to reach over 500 individuals living in these homes

You can find out more about the ‘how, who and what’ of TIS activities evaluation in Box 2.

Box 2: The how, who and what of evaluation

How to plan an evaluation?

It’s a good idea to work out an evaluation plan for your activities, so you can keep track of the evaluation. In your plan consider the following points:

  • What are the reasons for evaluating your activities?
  • Who will read or listen to your evaluation?
  • What sort of resources will you need for your evaluation?
  • What are your evaluation questions?

Who will do the evaluation?

Work out who will be responsible for organising and writing up the evaluation.

Who should participate in an evaluation?

Good evaluations involve those who are interested and affected by your activities. Involving people from the earliest stages of the activity’s development to the final evaluation can encourage local communities to set up, control and own an activity. Putting together a working group (a group appointed to study or report on a particular question) of community members also helps the evaluation process. A working group brings the values and shared interests of the community into the process. Often they are also the ones who are best placed to talk about the needs of their community.

What should you evaluate?

What aspect of the activities do you want to evaluate? For example, do you want  to know if the activities’ objectives were met? Or do you want to know what people liked about the activities? There are lots of evaluation questions you could use. Examples of the sorts of questions you might use to answer different questions are provided in the

.

What sort of resources will you need for your evaluation?

Be realistic about what resources are available to undertake the evaluation. This includes funding, time, staff, salaries, material, equipment and operating costs.

Source: material adapted from Kruger K, McMillan N, Russ P and Smallwood H (2007) Talkin’ up good air: Australian Indigenous tobacco control resource kit. Melbourne: Centre for Excellence in Indigenous Tobacco Control

[14810]

Or for a more in-depth general discussion of behaviour change evaluation and definitions of some other terms used in monitoring and evaluation, you might find the Evaluation toolbox useful.

When should I think about monitoring and evaluation?

Monitoring and evaluation should be built into your activities from the start. It is also a requirement of TIS program funding to measure your outcomes every six months for Indigenous Australians’ Health Programme (IAHP) grant agreement progress reporting. The

developed by HealthInfoNet on behalf of the Australian Government (2012), describes a simple six step process to building evaluation into an activities plan:

    1. Plan the activities – what do you want to achieve, who do you want to involve, what will you do and how will you do it?
    2. Plan an evaluation – decide and consult about what you will measure, identify key questions, identify local sensitivities, identify good processes, and allocate necessary resources.
    3. Design the evaluation – decide on the methods you will use to collect the information you want, such as counting how many attend a smoke-free community event (output), giving out a short questionnaire asking what people thought of the event, and whether those who attended learnt anything about smoking harms, benefits of quitting, or quit support available (outcomes).
    4. Collect and record your information – do this systemically to get a true picture of what your activity is achieving.
    5. Analyse your information – see if the program is achieving what you intended, or whether are are any unexpected outcomes, identify the lessons to learn. Decide if you need to make any changes to your activities to keep on track.
    6. Provide feedback on your findings – let the people involved in the activity such as the community, your organisation and the participants know about what was achieved.

Examples of ways to present your data using charts and dashboards for good visual impacts can be found in this monitoring and evaluation resource or on the Resources to monitor and evaluate your program page here.

Figure 1: Keeping your activities on track

 

 

Figure 1 shows how monitoring and evaluation forms a central part of your activities. You can use the information that you collect to make sure activities are on track and that you are achieving the outcomes you intended, adapting and improving your activities as you go. This process of using monitoring and evaluation data to improve your activities as you go is known as continual quality improvement (CQI). For support in undertaking CQI, you can download Reach, implementation, satisfaction and effectiveness (RISE): a tool for supporting CQI for TIS. More tips for how to use monitoring and evaluation data are provided in Box 3.

 

 

 

 

Box 3: Using monitoring and evaluation data

Activities do not always run as planned or as expected, and are a learning experience for all those involved. Monitoring your activities and evaluating outcomes will help you keep track of progress, identify important changes and make it easier to see what you have achieved.  Answering the following questions may help keep your activities on track:

  • What do the early results mean in terms of what you are trying to achieve?
  • Should you be changing the way you are doing things?
  • How do you continue the positive changes you are achieving?
  • Are you doing things in the best possible way?
  • Are you talking to the right people about the progress of your activities?
  • Are you making progress, making a difference?

Source: adapted from Australian Indigenous HealthInfoNet (2012) [2]


[1] This material is used with permission from the Commonwealth of Australia. Talkin’ up good air contains materials that were contributed by Quit Victoria (Anti-Cancer Council of Victoria), Apunipima Cape York Health Council, National Heart Foundation of Australia (NSW Division), Council of Social Services of New South Wales and Queensland Health and which remain their property.

[2] Australian Indigenous HealthInfoNet (2012) Healthy, Deadly and Strong: Healthy Lifestyle Worker Toolkit. Perth, WA: Australian Indigenous HealthInfoNet

Resources

{banner}

This section will provide you with tools and resources to plan and support your tobacco control activities. Click on the links below for more information.

{bottom image}

Activities Guidance

Once you have decided what you want to accomplish through your program, you will need to choose activities that will help you to achieve these aims. It is important that Tackling Indigenous Smoking (TIS) activities are effective for achieving the proposed outcomes of the TIS program (e.g. increasing awareness of the benefits of not smoking, more smoke-free environments, increased quit attempts, reduced uptake of smoking). The best way of ensuring that an activity is effective is to use ones which have been tried and tested, so we have evidence that they work. It is also important that you choose the right activities for your local population needs and your local community context. Your role is to use your understanding of the communities in your region, your professional experience and expertise, along with your knowledge of the evidence to put together a suitable set of activities.

Population health promotion

The TIS program uses population health promotion activities to reduce the prevalence of smoking in Aboriginal and Torres Strait Islander communities. TIS is a tobacco control program, not a smoking cessation program. It aims to reduce smoking prevalence by preventing the uptake of smoking, as well as encouraging people who smoke to quit. Regional Tobacco Control Grant (RTCG) teams are funded to provide community outreach, rather than clinical individual level action to improve the health and wellbeing of all Aboriginal and Torres Strait Islander people.

Click on the image on the left to read about the principles of population health promotion that should underpin all of your TIS activities.

 

What do we mean by evidence?

Evidence is the information or knowledge about ‘what works’ which can help you decide which activities you will use. Evidence comes from many different sources including published research, and many professionals value this kind of evidence the most. However published research about TIS activities is not always available. Local evidence that an activity works also has an important role to play in the development of TIS activities.

Figure 1: Using the evidence to develop locally relevant services

This is one reason why the careful monitoring and evaluation of local activities is an essential part of the TIS program. Collecting accurate and thorough local data will help you to see what works best and this information can then be used to improve your activities. You can also share this with other TIS-funded organisations and contribute to the evidence on what works for TIS.

Sometimes collecting data can be challenging – the number of participants in a program may be small, or it may be that it can take a long time to see changes in smoking behaviour in the wider community. The NBPU TIS can support teams to address these and other challenges in order to develop the evidence base for TIS.

It is important to remember that the evidence does not provide a set of fixed solutions (it is not a ‘recipe book’). It is one element in an ongoing process. Your decision making will draw on your professional expertise about TIS and the local community with the evolving local and research evidence to develop a locally relevant service (Figure 1).

 

What is the current best evidence to support the reduction of tobacco use?

Evidence from a large-scale cohort study recently showed that Aboriginal and Torres Strait Islander adults living in TIS-funded areas were more likely to be smoke free at home, to smoke less tobacco and show lower rates of nicotine dependency compared to Aboriginal and Torres Strait Islander adults who lived in non-TIS areas.

Quitting smoking is the best way to reduce tobacco-related harm. Individuals may make several quit attempts before successfully stopping smoking for good. This is why health promotion activities and community development to support quitting are so important; it’s about continually reminding people of the importance of quitting and informing them about where to get help. It is important that as well as providing education about tobacco harms, organisations providing health promotion activities are able to refer people who want to quit to smoking cessation support services. Being part of a wider health service system or network is therefore essential for TIS teams.

Eligible and ineligible TIS activities

The table below outlines the eligible and ineligible grant activities for TIS teams as described in the Tackling Indigenous Smoking Regional Tobacco Control Grants (2023-24 to 2026-27) Grant Opportunity Guidelines (GO6775). You can read the previous version of the guidelines here: Indigenous Australians’ Health Programme Tackling Indigenous Smoking Regional Tobacco Control Grants (2023-24 to 2025-26) Grant Opportunity Guidelines (GO5803). For examples of these activities and how to monitor and evaluate them please download this factsheet.

Individual level activities for smoking cessation

TIS teams are funded to carry out population health promotion activities. They are not funded to provide smoking cessation support. TIS workers still need up-to-date knowledge of the support available to individuals wanting to quit, as this can inform TIS population health promotion campaigns and educational activities. The factsheets below provide an overview of how different individual level activities that can help support smoking cessation.