Monitoring and evaluation methods

There are two types of data that you can collect, quantitative and qualitative:

Quantitative data is numerical and includes audits or counts. For example you could count the number of attendees at an activity, the number of activities carried out, the number of referrals to a quit support group.

Qualitative data is the information you get when you gather people’s thoughts or feelings about an activity. You might gather this through an interview when you ask people to tell you in their own words what they thought about an activity, or what the benefits of going to a quit support group have been. However qualitative data can take many forms – people might also be asked to express their feelings through photographs, paintings, drama, or other imagery.

The type of data you collect will depend partly on the question you want answered but also on the sort of data collection methods that will work best in your community.

Data collection methods

Counts

Counting is the most basic (and easiest) type of data collection. You can count the number (N) of participants, the kind of participants (e.g. age, smokers, non-smokers, male / female), and the outcome of participation (e.g. N who were referred to Quitline or quit support services). It is important to make sure you collect good quality data by accurate and consistent counting (making sure you count the same thing in the same way each time).

Box 1: Example of a case study

Newest family member makes grandma think about quitting again

Quitting smoking is a tough job, and not every attempt to do so will be a successful one. The arrival of Karen West’s newest grandson, Tarnah, has made this grandmother rethink her smoking habit. Karen is from the Kaladoon mob near Mt Isa, and, although not a heavy smoker, she has smoked about 15 cigarettes a day for nearly 40 years. Karen has tried once before to quit and managed to cut her smoking down to two cigarettes a day using the support of Zyban. ‘It was really easy, and I didn’t feel like I needed any more than those two a day’.

While Zyban helped he with her cravings, her first attempt at quitting only lasted a short while. She can’t recall what it was exactly that led her to smoking again, but remembers feeling that once she had started she might as well keep going. Apart from Zyban, she didn’t use any other strategies to support her quitting attempt and doesn’t know if this would have made any difference.

Asked what impact smoking has had on her health, Karen feels that her fitness has been affected by breathlessness and she suffers from high blood pressure. She realises that if she can give up smoking, her fitness and blood pressure would improve.

It is the impact her smoking has on those around her that concerns her most. While she doesn’t smoke around her grandchildren, she would like to be a non-smoking role model for them as well. Her daughter would be Karen’s biggest support should she make another quitting attempt, and has strictly enforced the no-smoking rule around baby Tarnah.

Although Karen does not have a particular quitting strategy in mind as yet, she is beginning to think about how quitting would improve her life and the importance of having a good reason to motivate her in this difficult challenge. She doesn’t think that she will qualify to get Zyban on the PBS again this year, since she has already used it, and will have to think about other ways she can cope with the cravings she might feel both physically and behaviourally.

We thank Karen for sharing her story and wish her luck and encouragement in making the next step along the road to quitting.

Material adapted from:

Case study

Case studies are a useful way of understanding how different parts of a project have worked together to produce a particular outcome. Case studies focus on an individual person, workplace, process or type of activity. For example you might focus on how one person in the community has managed to successfully quit:

  • What helped them decide to quit?
  • What supports did they use to quit?
  • How did your project help in this journey?

Or you could reflect on how you were able to increase smoke-free environments in your region:

  • Where have smoke-free environments increased? (e.g. homes, workplaces, community events or venues)
  • Which of your activities helped this change happen? (e.g. running smoke-free events, supporting workplaces to put smoke-free policies in place, working with families to go smoke-free)
  • What were the challenges you faced and how did you overcome them? (e.g. where workers were reluctant to go smoke-free, gaining support from CEOs and providing education on benefits of being smoke-free and providing tips for staying smoke-free were most effective).

An example of a case study is provided in Box 1. You can find out more information in this 

resource.

Focus group/yarning group

A focus group is a planned discussion with selected individuals, which you use to gather information, or opinions – for example on your activities – or to measure local knowledge of the benefits of quitting, what quit support is available and so on.

The group should be held in a familiar and comfortable environment so participants feel at ease. It is good practice to provide snack and drinks so that participants feel welcome and valued. This also sets a friendly and informal tone for the session. The group should be run by someone who can keep the conversation ‘on topic’ without influencing what people say (they should be impartial).

It is important to know what you want to ask before going into the focus group – this

resource provides a guide to writing focus group questions. You can also find an example of questions for focus groups on TIS topics such as the second module in the 

on workplace smoke-free policies. You also need to think about how you will record what people say. Will a second person make notes on the discussion, or will you make a video or audio recording? If you use any kind of recording device, you must ask people’s permission first.

A more detailed guide on running a focus group can be found here 

.

Interviews

Interviews are one-to-one question and answer sessions. Interviews are usually best carried out face-to-face, although telephone interviews are also popular. Instant messaging can also be used if your participants have the technology and are happy to use it.

As with a focus group you need to choose a setting for the interview where you and your interviewee feel comfortable, and where there will be no interruptions. You also need to have planned your questions, and thought about how to record what the person you are interviewing says. Will you take notes, or make an audio recording? Again you must get the person’s agreement if you want to record them.

This

resource can be a helpful guide to developing and writing interview questions for evaluation. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found on the Resources to monitor and evaluate your program page. More general information can be found in this 

factsheet.

Observation

There are two types of observation:

  • direct observation – this is where the observer is an unidentified ‘fly on the wall’ (you will need permission if you want to observe people, whether in a workplace or school or other venue/location)
  • participant observation – where the observer takes part in an activity with the participants and asks questions.

Observation can be useful when you want to find out if people are keeping to smoke-free policies. A checklist, such as the example provided by the Aboriginal Health & Medical Research Council (AH&MRC) in its Aboriginal tobacco resistance toolkit is a good way of recording your observations:

Questionnaire survey

This is a good method of gathering information from a large number of people. This information is collected by asking everyone the same questions. The kind of data you get will depend on the questions you ask. For example questions which give people a fixed set of answers (closed questions) will give you quantitative data:

Question: Did you find the quit support group helpful?
Answer: Yes/No

You can then count how many people say ‘yes’ and how many say ‘no’. Or you could ask them to rate how satisfied they are with the support group on a scale of 1-5:

Question: How helpful did you find the support group?
Answer: 1 2 3 4 5
Very Helpful Somewhat Helpful  Neither helpful nor unhelpful Somewhat unhelpful Very unhelpful

You can then count how many found the group very helpful, somewhat helpful and so on. However questions with yes/no answers or simple rating scales like this might not tell you everything you want to know. You might want to ask a question that allows people to give a free written response;

Question: What did you think of the quit support group?
Answer: ____________________________________

People can write what they like and this will give qualitative data. An advantage of this is that you will get much more detailed information about what people think. The disadvantage is that you might have to spend a lot of time organising answers so as to make sense of what is being said. The type of question you ask and the response choices you give people are therefore very important.

A helpful guide to developing and writing survey questions for evaluation can be found here

. Examples of the kinds of questions you might want to ask about knowledge of smoking harms and tobacco use can be found in the 

 on the Resources to monitor and evaluate your program page along with a template for a post workshop feedback questionnaire that you could use to find out about participant satisfaction with activities. An example of using a questionnaire for evaluation can be seen in Box 2.

You might also want to find out more about what members of your project team and experts in the community think about the importance of your organisation’s activities for the community as a whole. To do this you can carry out something known as a 

.

Box 2: Using a questionnaire to evaluate an activity

During the delivery of ‘Operation Smoke Signals’, an evaluation was done to see whether it was meeting the project’s objectives. Using a questionnaire, participants were asked for their feedback about the quitting course to measure their satisfaction levels. They were asked questions, such as:

  • Was the venue suitable, the session times convenient and the facilities adequate?
  • Did they find the content useful?
  • What did they like about the course?
  • What didn’t they like about the course?
  • How would they improve the course?

The results of the questionnaire showed that participants felt comfortable taking part in the course, that they found the staff friendly and the topics covered helpful. However, some had to drop out because they had problems with transportation and child care. To improve the running of the course, the local community bus is now picking up those participants who need a lift, and child care is being offered by the health service.

Material adapted from:

Story telling

Stories are a good way of capturing individual experiences. They are usually verbal, but can use a range of other methods such as writing, drawing, drama, web sites/blogs. When using storytelling for evaluation the focus is on how individuals or groups make sense of their experiences. Story telling is increasingly being recognised as a way of capturing significant changes in people’s lives.

A guide to using verbal story telling for evaluation, and how to use templates to collect significant change stories can be found here

. You can also find a significant change story template here or on the Resources to monitor and evaluate your program page. Section four of the

also has some useful information and advice on gathering stories.

Examples of other creative approaches to gathering data including playing games, creating a storyboard or using photography can be found in this 

resource.