Whenever possible, use standardized tests, especially when measuring concepts that are not directly observable. Standardized tests are also questionnaires, but consist of multiple questions to measure an underlying construct and have been tested for psychometric properties. By psychometric properties we are referring to the reliability, validity and scoring of the instrument. Some questionnaires may be developed by agencies but have not been tested psychometrically, which means you do not know how accurate they are. Reliability is an estimate of the accuracy of the test. There are several different types of reliability but the two main ones are (1) test re-test reliability (the extent to which you can expect to get the same score if you asked the same person to answer the questions a second time) and (2) internal reliability (the extent to which each question is related to each of the other questions in the test). Validity is a measure of how well the test measures what you want it to measure. This is usually determined by (1) how well the test result agrees with other related measures such as clinical interviews or related psychometric tests (e.g., a correlation between a measure of gambling problems might be validated by correlating it with financial problems), (2) its ability to predict behaviour (e.g., who will do well in treatment vs. who will not do well), or (3) its ability to discriminate between populations (e.g., clients in treatment vs. the general population, e.g., Lesieur & Blume, 1987). For more information on test quality, see Shaughnessy & Zechmeister (1997) or another introductory text on statistics and methodology. The standardized scoring key is essential in that everyone using the test interprets a similar score in the same way (e.g., a score of 0 on the sogs indicates no problem; a score of 10 indicates a severe problem).

The method is how the data are collected. The main methods of collecting quantitative data are:

• by telephone
• in an interview, with questions asked by the interviewer (e.g., an assessment, termination or follow-up interview)
• self-report, where respondents complete the questionnaire on their own, without help from an interviewer. This can be done in person or by mail.

Quantitative methods are good for collecting a large amount of data from a large number of people, because the scoring, data entry and data analysis are relatively easy and quick.

Whenever possible, use standard tests or existing questionnaires to collect data. However, when that is not possible, a new questionnaire may need to be constructed.

Writing Questions for Questionnaires
Collecting quantitative data usually requires the use of questions (but not always: urine drug screen results are also quantitative data, for example).

The following is a guide to writing good questions (Shaughnessy & Zechmeister, 1997).

1. Questions should use simple, direct and familiar words.

• Keep words as simple as possible
• Use abbreviations sparingly and only after using the full name in conjunction with the abbreviation on the same page.
• Avoid foreign phrases, slang words and technical jargon

2. Always prepare clear and simple questions.

• The words you choose must not be ambiguous. They must have the same meaning for all respondents.

3. Avoid double questions.

• Don’t use questions that refer to two ideas at the same time.
• Each question should only ask about one particular idea.
• Questions should not contain words such as and, or, also, either or but.

4. Avoid using double negatives within questions.

• Negatives are in general harder to read. Many people will misread the sentence and answer incorrectly.

5. Don’t use long and extensive questions.

• Long and extensive questions often confuse and tire the respondent.

6. Avoid leading or biased questions.

• Do not use words that suggest to the respondent a certain way of answering or a desirable response.
• Do not use words that result in the respondent being embarrassed, uncomfortable or ashamed.

7. Try to use questions that are applicable to all respondents.

• Don’t assume that all of the questions will apply to all respondents.
• Not only will some questions not apply, but also certain response categories may not apply to all respondents.
• Do not assume too much about the respondent’s behaviour and/or situation.
• Use screening questions if a large number of questions apply to specific groups of clients (e.g., If you have never gambled at a casino, skip to question 5).

8. Don’t use objectionable questions.

• Personal questions, questions that convey negative implications and incriminating questions should be avoided.
• If you must ask an objectionable question or a sensitive question, leave the question until late in the questionnaire, and ask for responses in broad categories.
• Use a series of questions to overcome the objectionable nature of the question.

9. Ensure that you have exhausted all response categories.

• Unless you are sure that a set of response alternatives completely covers all the options, use partially close-ended questions (e.g., other).

10. Since most people pay more attention to the first part of a sentence, if the sentence includes conditional statements (e.g., if, when) state the condition first. For example, “If you were forced to leave your present job, what type of work would you seek” is better than “What type of work would you seek if you were forced to leave your present job.”

11. Carefully consider the order of the questions

• The first few questions often set the tone. It is often best to begin questionnaires with the most interesting questions and leave questions about demographics to the end.
• In a telephone interview, start with general questions to help establish rapport.
• Generally, it is preferable to funnel from more general issues to more specific issues.
• Never begin with controversial issues.
• Consider very carefully how responses to one question might affect another. If you begin with questions about problem gambling and then ask about gambling frequency, people might report less gambling than if you start with questions about frequency and follow that with questions about problems.

Steps to Develop a Questionnaire

1. Choose the method you will use to gather the information.

• Will it be by phone, personal interview or self-report questionnaire?

2. Identify the content areas for the questionnaire.

• What are the topics you want to ask about?

3. Take the time to review existing questionnaires.

• You may not have to create something from scratch.
• You may be able to use questions from existing surveys that have already been shown to be reliable and valid.

4. Once you have prepared the draft, circulate it for comment and review.

• Share the questionnaire with staff and colleagues.
• Consider sharing it with a sample of clients.

5. Pre-test the questionnaire.

• Pre-test the questionnaire with a small sample to clients.
• Ask clients for feedback on the process and the content.
• Make changes accordingly.

6. Develop the final version of the questionnaire.

• Once you have revised the instrument, you are ready to begin your data collection.

Qualitative Data

Qualitative data cover a wide variety of information and research methods that most often take the form of written or verbal statements that are collected using unstructured or semi-structured interviews or questionnaires (Crabtree & Miller, 1992).Asking clients “What do you think helped you stop gambling the most?” is an example of a question that will yield qualitative data. In some cases such open-ended questions may be quantified, but in other cases these responses are analyzed using qualitative methods (see Crabtree & Miller, 1992).

Qualitative methods can be an important adjunct to an evaluation. When collecting quantitative data one often feels too remote from the participants to understand what their responses mean. Qualitative methods, in contrast, try to obtain a rich amount of information about an individual so that the individual’s ideas, opinions and experiences can be understood in more detail.

Qualitative research tends to be very labour-intensive and is usually only good for small studies. A problem with qualitative research is that the findings are often a subjective interpretation of the data by the researcher so that two researchers might interpret the same data somewhat differently and draw different conclusions. The problem can be overcome by creating a standard “coding dictionary” and having more than one person code the same data. A comparison of the scoring can be used to determine reliability of the coding system. These codes are sometimes treated as quantitative data; other times responses are grouped according to general themes. Thematic organization of the data usually reaches a saturation point where no new themes are uncovered after a relatively small number of cases are analyzed. Consequently, once the coding dictionary has been created, the subsequent analysis can progress more rapidly.

Qualitative methods provide the best opportunity to uncover new ideas and explore processes that may be poorly understood (e.g., “What aspects of this program helped you the most?” “Was there anything else going on in your life at the time that also seemed to help?”).

The main tools for collecting qualitative data are:

• Structured questionnaires
• Semi-structured questionnaires

The main methods for collecting qualitative data are:

• Focus groups
• Individual interviews (in person, or by telephone)

Focus Groups: A focus group is a small group of people that is chosen (not randomly) because they are typical of the people that you are concerned about or who have a stake in the program (e.g., members of the target group, family members, treatment professionals). The focus group might begin with a general discussion that includes all participants, followed by a breakout procedure in which subgroups are put together to focus on specific issues. The smaller groups are then usually brought back together and the ideas from each subgroup as discussed. Ideas can be recorded on a flip chart, then the participants can discuss and evaluate the importance of each idea. A focus group for a gambling treatment process might address issues such as perceived barriers to treatment, client retention, addressing the clients’ needs and developing appropriate outcome measures.

When collecting qualitative data it is recommended that the interview or focus group be audiotaped. This allows the interviewer or focus-group leader to make other “field notes” during the interview or focus group, while being able to review the audiotape at a later date.

There are excellent manuals that describe the process of conducting a focus group in a step by step fashion (e.g.,Morgan & Krueger, The Focus Group Kit). Individual Interviews: Qualitative data can also be collected by doing individual interviews with former clients, family members or other professionals.

Individual interviews are face-to-face (or telephone) meetings conducted according to an interview schedule that keeps the interview on track. This method allows for a very detailed exploration of the evaluation issue in question. Vague responses can be probed for greater specificity. However, this is a time-intensive data collection method, as one or two hours may be required to collect qualitative data from one person.

Back to Program Evaluation for Problem Gambling Services