View
1
Download
0
Category
Preview:
Citation preview
Study the scenario carefully before you consider the discussion questions. Always unpack an assessment scenario in terms of the following:
The assessment context The school community
The concern raised in the assessment context Illegal drug use amongst learners at the school
The owner of the concern The school board and parent association
The consultant The psychologist who is responsible to address the concern
The intention of the consultant To develop and tailor prevention programmes to accommodate the needs and circumstances of the learners
The test user The psychologist who uses the questionnaire (In this scenario the consultant and the test user is the same person, namely the psychologist)
The intention of the test user To identify and target learners who are at risk for illegal drug use
The group targeted for assessment The learners at the school
The questionnaire The Alcohol use Intention Questionnaire (AIQ)
The intention of the questionnaire To determine the level of an individual’s intention to use alcohol
Question 1 is your statement that it is your own work.
Question Discussion
2 Lindi: I presume a good place to begin this discussion is the intention of the AIQ
Banti: Well, the intention of the questionnaire clearly is to
determine the level of an individual’s intention to use alcohol Carl: No, the questionnaire’s intention is to identify and target
learners who are at risk for illegal drug use Jan: Actually the intention of the questionnaire is to assess
learners at the school You agree most with, second most with and least with: Banti Carl Jan
Banit’s response is correct. She indicates the intention of the questionnaire correctly. Carl identifies the intention of the test user (the psychologist) as the intention of the questionnaire. This is not correct. Although Carl refers to a reason for using the test, in this case the test user’s intention is not identical to the intention of the questionnaire. But Jan’s response is worse than Carl’s. Jan’s interpretation of the intention of the questionnaire does not refer to a reason for using the questionnaire. He refers to the target group of the assessment exercise.
3 Gcina: Do you think the questionnaire is fit for purpose in light of the intention of the test user?
Jan: No I do not think so, because the intention of the test user is
to identify and target learners who are at risk for illegal drug use whereas the intention of the AIQ is to determine the level of an individual’s intention to use alcohol
Lindi: No, because the intention of the questionnaire is to identify and target learners who are at risk for illegal drug use which falls short of the intention of the test user namely to tailor prevention programmes to accommodate the needs and circumstances of these learners
Carl: I think the questionnaire is fit for purpose because there is a relationship between the level of an individual’s intention to use alcohol, which is the intention of the AIQ, and the individual’s risk for illegal drug use, which is what the test user wants to find out
You agree most with, second most with and least with: Carl Jan Lindi
Carl’s response is correct. He identifies the intention of the questionnaire and the intention of the test user correctly, and he draws attention to the relationship between the intention of the questionnaire and how the test user wants to use the results. This relationship was reported in subject literature. Jan offers the correct information about the intention of the questionnaire and the intention of the test user, but because these are different he concludes that the test user cannot use questionnaire as he/she intends to do. In other words, Jan forgets (or does not know) about the subject literature that Carl refers to. Lindi’s response is incorrect. She not only confuses the intention of the questionnaire and the intention of the test user (i.e. what the psychologist wants to find out), she also confuses the intention of the test user and the intention of the consultant (i.e. how the psychologist wants to react to the concern that sparked the assessment).
4 Gcina Do you think the questionnaire can help to address the concern of the school board and parent association?
Carl: Yes, because we can use the AIQ to identify and target
learners who are at risk for illegal drug use Lindi: Yes, but then we have to know how to tailor prevention
programmes to accommodate the needs and circumstances of these learners after we identified and targeted learners who are at risk for illegal drug use
Banti: No I disagree, because the intention of the AIQ is to determine the level of an individual’s intention to use alcohol, which does not mean we can identify and target learners who are at risk for illegal drug use
You agree most with, second most with and least with: Lindi Carl Banti
Lindi’s response is most correct. She understands that the result of the questionnaire can be used to identify and target learners who are at risk of illegal drug use. But she also realises that this result does not automatically lead to prevention programmes. One has to know how to tailor such programmes to accommodate the needs and circumstances of the identified learners. Carl’s response is correct, but he does not consider the fact the assessment result does not automatically result in implemented. One has to know how to implement the result (as Lindi indicates). Banti’s realises that that there is a difference between the intention of the questionnaire and the intention of the test user, but she forgets (or does not know) about the relationship between the intention to use alcohol and drug abuse reported in subject literature. Thus she comes to an incorrect conclusion.
5
Gcina Do you think the AIQ is the correct type of questionnaire for our assessment purpose?
Jan: Yes, I think so. the AIQ is a survey type questionnaire and we
need a survey type questionnaire to gather information that allows us to identify and target learners who are at risk for illegal drug use
Lindi: No, the AIQ is not a survey type questionnaire, and it is not the type of questionnaire that is suitable to identify and target learners who are at risk for illegal drug use
Carl: Actually the AIQ is a survey type questionnaire, and it is suitable to assess learners at the school
You agree most with, second most with and least with: Jan Carl Lindi
The questionnaire is an example of a survey type questionnaire, and a survey type questionnaire is the correct type of questionnaire to use to determine how people perceive and feel about matters. Thus Jan’s response is most correct. Carl’s response is half correct and half incorrect. He is correct in identifying the AIQ as a survey type questionnaire, but he is incorrect to identify the assessment of learners at the school as the purpose of the assessment. He confuses the group to be assessed with the purpose of the assessment. Lindi’s response is incorrect on two accounts. She wrongly indicates that the AIQ is not a survey type questionnaire, and she is incorrect when she says that the AIQ cannot be used to identify learners who are at risk of illegal drug use.
6
Jan: Given the intention of the AIQ, what is its content domain? Carl: I would say the content domain of the AIQ is learners at the
school Banti: I think the content domain of the AIQ is attitude towards
using alcohol Lindi: No, I think the content domain of the AIQ is the psychological
factors associated with the intention to use alcohol You agree most with, second most with and least with: Lindi Banti Carl
The content domain is the area defined by the psychological construct that underlies the questionnaire. The intention of the AIQ is to determine the level of a person’s intention to use alcohol. Therefore Lindi’s response is most correct. The content domain of the AIQ (given its intention) is the psychological factors associated with the intention to use alcohol. Banti’s response is correct but incomplete. Attitude towards using alcohol is part (but not the whole story) of the intention to use alcohol. Carl’s response is incorrect. The group targeted for assessment is not the content domain of the questionnaire.
7 Banti: I am concerned about the content domain of the AIQ in light of the intention of the AIQ
Jan: I do not think there is need for concern. The questionnaire’s
content domain is theoretically determined as attitude towards using alcohol, subjective norm concerning the use of alcohol, and perceived control over using alcohol
Gcina: No I think there is need for concern. The questionnaire’s content domain is over determined. Perceived control over the use of alcohol should be deleted from the questionnaire’s content domain
The intention of the AIQ is to determine the intention to use alcohol. The intention to use alcohol is theoretically determined by the theory of planned behaviour. The intention to execute behaviour involves three factors as indicated in the scenario. Therefore there is no concern about the content domain of the AIQ. One could be concerned about the AIQ itself though because it only measures two of the three factors, namely attitude towards alcohol consumption and subjective norm concerning the use of alcohol. Biographical information and recreational behaviour are additional components of the AIQ. They may be included for other reasons, but they do not tap into the content domain associated with intention to use alcohol (which the questionnaire intends to measure). Therefore Jan’s response is correct. Gcina’s response is incorrect. He wants to limit the questionnaire’s content domain to the factors included in the questionnaire. This is not correct because the questionnaire is
Lindi: I agree, Gcina, but I want to add that perceived control over use of alcohol should be replace by biographical information and recreational behaviour
You agree most with, second most with and least with: Jan Gcina Lindi
determined by the content domain. The questionnaire does not determine the content domain. Lindi’s response is even worse because she wants to add factors to the content domain (biographical information and recreational behaviour) that do not concern the intention to use alcohol.
8 Carl: What is the scope of the AIQ? Banti: The scope of the questionnaire is the psychological factors
associated with the intention to use alcohol Jan: No, I think the scope is biographical information, attitude
towards alcohol consumption, subjective norm concerning the use of alcohol, recreational behaviour
Lindi: Actually the scope is attitude towards alcohol consumption and subjective norm concerning the use of alcohol
You agree most with, second most with and least with: Jan Lindi Banti
Jan’s response is most correct. The scope of the questionnaire refers to what the questionnaire measures, which are the factors he indicates in his response. Lindi’s response is partially correct, but incomplete. She limits the scope of the questionnaire to the content domain factors included in the questionnaire. The scope of the AIQ is broader than the factors indicated by Lindi. Banti’s response is incorrect. She identifies the content domain as the scope of the questionnaire. Note that a questionnaire should assess all the factors in its content domain, but the questionnaire is not necessarily limited to the factors in the content domain. It may include additional factors as for, example, biographical information.
9 Lindi: I am not convinced that the scope of the AIQ is sufficient given its intention.
Banti: I share your concern, Lindi. The items included in the AIQ are
not sufficient to determine the level of an individual’s intention to use alcohol
Jan: No, I am not concerned about the scope of the AIQ. It has sufficient scope to assess all learners at the school
Carl: Actually the scope of the AIQ is more than sufficient because it includes biographical information and recreational behaviour
You agree most with, second most with and least with: Banti Carl Jan
Banti’s response is correct. The items included in the AIQ are not sufficient because the AIQ does not include items to assess perceived control with regard to the use of alcohol, which is one of the factors of the intention to use alcohol. Carl’s response is incorrect because he misses the point that despite its broader scope (also including biographical information and recreational behaviour) the AIQ assesses only two of the three content domain factors. Jan’s response is really poor. The scope of the questionnaire is not defined by the group targeted for assessment.
10 Banti: What psychological construct is assessed by the AIQ? Gcina: learners at the school Lindi: intention to execute behaviour Carl: attitude towards using alcohol You agree most with, second most with and least with: Lindi Carl Gcina
Lindi’s response is most correct. The psychological construct is intention to execute behaviour. Carl’s response does not refer to the construct itself but to the application of the construct in the explanation of a particular behaviour, namely the intention to use of alcohol. Gcina’s response is absolutely incorrect. The group targeted for assessment certainly is not a psychological construct.
11 Carl: Which items of the AIQ do not contribute to the intention of the questionnaire?
Jan: All items concerned with biographical information and with
recreational behaviour Gcina: All items concerned with attitude towards alcohol
consumption and with recreational behaviour Lindi: All items concerned with attitude towards alcohol
consumption and with subjective norm concerning the use of alcohol
You agree most with, second most with and least with: Jan Gcina Lindi
Jan’s response is most correct. Biographical information and recreational behaviour are not content domain factors. Therefore items assessing these aspects do not contribute to assessment of the intention to use alcohol. (Note that this does not mean that these items may not be included in the AIQ. There may be other reasons for their inclusion). Gcina’s response is partially correct, but also contains an incorrect component. Items concerning recreational behaviour do not contribute to the assessment of intention to use alcohol. However, items concerning attitude towards alcohol consumption do contribute to the assessment of intention to use alcohol. Lindi’s response is incorrect. The two factors she refers to are factors that are part of the intention to use alcohol.
12 Gcina: People who complete the AIQ are informed that by answering the questions they indicate how they feel about the use of alcohol. Should this information be included in the questionnaire instructions?
Carl: It should be included because it refers to the reason for
completing the questionnaire Lindi: II think it should not be included because people who
complete the questionnaire should not be informed that by answering the questions they indicate how they feel about the use of alcohol
Jan: It can be included, but it is not necessary. The only information that should be included is to read all questions carefully and to select the answer that best describes the respondent’s thoughts and feelings
The instructions of a questionnaire should indicate to respondents why they complete the questionnaire. In this case it is to indicate how they feel about the use of alcohol. Thus Carl’s response is correct. Jan is correct in indicating that respondents should also be told how to complete the questionnaire, but he is incorrect when he suggests that it is optional to provide the reason for completing the questionnaire. Lindi offers a poor response. It is not correct to suggest that respondents should not be informed why they complete a questionnaire.
You agree most with, second most with and least with: Carl Jan Lindi
13 Lindi: Do you think the instructions of the AIQ encourage respondents to provide truthful answers?
Banti: Yes it does, because respondents are informed to read all
questions carefully and to select the answer that best describes the respondent’s thoughts and feelings
Gcina: No it does not, because they are informed that the school psychologist has access to the respondent’s answers
Carl: Yes it does, because they are informed that there are no right or wrong answers and that their opinion counts
You agree most with, second most with and least with: Carl Gcina Banti
Respondents are encouraged to provide truthful answers when they are told that there are no right or wrong answers and that their opinion counts. Carl’s response is most correct. Gcina’s response is incorrect. The fact that respondents are told that only the school psychologist has access to their responses should assure them that their responses will be treated confidentially, and therefore they should be more inclined to respond truthfully. Banti confuses truthfulness with accuracy. Telling respondents how to complete a questionnaire increases the accuracy because it minimises erroneous responses, but accuracy does not mean the respondents are truthful in their responses.
14 Gcina: Does the indication to the AIQ respondents that only the school psychologist has access to the respondent’s answers assure anonymity?
Lindi: No, it does not assure anonymity Banti: It is an assurance of confidentiality but not anonymity Carl: It is not an assurance of any kind. It merely encourages
respondents to complete the AIQ You agree most with, second most with and least with: Banti Lindi Carl
This part of the instruction is an indication that respondents’ information will not be available to anybody. It will only be seen by the psychologist. As such it is an assurance of confidentiality. But confidentiality should not be confused with anonymity. Confidentiality does not mean that the psychologist cannot identify the individual respondents. It simply means that he/she will not divulge the identities of the respondents. Both Banti and Lindi’s responses are correct, but Banti offers a better response than Lindi because she offers more information. She refers to confidentiality and anonymity, whereas Lindi only refers to anonymity. Carl’s response is incorrect. Contrary to his belief the instruction is an assurance rather than an encouragement to complete the AIQ.
15 Banti: We have to make sure the AIQ contains different types of questions. So I think we should identify some of the question types. For example, what kind of question is 2.8?
Carl: It is a rating scale item Gcina: It is a Likert scale item Lindi: It is semantic differential item
The item is a rating scale, and more precisely a Likert type rating scale. Gcina’s response is more correct than Carl’s because he indicates the exact nature of the item, whereas Carl merely indicates the more general nature of the item. Lindi’s response is incorrect. The item is not a semantic differential.
You agree most with, second most with and least with: Gcina Carl Lindi
16 Banti: What kind of question is 4.5? Carl: It is an open fill-in question Gcina: No, it is a closed fill-in question Lindi: I think it is a closed true/false question You agree most with, second most with and least with: Carl Gcina Lindi
The question is fill-in question type. In this regard Carl and Gcina are correct. But on the issue of it being an open or closed question, Carl is correct and Gcina incorrect. Lindi offers a poor response. The question is not a closed true/false question.
17 Banti: But what is the difference between questions 3.6 and 3.5? Carl: There is no difference. They both are checklist question types Gcina: No, I think 3.6 is a checklist question type and 3.5 is a MCQ
question type Lindi: Actually 3.6 is a MCQ question type, whereas 3.5 is an
inventory question type You agree most with, second most with and least with: Lindi Carl Gcina
Lindi is most correct. 3.6 is a MCQ type question and 3-5 is a checklist (also called an inventory) question type. A MCQ requires respondents to select one of the alternative responses. A checklist allows respondents to select one or more than one response. Carl’s response is partially correct because one of the questions is a checklist. However Gcina’s response is incorrect with regard to both items.
18
Banti: We need a semantic differential type question. Gcina: Then question 3.8 would be a good choice Lindi No, question 2.9 would be a better choice than 3.8 Carl: I disagree, Lindi. You cannot use question 2.9. it is Likert type
rating item, not a semantic differential type You agree most with, second most with and least with: Gcina Carl Lindi
Question 3.8 is a semantic differential question type. Thus Gcina’s response is most correct. Lindi’s response is incorrect. She confuses a Likert type rating item with a semantic differential question type. Carl’s response is correct. He correctly identifies question 2.9 as a Likert type question, but he does not suggest a semantic differential question type or indicate whether or not he agrees that 3.8 is a semantic differential question type.
19 Lindi: Table 4 shows 10 respondents’ responses to item 2.10. What do these responses tell us about the respondents’ attitudes towards alcohol consumption?
Gcina: The respondents’ responses are skewed towards the
disagree side of the scale
The responses to this item are mostly 1 or 2. Most respondents disagree or disagree strongly with the statement of the item, namely that one should not drink and drive. Thus most respondents think one can drink and drive, which reflects a positive attitude towards the use of alcohol. Both Jan and Gcina’s responses are correct, but Jan’s response is better than Gcina’s. Gcina simply indicates the obvious fact the responses are skewed towards the disagree side of the scale. Jan’s response indicates what the skewed responses mean in terms
Jan: The respondents express a positive attitude towards alcohol consumption
Banti: I think the respondents were indecisive about alcohol consumption
You agree most with, second most with and least with: Jan Gcina Banti
of what the item measures, namely attitude towards the use of alcohol. Banti’s response is incorrect. The group does not show indecisiveness because their responses are clustered towards the disagree side of the scale. If there responses were evenly spread across the scale one could say that the group’s response was indecisive.
So far our discussion was fairly general. But I think we should discuss the psychometric properties of the questionnaire and the questions in more detail. To begin with, how good is your knowledge about aspects such as validity and reliability?
20 Jan: So far our discussion was fairly general. But I think we should discuss the psychometric properties of the questionnaire and the questions in more detail. To begin with, how good is your knowledge about aspects such as validity and reliability? Gcina: A questionnaire is valid if it actually measures what it is
supposed to measure, and it is reliable if it provides consistent measures of whatever it measures
Lindi: I agree that a questionnaire is valid if it measures what it is supposed to measure, but reliability is not about consistency. A questionnaire is reliable if it is administered to the people it is supposed to be administered to
Carl: I do not agree with your understanding of validity. A questionnaire is valid if its questions are properly formulated. But I do agree that a questionnaire is reliable if it is administered to the correct people
You agree most with, second most with and least with: Gcina Lindi Carl
Gcina’s response is correct. He knows what the difference is between validity and reliability. Lindi knows what validity is, but she misunderstands reliability. Carl clearly has no idea of validity and reliability.
21 Jan: Ok, but I still not sure about validity and reliability when we talk about an actual questionnaire such as the AIQ
Lindi: The AIQ is valid if it really determines the level of an
individual’s intention to use alcohol, and it is reliable if it
Lindi’s offers the best response. She understands the meaning of validity and reliability and she is able to apply her knowledge. She identifies the intention of the questionnaire correctly, which is to determine the level of an individual’s intention to use alcohol. The AIQ is valid if it really measures an individual’s intention to use alcohol, and it is reliable if these measures are consistent. In his response Gcina refers to the intention of the test user, namely to identify
provides consistent measures of an individual’s intention to use alcohol
Gcina: The AIQ is valid if it identifies and targets learners who are at risk for illegal drug use, and it is reliable if it is used to measure all learners at the school
Banti: The AIQ is valid when there is a high correlation between its even items and its odd items, and it is reliable if it is used to assess all learners at the school
You agree most with, second most with and least with: Lindi Gcina Banti
and target learners who are at risk of illegal drug use. Although this is not a direct indication of the validity of the questionnaire the questionnaire has to be valid (i.e. provide a true measurement of the intention to use alcohol) in order for it to be used to identify and target learners who are at risk of illegal drug use. However, Gcina clearly does not know what the reliability of the AIQ refers to. Banti offers a similar response with regard to reliability, which means she is wrong in this regard. However, the first part of her response is also incorrect. Here she refers to the split-half reliability of the questionnaire, not to its validity.
22 Gcina: What is the difference between the information provided in Table 2 and in Table 3?
Carl: Table 2 shows the validity of the AIQ and Table 3 shows its
reliability Lindi: I think both tables provide information about the reliability
of the AIQ Banti: Actually Table 2 is about the AIQ’s internal inconsistency and
Table 3 is about its test-retest reliability You agree most with, second most with and least with: Banti Lindi Carl
Lindi is correct. Both tables are about reliability. The correlation between two halves of the questionnaire is a measure of internal consistency, which is a form of reliability, and the correlation between two applications of the questionnaire refers to its test-retest reliability. Banti’s response is better than Lindi’s because she correctly indicates specific information about reliability, which Lindi does not do. Carl’s offers the least correct response. He is correct about Table 3 showing reliability, but he is incorrect about Table 2 being about validity.
23 Carl: What does Table 2 in the manual tell us about the AIQ? Jan: The table shows that the AIQ was split into two halves, and it
indicates the total scores of five respondents for each half Lindi: The table shows a high correlation between the two halves
of the AIQ Banti: The table shows that the AIQ has a high degree of internal
consistency You agree most with, second most with and least with: Banti Lindi Jan
All three responses are correct, but they differ in terms of the value of the information they provide. Jan merely observes and describes the information provided in the table. Lindi adds information about the relationship between the two sets of values, namely that these values are positively correlated. Banti adds a further insight. She understands that the correlation means a high degree of internal consistency.
24 Gcina: I am unsure about correlations. How do you see whether there is a correlation between numbers?
Carl: Remember, correlation is between two groups of numbers.
There is a correlation if you can split the numbers into two groups
Lindi: You can see there is a correlation when high numbers in one group correspond to high numbers in the other group
Jan: And, there is no correlation when high numbers in one group correspond to low numbers in the other group
You agree most with, second most with and least with: Lindi Jan Carl
Carl is correct in suggesting that correlation is about the relationship between two groups of numbers, but he clearly does not understand what correlation really means. The fact that a set of numbers can be split into two groups does not mean that the two groups are related. Lindi offers the best response. She understands that the numbers in the two groups need to be paired. Correlation is about pairs of numbers. One member of each pair comes from one group and the other member of the pair comes from the other group. There is a correlation if the numbers in each pair has more or less similar values. Or, as Lindi puts it, if one member of a pair has a high value the other member’s value should also be high. There is an inverse correlation (i.e. a negative correlation) if in each pair of numbers one number is high and the other is low. This is what Jan refers to, but unfortunately Jan thinks this means no correlation. He is incorrect. What he refers to is a high but negative correlation. No correlation would be when there is no pattern in the values of the members of each pair of numbers. In other words, when there is no clear indication that high numbers in one group are paired with high numbers in the other group, or that high numbers are paired with low numbers.
25 Gcina: The other table in the manual I am unsure about is Table 3. What does the information in this table tell us about the AIQ?
Banti: The information in the table shows a poor correlation
between the two sets of scores, which means the AIQ did not demonstrate test-retest reliability
Jan: We should keep in mind that the poor correlation between the two sets of values refers to the criterion-related validity of the AIQ and not to the reliability of the AIQ
Carl: Actually the information in Table 2 does not reflect a poor correlation, which means the AIQ shows good criterion-related validity
You agree most with, second most with and least with: Banti Jan Carl
Table 3 shows a poor correlation between the two groups of scores. This correlation refers to two administrations of the questionnaire. As such it refers to the questionnaire’s test-retest reliability. A poor correlation means the test-retest reliability is low. Banti understands all of this. and offers the best response. Jan understands that the two groups of scores are poorly correlated, but he thinks the relationship between the two administrations of the questionnaire indicates criterion-related validity. This is incorrect. Carl really does not know what he is talking about. The correlation is poor, not good. And, it does not refer to criterion-related validity.
26 Gcina: So, if you compare the information from Table 2 with the information from Table 3, what do you think?
Carl: The AIQ has high test-retest reliability but its criterion
related validity is questionable Lindi: The items of the AIQ seem to measure the same thing, but
the questionnaire is vulnerable to external influences
Table 2 shows high internal consistency, and Table 3 shows low test-retest reliability. Lindi understand this, and she interprets the implication of this information correctly. Internal consistency refers to the items in the questionnaire measuring the same thing. This means the questionnaire is solid. Its items are bound together tightly. But when this solid questionnaire is administered on two separate occasions the scores correlate poorly. This means factors external to the questionnaire influence the scores. Banti and Carl do not demonstrate Lindi’s insight. Banti’s response holds some value because one of the two tables
Banti: The information obtained from these two tables supports the reliability of the AIQ
You agree most with, second most with and least with: Lindi Banti Carl
(but not both) supports the reliability of the questionnaire. Carl’s response is totally incorrect. The test-retest reliability is low, not high, and the none of the two tables provide information about the criterion-related validity of the questionnaire.
27 Jan: Will you keep item 2.3 in the AIQ or delete it? Banti: I will delete item 2.3 because it has low item-total correlation
and low item variance Lindi: I think the item shows high item variance, but one cannot
keep an item that has low item-total correlation Carl: I disagree. I will keep the item because it has low item
variance and high item-total correlation You agree most with, second most with and least with: Lindi Banti Carl
Item 2.3 has high item variance (the scores are distributed across the measurement scale), but low correlation (no clear pattern of the relationship between the item scores and the total scores). An item should have high variance and high correlation. Lindi’s response is correct. One cannot keep the item in the test because it has low item-total correlation. Banti’s is correct in suggesting that the item should be deleted from the questionnaire, but her reasons for doing so are only partially correct. The item does not show low item variance. Carl’s response is totally incorrect.
28 Lindi: What will you do about item 2.6 in the AIQ? Banti: I will keep the item in the AIQ because the negative
correlation indicates an inverse discrimination value Gcina: I will delete the item from the AIQ because the negative
correlation means a problem with the discrimination value of the item
Carl: I am not sure whether I will keep or delete the item. I will first check how the item was scored before I decide
You agree most with, second most with and least with: Carl Gcina Banti
Item 2.6 has high item variance and high negative item-total correlation. Gcina is correct in stating that the item should be deleted because the negative item-total correlation indicates a problem with the item’s discrimination value. Banti is incorrect because she should not keep the item. She also refers to an ‘inverse discrimination value’. Discrimination means differentiation between the respondents. An item can differentiate or not differentiate. There is no inverse differentiation. However, Carl provides the best response. He realises that the negative item-correlation may be because the item was scored incorrectly. An item that requires inversed scoring but is not scored as such may show an inverse relationship when the relationship should be positive. Therefore one should always check the scoring of an item with strong negative item-total correlation before one decides whether or not to delete the item. However, if its scoring is in order a negative correlating item should be deleted.
29 Carl: So, to summarise, which of the items in Table 4 we keep and which should we delete? Jan: We should keep item 2.8 and delete the others Lindi: We should keep item 2.8 and item 2.6 and delete the others Banti: We should keep 2.3 and delete the others
For the clever students: When answering this question simply consider the information provided in Table 4 and in the corresponding figures. Do not refer back to the original formulation of the item in the AIQ. Doing so would greatly complicate matters and may change the answers you have to provide. One would keep items that has positive item total correlation and high item variance. Only item 2.8 qualifies. The rest should be deleted. Therefore Jan’s response is most correct. Lindi is correct in identifying 2.8 as an item to be kept, but she is not correct in wanting to keep item 2.6. Banti’s response is totally incorrect.
You agree most with, second most with and least with: Jan Lindi Banti
30 Gcina: Given that we want to determine the level of an individual’s intention to use alcohol in order to identify and target learners who are at risk for illegal drug us, how should we think about the reliability of the AIQ?
Carl: It is important to make sure that the AIQ has high internal
consistency Lindi: I think test-retest reliability is more important than internal
consistency, given that we want to identify and target learners who are at risk for illegal drug use
Banti: I think criterion-related validity is the most important aspect of the AIQ’s reliability given our need to identify and target learners who are at risk for illegal drug use
You agree most with, second most with and least with: Carl Lindi Banti
In this scenario internal consistency is more important than test-retest reliability, because the questionnaire needs to be administered once only and one wants to be sure that the questionnaire provides a reliable result. Carl’s response is most correct. Lindi’s response is not correct. It looks like she associates the identification of learners at risk for illegal drug use as a second administration of the questionnaire, which it is not. Banti refers to validity and not to reliability. But Banti seems to be really confused. She indicates that validity is an aspect of reliability, which means she has no understanding of the difference between validity and reliability. Banti’s response is totally incorrect.
31 1. (0 point) Lindi: I presume the same holds for validity. Given how we want to
use the AIQ, how should we think about the validity of the questionnaire?
Banti: Because we want to determine the level of an individual’s
intention to use alcohol I think concurrent validity is the most important type of validity to consider
Carl: Given that we want to identify and target learners who are at risk for illegal drug use I think it is important to consider the AIQ’s criterion related validity
Jan: If we want to determine the level of an individual’s intention to use alcohol in order to identify and target learners who are at risk for illegal drug use I think we should consider the AIQ’s construct validity in addition to its criterion related validity
Jan offers the best response. He indicates that two kinds of validity, namely criterion related validity and construct validity are important given how the AIQ is going to be used in the present scenario. Subject literature suggests a relationship between the intention to use alcohol and future illegal drug use. Criterion related validity is important because the result of the questionnaire will be used to identify and target learners who may become illegal drug users. The AIQ If the result of the AIQ does not correlate with future illegal drug use the AIQ cannot be used to identify and target such learners. Construct validity is important because the AIQ needs to measure the construct that it is supposed to measure, namely the intention to use alcohol. If it does not measure this construct one cannot expect a relationship between the result of the AIQ and the future illegal drug use. Carl’s response is correct, but less valuable than Jan’s because he only refers to criterion related validity. Banti’s response is not correct because she sees the result of the AIQ (the level of intention to use alcohol) as a criterion score. Concurrent validity is a form of criterion related validity, and criterion related validity refers to the correlation between the result of a questionnaire and some measure of a criterion. Concurrent validity simply means the criterion scores coincide with the questionnaire scores.
You agree most with, second most with and least with: Jan Carl Bant
32 2. (0 point) Gcina: Considering the information provided in Table 5, what do
you think of the construct validity of the AIQ? Lindi: The high correlation between attitude towards alcohol
consumption and subjective norm concerning the use of alcohol supports the construct validity of the AIQ
Carl: The low correlation between the recreational behaviour and attitude towards alcohol consumption and subjective norm concerning the use of alcohol shows low construct validity of the AIQ
Jan: The information in Table 5 supports the construct validity of the AIQ , but the AIQ does not fully represent the content domain of the construct it purports to assess
You agree most with, second most with and least with: Jan Lindi Carl
Table 5 indicates the inter-correlations between the four factors of the AIQ. There is a high correlation (0.92) between the attitude factor and the norm factor, which means a strong relationship between these two factors. One would expect this to be the case as these two factors are part of the psychological construct measured by the AIQ, namely intention to execute behaviour. The biographical and recreation factors are not part of the psychological construct measured by the AIQ. They show low correlations with both the attitude and the norm constructs. Therefore the information in Table 5 supports the construct validity of the AIQ. But there is one quite serious problem with the construct validity of the AIQ namely that one of the factors of the intention to execute behaviour (the ‘perceived control’ factor) is not measured by the AIQ. In light of this discussion it is clear that Jan’s as well as Lindi’s responses are correct. But Jan’s response is better than Lindi’s because he realises that the AIQ does not include all factors of the content domain implicated by the ‘intention to execute behaviour’ construct. Carl’s response is incorrect. The low correlations between recreational behaviour and the two factors of the ‘intention to execute behaviour’ construct does not mean low construct validity.
33 3. (0 point) Gcina: Let us have a look at the convergent and discriminant validity
of the AIQ. What is the difference between these two kinds of validities?
Carl: Convergent and discriminant validity are two aspects of
construct validity Lindi: Convergent validity implies a high correlation between the
AIQ and another questionnaire, whereas discriminant validity means a low correlation between the AIQ and another questionnaire
Banti: I agree that convergent validity refers to the relationship between the AIQ and another questionnaire but discriminant validity is not a form of validity. It is a measure of reliability
Lindi’s offers the best response. Carl’s response is not incorrect, but note that he does not indicate how the two kinds of validities differ. He simply indicates that they are two aspects of construct validity. Banti’s response contains inaccuracies. Firstly, both convergent and discriminant validity (and not only convergent validity) refer to the relationship between the AIQ and other questionnaires, and secondly, it is not correct to state that discriminant validity is a measure of reliability.
You agree most with, second most with and least with: Lindi Carl Banti
34 4. (0 point) Carl: In Table 6, what does the correlation between the AIQ and
the AAQ mean? Lindi: It means there is a poor relationship between the AIQ and
the AAQ Gcina: It indicates a high degree of discrimination between the
measurements of the AIQ and the AAQ Banti: It means the AAQ should be grouped with the TSR rather
than with the AIQ You agree most with, second most with and least with: Gcina Lindi Banti
Both Gcina’s and Lindi’s responses are correct. However, whereas Lindi merely observes the poor correlation (0.04) between the two questionnaires, Gcina interprets the information correctly as an indication of discrimination between the two measures. Therefore Gcina’s offers a more valuable response than Lindi does. It is difficult to see what Banti is on about. It is not clear how the low correlation between the AAQ and the AIQ indicates that the AAQ should be grouped with the TSR, with which it has an equally poor correlation (0.08). It is also not clear what she means by ‘grouped with’. Therefore Banti’s response is really poor.
35 5. (0 point) Banti: I am wondering about the meaning of the correlation
between the TSR and the AIQ indicated in Table 6. Banti: It indicates a high correlation between TSR and the AIQ Gcina: It shows convergence between the measurements of the TSR
and the AIQ Lindi: I think it supports convergent as well as concurrent validity
for the AIQ You agree most with, second most with and least with: Lindi Gcina Banti
Although all three responses are correct they differ in quality. Banti observes the most basic fact, namely that the correlation is high (0.86). Gcina interprets the high correlation as an indication of convergence between the two measures. But the best response comes from Lindi. She realises that the TSR (a treatment suitability rating) is a criterion measure for the AIQ result. The AIQ provides a measure of the individual’s intention to use alcohol. If people in a treatment setting have high scores on the AIQ and they are rated suitable for treatment (the TSR result) it means the AIQ identifies those who end up in treatment. Note that the AIQ and the TSR results are obtained during the treatment process. Therefore one cannot consider the AIQ results as predictions of the TSR results. This is why Lindi indicates the criterion related validity as concurrent validity and not as predictive validity.
36 6. (0 point) Carl: To summarise: Does the information in Table 6 tell us
anything about the construct validity of the AIQ?
Gcina’s response is incorrect. The table does not provide information about the internal consistency of the AIQ. Banti’s response is only partly correct. The table indicates good convergent validity and good discriminant validity. Lindi offers the best response. The table
Gcina: The information in Table 6 shows good internal consistency
for the AIQ Banti: I think Table 6 indicates good convergent but poor
discriminant validity for the AIQ Lindi: Actually Table 6 shows good construct validity for the AIQ You agree most with, second most with and least with: Lindi Banti Gcina
indicates good construct validity for the AIQ.
This is a fruitful discussion, but I think we should move on now. We have to decide whether we can use the AIQ, which means we have to rate some aspects of the questionnaire and the manual. In each case we have to consider specific criteria to help us decide on a rating. We can express our rating on a scale of 1 to 4, where 1 reflects a bad rating and 4 is a good rating.
37 This is a fruitful discussion, but I think we should move on now. We have to decide whether we can use the AIQ, which means we have to rate some aspects of the questionnaire and the manual. In each case we have to consider specific criteria to help us decide on a rating. We can express our rating on a scale of 1 to 4, where 1 reflects a bad rating and 4 is a good rating. Rate the instructions of the AIQ The instructions of the AIQ are complete if: a the intention of the AIQ is indicated b the matter of confidentiality is addressed c instructions for completing the AIQ are indicated Rate 1 if: none of a, b, c Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 4 Criterion a (the intention of the AIQ is indicated) is met: “By answering these questions you indicate how you feel about the use of alcohol” Criterion b (the matter of confidentiality is addressed) is met: “Only the school psychologist has access to your answers” Criterion c (instructions for completing the AIQ are indicated) is met: “Please read the following questions carefully and provide the answer that best describes your thoughts and feelings” and also: “There are no right or wrong answers. Simply give your opinion. Be as honest as possible.”
38
Rate the formulation of question 1.3 of the AIQ Question 1.3 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e
Rate 2 Criterion a is met: The purpose of the item is to get biographical information Criterion b is met: The item is straightforward to understand Criterion c is not met: There is a problem with the configuration of the item. It is a filter question but its filtering is set incorrectly. It requires individuals who should answer questions 1.4 and 1.5 to jump to Section 2. Criterion d is met: The question is not leading Criterion e is met: The question is not likely to elicit a socially desirable response Note: Because criterion c is not met one cannot assign a rating of 3 or 4. Thus the highest rating that can be assigned is 2.
39
Rate the suitability of question 2.4 in the AIQ Question 2.4 is suitable if: a respondents are likely to find the question’s level of
language appropriate b respondents are likely to find the question relevant in
their context c respondents are likely to possess the knowledge
required to respond correctly to the question Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 2 Criterion a is met: The language level is appropriate Criterion b is not met: School level learners are most likely not married or living with a partner, and will not find the item relevant in their context Criterion c is not met: School level learners are most likely not married or living with a partner, and will not possess the knowledge required to respond to the question Note: Because criterion b is not met one cannot assign a rating of 3 or 4. Thus the highest rating that can be assigned is 2.
40
Rate the formulation of question 2.9 Question 2.9 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous
Rate 2 Criterion a is met: The item has relevant content and is appropriate with regard to the construct it aims to measure Criterion b is met: The item is clear and easy to understand
c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e
Criterion c is not met: The item refers to multiple matters that may require different responses. Yet the item allows only one response. Criterion d is met: The item does not encourages the respondent to respond in a particular way Criterion e is met: The item does not involve a socially desirable matter
41
Rate the suitability of question 2.7 in the AIQ Question 2.7 is suitable if: a respondents are likely to find the question’s level of
language appropriate b respondents are likely to find the question relevant in
their context c respondents are likely to possess the knowledge
required to respond correctly to the question Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 1 Criterion a is not met: The language level is too advanced. Only a professionally trained person knowing about psychodynamics and alcohol abuse is able to respond to this question. Criterion b is not met: School going children are not likely to understand the question and see how it relates to their experiences Criterion c is not met: It is unlikely that anyone who is not professionally trained will have the knowledge to respond correctly to this question. Note: Because criteria a, b and c are not met one cannot assign a rating of 2, 3 or 4. Thus the highest rating that can be assigned is 1.
42
Rate the formulation of question 3.2 Question 3.2 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response
Rate 2 Criterion a is met: The item has relevant content and is appropriate with regard to the construct it aims to measure Criterion b is not met: It is unclear what the differences are between maybe, perhaps and sometimes. It is also not clear whether one may tick more than one response. Criterion c is not met: The item is not configured correctly because the labels used to distinguish the different categories of the item (e.g. maybe, perhaps) do not clearly separate the categories from each other.
Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e
Criterion d is met: The item does not encourages the respondent to respond in a particular way Criterion e is met: The item does not involve a socially desirable matter
43
Rate the suitability of question 3.5 in the AIQ Question 3.5 is suitable if: a respondents are likely to find the question’s level of
language appropriate b respondents are likely to find the question relevant in
their context c respondents are likely to possess the knowledge
required to respond correctly to the question Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 1 Criterion a is not met: The average person in the street may not be able to assign the correct meaning to the terms used in the item Criterion b is not met: The question refers to symptoms of advanced alcoholism. School going children are not likely to understand the question and see how it relates to their experiences Criterion c is not met: It is unlikely that anyone who is not professionally trained will have the knowledge to respond correctly to this question.
44
Rate the functionality of the AIQ The AIQ is functional if: a the information gathered by the AIQ covers the
intended content domain b the AIQ produces information that is suitable for the
intended target population c the AIQ produces information that is practical to use Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 1 Criterion a is not met: The AIQ intends to measure the intention to use alcohol. But the AIQ does not include all the factors associated with the intention to execute behaviour. Therefore the information gathered by the AIQ does not cover the intended content domain. Criterion b is met: The information produced by the AIQ is the level of the intention to use alcohol. According to the manual this information is translated into a risk index using norm tables. This information can be used with the intended target population (school going learners). Criterion c is met: The result of the AIQ is provided in the form of a risk index. The index is easy to apply in practice because it only needs to be considered in terms of how high or low it is. No complicated interpretations are required. Note: Because criterion a is not met one cannot assign a rating of 2, 3 or 4. This rating
may seem low in light of the fact that criteria b and c are met. However consider the implication of not meeting criterion a. The AIQ does not provide an adequate measure of the construct it intends to measure. Therefore, although, it functions well in terms of the other criteria it is not a good questionnaire to use.
45
Rate the manual’s indication of the purpose of the AIQ The purpose of the AIQ is indicated if: a the aim of the AIQ is stated b the AIQ’s target population is defined c the design of the AIQ is described Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 3 Criterion a is met: The aim of the AIQ is described: “The Alcohol use Intention Questionnaire (AIQ) can be used to determine the level of a person’s intention to use alcohol.” Criterion b is met: The target population is indicated as: “The questionnaire is meant to be used in a preventive context – in other words, it should be used with individuals who have not been submitted to a treatment programme.” Criterion c is not met: The design (i.e. a brief indication of what the questionnaire looks like) is not described.
46
Rate the manual’s indication of the psychometric properties of the AIQ The psychometric properties of the AIQ are indicated if
information is provided concerning: a the sample used to test the AIQ b the procedures used to analyse and select items for
the AIQ c the reliability of the AIQ d the validity of the AIQ Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c or a + b + d Rate 5 if: a + b + c + d
Rate 2 Criterion a is met: The sample used for the development of the AIQ was 100 patients in a treatment clinic in Germany. Criterion b is not met: Although information is provided concerning item-total correlations and item variance the manual does not indicate what procedures were followed in selecting (or rejecting) items for the AIQ. Criterion c is not met: Although the manual includes information about reliability (split-half and test-retest) it does not provide and discuss reliability coefficients. Criterion d is not met: Although the manual includes information about validity (convergent and discriminant) it does not provide and discuss validity coefficients.
47
Rate the manual’s critical consideration of the psychometric properties of the AIQ
The psychometric properties of the AIQ are considered critically if:
a the sample used to pilot the AIQ is considered critically b item analysis and selection procedures used for the AIQ
are considered in terms of established practice c the reliability of the AIQ is considered critically d the validity of the AIQ is considered critically Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c or a + b + d Rate 5 if: a + b + c + d
Rate 1 Criterion a is not met: The sample is indicated but not discussed critically. Had they done so the test developers might have realised a fundamental issue about the sample. The sample consisted of people in treatment. But the manual indicates that the AIQ should not be used with people already in treatment. Because the AIQ was not developed using people who were not in treatment one wonders how valid and reliable it could actually be in assessing people who are in non-treatment situations (e.g. the learners in the school going context). Criterion b is not met: item analysis and selection procedures are not indicated and are not considered in terms of established practice. Criterion c is not met: Reliability coefficients are not indicated and there are no critical considerations of reliability. Criterion d is not met: Validity coefficients are not indicated and there are no critical consideration of validity.
48
Rate the manual’s description of the procedures for utilising the AIQ
The procedures for utilising the AIQ are described if: a instructions are provided for administering the AIQ b instruction are provided for scoring the AIQ c instructions are provided for interpreting the results Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Rate 4 Criterion a is met: (See Administration in Section 3 of the manual) Criterion b is met: (See Scoring in Section 3 of the manual) Criterion c is met: (See Interpretation in Section 3 of the manual)
49
Rate the structure of the manual The manual is structured properly if: a the text is clustered into topical sections b topics are presented in logical order c the information about each topic is relevant and
presented coherently
Rate 4 Criterion a is met: The manual is divided into clearly indicated sections that deal with particular topics Criterion b is met: The topics presented the manual is done so in a logical order Criterion c is met: The information provided in each section is relevant and presented coherently.
Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c
Note: This rating is only about the structure of the manual and how information is presented in it. The rating does not refer to the completeness or the quality of the information. The completeness and the quality of the information provided were already rated elsewhere.
50
Rate the text of the manual The manual is written properly if: a the text is factual and objective b the text is clear and to the point c the text is not inundated with spelling, grammar and
punctuation errors Rate 1 if: none of a, b, c Rate 2 if: one of a, b, c Rate 3 if: two of a, b, c Rate 4 if: a + b + c
Rate 4 Criterion a is met: The manual is written in factual and objective language. (As opposed to a text that is emotional and based on unfounded personal believes.) Criterion b is met: The text is clear and to the point Criterion c is met: The text is not inundated with spelling, grammar and punctuation errors. Note that this does not mean that no errors exist in the text. However such errors, if they exist, are minimal.
51
You: So, give me your final opinion. Do you think we can use the AIQ?
Jan: The AIQ is not perfect. But no questionnaire is ever going to
be absolutely perfect for the situation in which it is required. So I think we can use it
Lindi: I think we can use it if we correct the questions that are not formulated correctly
Carl: I think we cannot use the AIQ because there is a problem with its content validity, which is a fatal flaw
You agree most with, second most with and least with:
Carl Lindi Jan
Carl’s offers the best response. One cannot use a questionnaire if it does not adequately cover the content domain implicated by the questionnaire’s intention. Lindi’s suggestion is not without merit. Questions that are not formulated correctly have to be corrected. However, even if all the questions included in the AIQ are perfect one still cannot use the AIQ due to the flaw observed by Carl. Jan’s response is not in order. It does not reflect proper consideration of the AIQ in terms of its psychometric properties.
Recommended