Contemporary Research Practices Carlo Magno, PhD.
2nd NATIONAL CONFERENCE ON COMPUTERIZED STATISTICAL MODELING AND RESEARCH
METHODS
Tagbilaran City, BOHOL May 6-8, 2015
5th NATIONAL SEMINAR-WORKSHOP ON ACTION RESEARCH AND
COMPUTERIZED ITEM/TEST ANALYSIS AND STATISTICS
Contemporary Trends on 10 Perspectives
1. Research is written for an international audience 2. Research is becoming an interdisciplinary process 3. Research designs are based on the needs of the
researchers 4. Identify sample size based on power analysis 5. Adapting the instrument necessary for the sample 6. Instruments validity and reliability computed using the
sample data 7. Computation of the effect size with the NHST 8. Thesis and dissertation beyond the IPO 9. Publishing research reports in international abstracted
and indexed journals 10. Concern on author impact factor
Research written for an international audience
– Initiatives supporting ASEAN Integration
– Research databases easily accessible in the internet
– Implication:
• Specifying whether events are emic (part of the social group) or etic (perspective of an observer)
• Is there a need to separate local and foreign reviews?
• Using more of published studies than books…
• Country of origin in describing the participants
Research becoming an interdisciplinary process
• Looking at the complementary across and within fields: – education and psychology
– biology and education
– education and computer science
• “Communities of practice” generating knowledge
• New shared practices are discovered
• Apprenticeships, participations, and worldviews are developed.
Research designs based on the needs of the researchers
• Nonexperimental designs according to Johnson (2001)
Identifying sample size based on power analysis
No real effect Real Effect
Reject H0 Type 1 error α (.01, .05)
Ho not rejected Type 2 error β (small as possible)
1-β Statistical
power
Slim chance of concluding that the treatment is
effective, despite the fact that it is
Statistical Power
• β=.20 (the error of rejecting a true Ho is 4x more serious than the error of not rejecting a false Ho)
• .80=acceptable power
Statistical Power
• The probability of rejecting a false null hypothesis.
• The likelihood that a study will detect an effect when there is an effect to be detected.
• If statistical power is high, the probability of making a Type II error, (or concluding there is no effect when, in fact, there is one) goes down.
Statistical Power
• The power of any test of statistical significance will be affected by four main parameters:
– the effect size
– the sample size (N)
– the alpha significance criterion (α)
– statistical power, or the chosen or implied beta (β)
Statistics Power small medium Large
r .80 26 63 393
t .80 29 85 781
Statistical Power
http://danielsoper.com/statcalc3/default.aspx http://www.statisticalsolutions.net/pss_calc.php https://www.dssresearch.com/KnowledgeCenter/toolkitcalculators/statisticalpowercalculators.aspx http://homepage.stat.uiowa.edu/~rlenth/Power/
Influence of Effect Size on Power
High school n=65 College n=153
Influence of Effect Size on Power
Influence of Effect Size on Power
N=82 Taiwanese in Taiwan N=98 Taiwanese in the Philippines
Influence of Effect Size on Power
• What inference can be gained between effect size and power with fixed sample size and alpha level?
Influence of Significance Level on Power
• Study of De Frias, Dixon, and Strauss (2006)
• N=418
• r=.14 (not significant)
• α=.01 (power=.23)
α=.05 Power=.45
α=.10 Power=.58
α=.15 Power=.65
α=.20 Power=.71
• What inference can be gained between level of significance and power with fixed sample size?
Influence of Sample size on Power
Magno (2005) Monitoring and metacognition
N=280 r=.14 Power=.65
Magno, Mamauag, & Parinas (2007) Independence and self-esteem
N=373 r=.14 Power=.78
Chemers, Hu, & Garcia (2001) Challenge-threat and self-efficacy
N=381 r=.15 Power=.83
Influence of Sample size on Power
Adapting the instruments necessary for the sample
• Producing instruments that measure target constructs adequately in target cultures
• What is the American equivalent of the Dutch item “Hoe heet de koningin van Nederland?” (Suppose that
item is part of a test of crystallized intelligence) • Literal/close translation: What is the name of the queen
of the Netherlands?” – Problem: Item more difficult for American children than for
Dutch children • Adaptation: “What is the name of the president of the
USA?” – Problem: Queen and president are not equally known in their
respective countries
Construct equivalence
• Similarity of construct in source and target culture
Cultural equivalence
• Norms about interaction (modes of address)
• “Cultural fact sheet”
Linguistic equivalence
• Translation accuracy: Retention of denotation and connotation
Measurement equivalence
• Retention of psychometric features (response styles)
• Similarity of factors measured by a test and comparability of scores
Four Important Perspectives (Harkness & Van de Vijver, 2012):
Instruments’ validity and reliability recomputed using one’s data
• It is customary that in every instrument used, evidence of validity and reliability are reported for the instrument used.
• Confirmatory Factor Analysis is conducted for multidimensional measures.
• Internal consistency reported – Cronbach’s alpha – Coefficient Theta – Coefficient Omega
• Accuracy of the instrument is supported by using IRT and polytomous IRT models for scales
Computation of effect size with the NHST
• Issues on NHST • 1. NHST does not provide the information which the
researcher wants to obtain • 2. Logical problems derived from the probabilistic
nature of NHST. • 3. NHST does not enable psychological theories to be
tested. • 4. The fallacy of replication. • 5. NHST fails to provide useful information because
H0 is always false. • 6. Problems associated with the dichotomous
decision to reject/not reject the H0. • 7. NHST impedes the advance of knowledge.
1X
2X
n df SD1 SD2 t
p
value
30 28 4.6 4.1 2.3 2.3 0.60 0.56
60 58 4.6 4.1 2.3 2.3 0.84 0.4
100 98 4.6 4.1 2.3 2.3 1.09 0.28
500 498 4.6 4.1 2.3 2.3 2.43 0.02*
1000 998 4.6 4.1 2.3 2.3 3.44 0.00*
A researcher wanted to look at the effect of behavior modification technique on the aggression of clients. A group of participants in the experimental group were given behavior modification technique and no treatment in the control. The aggression of the two groups were measured after.
Effect Size
• Cohen (1988) defines the effect size as the extent to which the phenomenon is found within the population or, in the context of statistical significance testing, the degree to which the H0 is false.
• Snyder and Lawson (1993) argue that the effect size indicates the extent to which the dependent variable can be controlled, predicted and explained by the independent variable(s).
Effect Size Measures • Effect size measures of Two In/dependent Groups
– Cohen’s d – Hedges g – Glass Delta
• Correlation Measure of Effect Size – r – χ2 ►Φ; t ► r; F ► r; d ► r
• Effect size for Analysis of Variance – Eta Squared – Omega Square Index of Strength – Intercalss correlation
22
21
2
21
ss
MMd
2
22
1
12
21
ns
ns
MMt
Cohen’s d Formula t-test for independent Means Formula
Cohen's Standard Effect Size Percentile Standing Percent of
Nonoverlap
2 97.7 81.10%
1.9 97.1 79.40%
1.8 96.4 77.40%
1.7 95.5 75.40%
1.6 94.5 73.10%
1.5 93.3 70.70%
1.4 91.9 68.10%
1.3 90 65.30%
1.2 88 62.20%
1.1 86 58.90%
1 84 55.40%
0.9 82 51.60%
LARGE 0.8 79 47.40%
0.7 76 43.00%
0.6 73 38.20%
MEDIUM 0.5 69 33.00%
0.4 66 27.40%
0.3 62 21.30%
SMALL 0.2 58 14.70%
0.1 54 7.70%
0 50 0%
Thesis and dissertation beyond weighted means, difference, and
relationships
• Bad practices
– All students uses the INPUT-PROCESS-OUTPUT in their theoretical framework
– Research questions #1 determined the demographic profile of participants (gender, SES, age, family structure, residence, etc)
– The factors of the instrument are physical, emotional, social, spiritual …
Publishing research in abstracted and indexed journals
• An indexing and abstracting service is a service that provides shortening or summarizing of documents and assigning of descriptors for referencing documents.
• Examples: – Index Copernicus
– Scopus
– Ulrichs Web
– ISI
– Australian Education Index
– DOAJ
– www.africaknowledgeproject.org/akpjournalindexed.htm
Levels of Research
Publication
Public Presentation
Write reports
Our work is reviewed by a
panel of experts
Our work is evaluated whether we are
contributing to theory
Levels of Publication
ISI abstracted Journals
Abstracted journal
International Journal
Local journal
Conference proceeding
In house publication
What does it mean to publish?
Studies that will be published should have Significant contributions
• New argument or
conjecture
• New definition
• Clarification
• illustration or exemplar
• Elaboration
• refutation or rebuttal
• rephrasing
• Rebuttal of question
• Recasting of question
• Evaluation of an earlier
assertion
• New or alternative
interpretation
• Supportive evidence
• Contrary evidence
Concern on the authors impact factor
• The h-index quantifies the actual scientific productivity and the apparent impact of the scientist. The h-index is based on the author’s most cited papers and the number of citations they have received from other articles.
• "A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) papers have no more than h citations each." [For details in calculation, see Hirsch, 2005]
• An h-index of 16 means, for example, that a researcher has published 16 papers that each had at least 16 citations. Therefore, the h-index reflects both the number of articles as well as the number of citations per article.
• Google scholar • Web of science • Harzing’s Publish or Perish
Recommendations
• Review institutional research polices through various representations
• Guidelines in thesis and dissertation writing should be based on international standards on publication
• Libraries need to invest on access to high impact journals
• Get faculty who published in abstracted journals to teach research courses
• Train faculty in the discipline to teach statistics courses