Your browser does not support the PubReader view.
Go to this page to see a list of supporting browsers.
The Korean Society for Journalism & Communication (KSJCS)
[ Article ]
Korean Journal of Journalism & Communication Studies - Vol. 64, No. 5, pp.118-152
ISSN: 2586-7369 (Online)
Print publication date 31 Oct 2020
Received 03 Aug 2020 Revised 28 Sep 2020 Accepted 05 Oct 2020
DOI: https://doi.org/10.20879/kjjcs.2020.64.5.004

국내 커뮤니케이션 연구의 통계분석방법의 현황과 문제점 : 지난 10년간 <한국언론학보> 게재 논문의 내용분석

이병관** ; 김재민*** ; 김주환**** ; 장다연***** ; 권나현******
**한양대학교 광고홍보학과 교수 gogreen@hanyang.ac.kr
***한양대학교 광고홍보학과 석사과정 jmini369@naver.com
****한양대학교 광고홍보학과 석사과정 feelth1147@naver.com
*****한양대학교 광고홍보학과 석사과정 jdy6237@hanyang.ac.kr
******한양대학교 광고홍보학과 석사과정 nahyun0224@naver.com
Current Status and Problems of Statistical Analysis in Communication Research : Content Analysis of Studies in the Korean Journal of Journalism & Communication Studies over the Past 10 Years
Byoungkwan Lee** ; Jaemin Kim*** ; Juhwan Kim**** ; Dayeon Jang***** ; Nahyun Gwon******
**Professor, Dept. of AD & PR, Hanyang University gogreen@hanyang.ac.kr
***Graduate student, Dept. of AD & PR, Hanyang University jmini369@naver.com
****Graduate student, Dept. of AD & PR, Hanyang University feelth1147@naver.com
*****Graduate student, Dept. of AD & PR, Hanyang University, corresponding author jdy6237@hanyang.ac.kr
******Graduate student, Dept. of AD & PR, Hanyang University nahyun0224@naver.com

초록

본 연구는 2010년부터 2019년까지 <한국언론학보>에 게재된 양적 연구 논문을 대상으로 통계 분석 방법 사용의 현황과 경향을 살펴보고, 이를 통해 국내 커뮤니케이션 연구의 통계 사용의 관행과 문제점을 논의하는 것에 그 목적이 있다. 이를 위해 10년간 <한국언론학보>에 게재된 총 845편의 논문 중 통계분석방법을 사용한 612편을 대상으로 내용분석을 진행하였다. 분석 결과, 지나치게 특정 통계분석방법에 의존하거나, 해석의 문제, 인위적인 결과를 생산하기 위한 통계적 방법의 오용 현상이 발견되었다. 이와 함께, 대부분의 연구들에게서 통계 방법에 관한 중요한 정보를 제시하지 않는 잘못된 관행도 발견되었다. 본 연구는 통계 분석 방법 및 보고에 대해 지금까지 많은 통계 학자들로부터 지속적으로 제기 되어온 비판과 논의들을 실증적으로 확인했다는 점에서 그 의의를 가진다. 도출된 결과를 바탕으로 통계 분석의 오용, 남용, 관행적 사용 등에 대한 문제점이 논의되었다.

Abstract

Although statistical analysis methods are essential tools for quantitative researchers, controversy has persisted over the objectivity of inference through statistical procedures. The purpose of this study is to review the use and trends of statistical analysis methods in quantitative research papers published between 2010 to 2019 in the Korean Journal of Journalism and Communication Studies and to discuss its practices and problems of statistical use in communication research in Korea. For this purpose, 612 quantitative research papers using statistical analysis methods were content-analyzed out of a total of 845 papers published across 10 years. In this 612 quantitative research, the basic characteristics of the study, as well as the reliability between coders, the number of coders, and power for experimental design, were reviewed according to the methodology used. This study also checked for the use of null hypotheses and multi-item measures, and more specific details including the statistical packages, the basic assumptions of statistical analysis methods used in individual studies, such as t-test and regression, and the description of statistical results. Research papers were collected, reviewed, and analyzed by four graduate students majoring in advertising and public relations. The inter-coder reliability was measured by Kripendorff alpha, and the reliability for each item was between 0.71 and 1, thus ensuring a stable level of reliability. As a result of the analysis, research using surveys was the most common among the research methods, and regression was the most frequently used statistical method except for descriptive statistics. The most commonly used statistical package was SPSS. The current study found problems of over-reliance on a specific statistical package, erroneous interpretation of statistical analysis results, and misuse of statistical methods for yielding contrived results. In particular, over-reliance on a specific statistical package was related to over-reliance of specific statistical analysis methods such as partial eta squared and cronbach alpha. At the same time, in most studies, wrong practices were also found. For example, important information regarding the processes and results of statistical analysis was not provided, such as basic statistical assumptions and correlation between major variables, and neither was information on confidence intervals as supplementary indicators of null hypothesis significance testing. Given that this study empirically identified the criticisms and discussions that have been continuously raised about statistical analysis methods and reporting, this study holds implications. Through this study, it is hoped that future Korean communication researchers will actively carry out research on various problems and alternatives raised in individual statistical analysis methods, thereby enriching academic discussions.

Keywords:

Statistical analysis, Quantitative research, Communication research, Content analysis

키워드:

통계 분석 방법, 양적 연구, 커뮤니케이션 연구, 내용 분석

Acknowledgments

This study was supported by the Hanyang University(HY-2019-G)(본 연구는 2019년 한양대학교 교내연구비 지원으로 연구되었음(HY-2019-G)).

References

  • Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 10, 411-423. [https://doi.org/10.1037/0033-2909.103.3.411]
  • Bailar, J. C. & Mosteller, F. (1988). Guidelines for statistical reporting in articles for medical journals: amplifications and explanations. Annals of Internal Medicine, 108(2), 266-273. [https://doi.org/10.7326/0003-4819-108-2-266]
  • Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 1-29. [https://doi.org/10.1037/h0020412]
  • Bakeman, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379-384. [https://doi.org/10.3758/BF03192707]
  • Baumgartner, H., & Homburg, C. (1996). Applications of structural equation modeling in marketing and consumer research: A review. International Journal of Research in Marketing, 13, 139-161. [https://doi.org/10.1016/0167-8116(95)00038-0]
  • Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12(1), 41-75. [https://doi.org/10.1207/s15328007sem1201_3]
  • Bentler, P. M., & Woodward, J. A. (1980). Inequalities among lower bounds to reliability: With applications to test construction and factor analysis. Psychometrika, 45(2), 249-267. [https://doi.org/10.1007/BF02294079]
  • Boomsma, A. (2000). Reporting analyses of covariance structures. Structural Equation Modeling, 7, 461-483. [https://doi.org/10.1207/S15328007SEM0703_6]
  • Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425-440. [https://doi.org/10.1007/s11336-006-1447-6]
  • Bearden, W. O., Sharma, S., & Teel, J. E. (1982). Sample size effects on chi square and other statistics used in evaluating causal models. Journal of Marketing Research, 19, 425-430. [https://doi.org/10.1177/002224378201900404]
  • Boster, F. J. (2002). On making progress in communication science. Human Communication Research, 28(4), 473-490. [https://doi.org/10.1111/j.1468-2958.2002.tb00818.x]
  • Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of co-variance structures. British Journal of Mathematical and Statistical Psychology, 37, 62-83. [https://doi.org/10.1111/j.2044-8317.1984.tb00789.x]
  • Chou, C. P., & Bentler, P. M. (1995). Estimates and tests in structural equation modeling. In R. H. Hoyle (Ed.), Structural Equation Modeling (pp. 37-59). Thousand Oaks, CA: Sage.
  • Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997-1103. [https://doi.org/10.1037/0003-066X.49.12.997]
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. [https://doi.org/10.1007/BF02310555]
  • Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391-418. [https://doi.org/10.1177/0013164404266386]
  • Croon, M. (2002). Using predicted latent scores in general latent structure models. In G. Marcoulides & I. Moustaki (Eds.), Latent variable and latent structure modeling (pp. 195-223). Mahwah, NJ: Erlbaum.
  • Devlieger, I., Mayer, A., & Rosseel, Y. (2016). Hypothesis testing using factor score regression: A comparison of four methods. Educational and Psychological Measurement, 76(5), 741-770. [https://doi.org/10.1177/0013164415607618]
  • Ding, L., Velicer, W. F., & Harlow, L. L. (1995). Effects of estimation methods, number indicators per factor, and improper solutions on structural equation modeling fit indices. Structural Equation Modeling, 2, 119-144. [https://doi.org/10.1080/10705519509540000]
  • Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399-412. [https://doi.org/10.1111/bjop.12046]
  • Fisher, R. A. (1925). Statistical Methods for Research Workers. Edinburgh: Oliver and Boyd.
  • Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2-18. [https://doi.org/10.1037/a0024338]
  • Gardenier, J., & Resnik, D. (2002). The misuse of statistics: concepts, tools, and a research agenda. Accountability in Research: Policies and Quality Assurance, 9(2), 65-74. [https://doi.org/10.1080/08989620212968]
  • Graham, J. W. (2012). Missing data: Analysis and design. New York: Springer Science & Business Media. [https://doi.org/10.1007/978-1-4614-4018-5]
  • Holbert, R. L., & Stephenson, M. T. (2002). Structural equation modeling in the communication sciences, 1995-2000. Human Communication Research, 28(4), 531-551. [https://doi.org/10.1093/hcr/28.4.531]
  • Hotelling, H., Bartky, W., Deming, W. E., Friedman, M., & Hoel, P. (1948). The teaching of statistics [a Report of the Institute of Mathematical Statistics Committee on the Teaching of Statistics]. Annals of Mathematical Statistics, 19, 95-115. [https://doi.org/10.1214/aoms/1177730299]
  • Hsieh, F. Y. (1989). Sample size tables for logistic regression. Statistics in Medicine, 8(7), 795-802. [https://doi.org/10.1002/sim.4780080704]
  • Hu, L. T., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112(2), 351-362. [https://doi.org/10.1037/0033-2909.112.2.351]
  • Huang, Y., & Bentler, P. M. (2015). Behavior of asymptotically distribution free test statistics in covariance versus correlation structure analysis. Structural Equation Modeling: A Multidisciplinary Journal. 22(4), 489-503. [https://doi.org/10.1080/10705511.2014.954078]
  • Hunter, J. E. (1997). Needed: A ban on the significance test. Psychological Science, 8(1), 3-7. [https://doi.org/10.1111/j.1467-9280.1997.tb00534.x]
  • Jackson, P. H., & Agunwamba, C. C. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds. Psychometrika, 42(4), 567-578. [https://doi.org/10.1007/BF02295979]
  • Kang, N. (1999). The problems of using structural equation model(SEM) in communication research. Korean Journal of Journalism & Communication Studies, 44(1), 5-51.
  • Kelcey, B. (2019). A robust alternative estimator for small to moderate sample SEM: Bias-corrected factor score path analysis. Addictive Behaviors, 94, 83-98. [https://doi.org/10.1016/j.addbeh.2018.10.032]
  • Kline, R. B. (1998). Principles and practice of structural equation modeling. New York, NY: The Guilford Press.
  • Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
  • Lee, B., Sohn, Y., Oh, H., & Park, S. (2015). Current status and future direction on the use of statistical methods in quantitative research papers in public relations: A systematic analysis of journal of public relations, 1998~2004. Journal of Public Relations, 19(1), 270-298. [https://doi.org/10.15814/jpr.2015.19.1.270]
  • Lee, B., Sohn, Y., & Kang, K. (2018). How Reliable is the Reliability of the Celebrity Endorsement Scales?: Applying the Reliability Generalization Meta-Analysis. The Korean Journal of Advertising and Public Relations, 20(3), 35-77. [https://doi.org/10.16914/kjapr.2018.20.3.35]
  • Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625. [https://doi.org/10.1111/j.1468-2958.2002.tb00828.x]
  • Levine, T. R., Weber, R., Hullett, C., Park, H. S., & Lindsey, L. L. M. (2008). A critical assessment of null hypothesis significance testing in quantitative communication research. Human Communication Research, 34(2), 171-187. [https://doi.org/10.1111/j.1468-2958.2008.00317.x]
  • Lu, I. R., Kwan, E., Thomas, D. R., & Cedzynski, M. (2011). Two new methods for estimating structural equation models: An illustration and a comparison with two established methods. International Journal of Research in Marketing, 28, 258-268. [https://doi.org/10.1016/j.ijresmar.2011.03.006]
  • MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual review of psychology, 51(1), 201-226. [https://doi.org/10.1146/annurev.psych.51.1.201]
  • Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103(3), 391. [https://doi.org/10.1037/0033-2909.103.3.391]
  • McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates.
  • McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. New York, NY: The Guilford Press.
  • McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. [https://doi.org/10.1037/met0000144]
  • Nam, K. (2004). A study of choice for statistical method on social science. Journal of the Korean Data Analysis Society, 6(5), 1255-1265.
  • Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20A(1/2), 175-240. [https://doi.org/10.1093/biomet/20A.1-2.175]
  • Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5(2), 241-301. [https://doi.org/10.1037/1082-989X.5.2.241]
  • Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434-447. [https://doi.org/10.1037/1082-989X.8.4.434]
  • Park, J. (2015). Bayesian statistics as a solution to the problems of NHSTP: The case of psychology. The Korean Journal for the Philosophy of Science, 18(2), 135-147.
  • Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373-1379. [https://doi.org/10.1016/S0895-4356(96)00236-3]
  • Pelsma, J. R. (1937). Essentials of debate. New York, NY: Thomas Y. Crowell Company.
  • R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org, /
  • Reinsel, D., Gantz, J., & Rydning, J. (2018). Data age 2025: the digitization of the world from edge to core. Retrieved from https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf
  • Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145. [https://doi.org/10.1007/s11336-008-9102-z]
  • Rhee, K. (2016). Wrong applications with overall model evaluations and its corrections in structural equation modeling. Survey Research, 17(1), 71-83. [https://doi.org/10.20997/SR.17.1.4]
  • Richardson, J. T. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135-147. [https://doi.org/10.1016/j.edurev.2010.12.001]
  • Ritter, N. L. (2010). Understanding a widely misunderstood statistic: Cronbach's. Paper presented at the annual meeting of the Southwest Educational Research Association, New Orleans. Retrieved from https://files.eric.ed.gov/fulltext/ED526237.pdf, .
  • Rosenthal, R. (1991). Effect sizes: Pearson's correlation, its display via the BESD, and alternative indices. American Psychologist, 46, 1086-1087. [https://doi.org/10.1037/0003-066X.46.10.1086]
  • Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaic, & J. H. Steiger (Eds.), What if there were no significance tests (pp 37-64). London, UK: Psychology Press.
  • Shevlin, M., Miles, J. N. V., Davies, M. N. O., & Walker, S. (2000). Coefficient alpha: a useful indicator of reliability? Personality and Individual Differences, 28(2), 229-237. [https://doi.org/10.1016/S0191-8869(99)00093-8]
  • Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107-120. [https://doi.org/10.1007/s11336-008-9101-0]
  • Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350-353. [https://doi.org/10.1037/1040-3590.8.4.350]
  • Skrondal, A., & Laake, P. (2001). Regression among factor scores. Psychometrika, 66, 563-576. [https://doi.org/10.1007/BF02296196]
  • Thompson, B. (2002). “Statistical,”“practical,” and “clinical”: How many kinds of significance do counselors need to consider? Journal of Counseling & Development, 80(1), 64-71. [https://doi.org/10.1002/j.1556-6678.2002.tb00167.x]
  • Ten Berge, J. M., & Sočan, G. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613-625. [https://doi.org/10.1007/BF02289858]
  • Trafimow, D. & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1-2. [https://doi.org/10.1080/01973533.2015.1012991]
  • Wilkinson, L., & Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594-604. [https://doi.org/10.1037/0003-066X.54.8.594]
  • Yang, Y., and Green, S. B. (2011). Coefficient Alpha: a reliability coefficient for the 21st Century? Journal of Psychoeducational Assessment, 29, 377-392. [https://doi.org/10.1177/0734282911406668]

Appendix

부록

  • 강남준 (1999). 커뮤니케이션 연구에서 구조방정식 모형(SEM) 의 활용가능성: SEM 사용의 문제점을 중심으로. <한국언론학보>, 44권 1호, 5-51.
  • 남기성 (2004). 사회과학에서 통계분석방법의 선택에 관한 연구. <한국자료분석학회>, 6권 5호, 1255-1265.
  • 박준석 (2015). 영가설 유의성검증 절차의 문제점들에 대한 해결책으로서의 베이지언 통계학: 심리학의 경우. <과학철학>, 18권 2호, 135-147.
  • 이기종 (2016). 구조방정식모형의 모형평가 오, 남용과 교정. <조사연구>, 17권 1호, 71-83.
  • 이병관·손영곤·강경희 (2018). 유명인 모델의 공신력 척도는 얼마나 신뢰할 수 있는가?: 신뢰도 일반화 메타 분석의 적용. <한국광고홍보학보>, 20권 3호, 35-77.
  • 이병관·손영곤·오현정·박선화 (2015). 통계분석방법의 경향과 과제: 1998년 창간호부터 2014년 여름호까지 게재된 논문의 내용분석. <홍보학연구>, 19권 1호, 270-298.