Validity and Reliability Study in Turkish COVID-19 Literacy Scale

Validity and Reliability Study in Turkish COVID-19 Literacy Scale

The Open Nursing Journal 04 Oct 2023 RESEARCH ARTICLE DOI: 10.2174/18744346-v17-230927-2023-81



It is critical to conduct studies on the evaluation of COVID-19 health literacy in different cultures. Health literacy can make it easier to distinguish between reliable information about COVID-19 and misinformation, empowering individuals to make illuminated health decisions and to practice healthy and protective behaviours.


The aim of this study is to verify the validity and reliability of COVID-19 Health Literacy Scale (COVID-19-HLS) for Turkish.


This was a cross-sectional-descriptive study. The sample was randomly divided into two groups for factor analysis. A total of 712 individuals were included for exploratory (n=350) and confirmatory (n=362) factor analysis. The data were gathered by using the COVID-19-HLS and the Personal Variables Form. Language validity, content validity, item analyses, and exploratory and confirmatory factor analyses were performed to determine the validity and reliability of COVID-19-HLS.


Internal consistency reliability of the scale was found as 0.94. Item-total score correlation values of the scale were between 0.21 and 0.70, and the intra-class correlation coefficient was determined as 0.933. It was found that the item factor loads varied between 0.735 and 0.456 and the four-factor structure of the scale was found to be acceptable.


In line with the analyses, it was shown that the scale is a valid and reliable instrument for Turkish society.

Keywords: COVID 19, Health literacy, Information, Outbreak, Validation study.


With the rapid spread of COVID-19 all over the world, individuals required information regarding this new virus to change their behaviours accordingly [1]. During such a global pandemic, it is critical for the community to be able to procure new and particular health information [2]. Health authorities and policymakers are working together to minimize the risk of infection and the spread of the virus in the fight against the COVID-19 pandemic worldwide [3]. It is emphasized that some responsibilities should be taken individually and socially to prevent the COVID-19 pandemic, and therefore information, current recommendations, and practical solutions are shared for the prevention of infection [4]. The ability of individuals to access the right information sources, to make sense of this information and to apply it in daily life is related to health literacy (HL) [5]. The importance of health literacy emerges at this point. Low health literacy affects a large percentage of the population worldwide and brings along many problems such as difficulties in accessing and using health services and the inability to understand and apply medical advice [6]. People with high health literacy have health advantages over those with lower levels of health literacy, and health literacy is also an important indicator of hospital admission [7].

Health literacy can make it easier to distinguish between reliable information and misinformation about COVID-19, enabling people to make informed health decisions and adopt healthy and protective behaviors. Health literacy can also influence people's fear of COVID-19. According to a study conducted by Nguyen et al. [8], people with high levels of fear about COVID-19 had lower health literacy scores. Various studies have been conducted to determine the level of health literacy of the public and health professionals, highlighting the growing importance of health literacy in the age of COVID-19 [8-11]. According to a study conducted on undergraduate health students in South Korea by Hong et al. [12] scores in all dimensions of e-health literacy are associated with COVID 19-related infection preventative behaviours. Lawrence et al. applied the Health Literacy Scale they developed on adults in the United States and found that 29% of the participants had low health literacy associated with COVID 19 [2]. Different aspects of health literacy should be highlighted in different populations during the pandemic [13]. When the infodemic, which peaks during the pandemic periods, is added to the insufficient level of health literacy, the damage caused by the pandemic increases incrementally [14]. The aim of this study is to verify the validity and reliability of the COVID-19 Health Literacy Scale (COVID-19-HLS) for Turkish society.


2.1. Study Design

The data of this cross-sectional-descriptive study were collected from 712 individuals who were older than 18 years old, who were at least primary school graduates, who volunteered to participate in the study, who registered in a popular social network such as WhatsApp, by using the Snowball Method online between April and May 2022. Exclusion criteria were having a serious mental disease, post-traumatic stress disorder, or panic attacks. Voluntary participants had the right to back down from the study at any time. The general rule for determining the sample size in factor analysis is 10 participants and a 20% loss probability for each item [15]. Since the original scale item was 51 items in this study, at least 610 participants were needed in the sample size. The sample was randomly divided into two groups for factor analysis. A total of 712 individuals were included in the study for exploratory (n=350) and confirmatory (n=362) factor analysis [16, 17]. In the first phase of this study, which consisted of two stages, the scale was translated and culturally adapted; in the second stage, the psychometric evaluation of the scale was performed (Fig. 1).

2.2. Sociodemographic Characteristics and COVID 19-Health Literacy Scale

The study data were collected by using The Sociodemographic Variable Form and COVID-19-HLS.

The Sociodemographic Information Form was formed by researchers based on literature, and it contained questions about socio-demographic characteristics such as age, gender, education level, and COVID 19 status [9, 10, 14]. COVID-19-HLS was developed by Sanaeinasab et al. (2022) to find out the health literacy levels regarding COVID 19 [18]. The scale consists of 51 items. Each item of the original scale is graded with a 4-point Likert-type scale ranging from “never” (1) to “always” (4). The scale has five factors. Negative items in the scale are recorded and computed by dividing the total score by the number of items. A total of 1 to 4 points can be obtained from all sub-dimensions and the total scale. An increase in the total score obtained from the scale shows a higher level of HL.

2.3. Phase I. Translation and Transcultural Adaptation

The COVID-19-HLS was translated from English into Turkish by bilingual linguists and the authors independently. The translation of the scale was checked by a bilingual team of linguists consisting of seven experts. Later, it was independently back-translated from Turkish into English by a bilingual linguist. Conceptual and linguistic suitability was checked by three different experts.

2.3.1. Content Validity

The finalized COVID-19-HLS was sent to seven public health nurse specialists at different universities in Turkey for content validity, with both Turkish and English forms, and the experts who participated made minor corrections for cultural and language differences. Davis Technique was used for content validity of the scale. Each expert was asked to rate each item on a 4-point scale (1=not appropriate, 2= needs to be changed, 3= needs minor correction, 4=completely appropriate). When the Content Validity Index (CVI) is evaluated, a value greater than 0.80 suggests that the item is sufficient in terms of content validity (16). In the analyses performed, it was found that the CVI scores for all items ranged between 0.8 and 1.0. Due to content validity, no item was excluded from the scale.

Fig. (1). COVID-19-HLS flow diagram.

2.3.2. Pilot Study

The pilot study was conducted with 72 participants and none of them were included in the study. Before the pilot study, written information about the process was provided, the participants were informed about the purpose of the pilot study and necessary information was given. As the items were found to be comprehensible as a result of the pilot study, no changes were made to the scale and it was decided that the sample was sufficient for analysis.

2.4. Phase II. Psychometric Properties Evaluation

2.4.1. Construct Validity

Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were used to test and evaluate the factor construct validity of the scale. Before the factor structure, Kaiser-Meyer-Olkin (KMO) and Bartlett’s sphericity tests were performed to evaluate the sample size and the fitness of the scale for factor analysis [19].

2.4.2. Reliability of the Scale

Four different analysis methods that test the reliability were; (i) internal consistency analysis, (ii) split-half reliability analysis, (iii) test-retest and (iiii) intraclass correlation coefficient (17).

Item-total score correlations were made and the corrected item-total score correlations of the scale items were found to be between 0.21 and 0.70. Eight items (1, 2, 3, 4, 5, 42, 44, and 45) with item-total score correlations <0.30 were excluded from the scale and the scale was reanalysed. The Spearman-Brown Prophecy, Guttmann Split-Half, and Cronbach Alpha Coefficients were used to test reliability based on Split-Half Reliability Analysis. A value of ≥ 0.6 was used as the reliability criterion [20]. Hoteling’s T2 test was used to check whether the mean of the items were different from each other [21].

2.4.3. Time Invariance (test-retest)

The test-retest technique was used to find out the time-invariance criterion of reliability. It is performing the same test on the same group at different times. This was as follows: the participants were asked to choose a nicknames. After two weeks, the participants who took the first test were asked to take the second test using their nicknames. The scores obtained in the two questionnaires were calculated by Pearson Correlation Analysis. The Dependent t-test was performed to evaluate the difference in the mean scores between the test and retest [11, 22].

2.5. Data Collection

The study data were collected by using The Snowball Method. According to this method, the first data source was the students studying with the nursing faculty and their families. After completing the questionnaire, nursing students were asked to send it to their family members. On the first page of the questionnaire, there is a text explaining the purpose of the study to the individuals. The next section includes the study criteria. Pages with study questions are followed for individuals who meet all of the study criteria.

2.6. Ethic Statements

Permission was obtained by e-mail from the authors who developed the scale in accordance with ethical principles. Also, ethical approval was obtained from the Ethics Committee of Atatürk University Faculty of Medicine (B.30.2.ATA. 0.01.00/261).

2.7. Statistical Analyses

The Statistical Package for the Social Sciences (SPSS) version 20.0 (SPSS v.20.0, IBM Corp., Chicago, IL, USA) and AMOS version 20.0 were used to analyze the study data. Language and content validity, Item-Total Score Correlation, Test-Retest Reliability Methods, Explanatory and Confirmatory, Cronbach’s Alpha Coefficient, Split-Half Reliability Analysis, Factor Analysis, and criterion-concurrent validity were performed to the validity-reliability analysis.


It was found that 69.7% of the participants were women, 47.2% were primary school graduates, and 64% were single. In addition, it was found that 57% of the participants did not have COVID 19, and 79.2% did not have a COVID-19-positive individual in the family.

Table 1.
Factor loadings using exploratory factor analysis on COVID-19-HLS (n=350).
Items Factors -
- Analysis Behavior Information Seeking Practice Corrected Item-total Correlation
I20 0.727 - - - .653
I19 0.705 - - - .627
I18 0.696 - - - .531
I23 0.693 - - - .626
I21 0.682 - - - .557
I26 0.610 - - - .700
I15 0.609 - - - .581
I14 0.608 - - - .594
I22 0.573 - - - .589
I16 0.566 - - - .474
I34 0.532 - - - .664
I17 0.529 - - - .557
I27 0.510 - - - .645
I25 0.494 - - - .598
I8 0.485 - - - .502
I24 0.463 - - - .599
I50 - 0.735 - - .542
I49 - 0.732 - - .581
I46 - 0.721 - - .549
I48 - 0.684 - - .436
I47 - 0.681 - - .517
I51 - 0.663 - - .542
I31 - 0.621 - - .530
I30 - 0.617 - - .578
I43 - 0.542 - - .541
I12 - - 0.722 - .552
I11 - - 0.688 - .590
I13 - - 0.661 - .568
I9 - - 0.659 - .582
I6 - - 0.621 - .444
I7 - - 0.550 - .460
I10 - - 0.542 - .592
I36 - - - 0.666 .405
I37 - - - 0.632 .223
I38 - - - 0.618 .221
I33 - - - 0.615 .406
I32 - - - 0.550 .484
I35 - - - 0.470 .529
I40 - - - 0.469 .308
I41 - - - 0.456 .429
Eigenvalue 7.318 5.853 4.312 3.588 -
Explained variance (%) 25.019 19.700 17.027 12.345 74.091
Table 2.
Results of confirmatory factor analysis for the four factor model (n = 362).
Fit indices Acceptable Fit Criterion Post Modification
X2/SD 3≤χ2/df≤5 4.196
RMSEA 0.05≤RMSEA≤0.08 0.067
CFI 0.90≤CFI 0.901
GFI 0.85≤GFI 0.863
AGFI 0.85≤AGFI 0.894
NFI 0.90≤NFI 0.971
IFI 0.90≤IFI 0.933
TLI 0.90≤TLI 0.919

3.1. Construct Validity

3.1.1. Exploratory Factor Analysis

KMO was used for sample adequacy, and the correlation between items for factor analysis was determined by using Bartlett’s Test of Sphericity. The 43-item scale has a KMO=0.931, Bartlett test value x2=14282.550, and p<0.001, indicating adequate sampling for principal component analysis and sufficient correlation between items for factor analysis. The Principal Component Analysis was used for the Exploratory Factor Analysis. Since 3 of 43 items (28, 29, and 39. items) showed overlapping item characteristics, they were excluded from the scale. Factor analysis was repeated for 40 items, and the factor load values of the items ranged between 0.735 and 0.456 (Table 1). In addition, four factors of the scale explained 74.091% of the total variance. In conclusion, COVİD 19-HLS consisted of four factors and 40 items.

3.1.2. Confirmatory Factor Analysis

CFA was used to determine whether the items represented the factors and explained the scale structure of the factors. It was found that the 40 items in the scale were related to the four-factor scale structure (Fig. 2). The model was improved. While improving, a covariance was created between the errors with high MI values. As a result, it was determined that it showed acceptable agreement with AGFI=0.894, CFI=0.901, RMSEA=0.069, GFI=0.863 and x2(Cmin/df) 4.429. In conclusion, it was shown that the scale had construct validity and the fit indices showed that the model was acceptable in its current state (Table 2).

3.2. Results on Reliability

The internal consistency coefficient of the four-factor scale was found to be 0.941, and internal consistency coefficients of the factors were 0.922 for the analysis factor, 0.890 for the behaviour factor, 0.846 for the information seeking factor, and 0.746 for the application factor. The Spearman-Brown Correlation Coefficient and Guttmann Split-Half Value were found to be 0.836 and 0.841, respectively. The Spearman-Brown Correlation Coefficients of the factors were found to be 0.861, 0.850, 0.781, and 0.685, respectively. Hoteling’s T2 value was 1615.931, p<0.001. The difference between the means of COVID-19-HLS items was found to be significant. In addition, the reliability of the data obtained was determined by the intra-class correlation coefficient. These values were found to be 0.919 for the analysis factor, 0.887 for the behaviour factor, 0.830 for the information seeking factor, and 0.742 for the application factor (Table 3).


4.1. Validity

Varimax Rotational EFA results showed a four-factor structure, unlike the original five-factor scale with more than one eigenvalue [22, 23]. The factor loads obtained for each item were >.30, as suggested in the literature. Factors explained 74.09% of the total variance. This was reported to be 47.3% in the original study of the scale. For multi-factor scales, the total variance is >40%, which is accepted as the lower limit value [11, 23]. This shows that the total variance explained by the factors is acceptable. The model appropriate for the four-factor structure of the scale was tested with CFA. It is a validation method specifically used in the application of measurement instruments developed in other cultures and samples [11, 24]. A series of goodness-of-fit measures were used to assess the model fit. When the values of the CFA fit indices were examined, it was determined that all values were within acceptable limits and they showed perfect fit [11, 19, 25]. It was also found that the factor loads of COVID-19-HLS varied between 0.45 and 0.74. These results confirmed the four-factor structure. The DFA results of this study could not be compared with those of the original study, as the original study did not include the DFA results [18].

Fig. (2). COVID-19-HLS path diagram.
Table 3.
Results of reliability for COVID-19-HLS.
Subscales - X±SS Internal Consistency Coefficient (Cronbach’s alpha) Correlation Coefficient (95% Confidence İnterval) Split-half Reliability Correlations*
Analysis I=16 3.25±0.56 0.922 0.919 (.909-.927) 0.861 -
Behavior I=9 3.51±0.54 0.890 0.887 (.873-.899) 0.850 -
Information seeking I=7 3.13±0.60 0.846 0.830 (.802-.854) 0.781 -
Practice I=8 2.81±0.56 0.754 0.742 (.709-.771) 0.685 -
Total I=40 3.18±0.45 0.941 0.933 (0.925-0.941) 0.836 -
Test-retest value(n=122) - - - - - -
1. Application - 3.21±0.42 - - - 0.880**
2. Application - 3.13±0.47 - - - 0.000
- - t=1.142 - - - -
- - p=0.425 - - - -
Note: t = paired samples test. I = Number of items in the factor.
*Pearson’s correlation. **p < .01.

4.2. Reliability

In this study, four reliability methods, Internal Consistency, Split-Half Reliability, Test-Retest Reliability, and Intraclass Correlation Coefficient Methods, were used [22, 23]. The results obtained were high for the scale’s internal consistency, Split-Half Reliability, Test-Retest Reliability, and Intraclass Correlation Coefficient in terms of the overall scale and its factors. In this study, it was observed that the Cronbach Alpha Value of the total scale was 0.94, which was 0.89 in the original version of COVID-19-HLS [18]. These results showed that COVID-19-HLS and its factors had a high level of reliability for the Turkish society. In addition, the Split-Half Reliability Analysis in the present study showed that a strong and significant relationship was between the two halves of the scale and the scale had a high level of reliability for the Turkish population.

In general, the correlation value obtained from the test-retest in scale development and adaptation should be ≥ 0.70 [22]. In this study, the test-retest correlation value was found as 0.88 for the total scale and a positive and strong correlation was found between the test-retest scores. Test-retest correlation value of the original scale was reported to be 0.93 for the total scale [17]. It can be said that test-retest reliability results of both the original and Turkish versions of the scale are high. In addition, no significant difference was found between the average scores obtained from the two applications. This may be attributed to the comprehensibility of the scale. Hoteling’s T2 Test, which found out even if there is a significant difference between the scale items in terms of means, whether the difficulty levels of the questions are equal, whether the answers given by the participants to the items are similar, and shows the significance of all items for the scale, showed that the difference between the COVID-19-HLS item scores was significant in this study [21]. A comparison could not be made since this analysis was not available in the original scale study.

In validity and reliability studies, floor and ceiling effects show the measurability of the scale factors [26]. It can be seen that the COVID-19-HLS total and factor scores do not show a floor or ceiling effect and these effects of the scale are within the desired limits.

4.3. Limitations and Strengths

One of the strengths of this study is that in addition to standard scale adaptation analyses, test-retest application, EFA and CFA were also performed. The fact that the scale is easy to understand shows that it can be easily applied and interpreted, which provides convenience to researchers. Another strong aspect of the study is that it represents the general population of Turkey in terms of socio-demographic factors such as gender, age, and educational level. The fact that the study was conducted on an online platform, electronic devices were dependent on the Internet and the participants had to have internet skills, and the data collection process started with nursing students and their families are the limitations of the study.


As a result, it has been determined that COVID-19-HLS is a valid and reliable scale for Turkish society. This scale can be used to determine the health literacy levels of individuals and to measure their knowledge and attitudes about COVID 19. The scale also assesses individuals' ability to learn and understand about COVID 19, their knowledge, motivation and competence in finding, understanding and using health information.

It is necessary to increase the knowledge level of society against all infectious diseases, especially COVID-19, and the awareness of protection from diseases, and to gain literacy knowledge about COVID-19. Thus, individuals have the authority to make decisions about their health and can facilitate disease management. Therefore, ensuring that society is properly informed about an important epidemic such as COVID 19 can reduce anxiety, improve the attitudes and behaviors of individuals, reduce the risk of disease transmission, and contribute to greater participation in prevention measures by enabling individuals to do more.


NT and EK contributed to project conception and did the data acquisition and analysis, NT performed the statistical analysis, EK reviewed the literature, NT and EK wrote the manuscript and NT critically revised the successive drafts. All authors read and approved the final manuscript.


HL = Health literacy
COVID-19-HLS = COVID-19 Health Literacy Scale
CVI = Content Validity Index
EFA = Exploratory Factor Analysis
CFA = Confirmatory Factor Analysis
KMO = Kaiser-Meyer-Olkin
SPSS = Statistical Package for the Social Sciences


Ethical approval was obtained from the Ethics Committee of Atatürk University Faculty of Medicine (B.30.2.ATA.0.01.00/261).


No animals were used in this research. All procedures performed in studies involving human participants were in accordance with the ethical standards of institutional and/or research committees and with the 1975 Declaration of Helsinki, as revised in 2013.


Informed consent was obtained from all participants.


STROBE guidelines were followed.


The data and supportive information is available within the article.




The authors declare no conflict of interest financial or otherwise.


Declared none.


Akbal E, Gökler E. A fact that emerged in the COVID-19 outbreak process: Health literacy. Estud Public Heal J 2020; 5: 148-55.
An L, Bacon E, Hawley S, et al. Relationship between coronavirus-related ehealth literacy and covıd-19 knowledge, attitudes, and practices among us adults: Web-based survey study. J Med Internet Res 2021; 23(3): e25042.
Hemati S, Farhadkhani M, Sanami S, Mohammadi-Moghadam F. A review on insights and lessons from COVID-19 to the prevent of monkeypox pandemic. Travel Med Infect Dis 2022; 50: 102441.
Mukherjee S, Vincent CK, Jayasekera HW, Yekhe AS. Personal care formulations demonstrate virucidal efficacy against multiple SARS-CoV-2 variants of concern: Implications for hand hygiene and public health. PLOS Global Public Health 2022; 2(2): e0000228.
Leung AYM, Parial LL, Tolabing MC, et al. Sense of coherence mediates the relationship between digital health literacy and anxiety about the future in aging population during the COVID-19 pandemic: A path analysis. Aging Ment Health 2022; 26(3): 544-53.
Sariyildiz AY, Aksoy Y, Samanci M. The effect of mobile health applications on the health literacy of individuals. Pak J Med Health Sci 2022; 16(3): 552-.
Damian AJ, Gallo JJ. Promoting health literacy during the COVID-19 pandemic: A call to action for healthcare professionals. Harvard Kennedy Sch Misinformation Rev 2020; 1: 1-8.
Brørs G, Norman CD, Norekvål TM. Accelerated importance of eHealth literacy in the COVID-19 outbreak and beyond. Eur J Cardiovasc Nurs 2020; 19(6): 458-61.
Dastani M. E-Health literacy: A skill needed in the coronavirus outbreak crisis. Heal Technol Assess Action 2020; 4(1): 5861.
Robbins T, Hudson S, Ray P, Sankar S, Patel K, Randeva H. COVID-19: A new digital dawn? Digit Health 2020; 6: 2055207620920083.
Can Gür G, Altinbaş Y. COVID-19 literacy scale: Turkish validity and reliability study. Clin Nurs Res 2022; 31(3): 404-12.
Hong KJ, Park NL, Heo SY, Jung SH, Lee YB, Hwang JH. Effect of e-health literacy on Covıd-19 ınfection-preventive behaviors of undergraduate students majoring in healthcare. Healthcare 2021; 9(5): 573.
Jennings-Roche A, Adle M, Jean BS, Jaeger PT. The 1918 Influenza pandemic in popular media and the roles of public libraries in supporting health ınformation access, health literacy, and health justice during pandemics: learning from the past to prepare for the future. Libr Q 2023; 93(1): 86-109.
Daşlı Y, Bakırer A, Mısırlıoğlu A. The effect of the COVID-19 Outbreak on health literacy: A field study. J Econ Adm Sci 2022; 23(2): 585-97.
Robert F. Scale development: Theory and applications. 5th ed.. SAGE Publications 2021.
Hair JFJ, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7th ed. Pearson 2009.
Koyuncu İ, Kılıç AF. Use of exploratory and confirmatory factor analysis: A document review. Educ Sci 2019; 44(198): 361-88.
Sanaeinasab H, Saffari M, Rashidi-Jahan H, Rahmati F, Koenig HG, Lin CY. Development and psychometric assessment of the covıd-19 health literacy scale: Preliminary testing and factor structure. J Heal Lit 2022; 6(4): 32-46.
Seçer İ. Psychological test adaptation and development. Anı Publishing 2015.
Alpar R. Applied statistics and reliability-validity. 3rd ed. Detay Publishing 2014.
Aparisi F, Haro CL. Hotelling’s T2 control chart with variable sampling intervals. Int J Prod Res 2001; 39(14): 3127-40.
DeVellis RF, Thorpe CT. Scale development : theory and applications. SAGE Publications, Inc 2021.
Educational Research quantitative, qualitative, and mixed approaches. 5th ed.. SAGE Publications 2019.
Elangovan N, Sundaravel E. Method of preparing a document for survey instrument validation by experts. MethodsX 2021; 8: 101326.
Finch WH. Using fit statistic differences to determine the optimal number of factors to retain in an exploratory factor analysis. Educ Psychol Meas 2020; 80(2): 217-41.
Karagöz Y. = SPSS 23 and AMOS 23 applied statistical analysis.