Development and Psychometric Evaluation of the Instrument: Attitudes Towards Organ Donor Advocacy Scale (ATODAS)
Anne Flodén*, 1, Anna Forsberg2
Identifiers and Pagination:Year: 2012
First Page: 97
Last Page: 99
Publisher Id: TONURSJ-6-97
Article History:Received Date: 13/3/2012
Revision Received Date: 19/3/2012
Acceptance Date: 19/3/2012
Electronic publication date: 6/9/2012
Collection year: 2012
open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.
We appreciate the current discussion regarding our publication entitled Development and Psychometric Evaluation of the Instrument: Attitudes Towards Organ Donor Advocacy Scale (ATODAS) . This is what research and publication is all about, to criticize and discuss the rigor and trustworthiness of scientific results. However, the discussion seldom takes place in public which hampers the possibilities for others to learn more about methodological issues. We welcome the comments from Professor Loyd Lee Glenn and Dr. Jessica Stamey regarding the lack of psychometric soundness and the lack of clarity about content validity.
1. FORMING THE EXPERT GROUP TO GUARANTEE ITEM QUALITY AND CONTENT VALIDITY
We selected seven experts to assess the content validity of the initial 55-item scale. The number of experts was based on the guidance of Lynn  who states that a minimum of five experts would provide a sufficient level of control for change agreement and that the maximum numbers should be no more than ten.
There are a limited number of experts in Sweden with the prerequisite experience and knowledge in the field. The professional role of the experts was described in the article. The two intensive and critical care research nurses were selected on the basis of their extensive experience in this context, their methodological knowledge and expertise in developing instruments. They were known to us from their research. The four clinical nursing experts were selected due to their extensive experience of taking care of potential organ donors. Two of them worked in a neuro intensive care unit (NICU), which statistically cares for most of the brain dead patients occurring within intensive care. The other two worked in a large general ICU. We identified these four experts on the basis of their reputation as clinical supervisors. These four experts knew us by name as established researchers at our university but had no other relationship with us. To complement the clinical experts on the panel, a senior academic researcher an expert in item construction was invited to join the group.
All experts on the panel were sent a rating form with the theoretical definition as well as a delineation of the three dimensions, objectives and items. The experts discussed the instrument thoroughly. The experts were asked to rate each item for its relevance to the content domain and were invited to suggest revisions that would improve its clarity and to identify content not indexed by the existing set of scale items. Everyone rated each item individually.
Professor Loyd Lee Glenn and Dr. Jessica Stamey imply that there might have been potential bias involved in the selection process of the expert group, leading to errors in item construction. However, we argue that the experts’ long experience from intensive care and the critical situations of caring for potential organ donors, improved content validity. As presented, the content validity index (CVI) for the entire ATODA scale was 82%. A CVI score of 80% or higher is generally considered reasonable content validity .
The item construction process involved several basic steps. First we conducted two qualitative studies with in-depth interviews [4,5]. We further used the results from a large survey  to find further areas of concern. After that we consulted the literature about advocacy and started to develop the theoretical framework behind organ donor advocacy. When this preparatory work was finalized we started to generate and construct items. By this rather rigorous procedure, we were confident that the content reflected most of the important aspects of organ donor advocacy.
However, in our view the main weakness in the results, was the total scale variance of 41.9%. This result indicates that important areas of concern may not be covered by this instrument, which could have contributed to the low response rate.
FACTORS AFFECTING LOW RESPONSE RATE
The response rate of 502 in the present study is acceptable, if using Tabachnick and Fidell  reference that at least 300 cases are required for a factor analysis and MacCallum et al.  recommended that a sample size of 500 or more is preferable. The whole subject is complex, involving many aspects of care and professional responsibilities where attitudes are the sum of thoughts and feelings that lead ultimately to action. Actions constitute a central part of any attitude, so we chose to construct the instrument in the form of statements on how the informants would act in a certain situation. We are aware of the difficulties and complexity of the ATODAS that involves illuminating one’s own moral obligations and standpoints. Therefore, almost every item in the ATODAS demands deep reflection on one’s professional and moral responsibility. We assume this contributed to the low response rate whereby only nurses with a firm opinion about organ donor advocacy chose to participate. As identified in the article, this aspect requires further testing and revision before using the ATODAS more widely.
We raised the issue “that this instrument could be very useful when attempting to grasp the complex content of attitudes towards organ donor advocacy, which at times involves ethical and delicate aspects of ICU-nurses’ professional actions” (p,71). We do not imply that the instrument does not need further testing rather, that any effort to grasp a complex matter such as organ donor advocacy, might be facilitated by an instrument that rests on a strong theoretical framework.
ITEM CORRELATION AND CONSTRUCT VALIDITY
Professor Glenn and Dr. Stamey state that there are a large number of items with correlations under 0.60. All scientific work is about making choices. We argue that construct validity can be approached in several ways. We chose to construct validity by principal component analysis since it is the method for identifying a cluster of related items on a scale. According to Hays , item convergent validity is supported if an item correlates substantially (i.e. corrected correlation of 0.30 or above) with the scale it is hypothesized to represent. Hays advocate a more stringent convergent validity criterion if refining and adding development of a previous scale. In our study we employed this more stringent convergent validity criterion by Hays of 0.40 or above. We choose this five-factor solution since we judged it to fit the theoretical construct best and thereby we also accepted a few correlations close to 0.40.
In their critique Professor Loyd Lee Glenn and Dr. Jessica Stamey state that a low correlation to factors might not be a problem if the factors were orthogonal to each other. Orthogonal rotation is a subset of oblique rotations1. If the clusters of relationships are in fact uncorrelated, then oblique rotation will result in orthogonal factors. Therefore, the difference between orthogonal and oblique rotation is not in discriminating uncorrelated or correlated factors. As factor correlation decrease, the result from oblique and orthogonal rotation tends to become more similar .
The pilot study was conducted as a part of instrument evaluation and refinement of the ATODAS. As Professor Loyd Lee Glenn and Dr. Jessica Stamey point out, we described a random sampling of ICU nurses to the pilot study and this was a mistake by us. However, the effect of this selection has probably low impact of the validity and reliability of the study as this error was made in the pilot study and corrected in the main study.
Finally, Professor Glenn and Dr. Stamey suggest that our results don’t support the idea that the factors in the ATODAS really separate entities from each other. In our study, based on principal component analysis to obtain a solution with the most optimal scale variance, the five-factor solution fitted the best. The rotated five-factor solution revealed the presence of a simple structure, with each factor showing a number of strong loadings and most variables loading substantially (>.35) on only one factor. Therefore, we argue that the results from this initial testing of the ATODAS supports that the factors are separated from each other.
We argue many more tests are needed for generalisability e.g. reliability and validity testing of the instrument on other nursing populations outside Sweden. The context and organization surrounding the care of potential organ donors does differ considerably by country. The planning for further research has already started including validating the ATODAS instrument for different nursing populations including translation and adding culture specific items.
In conclusion, although we acknowledge the helpful comments made by Professor Glenn and Dr. Stamey we maintain that this context specific self-assessment instrument aimed at measuring attitudes towards organ donor advocacy can be used in future research.
CONFLICT OF INTEREST
The authors confirm that this article content has no conflict of interest.
1 Using oblique rotation provided the best definition of the uncorrelated and correlated cluster patterns of interrelated variables. Orthogonal rotation defines only uncorrelated patterns. Oblique rotation has greater flexibility in searching out patterns regardless of their correlation and the factors are rotated individually to fit each distinct cluster. The relationship between the resulting factors then reflects the relationship between the clusters.