The Development of an e-platform to Strengthen Nursing in Kazakhstan: A Systematic Review and a Delphi Study to Define Requirements

The Development of an e-platform to Strengthen Nursing in Kazakhstan: A Systematic Review and a Delphi Study to Define Requirements

The Open Nursing Journal 05 Oct 2021 RESEARCH ARTICLE DOI: 10.2174/1874434602115010156



To strengthen nursing in Kazakhstan, an e-platform has been developed to support knowledge sharing and collaboration in nursing education, nursing research, and clinical practice.


This study aims to identify and prioritize requirements for an e-platform to strengthen nursing in the trinity of nursing education, nursing research, and clinical practice.


The research was conducted using an exploratory sequential mixed-method design, including a systematic review and a Delphi study. The systematic review utilized search terms concerning nursing, e-platform, research, and education. The Delphi study included two rounds of questionnaires among a panel of Kazakh and European experts. Outcomes were analyzed using content validity ratios and level of consensus thresholds.


The systematic review generated eight studies and identified nine categories of requirements to be covered by four groups in the Delphi study. These four groups were content-related, functional, usability, and technical requirements. The study generated 52 essential and 15 useful requirements, which are common in the literature and in experts’ opinions.


The list of requirements is well embedded in the literature as well as in the national context of nursing in Kazakhstan. Therefore, the e-platform developed according to these requirements will contribute to improving the trinity of nursing education, research, and practice. Although the list is context-specific, when validated by a panel of experts, it is universally applicable when developing e-platforms for nursing.

Keywords: Nursing, E-platform, Requirements analysis, Knowledge sharing, Collaboration, Nursing education, Nursing science, Nursing practice, Kazakhstan.


This study is part of a larger project addressing nursing innovation in Kazakhstan. As one of the countries in the Central-Asian region that gained independence in 1991, Kazakhstan has undertaken major efforts to reform its post-Soviet healthcare system [1]. Still, increasing healthcare utilization and improving health outcomes are among the most challenging aspects of healthcare in the country [2]. In alignment with this transition in healthcare, the role of nursing care professionals is increasingly important, and their organizational authority and responsibility are also growing [3]. The developments in healthcare and changes for nursing professionals have a considerable impact on nursing education. Consequently, Kazakhstan is facing demanding innovations in nursing higher education.

For this reason, the project ‘Promoting the Innovation Capacity of Higher Education in Nursing During Health Services Transition’ (ProInCa) was launched in Kazakhstan [4]. This European-funded project operated from 2017–2020 and was supported by four European Higher Educational Institutes (HEIs) – two in Finland, one in the Netherlands, and one in Slovenia – that worked cohesively with Kazakh HEIs and associated partners, including Medical Colleges, health care organizations, and the Ministries of Health and Education. To guarantee the continuation of the transition process after the project was completed in 2020, communities of nurse educators and students, as well as researchers and clinical nurses, were encouraged to collaborate and exchange their expertise. For this reason, an e-platform was developed to support knowledge sharing and collaboration among all clinical nurses, nursing students, educators, and researchers [4]. Improved collaborative structures for clinical nursing practice, education, and research are at the core of the innovation capacity that is needed to realize the transition process [4].

The challenges in nursing higher education explained above and the need for collaboration and knowledge exchange in all domains of nursing, including nursing practice and nursing research, can be successfully addressed with the features of an e-platform. The use of e-platforms in online learning and collaboration has increased worldwide, and the Kazakh infrastructure is well equipped to provide proper Information and Communication Technology (ICT) tools. These ICT tools facilitate the ability to learn, work, meet, and network virtually and promote collaboration among persons with similar professional interests [5]. In the nursing profession, it has become common practice for educational or research institutes and clinical nursing associations to use the Internet, social media, and e-platforms. Collaboration and knowledge exchange within the three domains of nursing – education, research and practice – is crucial to improve the quality of nursing, which are prerequisites for accomplishing the national healthcare transition [6].

First, in the domain of nursing education, e-platforms for educational purposes generally include interactive online services that provide trainers, learners, and others involved in education with information, tools, and resources to support and enhance education delivery and management [7]. These e-platforms combine the advantages of Learning Management Systems (LMSs) with social software, thus contributing to interaction and collaboration between learners and educators in the cognition, construction, and socialization aspects of learning processes [8].

Second, in the domain of nursing research, developments in ICT and global health have contributed to a considerable increase in the number of nursing research projects. Although geographic proximity is known to facilitate successful collaborations, ICT enhances distant cooperation as international nursing science collaborations continue to grow [9]. The application of ICT has become increasingly important for informing and educating nurses about research findings. Devices such as smartphones are employed to distribute information that increases nurses’ awareness concerning the importance of nursing science and the accessibility of research evidence [10, 11].

Third, concerning the domain of nursing practice, the literature indicates that clinical nurses have increasing access to ICT for retrieving information related to Evidence-Based Practice (EBP) and clinical guidelines that are evidence-based. Nurses have developed skills to search health-related websites and databases to expand their knowledge, improve their clinical judgement, and consequently enhance the quality of care [12]. Social media applications that have been developed as teaching tools are also currently used by clinical nurses in practice. Several studies report positive outcomes in building best practices by teaching through social media, which contributes to clinical practice [13, 14].

The above evidence indicates that the development of an e-platform should enhance knowledge sharing and collaboration in the fields of nursing education, nursing research, and clinical practice. To this end, the e-platform should contain online learning courses that include access to research results and clinical guidelines. To ensure that the e-platform corresponds the future users’ needs and requirements, stakeholders from the fields of nursing education, nursing practice, and nursing research should be consulted. Therefore, a requirements assessment was realized by conducting a Delphi study among a panel of experts to prioritize e-platform features, preceded by a systematic review to identify which requirements have been reported in international scientific publications. The results of this combined study contributed to identifying and prioritizing the essential and useful requirements for an e-platform to improve the quality of nursing as part of the Kazakh healthcare transition.


2.1. Design

This study has an exploratory sequential mixed-method design, including a qualitative systematic review followed by a quantitative Delphi study [15]. The systematic review aimed to identify requirements for an e-platform for collaborative developments in nursing science and nursing research education. The findings of the systematic review generated the input for the Delphi study, which aimed to achieve consensus among experts on the requirements to be prioritized.

2.2. Systematic Review Methods

A systematic literature search was conducted according to the PRISMA guidelines by researchers collaborating internationally from Kazakhstan, the Netherlands, and Finland. Consensus on search strategy and screening, as well as the selection of results, was reached through online meetings. A series of search terms were identified in relation to e-platforms or online environment, nursing, research or evidence, and education or learning. From these search terms, different search strings were developed to browse the electronic databases IEEE, TRIP, EBSCO, ABI, PubMed, and CINAHL (Table 1). The searches were conducted between February 10 and April 12, 2018. In addition to this systematic approach, literature introduced in the included studies was also examined (also known as the snowball technique) to minimize the possibility of overlooking important research. The studies identified were required to involve humans, be published in peer-reviewed academic journals not older than six years, and be written in the English or Russian language. Further inclusion criteria were that studies should concern nursing students or clinical nurses who had participated in an online platform or learning program concerning research or evidence-based nursing. The studies should have evaluated the efficacy or impact of the online platform or program using an evaluation instrument or should have evaluated the platform’s development process. The students’ perceptions, experiences, or opinions should have been an element of the evaluation.

Table 1.
Search strings.
Electronic Database Search Strings
IEEE e platform AND nursing AND research AND education
Trip e-platform AND nursing AND research AND practice
EBSCO (eplatform OR portal) AND “nursing education”
ABI (eplatform OR portal) AND (healthcare OR nursing)
PubMed (“evidence-based nursing” (MeSH) OR nursing) AND (“computer communication networks” (MeSH) OR “virtual learning environment”)
CINAHL (nursing OR nurses) AND (research OR evidence-based) AND (education OR learning) AND (online OR internet)

2.3. Systematic Review Analysis

Initially, two assessors (ATO, AA) completed the first round of assessments on titles and abstracts in accordance with the PRISMA guideline. Subsequently, one reviewer (ATO) screened and reviewed the full text of articles as needed to determine eligibility. Eligible articles were discussed with two other reviewers (WP, AA) and only included if the researchers reached an agreement. Two researchers (ATO, WP) graded the evidence level of included studies according to the Oxford Centre for Evidence-based Medicine [16] and conducted the methodological quality appraisal according to the Joanna Briggs Institute checklists [17]. Data extraction and data synthesis from the articles took place by assessing the objectives, types of platforms or programs, evaluation tools, key findings, and major recommendations. Three researchers (ATO, WP, AA) analyzed the included studies to identify essential requirements for an e-platform. These independent analyses were compared in two consensus meetings to arrive at a definitive overview of the results and key findings of the studies.

2.4. Delphi Study Methods

A Delphi study was conducted to prioritize e-platform requirements since this consensus-building tool has been applied in a variety of fields, including innovations in health care [18]. It is an appropriate method when respondents are located at considerable geographic distances, as was the case in this study.

The Delphi panel in this study is quantitative in nature. The respondents were asked to score items on a three-point scale to facilitate the calculation of content validity ratios as well as percentages. A panel of experts was established, including 59 respondents originating from Kazakhstan, Finland, and the Netherlands. Since the e-platform aims to match the context of Kazakh nursing developments, most experts (75%) were from Kazakhstan. Experts in education, research, clinical practice, and information technology as well as professional leadership and the government, were included. The respondents were either participant in ProInCa activities or connected to the project as associated partners or international project members. They were also professionally related to the field of nursing and had adequate understanding or experience in at least one of the following four domains: nursing education at the vocational, bachelor, or master level; nursing research; ICT expertise in nursing; or ICT expertise in online collaboration. Project members engaged in the e-platform development were excluded because they would be too closely involved.

The Delphi study included two rounds of questionnaires containing statements concerning e-platform requirements. For the preparation of the questionnaire, the key findings of the systematic review were discussed at a ProInCa project meeting and translated to requirement statements for the e-platform for nursing in Kazakhstan. This resulted in a list of 62 requirement statements to be included in the Delphi questionnaire. These statements were clustered into four groups: “content requirements,” “functional requirements,” “usability requirements,” and “technical requirements.” The group “content requirements” also included nursing content. For prioritizing nursing content issues, the seven domains of the Nursing Intervention Classification (NIC) were used, which are physiological basic, physiological complex, behavior, safety, family, community, and health system [19].

The first questionnaire was transferred to the SurveyMonkey tool in order to electronically aggregate the survey data [20]. Respondents received a link to the questionnaire by e-mail and were requested to provide informed consent. They were invited to prioritize all requirements on a three-point scale: 1) essential, 2) useful but not essential, or 3) not necessary [21]. Rating was based on the respondents’ own opinions, and anonymity was guaranteed. In addition, two open-ended questions were included through which respondents could contribute additional requirements. A reminder was sent after two weeks and a second reminder after four weeks.

As described by Linstone and Turoff (1975), the Delphi procedure enables individual experts to provide feedback on the results of the first round by offering the opportunity to modify their previous ratings. Therefore, in the second-round questionnaire, all requirements were presented again either as 1) essential or 2) useful but not essential. In addition, based on responses to the open-ended questions in the first round, some minor linguistic changes were made and five additional statements were included. Experts were invited to agree or to disagree with the lists of “essential” and “useful but not essential” requirements. Both questionnaires were provided in English and Russian. Anonymity for individual responses was guaranteed [22].

2.5. Delphi Study Analysis

For the analysis of the ratings in the first round, a content validity ratio (cvr) was applied, which is the appropriate technique for determining consensus among a panel of experts [21]. The cvr should be determined as (ne - N/2)/(N/2) in which ne is the number of panelists indicating “essential,” and N is the total number of panelists. In a panel of 40 or more experts, a cvr of 0.29 is considered the minimum [21]. The outcomes of the second round in which requirements could be rated as “essential” or “useful but not essential” were analyzed by allocating percentages. The consensus among experts was determined by applying a threshold of 75%, which is the appropriate analysis technique at this stage in a Delphi study [23].


3.1. Systematic Review Results

After removing duplicates and studies that did not meet the inclusion criteria, a further selection was accomplished. This selection included the availability of the full text and screening based on title and abstract. Additional consideration of these criteria resulted in six relevant studies. After reading the full text of the six studies, the researchers determined that references of two studies were relevant to include, although not included by the first queries [snowball searching]. As a result, the literature search strategy generated eight studies. The selection and screening process are presented in the PRISMA flowchart in Fig. (1).

Fig. (1). PRISMA flowchart.

The studies identified are presented in Table 2, representing a variety in country and study participants, study design, and evidence grading. The methodological quality of all studies was good to excellent. In one cross-sectional study, excellence was achieved since confounding factors were addressed [24]. Excellence was also achieved in one qualitative study by specifically addressing the researchers’ roles [25].

In all studies, the e-platform users’ perceptions and experiences were assessed. Objectives were to evaluate, assess, or discuss either the efficacy and impact or the development of an online platform or program. The objectives, types of platforms or programs, and evaluation tools are presented in Table 3, including the key findings and major recommendations. The key findings and recommendations generated an overview of e-platform requirements that were classified into nine categories [1]: aim and users [2], educational content and materials [3], nursing skills [4], platform structure [5], communication, interaction, and feedback [6], knowledge acquisition and achievement of learning outcomes [7], management [8], infrastructural and technological support and [9] usability.

3.2. Delphi Study Results

In the first round of the Delphi study, 59 experts completed the online questionnaire, including 48 respondents from Kazakhstan (81%) and 11 from European countries (19%), as illustrated in Table 4. The table also shows the respondents’ employment and expertise, with the highest percentage of respondents (46%) being employed in a health care organization and the highest percentage of respondents’ expertise (46%) being in nursing practice. In the second round, 17 experts completed the questionnaire: 65% from Kazakhstan and 35% from European countries. In contrast to the first round, most respondents (76%) in the second round were employed at a university, and most respondents’ expertise (47%) was in nursing education.

Table 2.
Included studies, country and study participants, study design, evidence grade, and methodological quality appraisal score.
Included Study Country and Study Participants Study Design Evidence Grade1 Methodological Quality Appraisal Score2
Davidson, SJ. et al. Teaching EBP Using Game-Based Learning: Improving the Student Experience. Worldviews on Evidence-Based Nursing, 2016, 13:4, 285-293. Canada
Undergraduate nursing students n=30
B 6/8 (Good)
Du, S. et al. Web-based distance learning for nurse education: a systematic review. International Nursing Review, 2013, 60, 167-177.
9 RCT’s
Nursing students and employed students
Systematic Review of RCTs A 10/11 (Good)
Gagnon, J. et al. Adaptation and Evaluation of Online Self-learning Modules to Teach Critical Appraisal and Evidence-Based Practice in nursing. Computers, Informatics, Nursing, 2015, 7, 285-294. Canada & Spain
Clinical nurses n=83
Quebec (36) and Basque country (47)
Pre-post study B 8/9 (Good)
Karaman, S. et al. Evaluation of an online continuing education program from the perspective of new graduate nurses. Nurse Education Today, 2014, 34, 836-841. Turkey
Registered nurses
B 6/8 (Good)
Kowitlawakul, Y. et al. Development of an e-Learning Research Module Using Multimedia Instruction Approach. Computers, Informatics, Nursing, 2017, 3, 158-166. Singapore
Master of Nursing students n=8
(2 focus groups of 4 students)
Faculty members n=2
Qualitative study D 8/10 (Good)
League, K. et al. Increasing Nurses’ Access to Evidence Through a Web-Based Resource. Journal of Nursing Administration, 2012, 42:11, 531-535. United States
Staff nurses
744 pretest
1164 posttest
Pre-post study B 8/9 (Good)
McIntyre M. et al. A critical Analysis of Online Nursing Education: Balancing Optimistic and Cautionary Perspectives. Canadian Journal of Nursing Research, 2013, 45:1, 36-53. Canada
undergraduate and graduate nursing students
Qualitative study D 10/10 (Excellent)
Seixas, CA. et al. Usability Assessment of Moodle by Brazilian and Portuguese Nursing Students. Computer, Informatics, Nursing, 2016, 6, 266-271.
Portugal & Brazil
Undergraduate and diploma nursing students n=79
Brazilian (53) and Portuguese (26)
B 8/8 (Excellent)
Table 3.
Included studies, objectives, types of platform, evaluation tools, key findings and recommendations, and requirements category.
Included Studies Objectives Types of Platform or Programme Evaluation Tools Key Findings and Recommendations Requirements Category
Davidson, SJ. et al. (2016) Describe the development and evaluation of game-based learning to better prepare nursing students to engage in EBP Quest-based learning platform (3D Game Lab) Survey with 12 questions 1-5 pt. Likert scale,
platform analytics,
thematic analysis of students’ comments
Provide timely and individualized feedback, create student choice in selection of learning quests, create customization of learning, provide visible milestones, level learning activities to build on each other Aim and users; Communication, interaction and feedback; Knowledge acquisition and achievement of learning outcomes
Du, S. et al.
Examine the efficacy of web-based distance education for nursing students and employed nurses Websites containing distance education as experimental teaching strategies Web-based distance nursing education compared to traditional teaching or blank control Develop course information, a curriculum map, teaching materials, communication tools, formative and summative assessments, student management tools, links to databases, teleconferencing, case-based learning. Content in text/video/audio formats, interactive designs. Minimize hardware and software problems. Inadequate time/skills to develop materials, expenses involved, lack of research-produced proof Aim and users; Nursing skills; Platform structure; Communication, interaction and feedback; Knowledge acquisition and achievement of learning outcomes; Management; Infrastructural and technological support; Usability
Gagnon, J. et al. (2015) Evaluate online self-learning modules on critical appraisal skills to promote the use of research in clinical practice Online self-learning course (InfoCritique Program) Two questionnaires (pre and post course)
(SDLRNE and knowledge questionnaire), one satisfaction questionnaire
Content satisfaction involves difficulty, number of examples, language of materials, complexity of content, number of modules addressing literature searches, experimental designs, and statistics. Participation satisfaction involves motivation, commitment. Platform satisfaction involves user-friendliness, rapid load, feedback, interaction, structure, easy navigation. Prevent lack of space for interaction, enhance transfer of knowledge into practice and to colleagues Aim and users; Educational content and materials; Knowledge acquisition and achievement of learning outcomes; Infrastructural and technological support; Usability
Karaman, S. et al. (2014) Evaluate the online continuing education program from the perspective of nurses Online degree in nursing (HELITAM:
first online bachelor nursing program in Turkey)
Survey with 1-5 pt. Likert scale, open-ended questions Monitor programme and course structure, including quality of design of asynchronous online education, quality of effective support, quality of technological infrastructure, and quality of teaching Aim and users; Platform structure; Knowledge acquisition and achievement of learning outcomes; Infrastructural and technological support
Kowitlawakul, Y. et al. (2017) Discuss the development and piloting process, including the variety of evaluation perspectives e-learning research project module using interactive multimedia Two semi-structured focus group sessions, two individual interviews Focus on impact of the e-learning module on students’ performance and students’ learning outcomes Aim and users; Knowledge acquisition and achievement of learning outcomes
League, K. et al
Describe the development and impact of a Web-based tool to improve nurses’ access to evidence Website
Launched 2009
Survey with 1-4 pt. Likert scale (pretest 2008 and posttest 2010) Support increasing use of EBP, improve perception of supervisors’ support for EBP, allocate time for staff to use evidence at work Aim and users; Educational content and materials; Nursing skills
McIntyre, M. et al. (2013) Discuss how peer dynamics influence student learning in an online environment Two online nursing programs Individual face-to-face or telephone interviews (N=30), one focus group interview (eight participants),
Consider pros and cons of flexible access to education, consider differences in expectations and limits of electronic media, online skills development should also address clinical practice, take into account the specific nature of online environments Aim and users; Nursing skills; Communication, interaction and feedback
Seixas, CA. et al. (2016) Assess the usability of a virtual learning environment for nursing students Distance education platform
Questionnaire, log records analysis Consider usability and variables such as country of origin, previous experience in distance education Aim and users; Management; Infrastructural and technological support; Usability
Table 4.
Experts’ characteristics: country, employment and expertise.
Experts’ Characteristics First Round
Second Round
n % n %
Country - - - -
• Kazakhstan 48 81% 11 65%
• Europe 11 19% 6 35%
Employment1 - - - -
• University 26 44% 13 76%
• Health care organization 27 46% 2 12%
• Other organization 6 10% 3 18%
• Missing value - - 1 6%
Expertise1 - - - -
• Nursing practice 27 46% 1 6%
• Nursing education 19 32% 8 47%
• Nursing research 14 24% 6 35%
• Development of e-platform 3 5% 3 18 %
• Missing value 1 2% 1 6%

Most requirements identified from the systematic review were included as statements in the Delphi study for prioritizing. These requirements, which were divided into nine categories, were covered by four groups of statements for the Delphi study, as shown in Table 4.

Table 5.
Nine categories of requirements covered by four groups.
Category GROUP
1. Aim and users CONTENT
2. Educational content and materials
3. Nursing skills
4. Platform structure
5. Communication, interaction and feedback FUNCTIONAL
6. Knowledge acquisition and achievement of learning outcomes
7. Management
8. Infrastructural and technological support TECHNICAL
9. Usability USABILITY

3.3. Combined Results

Requirements identified from the systematic review and prioritized in the Delphi study are presented below by group and by category, with ratings shown in Table 6 and Table 7. Table 6 presents a list of 52 requirements that were rated as “essential,” with a cvr of 0.29 or higher. Table 7 presents a list of 15 requirements that were rated as “useful but not essential,” with a cvr below 0.29.


4.1. Aim and Users

The aim of the platforms studied in the systematic review is to enhance the ability to put acquired nursing knowledge into practice [26]. In the systematic review of Du, clinical nursing skills performance was one of the main outcomes [27]. All authors stated that the acquisition of research skills should enhance the transfer of knowledge to clinical practice and should also increase the access to and use of evidence in the clinical nursing profession [28]. A further requirement is that knowledge should be disseminated among colleagues [26]. It was suggested that it is important to allocate additional time for staff to realize knowledge transfer at work [28].

The Delphi study showed that all statements concerning aim and users were rated as essential (Table 6, Statements No. 1–4), with the highest scores given to enhancing the ability to put nursing knowledge into practice and enhancing nursing skills performance. Enhancing nursing management and leadership (Table 6, No. 5) was added by respondents in the first round as an essential requirement and was confirmed in the second round.

Table 6.
Essential requirements.
     It is ‘essential’ that the e-platform cvr
first round (n=59)
agreement second round (n=17)1
  Aim and users      enhances
  1. To put nursing knowledge into practice 0.83 93.75%
  2. Nursing skills performance 0.69 76.47%
  3. Increasing usability of evidence in clinical nursing professions 0.47 87.50%
  4. Transfer of knowledge to colleagues 0.46 81.25%
  5. Nursing management and leadership2 81.25%
     has the following characteristics
  6. Information is available in Russian and English and Kazakh 0.63 100.00%
includes for educational purposes
  7. Materials for different target groups (bachelor, master) 0.66 100.00%
  Educational content and materials      includes educational materials about
  8. Evidence based nursing 0.90 94.12%
  9. Evidence based nursing clinical guidelines for use in practice 0.82 100.00%
  10. Nursing research skills 0.55 100.00%
  11. Nursing leadership 0.42 93.75%
includes information concerning nursing research skills such as
  12. To apply evidence in the clinical practice 0.64 93.33%
  13. Basic research principles 0.61 87.50%
  14. Skills to use electronic databases (information systems in nursing) 0.61 87.50%
  15. Literature searches 0.44 86.67%
  16. Qualitative designs 0.36 80.00%
  17. Formulate quests/ research questions e.g. in PICO format 0.35 86.67%
  18. Quantitative designs (including surveys and cross-sectional designs) 0.32 82.35%
  Nursing skills      includes information concerning nursing domains such as
  19. Patient safety (such as hygiene, prevention of adverse events) 0.73 100.00%
  20. Physiological-complex (such as care for cancer; optimize neurological function) 0.56 93.75%
  21. Family (such as child baring care and care to support families) 0.52 93.33%
  22. Health promotion and prevention of health risks 0.45 93.75%
  23. Behavioural (such as dementia and depression) 0.39 93.33%
  24. Health System 0.39 85.71%
  25. Nursing documentation (development and practical use)2 93.33%
  26. Quality management (tools, techniques for improving nursing practice)2 93.33%
  27. Standard operating procedures (SOP) and complex routine operations2 93.33%
  28. Laws and regulations for nursing practice2 93.33%
  Platform structure      includes
  29. An agenda/calendar with relevant nursing research and nursing educational activities in Kazakhstan and beyond 0.34 missing value
  Communication interaction and feedback      includes functionalities for
  30. Online collaboration 0.30 81.25%
     includes distance learning courses which
  31. Provide timely feedback from tutors and peers 0.62 88.24%
  Knowledge acquisition and achievement of learning outcomes      includes for educational purposes
  32. Distance learning courses 0.47 100.00%
  33. Instruction films 0.41 93.75%
     includes distance learning courses which
  34. Include clear learning goals and learning outcomes 0.86 87.50%
  35. Have a clear structure and a course map 0.83 100.00%
  36. Include instructions, examples and assignments 0.73 100.00%
  37. Support knowledge acquisition by enabling self-directed learning 0.61 100.00%
  38. Include summative assessment with certification 0.49 87.50%
  39. Include formative assessment to assess knowledge acquisition 0.40 81.25%
  Management       includes tools
  40. To secure personal data and administration of courses 0.44 93.75%
  41. To manage user management accounts 0.40 68.75%
  42. For content management so it is easy to add and change information 0.29 82.35%
  Infrastructural and technological support      should technically support
  43. Use on a computer, tablet, and mobile phone. 0.83 100.00%
  44. Guaranteed technological support (not restricted to time, place, or cost) 0.53 87.50%
  45. Use in different browsers 0.51 93.75%
  Usability has the following characteristics
  46. Well designed and attractive 0.82 100.00%
  47. Open and free access to information 0.80 100.00%
  48. Clear and simple site navigation 0.75 100.00%
  49. Text on each page is not too much and easy to read 0.62 81.25%
  50. Contact information is easy to be found 0.58 100.00%
  51. Login for educational and collaboration purposes 0.44 93.75%
  52. Clear site structure2 100.00%

The systematic review indicated that the choice of language should be considered thoroughly before developing the e-platform [26]. In the Delphi study, it was considered essential that information be available in Russian and English as well as in Kazakh (No. 6).

Respondents in the Delphi study also considered it essential that the e-platform include materials for different target groups (No. 7). The platforms in the literature identified addressed a variety of users and levels varying from undergraduates and graduates to clinical nurses or combinations of these. McIntyre et al. (2013) addressed undergraduate as well as graduate nurses, including student levels of master’s degree, bachelor’s or baccalaureate’s degree, or associate or diploma degree [25]. Some platforms explicitly targeted clinical nurses [26, 28, 29], or employed students [27].

4.2. Educational Content and Materials

The primary requirement related to educational content and materials identified by the systematic review was access to evidence-based guidelines [28]. The literature also indicated the importance of basic nursing research skills. Three studies explicitly explained what educational content concerning nursing research skills should be included [28, 30, 31], such as expertise in basic research principles, expertise in research paradigms like quantitative and qualitative methods [30], and understanding of the major characteristics, strengths, and limitations of quantitative and qualitative research designs [31]. Research skills also involve implementing quest-based search strategies and formulating clinical research questions [31] and using the Patient, Intervention, Comparison, Outcome (PICO) structure [28, 31]. Subsequent skills include performing literature search strategies using a variety of library databases [26, 31]; all studies mentioned the importance of the ability to use electronic databases to access scientific nursing publications. Most statements in the Delphi concerning educational content and materials were rated as essential, as demonstrated in Table 6 (No. 8–18).

The systematic review indicated that additional nursing research skills concern statistical data analysis [26], critical appraisal of scientific reports [31], and academic writing [25]. These research skills were rated as useful but not essential (Table 7, No. 8–10).

Table 7.
Useful but not essential requirements.
     It is ‘useful but not essential’ that the e-platform cvr
first round (n=59)
  agreement second round (n=17)1
Platform structure includes
  1. Webinars about nursing topics 0.28   75.00%
  2. A blog about several nursing topics, updated once a month 0.22   82.35%
  3. Links to electronic databases with scientific nursing publications 0.14   86.67%
  4. Forum discussions for nurses on nursing topics 0.07   93.75%
  5. Links to international websites with additional nursing guidelines 0.07   73.33%
  6. An overview of important stakeholders for nursing in Kazakhstan such as universities with bachelor and/or master programmes in nursing, university medical hospitals 0.00   81.25%
  7. An overview of national and international institutions -0.09   87.50%
Educational content and materials includes information concerning nursing research skills such as
  8. Analysis, such as statistics or qualitative analysis 0.16   73.33%
  9. Critical appraisal of scientific literature 0.04   80.00%
  10. Academic writing -0.05   81.25%
Nursing skills       includes information concerning nursing domains such as
  11. Physiological-basic (such as physical activity, nutrition support) 0.19   53.33%
Communication interaction and feedback includes functionalities for
  12. Online discussions 0.11   80.00%
includes distance learning courses which
  13. Enable communication and interaction with tutors and peer students 0.05   88.24%
Knowledge acquisition and achievement of learning outcomes includes for educational purposes
  14. Educational materials for use in classroom settings to download 0.28   81.25%
Management includes tools
  15. To report functions with the option to monitor the number of users, popularity of the different items, and improve the site content 0.00   81.25%

4.3. Nursing Skills

Statements concerning nursing skills were obtained from the seven NIC domains, and six of these were rated as essential, as shown in Table 6 (Statements No. 19–24). The domain physiological-basic was rated as useful but not essential, as shown in Table 6 (No. 8), although no agreement was achieved in the second round.

Nursing skills concerning documentation, quality management, standard operating procedures, and laws and regulations (No. 25–28) were added by respondents as essential requirements after the first round, which was confirmed by high levels of agreement in the second round.

4.4. Platform Structure

A variety of platform structures was reported in the studies identified by the systematic review. All platforms or websites were structured in a way to contain distance learning courses or series of nursing learning modules [27]. Some of the platforms included additional structures that enabled content and tools such as forum discussions, webinars about nursing topics, blogs or podcasts [25], or teleconferencing [27]. The content can be provided in text as well as in audio and video formats [27]. These formats may also include audio-visual presentations as well as graphics and animations, usually with multimedia and interactive elements [29]. The platform structure should allow links to electronic databases, as is mentioned by all the studies.

Although identified as crucial in the systematic review, all statements in the Delphi study concerning platform structure were rated useful but not essential. Only one statement was prioritized as essential – an agenda or calendar with relevant activities (Table 6, No. 29).


5.1. Communication, Interaction, and Feedback

The review revealed that interactive functionalities for online collaboration and online discussions are the basic characteristics of a successful educational e-platform. This includes interaction between students, teachers, and content, which must always be maintained [25]. This type of e-platform design is explicitly referred to as an interactive design [27], and all modules or courses that are included should be interactive [30]. Interaction is aimed at achieving learning outcomes using feedback from tutors and peers. In addition, feedback should be individual-based and prompt or timely [31].

In the Delphi study, the statements concerning online collaboration and feedback were rated as essential (Table 6, No. 30–31). However, the statements concerning online discussions and communication and interaction with tutors and peers were rated as useful but not essential (Table 7, No. 12–13).

5.2. Knowledge Acquisition and Achievement of Learning Outcomes

The review demonstrated that requirements for a successful e-platform include clear learning goals and learning outcomes to support knowledge acquisition through self-directed learning. Consequently, courses should provide explicit instructions, examples, and assignments. Davidson et al. (2016) demonstrated the need for guaranteeing progression by increasing levels of learning activities [31]. The specific characteristics of knowledge acquisition through e-learning were considered thoroughly in all studies. Specific consideration should be given to assessments for measuring progression in learning. Specifically, in self-directed learning, it is important to distinguish between formative and summative assessments [27]. In testing the achievement of learning outcomes, priority was on formative assessment, which corresponds with the typical features of self-directed learning [26]. It is recommended that the e-platform includes materials to download for use in classroom settings as well.

In the Delphi study, all statements concerning knowledge acquisition were rated as essential (Table 6, No. 32–39), apart from the statement concerning downloadable educational materials for classroom use, which was rated as useful but not essential (Table 7, No.14).

5.3. Management

A platform should include management tools for developing and maintaining structures and materials and for secure administration of courses. It should be simple to further develop the platform and to add and change information. Management requirements involve monitoring tools to track students [27], check logins, and ensure secure personal data and user accounts. It may also be necessary to monitor the number of users and the popularity of different items.

The experts in the Delphi panel rated most management requirements as essential (Table 6, No. 40–42), although monitoring the number of users and popularity of different items was rated as useful but not essential (Table 7, No. 15).


6.1. Infrastructural and Technological Support

All studies in the review demonstrate that the e-platform must be compatible for use with different devices, such as a laptop, tablet, or smartphone. Requirements concerning infrastructural and technological support also stipulate that access to the Internet should not be restricted in terms of time, place, or costs [27]. The unlimited availability of technological support by a support center or helpdesk is a basic requirement because hardware and software problems commonly occur [27]. The effectiveness and quality of the technological support provided are essential [29]. Testing by information technology personnel is required to check for viruses in the system [30]. All studies reported that the technological infrastructure should enable unrestricted access to other websites or electronic databases.

All statements in the Delphi study concerning technical requirements (Table 6, No. 43–45) were rated as essential.


7.1. Usability

In all studies reviewed, usability was an important feature that involves user-friendliness, student motivation, and engagement. Usability requirements include easy site navigation and a rapid load time [26] as well as an attractive design with an adequate amount of text that is easy to read. In addition, ergonomic usability is required [24]. Another feature of usability is accessibility, which may be open, closed, or a mixture. In some of the studies, access was limited to registered members of a specific learning community [28]. In all nine studies of Du’s systematic review (2013), users had open access [27]. Usability of the platform should enhance and support flexibility, which is the key to distance learning and includes flexibility in time, place, and learning activities [31]. Flexibility in accessing education encourages students to work across time zones [25]. Usability also means information should be easy to locate, including clear course information by curriculum mapping [27] and useful instructions [26].

In the Delphi study, all usability requirements were rated as essential (Table 6, No. 46–52).


The aim of this research was to identify and prioritize the essential and useful requirements for an e-platform to improve the quality of nursing as part of the Kazakh healthcare transition. The results provide a valid list of 52 essential and 15 useful requirements with a high level of consensus. The input for the Delphi study that was derived from the literature and translated into a list of 62 requirement statements for the e-platform of nursing in Kazakhstan appeared to be relevant and corresponded with the perceptions of experts concerning the needs of future users of the e-platform to be developed.

The developers of the e-platform in Kazakhstan were advised to give priority to the essential requirements. This does not mean that the other useful requirements are to be ignored. These requirements should be added in time as well, but possibly with a smaller scope or depth. For example, although webinars, blogs, and discussion forums were not reported as “essential” by the experts, it is still recommended to design the platform with this functionality. In addition, regarding the different skills taught through the platform, the choice has been made to include them all but to give certain skills more attention than others. Our research design did not include a way to ask experts about the reason, why they made a choice to label, for example, physical skills as essential and psychological skills as useful but not essential. This could be seen as a limitation of the study and an opening for further research.

Within the category “educational content and materials,” the basic research skills were rated as essential, whereas the advanced research skills were rated as useful but not essential. Advanced research skills refer to analysis methods, critical appraisal of scientific literature, and academic writing. An explanation for the different rating of basic and advanced skills could be that – although there are nursing professionals at a master level among the future users of the e-platform [4] – nursing research developments are quite recent in Kazakhstan, and, therefore, research skills at an advanced level are not a priority.

Close examination of the categories and groups indicates that requirements concerning platform structure are useful but not essential, apart from the requirement for “an agenda or calendar with relevant nursing research and nursing education activities in Kazakhstan and beyond,” which were considered essential. Although this requirement was not obtained from the systematic review but from a project meeting [4], it was included in the Delphi study and was rated as essential. Apparently, the experts anticipated that platform users would appreciate being well informed concerning nursing activities. Although this requirement was not included in the second round of the Delphi, it could be a topic of interest in consulting future users when developing an e-platform.

One strength of this study is that the ability of experts to represent future users’ perceptions was reinforced by inviting the experts to add more statements. This resulted in five additional essential content-related requirements for the Delphi and thus contributed to the embedding of results in the experts’ perspectives. In the first round, for example, experts added that the e-platform should enhance nursing leadership. This input was relevant since nursing leadership is considered a key factor in implementing evidence-based nursing [32].

8.1. Methods Used

The exploratory sequential mixed-method design, which has also been applied in other health-related studies [18], appeared to be an appropriate method for identifying and prioritizing requirements for an e-platform in nursing. The requirements ascertained from the literature are translated and validated for the specific context using the Delphi method.

A strength of the Delphi study was the panel composition with most experts from Kazakhstan and others from European countries. Consequently, the results are well embedded in the national context. At the same time, a limitation is that the panel composition varied from experienced e-platform users to non-users, whereas the results from the systematic review were derived from samples of study participants who were all experienced platform users.

Another limitation is the low response rate in the second round of the Delphi, with only 17 respondents compared to 59 respondents in the first round, and a higher percentage (76%) of respondents employed at universities in the second round compared to the first round (44%).

In this study, the cvr was used to analyze ratings in the first round, and frequencies were utilized in the second round, applying a consensus threshold of 75%. One strength of this combined analysis technique is that it enables panel members to provide feedback and adjust scores during the study. Limitations of the technique were that no explicit criteria for removing items from the list of requirements were determined and the number of Delphi rounds were established beforehand. Although these limitations appear in most Delphi studies [23], it is recommended to prevent this when conducting a Delphi study.

Beyond the scope of this study is the organizational structure of monitoring the e-platform. In general, to guarantee the long-term success of an e-platform, it is important to delineate responsibilities related to monitoring the content, functionality, usability, and technology.

8.2. Implications for Practice and Research

To strengthen and substantiate the nursing profession in Kazakhstan, it was recommended to develop an e-platform following the requirements established in the study. The e-platform, launched in 2020, is an innovative resource on a national level in Kazakhstan. It provides a permanent platform for nursing stakeholders from clinical practice, education, and research to strengthen cooperation and collaboration to enhance nursing and to enforce the quality of nursing care [32]. The current e-platform will be further enriched and will contribute to substantive innovations in the near future. For example, recently developed national clinical nursing guidelines are now disseminated through the e-platform. The platform will also be used to support Masters and PhD programs in nursing [33].

Regarding follow-up research, we suggest measuring the actual medium- and long-term use and effects of the e-platform. It is recommended to investigate different effects, such as the influence of the e-platform on the use of evidence-based interventions in clinical practice. Effects on quality of care and patient satisfaction must ultimately also be included in the research.

Our research offers new insights into the requirements for an e-platform to successfully strengthen nursing on a national level in the trinity of nursing education, nursing research, and clinical practice. Based on this new understanding, the current platform can be applied even more widely and can grow further in terms of content. In addition, the nine categories and four groups of essential and useful requirements may provide a starting point for developing other e-platforms of national scope in other countries or for other health care professionals. It is recommended to consult a panel of experts who represent future users’ perceptions and are well embedded in the specific context. Since the application of ICT is increasing rapidly, it is crucial to continue studying how these developments affect knowledge sharing and collaboration in the nursing profession.


For the development of an e-platform to improve the quality of nursing in Kazakhstan as part of the healthcare transition in Kazakhstan, 52 essential and 15 useful requirements have been identified and prioritized, with a high level of consensus. The e-platform is expected to contribute to the improvement of nursing education, nursing research, and nursing practice in Kazakhstan. Most content-related requirements are essential, although some advanced research skills are useful but not essential. In addition, most functional, usability, and technical requirements are essential. The results of a systematic review translated into requirements for an e-platform for nursing in Kazakhstan generated the pertinent input for achieving consensus on identifying and prioritizing requirements. The list of requirements found in this study is context-bound but may also be applicable as a reference framework when prioritizing requirements for the development of other e-platforms in nursing education, nursing research, and nursing practice.


Cvr = Content validity ratio
EBP = Evidence-Based Practice
HEI = Higher Educational Institute
ICT = Information and Communication Technology
LMS = Learning Management Systems
NIC = Nursing Intervention Classification
PICO = Patient, Intervention, Comparison, Outcome
ProInCa = Promoting the Innovation Capacity of Higher Education in Nursing During Health Services Transition


BLD and WP: Initiation and design of the research. ATO conducted the systematic review with input from AA and WP. BLD, WP and AA developed the Delphi questionnaires, and AA collected the data. BLD and ATO performed the data analysis and wrote the paper. WP performed an editorial revision of the paper. All authors read and approved the final manuscript.


Ethical approval was not necessary for this study. Participant anonymity was maintained throughout both rounds of the Delphi study.


Informed consent was obtained from the participants.


Not applicable.


This project was funded with support from the European Commission. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained herein. The research was conducted as a part of the ProInCa project. Grant number: No2017-3107/001-001


The authors declare that they have no competing interests.


The authors thank Gulim Aimagambetova for her assistance with the online questionnaires and English-Russian translations. We thank Jari Hautamaki for contributing to the literature search. Thanks also to the ProInCa project members and the respondents of the Delphi questionnaire for their contributions.


Katsaga A, Kulzhanov M, Karanikolos M, Rechel B. Kazakhstan. Health Syst Rev 2012.
Heikkila J, Hopia H, Hasselberg J, Tiittanen H, Biaghorzina Z. A cross sectional study of nurses’ and nurse educators’ perceptions of evidence-based practice in Kazakhstan. Annals of Nursing Research and Practice 2017; 2(1): 1016.
Nurakhova A, Heikkilä J, Ospanova D. Importance of developing the image of the nursing profession in Kazakhstan. Наука и здравоохранение 2019; 1(1)
Promoting the innovation capacity of higher education in nursing during health services transition (ProInCa). Project plan 2017. available from:
Frisch NC, Borycki EM, Mickelson G, Atherton P, Novak-Lauscher H, Hooker D, et al. Use of social media and web 2.0 technologies to increase knowledge and skills of british columbia nurses. NI 2012 2012; 2012: 117.
Stevens KR. The impact of evidence-based practice in nursing and the next big ideas. Online J Issues Nurs 2013; 18(2): 4. [2].
Neimann T, Wang VX. Harnessing the tiger of emerging E-learning platforms Handbook of research on program development and assessment methodologies in K-20 education 2018; 147-70.
Interactive and collaborative e-learning platform with integrated social software and learning management system 2013.
Ulrich CM, Wallen GR, Cui N, Chittams J, Sweet M, Plemmons D. Establishing good collaborative research practices in the responsible conduct of research in nursing science. Nurs Outlook 2015; 63(2): 171-80.
Weng YH, Kuo KN, Yang CY, et al. Increasing utilization of Internet-based resources following efforts to promote evidence-based medicine: A national study in Taiwan. BMC Med Inform Decis Mak 2013; 13(1): 4.
Weng YH, Kuo KN, Yang CY, Lo HL, Shih YH, Chiu YW. Information-searching behaviors of main and allied health professionals: A nationwide survey in Taiwan. J Eval Clin Pract 2013; 19(5): 902-8.
Ahmad MM, Musallam R, Allah AH. Nurses and internet health-related information: Review on access and utility. Clujul Med 2018; 91(3): 266-73.
Bassell K. Social media and the implications for nursing faculty mentoring: A review of the literature. Teach Learn Nurs 2010; 5(4): 143-8.
Bristol TJ. Twitter: Consider the possibilities for continuing nursing education. J Contin Educ Nurs 2010; 41(5): 199-200.
Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013; 48(6 Pt 2): 2134-56.
Howick J, Chalmers I, Glasziou P, Greenhalgh T, Heneghan C, Liberati A, et al. The Oxford 2011 table of evidence 2011.
Joanna Briggs Institute. Critical appraisal tools 2018. Available from:
Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care 2004; 16(2): 107-23.
Butcher HK, Bulechek GM, Dochterman JMM, Wagner CM. Nursing Interventions Classification ([NIC)]-E-Book. Elsevier Health Sciences 2018.
Survey Monkey. 2018. Available at:
Lawshe CH. A quantitative approach to content validity. Person Psychol 1975; 28(4): 563-75.
Linstone HA, Turoff M. The delphi method 1975.
Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67(4): 401-9.
Seixas CA, de Godoy S, Martins JCA, Mazzo A, Baptista RCN, Mendes IAC. Usability assessment of moodle by Brazilian and Portuguese nursing students. CIN: Computers, Informatics. Comput Inform Nurs 2016; 34(6): 266-71.
McIntyre M, McDonald C, Racine L. A critical analysis of online nursing education: balancing optimistic and cautionary perspectives. Can J Nurs Res 2013; 45(1): 36-53.
Gagnon J, Gagnon MP, Buteau RA, et al. Adaptation and evaluation of online self-learning modules to teach critical appraisal and evidence-based practice in nursing: An international collaboration. Comput Inform Nurs 2015; 33(7): 285-94.
Du S, Liu Z, Liu S, et al. Web-based distance learning for nurse education: a systematic review. Int Nurs Rev 2013; 60(2): 167-77.
League K, Christenbery T, Sandlin V, Arnow D, Moss K, Wells N. Increasing nurses’ access to evidence through a web-based resource. J Nurs Adm 2012; 42(11): 531-5.
Karaman S, Kucuk S, Aydemir M. Evaluation of an online continuing education program from the perspective of new graduate nurses. Nurse Educ Today 2014; 34(5): 836-41.
Kowitlawakul Y, Chan MF, Tan SSL, Soong ASK, Chan SWC. Development of an e-Learning research module using multimedia instruction approach. CIN: Computers, Informatics. Comput Inform Nurs 2017; 35(3): 158-68.
Davidson SJ, Candy L. Teaching EBP using game-based learning: Improving the student experience. Worldviews Evid Based Nurs 2016; 13(4): 285-93.
Sandström B, Borglin G, Nilsson R, Willman A. Promoting the implementation of evidence-based practice: A literature review focusing on the role of nursing leadership. Worldviews Evid Based Nurs 2011; 8(4): 212-23.
Center of Nursing Excellence launched in Kazakhstan – the ProInCa project created new ways of knowledge-sharing Available at: