MARTÍN MAURICIO BRETÓN DE LA LOZA
Departamento de Transferencia de Conocimiento, Facultad de Nutrición, Universidad Autónoma del Estado de Morelos, México
martin.breton@uaem.mx
orcid: 0000-0001-9921-4014
OFMARA YADIRA ZÚÑIGA HERNÁNDEZ
Profesor Investigador de tiempo completo, Instituto de Ciencias de la Educación, Universidad Autónoma del Estado de Morelos, México
ofmara.zuniga@uaem.mx
orcid: 0000-0002-4773-0102
MARÍA ALEJANDRA TERRAZAS MERAZ
Profesor Investigador de tiempo Completo, Facultad de Nutrición, Universidad Autónoma del Estado de Morelos, México
maria.alejandra@uaem.m
orcid: 0000-0002-6821-5732
SARA DIANA GARDUÑO DÍAZ
Departamento de Investigación y Desarrollo, NM-Invincible Brands, Alemania
sdgarduno@gmail.com
orcid: 0000-0002-5764-7409
JAQUELINE GUADALUPE GUERRERO CEH
Profesor investigador de tiempo completo, Universidad Autónoma de Campeche, México
jgguerre@uacam.mx
orcid: 0000-0002-2913-1309
SONALÍ CARRANCO GÓMEZ
Profesor investigador de tiempo completo, Instituto de Ciencias de la Educación,
Universidad Autónoma del Estado de Morelos, México
sony@uaem.mx
orcid: 0000-0003-3803-5420
ABSTRACT:
Objetive: to evaluate the effectiveness of an educational intervention, with inverted classroom strategies, at the level of clinical competencies of sixthsemester students of the Bachelor of Science in Nutrition program.
Method: Community trial, pre and post intervention, with control group. Setting in the Faculty of Nutrition of the Autonomous University of the State of Morelos. Participants: 137 students included in two groups by self-selection, group A (control, n=64) and group B (intervention, n=73). Intervention: the flipped classroom strategy was implemented organized with digital platforms and the use of simulators. Main Outcome Measure: Evaluation of competencies in medical history, physiopathology and procedures in nutrition. Descriptive statistics, contingency tables, Wilcoxon test and robust linear regression performed with Stata Program release 14.
Results: Significant and positive differences were found in the increase of knowledge about physiopathology in the control group, while in the intervention group, this condition occurred in all the competencies evaluated and in the total sum. The highest socioeconomic level group reflected an increase in knowledge with a significant association force ( p<.05 value), in addition to the previous knowledge and having participated in the intervention.
Conclusions: It is suggested that the use of simulators that allow for the improvement of the learning of the students for clinical nutrition be included as part of their curriculum.
PALABRAS CLAVE: Nutritional Sciences/education, Students, Nutrition, clinical nutrition, Community trial.
RESUMEN:
Modelo metodológico para el desarrollo de competencias clínicas en estudiantes de nutrición
Objetivo: Evaluar la efectividad de una intervención educativa, con estrategias de aula invertida, a nivel de competencias clínicas de estudiantes de sexto semestre de la Licenciatura en Nutrición.
Método: Ensayo comunitario, pre y post intervención, con grupo control. Realizado en la Facultad de Nutrición de la Universidad Autónoma del Estado de Morelos. Participantes: 137 alumnos incluidos en dos grupos por autoselección: grupo A o control (n=64) y grupo B o intervención (n=73). Intervención: se implementó la estrategia de aula invertida organizada con plataformas digitales y el uso de simuladores. Medida de resultado principal: Evaluación de competencias en historia médica, fisiopatología y procedimientos en nutrición. Se muestran estadísticas descriptivas, tablas de contingencia, prueba de Wilcoxon y regresión lineal robusta realizadas con programa Stata®v14.
Resultados: se encontraron diferencias significativas y positivas en el aumento del conocimiento sobre fisiopatología en el grupo control, mientras que en el grupo intervención esta condición se presentó en todas las competencias evaluadas y en la suma total. El grupo de nivel socioeconómico más alto reflejó un aumento de conocimientos con una fuerza de asociación significativa (valor p<0.05), además de los conocimientos previos y haber participado en la intervención.
Conclusiones: Se sugiere incluir como parte del plan de estudios el uso de simuladores que permitan mejorar las competencias y el aprendizaje de los estudiantes en aspectos importantes para la nutrición clínica y hospitalaria.
KEYWORDS: Ciencia de la nutrición/educación, estudiantes, nutrición, nutrición clínica, ensayo comunitario.
1. Background
This study is guided by the interest to design and implement a methodological model of teaching-learning, which allows for the development of the basic competences in clinical nutrition. The model starts from a situational diagnosis that facilitates the knowledge of the level of basic clinical competencies that students have at the moment of pursuing a nutrition degree in the different semesters, as proposed by means of the teaching-learning model based on the significant, situated, collaborative learning and the inverted classroom (Flipped classroom). Based on this analysis, suggestions are proposed to improve the curriculum of the Bachelor's Degree in Nutrition.
According to Villaroel and Bruna (2014), University education has focused on providing knowledge through lectures, aiming to teach large amounts of theoretical information, delegating the responsibility of acquiring that knowledge to the student. Gómez Mendoza (2010) mentions that this way of conceptualizing the teaching andlearning process has implied that often university students have not been able to integrate or articulate the theoretical knowledge obtained in the classroom with the technical procedures of each professional discipline, which has generated superficial learning.
Epstein & Hundert (2002) define competencies in clinical practice as "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions and values, which are reflected in daily practice for the benefit of the individuals and communities they serve" and retomed by Gómez Mendoza (2010, p. 462) and Garcia (2010). For this reason, it is important to mention that the degree of competence of graduates in nutrition, medicine and nursing is heterogeneous in the different areas of care for the sick population and, for this reason, health personnel should have clear competencies in both the clinical and educational areas.
Epstein & Hundert (2002) define competencies in clinical practice as "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions and values, which are reflected in daily practice for the benefit of the individuals and communities they serve" and retomed by Gómez Mendoza (2010, p. 462) and Garcia (2010). For this reason, it is important to mention that the degree of competence of graduates in nutrition, medicine and nursing is heterogeneous in the different areas of care for the sick population and, for this reason, health personnel should have clear competencies in both the clinical and educational areas.
Epstein & Hundert (2002) define competencies in clinical practice as "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions and values, which are reflected in daily practice for the benefit of the individuals and communities they serve" and retomed by Gómez Mendoza (2010, p. 462) and Garcia (2010). For this reason, it is important to mention that the degree of competence of graduates in nutrition, medicine and nursing is heterogeneous in the different areas of care for the sick population and, for this reason, health personnel should have clear competencies in both the clinical and educational areas.
Similarly, Olivares et al. 2018; Canicoba, 2013) establish that the competencies of a professional (nutritionist and/or Bachelor in Nutrition with clinical orientation) allow the acquisition of the necessary knowledge, both theoretical and practical, that promotes the scientific development of the professional and guarantees the correct management of clinical nutrition and metabolic nutritional support; establishing criteria with scientific character for the application of clinical nutrition. This will encourage students to perform successfully in the professional field in what is known today as nutritional therapy.
In the present study, a methodological model of teaching and learning was designed, developed, implemented and evaluated, in which different pedagogical strategies are integrated. Those strategies include significant learning, situated learning, collaborative learning, problem-based learning (ABP), and the flipped classroom, which facilitate the deepening of theoretical knowledge and the development of technical skills. All of this is supported by a virtual platform of free access and the Cornell Method, which allows the student to hierarchize and organize large amounts of information. Finally, in order to ensure the acquisition of the technical skills typical of clinical nutrition, three clinical simulators were developed and implemented, which are described in the methods section.
This study seeks to evaluate the effectiveness of an educational intervention, with inverted classroom strategies, at the level of clinical competencies, of students in the sixth semester (third year) of the Bachelor's Degree in Nutrition at the Autonomous University of the State of Morelos, during the period from January to June 2019.
2. Methods
A community trial was conducted before and after with a control group (Gerrish & Lacey,2006; Valvé, Artés &cobo,2005), in which 137 students enrolled in the sixth semester of the Bachelor of Nutrition were included in two groups by self-selection, group A (control n=64) and group B (experiment n=73). The sample was non-probabilistic, since all the students who studied the sixth semester of the Bachelor's degree were invited to participate as they corresponded to the level at which the learning unit, which is optional, is taught. The intervention was by self-selection, each participating student selected the group in which he or she wanted to participate: A) Control, with or without participation in the traditional class of Enteral and Parenteral Nutrition Option 3, or B) Intervention by choice of participation in the group. There was no masking; however, the students did not know that alternative educational techniques would be used in the intervention group, they only knew the name of the teachers beforehand as a reference.
Group A (control) was made up of 16 students who did not choose the optional learning unit 3, which is described below, as well as 48 students who did choose the subject, but took it with the traditional theoretical method. Group B (intervention) was made up of 73 students who chose to participate in the group with the experimental method.
Within the 2010 curriculum of the Bachelor of Science in Nutrition in the sixth semester, students must take an elective course, within which the elective 3 Enteral and Parenteral Nutrition is, divided into two groups with different schedules. Students choose an elective and the schedule that best suits them. They are also given the names of the teachers who teach the subjects, the program has flexibility in this aspect, so students are free to choose the group they are interested in.
At the beginning of the semester, the objective of the study was explained to all of the students; some students had the option of changing groups in case they wanted to participate in the intervention group or not. The same applied to those students who decided not to take the optional course.
The students of both groups who chose the elective subject 3 Enteral and Parenteral Nutrition, took the same program of contents, which was programmed in advance, under the concept of free chair, by the teachers in charge of each group.
Both teachers met the profile and professional experience in the area of study. Group A took lectures, 3 hours on Tuesday and 2 hours on Thursday. Group B, took the same 5 hours on Saturdays and during the week the students went to the simulators of assisted practice, under the guidance of the professors and with the technique of inverted classroom assisted with technological platforms.
The strategy of an inverted classroom (Aguilar Trejo, 2016; Aguilar Trejo, 2015) or flipped classroom was implemented for the intervention, using information and communication techniques (ICT) through a virtual platform of free access (Google Classroom) within which folders were enabled. The folders contained the syllabus, the program of contents of the learning unit, audiovisual support materials, books and reference articles, as well as information search engines, formats for the use of references, the methods of hierarchization and organization of information and Mexican Official Standards related to Clinical Nutrition.
For the development of skills (techniques) three simulators (Amaya Afanador, 2008; Piña Jiménez, 2015) of assisted practice were built: 1) handling and application of clinical history, 2) central of artificial nutrition mixtures and 3) knowledge and technique of feeding tubes placement and identification of safe venous access and peripheral veins canalization. The unit of analysis was individual. For the measurement of the intervention, three partial evaluations were carried out by means of self-evaluation questionnaires on the platform, participation in the presentation of topics by team in face-to-face classes and practice in the simulators, each team of two students had to pass at least ten times each simulator and an individual record of the activities was kept All those who started in the intervention group finished the course with a good level of achievement (higher than 70 %).
The clinical competencies questionnaire was applied to students in a digital format, with prior informed consent. The students who responded at the beginning were 192 from groups A and B, while the post-intervention questionnaire was answered by 137, hence the resulting non-response rate was 28.6 %. With the usage of the Stata v.14 program, descriptive and inferential statistics were performed. The items corresponding to each competence (HC: medical history, FP: physiopathology and PR: procedures) were added up and the results were compared on average before and after. Bias-curtosis test was used to evaluate the normality of the summations; since the null hypothesis was rejected, non-parametric tests were used (Siegel, 2015). Reliability scales with Cronbach's Alpha test were good as shown in Table 1.
Table 1. Cronbach’s Alpha before and after the intervention
The study variables were: the sum of knowledge scores in clinical history, physiopathology and procedures (dependent), having participated in the intervention group at the beginning and end of the semester (independent). Covariates were evaluated such as: sex, age, socioeconomic level, average grades. It was carried out for the paired comparison of the means of the summations in ranges with the Wilcoxon's sign test; 4 models of robust linear regression were constructed to identify the factors associated to the score of competences, one for each competence and one for all the summations together.
3. Results
Out of the 137 students who answered the questionnaire before and after the course, 24.8 % (34) were men, with an average age between both genders of 21.6 years (SD=2.26), with no significant difference observed (value p=0.129). The sociodemographic data of the variables are shown in Table 2, shows that there were no significant differences by sex.
Table 2. Population distribution
For the summation of the scales, exploratory and confirmatory factor analyses were performed to define the items that contributed valuable information to the summations (Bretón de la loza, 2019), so that the differences found, in this study, between the before and after summations are shown in Table 3a and 3b, in which the Wilcoxon range and sign test was performed for paired data. The competencies where significant and positive differences were found (demonstrating that knowledge increased), were only with regards to physiopathology in the control group, while in the intervention group, this condition was presented in all the competencies evaluated and in the total sum (Figure 1).
Table 3a. Difference in means, group A (control)
Table 3b. Differences in mean, group B (intervention)
Figure 1
There was a lack of significant statistical difference between the intervention and control groups (Table 3a), both overall and per each competency evaluated. This may indicate a similar level of theoretical knowledge and abilities prior to the intervention. A statistically significant difference exists between the control and intervention group, both overall and per each individual competency evaluated. This may indicate not only a higher level on knowledge among the intervention group, but also an increase of such knowledge.
Table 4 shows the comparison of the association of the factors evaluated with robust regression models. Post-intervention summation was established as a response variable, and the regressors included were the summation of competencies prior to the course, being part of group A or B, sex, age, and belonging to a specific socioeconomic group. Model 1 evaluates the total sum of competencies; model 2, the sum of competencies on clinical history; model 3, the sum of competencies on physiopathology, and model 4, the final sum of competencies on procedures. An increase in the knowledge associated with the intervention was found in all models by adjusting with the aforementioned regressors. The group with the highest socioeconomic level also reflected an increase in knowledge with a significant strength of association (p<.05 value).
Table 4.Factors associated with knowledge following intervention
Note: Sum pre and sum post: sum total before and after the intervention. MR: Medical Record competencies, PP:
pathophysiology competencies and TP: technical procedures competencies
4. Discussion
Among students of the Nutrition Bachelor, a preferential female enrollment is observed. In terms of gender, 2 % of all female students in the country study nutrition, which in turn represents 76 % of the enrollment (about 45,000 women) of this specific career choice and, in contrast, only 14,430 men. Other studies carried out on the evaluation of competencies in students of the Nutrition Bachelor show this behavior in the distribution by gender (Smart, 2014; Buchholz, 2020).
As it can be seen in Table 3a, the control group (A) and the experimental group (B) did not show statistical significance when the means, both general and by competence before the intervention, were compared (difference between the general means (p=0.115), Clinical History p=0.463, Physiopathology (p=0.035) and for the competence of Technical Procedures (p=0.342). This indicates that both groups had the same level of theoretical knowledge and technical skills prior to the intervention. However, when the means of both groups were compared after the intervention, the experimental group (B) not only showed to have more knowledge, but also increased it. The group exposed to the intervention showed a statistically significant difference (p=0.000), for the Clinical History competence (p=0.005), in Physiopathology (p=0.000) and for the Technical Procedures variable (p=0.000). For more detail of the differences in the increase in knowledge see Table 3b.
The results obtained in this study coincide with those published by other authors where they explain that the articulation of teaching and learning strategies such as the inverted classroom, ICT supported in virtual environments improves student learning. It is important to point out that the implementation of management tools and information hierarchization such as the Cornell Method, help students to identify and extract relevant information on a particular topic, allowing them to generate questions and contributions of their own to the topics to be reviewed in class (Aguilar. ,2016, p .11).
This stimulates and encourages the student to resort to constant reading, to identify and select relevant information through the Internet supported by the appropriate use of information search engines. It was observed in this study that students improved in the use of the virtual environment. The inverted classroom strategy was supported in the Google Classroom platform, as it has free access, and is very user friendly. This virtual environment allowed students to strengthen their knowledge and skills development to study and generate knowledge supported by ICT (Novillo, 2018). The impact of technology in the classroom promotes a solid context, and the Information and Communication Technologies, help students acquire the skills necessary to become competent, searchers, analyzers and problem solvers (Piña Jiménez, 2015, p. 20; Novillo Lorenzo, 2018). Education requires a change in teaching methodology in general and in the health sciences in particular.
The inverted classroom methodology incorporates these types of necessary changes demanded by the society of the 21st century, where the student is self-motivated and becomes an actor in his or her own learning process. A personalized teaching system with immediate constructive feedback promotes the development of competencies (Novillo Lorenzo, 2018, p. 23). It should be noted that this methodology takes into account the different rhythms and types of learning and frees up space in the classroom for other types of activities, such as the use of simulators. There is currently a wide variety of free and easy-to-use tools that allow teachers to implement the inverted or flipped classroom methodology, without requiring exceptional equipment and software (Novillo, 2018, p. 24).
In this sense, it is essential to recognise that if a student is not used to working with different teaching techniques and is more accustomed to the exposure of topics by the professor, then he or she may be resistant to change. As Piña Jiménez (2015) affirms in order to solve this type of incident, the professor in charge of the experimental group explained on the first day of class how to work during the semester with the inverted classroom technique and other strategies, as well as the use of simulators. The professor communicated that the students were very interested in learning from the perspective of these techniques. However, it should be mentioned that some students were slow to adapt to the pace of work and in general to this new teaching-learning system.
As a result, the professor paid more attention to this group of students and the problem was solved through out two or three personalized tutoring sessions. As for evaluation in this research, it converges in concluding that it is a good idea to use short evaluations at the end of each session or topic, which can be printed or online. This helped to decrease the students' stress and as a consequence the group improved their grades in addition to favoring knowledge retention18. It was also observed that the general average of grades in the subject was satisfactory since the students showed better performance, worked in a collaborative environment, felt free of stress and became more autonomous.
The use and participation of students in the simulators helped students follow protocols established by the Mexican Official Standards for clinical nutrition and make conscious and thoughtful decisions, which translates into patient safety. It is worth mentioning that the students do not reflect immediately, but it is important to point out that the students noticed their progress throughout the semester (Piña Jiménez, 2015; Agudelo, 2015) Situated learning is achieved when the student develops activities with socio-cultural interactions inside or outside the classroom. The use of simulators encourages contextualized learning that involves activities related to the real needs of solving a problem without exposing students to unnecessary risks. In addition, it is possible to practice countless times in the different simulators, which offers an advantage that allows the improvement of different techniques without hurting the patient (Avila Juárez, 2018).
Based on this methodology, the educational process of the students can be dosed and planned in a didactic way, considering their previous knowledge, and the curricular purposes of each subject (Piña Jiménez, p158), in the case of this study in the field of clinical nutrition. Therefore, the old classroom models where the professor is the protagonist exposing and applying an exam are no longer adequate for the demands of current students and the labour market.
It is necessary to promote the change from the traditional teacher-centered model to a student-centered one considering the profile and learning needs. It becomes evident that it is key to supportstudents to focus on collaborative learning by promoting teamwork, and to intensify the promotion of reflection and critical skills of students. Situated learning can be seen through the use of logs and rubrics with which the development of skills for the execution of procedures based on skills and abilities during the stays in the simulators are evaluated. As Buchholz reflects, traditional education may not correctly prepare students for professional practice as opposed to the inclusion of simulators that improve the approach to reality (Buchholz, 2020). One of the limitations of longitudinal studies is the loss of follow-up. In this study 28 % did not answer the post-intervention questionnaire.
In conclusion, teaching based on competences has become a necessity nowadays. Thus, the implementation of the inverted classroom represents a challenge not only for the student, but also for the teacher, since he or she has to invest time in the analysis of the academic program and modify it to adapt the methodology to a different model that fundamentally benefits the student.
As for the articulation of the use of simulators with the methodology of inverted classrooms supported by ICT, the proposal of this study for the development of basic competencies managed to awaken the interest of the students of the intervention group. The students successfully adapted to the simulators, and they actively participated and elaborated a photographic evidence portfolio. Students developed the necessary skills to correctly execute the technical procedures linked to the exercise of clinical nutrition, specifically in artificial nutrition. In this sense, it is concluded that as the world trend is globalization, health aspects should not be left behind but on the contrary, improvements in teaching should be promoted so that nutrition personnel in training can obtain the knowledge, techniques and skills that meet the needs to care for the population, benefitting them.
5. References
Agudelo, S. M. (2015). Efecto de un sistema de enseñanza personzalizado. Educación Médica Superior, 29(4).
Aguilar Trejo, M. (2016). Aprendizaje activo: Prácticas innovadoras de la Universidad de Celaya (1ra Edición ed., Vol. I). Celaya, Guanajuato, México: Educación superior de Celaya.
Aguilar Trejo, M. U. (2015). El aula del nuevo milenio: Hacia una educación de vanguardia (1ra Edición ed., Vol. I). Celaya, Guanajuato, México: Eduación superior de Celaya A.C.
Amaya Afanador, A. (2008). Simulación clínica: Pretende la educación médica basada en la simulación remplazar la fromaciín tradicional en medicion y en otras ciencias de la salud en cuanto a la experiencia actual con los pacientes. Universidad Médica de Bogotá, 49(3), 399-405.
Avila Juárez, S. G. (2018). Simulación con paciente estandarizado y simuladores de baja fidelidad (PESiBaF) como primer acercamiento a un paciente en estudiantes de primer año de la carrera de médico cirujano. Educación Médica, 434(6). doi:10.16/j.edumed.2018.10.011
Bretón de la loza, M. G. (Dociembre de 2019). Validación de cuestionarios. International Journal of new edu(4), 123-32.
Buchholz, A. V. (2020). Patients Simulation imrpove dietetics students and intems communications. 52(4), 377-384. doi:10.1016/j.jneb.2019.09.022
Canicoba, M. B. (Enero- Junio de 2013). Funciones y competencias del nutricionista clínico. Documento de consenso. Revista Cubana de Alimentación y Nutrición, 23(1), 146-172.
Cárdenas Becerril, L. M. (21 de Mayo de 2017). Desarrollo de pensamiento reflexivo y crítico en estudiantes de enfermería: evidencia de una universidad pública mexicana. Revista Uruguaya de Enfermería, 12(1), 1-19. Recuperado el 07 de 12 de 2020, de http://rue.fenf.edu.uy/index.php/rue/article/view/211
Díaz Barriga, A. (Enero-Marzo de 2006). El enfoque de competencias en la educación. ¿Una alternativa o un disfraz de cambio? Perfiles Educativos, 28(111), 7-36. Recuperado el 19 de octubre de 20, de //www.redalyc.org/articulo.oa?id=13211102
Epstein, Ronald & Hundert, Edward. (2002). Defining and Assessing Professional Competence. JAMA: The Journal of the American Medical Association. 287. 226. 10.1001/jama.287.2.226.
García Carcía, E. (2010). Competencias éticas de, profesor y calidad de la educación. Revista Electrónica Interuniversitaria de Formación del Profesorado., 13(4), 29-41. Obtenido de https://dialnet.unirioja.es/servlet/articulo?codigo=3675461
Gerrish, K. L. (2006). Investigación de enfermería (Vol. I). Madrid, España: McGrow-Hill/Interamericana.
Gómez Mendoza, M. A. (2010). La alegre entrada y el irresistible ascenso de las competencias en la universidad. Educación y Educadores, 13(3), 453-474. Recuperado el 22 de octubre de 2020, de http://www.redalyc.org/articulo.oa?id=83417001008
Novillo Lorenzo, M. (7 de Enero de 2018). Informe final de la propuesta de intervención. Uso de la metodología flipped classroom en la asignatura de biología y geología, 30. Sevilla, España. Obtenido de http://openaccess.uoc.edu/webapps/o2/bitstream/10609/74367/7/enovilloTFM0118memoria.pdf
Olivares Olivares, S. A. (2018). Valor percibido de una experiencia de inmersión educativa para el desarrollo de competencias transversales: semana i. Educación Médica, 1-7. Obtenido de https://app.dimensions.ai/details/publication/pub.1104496455?and_facet_journal=jour.1295897
Piña Jiménez, I. A. (2015). Innovación para la práctica: La enseñanza de la enfermería con simuladores,. Enfermería Universitaria, 12(3), 152-159. doi:10.1016/j.reu.2015.04.007
Rivera Michelena, N. P. (2017). Un sistema de habilidades para la carrera de medicina , su relación con las competencias profesionales. Una mirada actualizada. Educación Médica Superior, 31(1), 215-238. Recuperado el 22 de Octubre de 2020, de http://scielo.sld.cu
Smart, H. C. (2014). Nutrition students gain skills from motivational interviewing curriculum. Journal Academy of Nutrition and Dietetics, 1-5. doi:10.1016/j.jand.2014.04.012
Vallvé, C. A. (2005). Estudios de intervención aleatorizados (TREND). Medicina Clínica (Bacr)(1), 38-42. doi:10.1016/S0025-7753(05)72208
Villaroel, V. B. (2014). Reflexiones en torno a las competencias genéricas en educación superior: Un desafío pendiente. Psicperpectivas individuo y sociedad, 13(1), 23-34. Recuperado el 21 de Octubre de 2020, de https://www.psicoperspectivas.cl/index.php/psicoperspectivas/article/viewFile/335/31