Abstract

Objective

To develop an assessment tool for evaluating the ability of otorhinolaryngology (ORL) residents to perform rhinomanometry and acoustic rhinometry (R-AR), and therewith to assess their learning processes in acquiring R-AR skills.

Methods

Firstly, a “procedural skill rating scale (PSRS)” was designed for the assessment process. The R-AR performances of 10 residents, each performing 20 procedures, were observed and scored using the PSRS. The correlation between the increase in the number of procedures performed, the improvement in scores, and the reduction in procedure duration was studied. Interrater reliability was evaluated independently by two raters using 20 R-AR procedures. Cronbach’s alpha was calculated for reliability.

Results

The correlation between the number of procedures performed and the increase in scores was r=0.911 (p<0.001) for rhinomanometry-PSRS and r=0.832 (p<0.001) for acoustic rhinometry-PSRS. Mean procedure duration was 2.11±0.39 minutes (min: 1.17, max: 3.95) for rhinomanometry and 1.55±0.34 minutes (min: 1.07, max: 3.19) for acoustic rhinometry. The correlation between the increase in the number of procedures performed and the reduction in duration was r=-0.937 (p<0.001). Interrater reliability was r=0.788 for rhinomanometry-PSRS and r=0.795 for acoustic rhinometry-PSRS (p<0.001). Cronbach’s alpha was 0.971 for rhinomanometry-PSRS and 0.969 for acoustic rhinometry-PSRS.

Conclusion

A valid and reliable assessment tool has been developed to evaluate ORL residents’ skills in performing R-AR. Both tools are recommended for assessing ORL residents’ proficiency and determining their achievement of competence.

Introduction

During residency training, otorhinolaryngology (ORL) residents learn to medically or surgically treat patients presenting with nasal obstruction and similar nasal complaints. Rhinomanometry and acoustic rhinometry (R-AR) are tools that allow for the objective assessment of nasal obstruction symptoms and identifying anatomical pathologies. These instruments are tools that every ORL resident should be able to use after completing residency training (1).

Rhinomanometry provides an objective measurement of nasal airway resistance. A pressure-flow curve is obtained by simultaneously recording intranasal air pressure and airflow. Consequently, inspiratory and expiratory resistance values, as well as mean resistance, are determined (2). In acoustic rhinometry, acoustic signals are transmitted into the nasal cavity via a tube, and the reflections from the intranasal structures are amplified and digitized. The cross-sectional area of each point in the nasal cavity can be calculated with the area-distance graphs obtained (3). Both tools enable an objective evaluation of the human nose in two distinct ways. In competency-based residency training, the acquisition of professional skills by ORL residents is expected to be documented through assessment and evaluation processes (4-6). In literature, assessment tools are frequently reported for evaluating professional, particularly surgical, skills (7-10). However, no assessment tool has been identified for assessing ORL residents’ ability to learn and perform rhinomanometry and/or acoustic rhinometry.

The purpose of this study was to develop an assessment tool [procedural skill rating scale (PSRS)] to evaluate ORL residents’ ability to perform R-AR and, through this tool, to assess their learning processes in acquiring these skills.

Methods

Firstly, for the planned assessment process, a “PSRS” was developed (Tables 1 and 2). The workplace-based assessment tools reported in the literature and user manuals for R-AR were reviewed, and the Turkish scale items and the PSRSs were developed accordingly (2-4,10-12). Before data collection, written consent was obtained from all residents who volunteered and were observed in the study. A total of 10 residents participated in the study, with the following seniority distribution: 2 fourth-year, 1 third-year, 3 second-year, and 4 first-year residents. All residents received theoretical training on the indications for performing R-AR, how to use the instruments, and how to interpret results. Clinical diagnosis, identification of pathology, and treatment decisions were not included in the evaluation criteria. Additionally, demonstration-based skills training was conducted on how to perform R-AR and evaluate the outcomes. Evaluations were conducted by two ORL specialists (one professor and one associate professor) with experience in R-AR.

For a tool to be valid, it must be appropriate for its intended purpose. Accordingly, to demonstrate the progression of residents’ R-AR application skills, those learning the procedure for the first time were observed from their initial attempts, and their performances were evaluated using the PSRS. As residents performed more procedures, increases in their PSRS scores were expected, thereby demonstrating the validity of the PSRS. The correlation between the number of R-AR procedures performed and the corresponding increase in scores was analyzed statistically. For reliability, Cronbach’s alpha was calculated to assess internal consistency.

Competence was defined as achieving a performance level considered “independently acceptable” for each step. The number of procedures in which a resident scored three or higher on all steps, without subsequently dropping below 3, was defined as the “number of cases required to achieve competence.”

During data collection, residents’ R-AR performances were assessed and scored during “routine patient services.” The measurement of procedure duration began with the initiation of the R-AR application and ended with the interpretation of test results. Ethical approval for this study was obtained from the Pamukkale University Ethics Committee (approval no: E-60116787-020-561697, date: 06.08.2024).

Statistical Analysis

For statistical analysis, SPSS version 25 (IBM Corp., Armonk, NY, USA) was used. Continuous variables were presented as mean ± standard deviation, along with minimum and maximum values. Categorical variables were presented as frequency and percentage. Cronbach’s alpha coefficients were calculated to demonstrate internal consistency. Pearson correlation analysis was used for correlation testing. A p-value <0.05 was considered statistically significant.

Results

Two hundred and twenty adult patients who presented to our outpatient clinic with complaints of nasal obstruction were included in the study. As in routine practice, the procedure was explained in detail to the patients before R-AR measurement. Of the patients, 112 (51%) were male and 108 (49%) were female, and their mean age was 32.64±11.80 years (min: 18, max: 65).

Each of the ten residents performed 20 R-AR procedures. To facilitate graphical evaluation, learning curves were generated for each resident (Figure 1). Examination of the learning curves revealed that, although the pace varied among residents, they generally began to acquire the skill in the initial cases, subsequently demonstrated rapid improvement, and reached maximum scores in the later cases. In the graphs, the plateau line at full score represents the maximum score obtainable from the PSRS. The correlation between the increase in the number of procedures performed and the increase in PSRS scores was statistically significant, with r=0.911 (p<0.001) for rhinomanometry-PSRS and r=0.832 (p<0.001) for acoustic rhinometry-PSRS.

The number of cases required for residents to achieve competence was median: 8 (min: 4, max: 9) for rhinomanometry and median: 8.5 (min: 4, max: 10) for acoustic rhinometry. A statistically significant negative correlation was observed between residents’ year of seniority and the number of cases required to achieve competence for both rhinomanometry (r=-0.773, p=0.009) and acoustic rhinometry (r=-0.830, p=0.003).

Cronbach’s alpha was found to be 0.971 for rhinomanometry-PSRS and 0.969 for acoustic rhinometry-PSRS. With both values exceeding 0.95, the reliability of both PSRSs was found to be very high.

The mean procedure duration was 2.11±0.39 minutes (min: 1.17, max: 3.95) for rhinomanometry and 1.55±0.34 minutes (min: 1.07, max: 3.19) for acoustic rhinometry. As procedures were repeated, a reduction in procedure duration was also observed graphically (Figure 2). The correlation between the increase in the number of procedures and the reduction in duration was statistically significant (r=-0.937, p<0.001).

To evaluate interrater reliability, 20 R-AR procedures performed by different residents were simultaneously assessed by two raters. Interrater reliability coefficients were r=0.788 (p<0.001) for the rhinomanometry PSRS and r=0.795 (p<0.001) for the acoustic rhinometry PSRS. These results demonstrate a strong, statistically significant positive correlation between raters for the two PSRS instruments.

When all evaluation results were considered, comparisons of initial and final scores and durations revealed reduced procedure times and a statistically significant increase in total scores. These findings provide evidence supporting the validity of the developed PSRS.

Discussion

Nasal obstruction is among the most common reasons for presenting to ORL outpatient clinics. As future specialists, ORL residents will encounter patients with nasal obstruction, diagnose them, and provide treatment. In this process, they will use R-AR as objective measurement tools. The results of these tests can be employed in both the diagnosis and the post-treatment follow-up of patients presenting with nasal obstruction. Therefore, ORL residents must learn to perform and interpret R-AR during their training. In this study, R-AR PSRS was developed to guide residents in learning R-AR and to evaluate their skills.

Competency-based residency education is becoming increasingly widespread worldwide, as well as in our country (4-6). The Turkish Medical Specialty Board’s efforts are directed toward improving this education. These efforts began with the development of a core curriculum that is continuously updated. In competency-based programs, it is necessary to evaluate whether residents have achieved the expected competencies using valid assessment tools. Accordingly, various assessment tools have been developed to demonstrate residents’ achievement of expected competencies. A review of the literature reveals that most workplace-based assessment tools primarily focus on surgical skills, particularly operative procedures (6-10). However, assessment tools have also been developed for outpatient procedures such as R-AR, which are included in the core curricula (11-14).

This study also aimed to determine the time it takes for an ORL resident to learn to perform and interpret R-AR. On average, residents reached the level of competence by the eighth case. However, the learning process is individual for each resident. Both the learning curves and the numerical results demonstrated that residents achieved competence with different numbers of cases. This result supports the fundamental principle of competency-based residency education. There is no standard number of cases required to achieve competence, and each resident must continue practicing until they attain competence. Furthermore, a significant correlation was observed between residents’ years of experience and the number of cases required to reach competence, suggesting that senior residents’ greater familiarity with nasal anatomy and endoscopic or instrumental examinations positively influences their learning curves.

Study Limitations

There are some limitations in our study. Since skills training and assessments were conducted in real-life patient care settings, numerous variables, such as the clinical environment, examination times, and communication difficulties with patients, may have influenced the results. Additionally, blinding the evaluators to the residents’ level of experience was not feasible due to the nature of direct, competency-based procedural assessment. This lack of blinding may represent a potential source of observer bias and should be considered when interpreting the results.

Conclusion

In this study, valid and reliable assessment tools (PSRS) were developed for evaluating ORL residents’ skills in performing R-AR. Both PSRSs can be used to assess ORL residents’ skills in R-AR and to determine their achievement of competence. Based on the data obtained, it can be said that residents generally reach the expected level of competence in performing and interpreting R-AR after approximately eight applications. However, it is essential to acknowledge that learning processes vary individually. The achievement of competence should be assessed separately for each resident.