The development and validation of a rehabilitation capability questionnaire for injured athletes

Participants

The participants in the study consisted of 240 injured athletes (exploratory factor analysis: 120, secondary analysis: 120) who participated in a physical rehabilitation program. The results of 120 injured athletes were utilized to conduct the descriptive statistics analysis, item analysis, and exploratory factor analysis. The results of another 120 injured athletes were utilized to conduct the confirmatory factor analysis to examine inter validity of the RCQ.

A total of 240 injured athletes (72% male, 28% female), from 13 to 25 years of age, participated in this study. These athletes were competing in international (5%), national (13%), provincial (40%), and regional (42%) events at the time of the study. The athletes participated in a variety of sports, such as soccer (46.5%), ice hockey (5.4%), baseball (13.6%), basketball (21%), volleyball (5.9%), and track and field events (7.6%). The sample consisted of 79 middle school students, 82 high school students, 54 university students, and 25 professional athletes.

The participation criteria were as follows: First, participants were required to enroll in a clinic-based rehabilitation program; Second, participants had to attend physical therapy sessions for at least four weeks; Third, participants must have had at least four weeks of rehabilitation remaining; Lastly, participants had to be competitive athletes.

Procedures

Item responses to the RCQ’s 95 items were measured via a five-point agree/disagree Likert scale. Expert consultations were used to test the inter validity of the RCQ test and were conducted using descriptive statistics analyses (mean, standard deviation, kurtosis, skewness, response rate). Analyses of the collected data were conducted via item analysis, including item-to-total correlations. Following the reliability analysis, an exploratory factor analysis (maximum likelihood estimation with oblique rotation) was applied to the data. A confirmatory factor analysis was applied to the measurement model to test the dimensionality of the RCQ.Convergent, discriminant validity, cross validity and predictive validity analyses were also performed to test external validity of the RCQ. All data was analyzed using Statistical Package for the Social Sciences (SPSS 18.0) and AMOS.

Participants

The sample group comprised 52 injured athletes, between 13 to 25 years of age, who were treated and diagnosed at the Sporting Medicine Unit of Sol-hospital. These athletes were competing in international (7.7%), national (11.5%), provincial (32.7%), and regional (48.1%) events at the time of the study. The sports the athletes participated in were soccer (35.5%), ice hockey (8.3%), baseball (12.7%), basketball (16.4%), volleyball (12.7%), and track and field events (15.4%). The sample consisted of 18 middle school students, 21 high school students, eight university students, and five professional athletes. All participants had sustained moderate to severe injuries, which had kept them from attending practices and/or competitions for at least four consecutive weeks. For this study, qualifying participants must have missed a minimum of four weeks out of practice/competition due to their injuries. Injury severity was determined by using the Colorado Injury Reporting System (Blackwell & McCullagh, 1990).

Measures

Convergent Validity Measure

The Pearson correlation between RCQ subscales and the measures of theoretically related constructs were used to assess the convergent and discriminant validity of the RCQ. The Competitive State Anxiety Inventory-2 (CSAI-2; Martens, Vealey, & Burton, 1990), Profile of Mood States (POMS; McNair et al., 1971), and Rehabilitation Adherence Questionnaire (Fisher et al., 1988) were used to examine evidence for the convergent validity of the RCQ.

The CSAI-2, a multidimensional measure of competitive state anxiety, includes three scales of nine items that measures cognitive state-anxiety, somatic state-anxiety, and state-confidence on a five-point, Likert scale (0=not at all; 4=very much so).

In order to assess the injured athletes' emotions, the POMS scale was used. The POMS included scales for tension (T), depression (D), anger (A), vigor (V), fatigue (F), and confusion (C). All questionnaires used in this study were translated using a double back-translation procedure. Bivariate Pearson correlations were used to assess the hypotheses, bearing on the convergent relationship among the RAQ, CSAI-2 and POMS subscales.

RAQ was a self-report inventory with subscales designed to assess (a) perceived exertion, (b) pain tolerance, (c) self-motivation, (d) support from significant others, (e) scheduling, and (f) environmental conditions. Fisher et al. (1988) found that high scores on each of the subscales were associated with better adherence to clinic-based sport injury rehabilitation protocols.

Discriminant Validity Measure

The discriminant validity of the RCQ was assessed by computing the Pearson correlations between the RCQ subscales and the measures of theoretically unrelated constructs. The Task and Ego Orientation in Sport Questionnaire (TEOSQ; Duda & Nicholls, 1992) was used to examine evidence for the discriminant validity of the RCQ. The TEOSQ used a five-point Likert scale and contained a seven-item task orientation subscale, which assessed the extent to which an individual defined success in terms of learning, and a six-item ego orientation subscale, which assessed the extent to which success was viewed in terms of outperforming others.

Predictive Validity Measure

The participation levels of injured athletes in rehabilitation programs (initial, intermediate, final) were used to examine evidence for the predictive validity of the RCQ. Participation levels were defined as the participant’s level of compliance with the medically prescribed rehabilitation plans. To assess this variable, the researchers used a personalized record sheet, designed specifically for each participant in accordance with the rehabilitation plans recommended by the attending doctor. Athletes' intensity of effort during rehabilitation exercises, the frequency of which they followed the practitioner’s instructions and advice, and their receptivity to changes in the physical therapy program were calculated by the researcher, athletic trainers, and a doctor in three phases (initial, intermediate, and final). A Bivariate Pearson correlation was used to assess the hypotheses’ bearing on the significance of the relationship between the RCQ and participation level in rehabilitation program.

Procedures

During their first visits to the clinic following their injuries, the athletes were invited to participate in this research project. All those who agreed were asked to complete the RCQ, CSAI-2, POMS, RAQ, and TEOSQ questionnaires. During their initial visits, the attending doctor estimated the optimum recovery period for each injury. This information was used as a means of assessing the seriousness of the patients’ injuries and as a criterion for participation in the study. Subsequent psychological assessments were arranged to coincide with the check-up dates agreed upon by the doctor and the participants. In this way, every time the athletes came to the clinic for a medical check-up, they would complete the aforementioned questionnaires. This continued until the injured person was declared medically fit.

From the first visit, the injured athletes' participation levels in the rehabilitation program were also assessed. Of all the data collected, only data coincided within the initial, intermediate, and final assessments was selected for analysis. The initial assessment was carried out when the injured person visited the clinic for the first time; the intermediate assessment was carried out based on the length of the recovery period; and the final assessment was carried out on the day the injured person was declared medically fit.

Data Analysis

The results were presented in three different sections. The first section assessed the convergent relationship between the RCQ subscales and convergent validity measures through the Bivariate Pearson correlation. The second section focused on the relationship between the RCQ subscales and discriminant validity measures through the Bivariate Pearson correlation. The third section assessed the correlation between subscales of the RCQ between the participation levels of injured athletes in the rehabilitation program (initial, intermediate, final) to examine the evidence for the predictive validity of the RCQ.

Scale Development

The main goal of this research was to develop a self-administered questionnaire that had both theoretical and conceptual relevance in rehabilitation settings; this was accomplished by reviewing sports-centric, qualitative literature of psychological responses, behavior responses to the injury, and the rehabilitation process. In accordance with previous research in this area (Evans & Hardy, 1995; Gould, Udry, Bridges, & Beck, 1997a; Podlog, Heil, Schulte, 2014; Shin, 2011; Tracey, 2003; Udry, 1997; Wiese-Bjornstal, Smith, & LaMott, 1995; Wiese-Bjornstal, Smith, Shaffer, & Morrey, 1998), psychological factors, such as personal characteristics, situational variables, cognitive responses, and emotional responses, were thought to influence both rehabilitation behavior and rehabilitation outcome. Theoretically, psychological factors can affect the rehabilitation outcome of a patient both directly and indirectly, with the latter relationship mediated by rehabilitation adherence. It was necessary to assess the psychological responses to injuries and mediated variables, such as social support and personal resources, because psychological responses and mediated variables of athletes during the rehabilitation process have been found to be associated with successful recovery. In order to ensure the rehabilitation process-specificity and the content validity of the measure, a two-step procedure was followed.

First, a broad review of mainstream and rehabilitation psychology literature was conducted in order to cover the relevant domains of psychological responses, behavioral responses, and mediated variables that could affect rehabilitation outcomes. Secondly, four qualitative rehabilitation psychological studies were examined. The researcher reviewed published research pertaining specifically to rehabilitation process of athletes (Gould et al., 1997a; Gould, Udry, Bridges, & Beck, 1997b; Podlog, Heil, Schulte, 2014; Shin, 2011; Tracey, 2003). Other studies investigating factors associated with social support (Bianco, 2001; Sherman, DeVinney, & Sperling, 2004; Udry, Gould, Bridge, & Beck, 1997) and personal variables (Malec, Brown, & Moessner, 2004) were also reviewed.

The breadth of the literature review maximized the content broadness of the measure and examined participants’ emotions, mental states, behavior, social support, and personal resources. Integrated models depicting psychological responses to sport injury and rehabilitation processes, developed in rehabilitation psychology (Wiese-Bjornstal et al., 1995; Wiese-Bjornstal et al., 1998), have considered the situational factors, as well as cognitive, motivational, and affective states, as determinants of rehabilitation behaviors. Thus, constructs such as social support (Bianco, 2001; Sherman et al., 2004; Udry et al., 1997) were seen as potential determinants of psychological responses. Also, Shin (2011) examined four factors that fit with the parameters of the construct of rehabilitation capabilities of athletes: Psychological factors, behavioral factors, social support factors, and personal resource factors. Based on previous research (Bianco, 2001; Fisher et al., 1988; Shin, 2011; Gould et al, 1997a; Gould et al, 1997b; Tracey, 2003; Wiese-Bjornstal et al., 1998), a questionnaire encompassing five factors that matched the necessary parameters of the construct was developed. Using a theoretically based approach to scale development (Clark & Watson, 1995; DeVellis, 1991), six sport-relevant, homogeneous, and conceptually distinct subfactors associated the rehabilitation capability of rehabilitating athletes were identified and used to form the core of the questionnaire.

Creation of Items

95 items with five factors were developed by a quantitative assessment of literature related to rehabilitation. Experts referred to research to maximize their clarity, their specificity, and the shortness of items based on guidelines for wording items (Clark & Watson, 1995). Each item was written so that 13-year-old athletes could understand them.

The adequacy of the 95 items made through the inductive method and literature review were tested by two sports psychology professors and a psychometric professor in the first stage. In the second stage, three researchers agreed on 74 of these items, yielding an inter-judge reliability coefficient of .78. After closer examination, three researchers agreed that 9 of the 74 items overlapped significantly. After eliminating highly redundant items, this reexamination resulted in a total of 60 items. In the final stage, three researchers were asked to categorize the 60 items based on content similarities.

Assessment of Items

Participants were instructed to rate the clarity of each item and to give comments, along with alternative formulations, for items that were not totally clear. Items were also presented to one rehabilitation psychology consultant and one doctor working in the rehabilitation hospital. Using a dichotomous scale, they were instructed to assess the applicability of each item to the rehabilitation process. Based on the ratings provided by athletes, the rehabilitation psychology consultant, the doctor, and their numerous comments, some items were rewritten in order to improve their clarity and to broaden their applicability across sports type and phases of rehabilitation. Items of the RCQ are presented in table 1.

Procedures

The 240 athletes were contacted by researchers, who provided them with an explanation of the purposes and procedures of the research project. The athletes allowed researchers to meet them under a trainer’s supervision at the end of a training session. Athletes were asked to participate in the study during these meetings. They were told the exact purpose of the study and that the questionnaire would take 10–20 minutes to complete. A strong emphasis was put on the confidentiality of data, and athletes were instructed not to write their names on the questionnaire and to put it in an unmarked envelope before returning it to their trainers. Participants were instructed to be sincere, serious, and to complete their questionnaire individually in a quiet environment when they had enough time.

Descriptive Statistics Analysis

The means of the RCQ items ranged from 2.338 to 4.617 and their standard deviations ranged from .52 to 1.299, thus suggesting that some items could not accurately discriminate the emotions, mental states, behaviors, and social support resources of injured athletes. Consequently, 14 items (Q10, Q12, Q14, Q15, Q17, Q20, Q26, Q32, Q44, Q45, Q51, Q53, Q59, Q60) with a mean greater than 4.5 and a standard deviation of less than .10 were omitted from further analyses (Tak, 1996).

The univariate skewness values of the RCQ items ranged from -1.312 to 5.782 and their univariate kurtosis values ranged from -1.250 to 11.471, thus suggesting that some items deviated severely from normal distribution. Consequently, six items (Q3, Q8, Q18, Q30, Q43, Q46) with a skewness greater than ±3.0 and a kurtosis greater than ±8.0 were omitted from further analyses (West, Finch, & Curran, 1995). Also, two items (Q6, Q21) with a response rate greater than 50% on a scale were excluded from further analyses (Eum, 1996).

Item Analysis

Inter-item correlations were calculated for all 38 items. The goal of item analysis was to identify redundant items, because large scales often place severe limitations on clinical utility, particularly in busy medical settings. Consistent with other research and recommendations (DeVellis, 1991; Rapee, 1994), redundancy was defined in two ways: (1) A high level of inter-item correlation (r>0.45), and (2) similar content. When items met both criteria, we deleted the item with the lower item-total correlation coefficient and retained the item with the higher coefficient (DeVellis, 1991).

Item-total correlations were calculated for the remaining 38 items in order to produce a scale measuring a relatively specific construct and following DeVellis’ (1991) suggestion, items with the lowest corrected item-total correlations (rit≦.30) were excluded. This process was repeated until the internal consistency of the scale could not be increased further by additional item deletion (Gregory, 1996). This procedure resulted in the exclusion of nine items (Q2, Q11, Q23, Q29, Q37, Q38, Q42, Q48, Q58), reducing the total to 29 items. The corrected item-total correlations were assessed for 29 items of the RCQ, with correlations for all items being greater than 0.30 (see Table 3). As indicated, there were no more items that would have increased the scale’s reliability if they were deleted. Cronbach’s alpha for the overall scale was high (alpha=.925), as were alpha coefficients for each of the subscales (emotive factor r=.86; cognitive factor r=.81, behavioral factor r=.83, social support factor r=.78, and personal resources factor r=.80).

Exploratory Factor Analysis

Following the reliability analysis, an exploratory factor analysis (maximum likelihood estimation with oblique rotation) was applied to the data in order to investigate the factor structure of the 29-item RCQ. The resulting factors were subjected to oblique (Oblimin) rotation because we expected the subfactors of the RCQ to correlate with one another. The number of factor to retain was evaluated using (a) Kaiser’s (1961) eigenvalue >1 factor extraction rule, (b) scree plot analysis (Cattell, 1966), and (c) the interpretability of the resulting factor structures (Gorsuch, 1983).

Kaiser-Meyer-Olkin index of sampling adequacy was .83, indicating that the items were appropriate for common factor analysis using maximum likelihood estimation. Cattell’s scree plot analysis identified adequate factor number. Result of Cattell’s scree test indicated that a four-factor solution was the most interpretable. Consequently, four items (Q22, Q54, Q55, Q56) representing personal resources factor did not emerge as viable factors, as indicated by low factor loadings and/or high cross-loadings. In light of factor analysis results, the personal resources factor was discarded. In addition, one item each from emotional factor (Q28), cognitive factor (Q27), and social support factor (Q47) was deleted due to high cross-loadings or factor loadings below .50(Tak, 1996). The four factors retained were: Emotional factor (six items), cognitive factor (six items), behavioral factor (five items), and social support factor (five items). The four-factor solution of the 22-item RCQ is presented in Table 4.

Confirmatory Factor Analysis

The fit of the three measurement models, shown in tables 5 and 6, were tested at this stage. Four-factor measurement model (22 items) exhibited a non-significant chi-square as well as a GFI, AGFI, and CFI greater than .90, and an RMSEA below .08. This result indicated that the RCQ had four factors: An emotional factor, cognitive factor, behavioral factor, and social support factor. Furthermore, the magnitudes of the factor loading estimates of four-factor measurement model (22 items) shown figure 1 ranged from .594 to .820, where a majority of the factor loadings were higher than .60. Also, internal consistency reliability estimates exceeded the .70 cut-off value recommended by Nunnally (1978).

View original

Figure 1.

Model of Confirmatory Factor Analysis of RCQ

Convergent Validity and Discriminant Validity

Convergent and divergent validity of the RCQ were assessed by computing the Pearson correlation between RCQ subscales and measures of theoretically related and unrelated constructs. The Pearson correlation analysis revealed moderate to high correlation between the emotion score of the RCQ and the subscale score of POMS-related emotion measurement (range, r=-.56-.79). As expected, the correlation between the emotion score of the RCQ and the POMS subscales score were high and significant; both instruments were designed to assess emotion. The Pearson correlation analysis revealed a moderate to high correlation between the cognition score of the RCQ and subscales score of CSAI-2-related cognition measurement (range, r=-.82-.78). The correlation between the cognition score of the RCQ and POMS subscales score were high and significant; both instruments were designed to assess cognition. The Pearson correlation analysis revealed a moderate to high correlation between the behavior score of the RCQ and subscales (perceived exertion, scheduling) score of the RAQ-related behavior measurement of injured athletes (range, r=.49-.55). As expected, the correlation between the behavior score of the RCQ and the subscales score of the RAQ were high and significant; both instruments were designed to assess the behavior of injured athletes. Pearson correlation analysis revealed a moderate to high correlation between social support score of the RCQ and subscales (support from significant others) score of the RAQ-related social support level of injured athletes (range, r=.81). As expected, the correlation between social support score of the RCQ and subscales score of RAQ were high and significant; both instruments were designed to assess the social support level of injured athletes.

Predictive Validity

In psychometrics, “predictive validity” refers to empirical evidence that demonstrates a scale has predictive power over the unobservable construct that it is intended to measure. For example, the validity of a cognitive anxiety test for performance is the correlation between cognitive anxiety scores and performance level. Correlations between subscales of the RCQ between the participation levels of injured athletes in rehabilitation programs (initial, intermediate, final) were significant, with the exception of the social support factor. There was no relationship between the social support factor and the participation level of injured athletes in rehabilitation program in the intermediate phase. The predictive validity's results are presented in table 8.