Key Words: child, adolescent, mental health services, outcome assessment, Psychiatric Department, Hospital, psychometrics/instrumentation

Introduction

Routine outcome assessment in mental health requires instruments that are easily available, brief, and easy to use in the clinical setting[1]. Implementation of these types of instruments allows for reliable and reproducible outcome assessment through quantitative measure of therapeutic change. Routine outcome assessment can help inform decision-making, resource allocation, and policy design[2],[3].

One of the most widely accepted instruments for mental health outcome assessment is  HoNOS (Health of the Nation Outcome Scales)[4]. HoNOS is useful for measuring behavior, impairment, symptoms, and social functioning in mental health patients. A tool specifically designed for children and adolescents, HoNOSCA, has also been developed[5]. HoNOSCA allows researchers to collect assessments from three different stakeholder groups (patients, their parents or legal guardians, and clinicians) through the use of three different scales. The clinician version of the instrument, known as HoNOSCA–Glossary, has two additional items for appraising patients’ and parents/legal guardians’ understanding of the diagnosis and the available management options. HoNOSCA can be administered by various types of professionals, including psychiatrists, nurses, psychologists, counselors, social workers/ educators, occupational therapists, and teachers. Since its publication, HoNOSCA has been translated into several languages, including Danish, French, Lithuanian, and Norwegian[6],[7],[8],[9],[10]. More recently, the three HoNOSCA scales have become available in Catalan and Spanish. These new versions, translated using a reverse translation approach, were found to have satisfactory internal consistency and inter-rater reliability[11]. The Spanish version of HoNOSCA–Glossary  also had good concurrent validity.

Interesting similarities and differences can be found when looking at the perspectives of different groups of informants in health care[12]. In a study exploring treatment preferences in the context of anticoagulation therapy, authors reported wide differences between patients and clinicians on risk of bleeding (threshold risk) for justifying warfarin treatment over aspirin[13]. A similar study found that, compared to clinicians, patients  saw more value in avoiding a stroke than in avoiding bleeding[14]. A systematic review of qualitative studies on the use of vitamin K antagonists for atrial fibrillation also showed differences by evaluator group, with patients reporting lack of information and understanding as the main difficulties with anticoagulant treatment, while physicians reported uncertainty about its use, the need for individualized decision-making, and the delegation of responsibility for it (from specialists to family practitioners, for example) as their biggest concern[15].

Comparisons of multiple perspectives have also been conducted in the context of mental health. A meta-analysis of 341 studies reporting cross-informant correspondence of estimates assessing child and adolescent mental health found low-to-moderate correspondence between stakeholders (parents, children, or teachers); assessments in which the stakeholders being compared observed the child/adolescent in the same setting (e.g., a mother and a father) correlated better than those that did not (e.g., a parent and a teacher)[16]. A study on the perspectives of young migrant mental health patients, their parents, and clinicians on optimal health services provision found that patients stressed the importance of ensuring privacy with respect to school or friends, whereas families highlighted addressing measures to decrease stigma[17]. A study of HoNOSCA evaluations by Gowers et al. reported limited agreement between patient and clinician assessments, with Pearson’s correlation coefficients of 0.27 at admission and 0.58 at discharge[18]. Lastly, a recent retrospective pre–post observational study found that patients considered their mental health status to be worse than clinicians did at both admission and discharge[19].

Despite a growing body of evidence comparing the perspectives of different stakeholders in health care, including mental health, research in this area is still sparse. Assessment of differences in HoNOSCA evaluations from different stakeholders(patients, parents or legal guardians, and clinicians) is also limited, especially for the non-English versions of the instrument. Therefore, this study aimed to assess mental health outcomes in pediatric patients at psychiatric day hospitals from the perspective of patients, parents or legal guardians, and clinicians, using the Spanish and Catalan version of HoNOSCA.

Methods

Design
Longitudinal study.

Participants
Pediatric patients (18 years of age or less) with at least one of the following diagnoses from the 2017/18 International Classification of Diseases, 10th Revision, Clinical Modification

(ICD-10-CM), section F01–F99 (“Mental, Behavioral and Neurodevelopmental disorders”): psychotic disorders; mood disorders (including major depression disorders, single or recurrent episodes); anxiety disorders; eating disorders; autistic spectrum disorder; and behavioral and emotional disorders. Study inclusion criteria for patients were having parents or legal guardians that were available to participate and the ability of both patients and parents / legal guardians to communicate in Spanish or Catalan.

All patients were recruited at the day unit of Parc Taulí Hospital Universitari (Sabadell, Catalonia, Spain), an outpatient short- to medium-term facility for children and adolescents. This unit has an inter-disciplinary team of social workers/educators, nurses, occupational therapists, counselors, teachers, psychologists, and psychiatrists. In general, patients stay at the unit throughout the whole week, although increasing periods of leave are encouraged throughout the course of admission.

Patients were recruited between January 2015 and December 2017.

Measures
We conducted all outcomes assessments using three versions of HoNOSCA: clinician (“HoNOSCA–Glossary”), patient (“HoNOSCA–Patients”), and parent/legal guardian (“HoNOSCA–Parents”).

HoNOSCA measures the following 13 areas of health and psychosocial functioning during the  two-week period preceding the assessment: 1) problems with disruptive, antisocial, or aggressive behavior; 2) problems of hyperactivity, attention, or concentration; 3) non-accidental self-harm; 4) problems with alcohol, abuse of substances/solvents; 5) problems with school or language skills; 6) physical illness or disability problems; 7) problems associated with hallucinations, delusions, or abnormal perceptions; 8) problems with non-organic somatic symptoms; 9) problems with emotional and related symptoms; 10) problems with peer relationships; 11) problems with personal care and independence; 12) problems with family life and relationships; and 13) low school attendance. The clinician version of the instrument, HoNOSCA–Glossary, has two additional items: 14) problems in knowing or understanding the nature of the difficulties of the child / adolescent, and 15) problems with lack of information about services or management of child / adolescent difficulties. Each HoNOSCA item is assessed using a 5-point scale ranging from 0 (“no problem”) to 4 (“severe problem”). Higher scores therefore represent greater severity of dysfunction. The final (total) HoNOSCA score is the sum of the scores for all items[20].

Administration of HoNOSCA and data collection
The HoNOSCA–Glossary instrument was administered by a hospital day unit physician or psychologist once after the first multidisciplinary team case discussion meeting post-admission, where the patient’s background history and recent progress were discussed, and again upon discharge. The HoNOSCA-Parents and HoNOSCA-Patients instruments were also administered twice, to each of those two stakeholder groups (immediately after admission, and upon discharge). Various hospital day unit staff administered these instruments, including psychiatrists, nurses, psychologists, counselors, social workers/ educators, occupational therapists, and teachers.

After the administration of each of the three different HoNOSCA instruments, the interviewers reviewed the medical history of the patient and collected additional information, including demographics and diagnosis, assessed the answers collected, and calculated the final scores.

Training in use of the scales
Participating staff had been trained in the use of the scales during a HoNOSCA validation study[11]. They received all available training material for HoNOSCA, including a workshop video and a training guide. Throughout the course of the study, the lead author (AB) was available to answer any questions about HoNOSCA and address any uncertainty about how to administer the different assessments.

Statistical analyses
We defined HoNOSCA improvement scores as the difference between measures taken at discharge minus those taken at admission and negative improvement scores as decreases in HoNOSCA scores from admission to discharge (reflecting overall improvement in mental health status). Statistical analyses were performed using repeated-measures ANOVA, with HoNOSCA scores (pre- and post-treatment) and stakeholder group (patient, parent or legal guardian, and clinician) as factors. Subsequently, Bonferroni-corrected pairwise comparisons were conducted. Differences between groups with p values less than 0.05 were considered statistically significant.

The sample size for this study was calculated using Epidat 3.1, accepting an alpha risk of 0.05, a power of 98% in a bilateral contrast, and a correlation coefficient of 0.5. Using this approach, it was found that a total of 80 participants would be needed. Accounting for a lost-to-follow-up rate of 10%, the target sample size was 88.

Ethical considerations
We obtained approval to conduct this study from the Ethics Committee of the Hospital Clinic of Barcelona, the primary affiliation of the principal investigator of this study (AB). All participants (the pediatric patients and their parents or legal guardians) signed consent forms to participate. All data collected were anonymous and stored in a password-secured database. Data were entered using an anonymous code for each participant in an Excel spreadsheet to facilitate statistical analysis.

Results

Demographics
A total of 99 children and adolescents were recruited for the study, and 11 of them were lost during follow-up. The characteristics of the 88 patients included in the analysis are shown in Table 1. There were equal numbers of females and males, with a mean age of 14.4 years (standard deviation (SD): 2.00, range 5 to 18) and a median age of 14 years. The most prevalent diagnoses were autistic spectrum disorder, eating disorders, and mood disorders. Mean length of treatment at the day hospital unit was 95.7 days (SD: 73.8, range 11 to 385) with a median of 69 days (Table 1).

Table 1. Characteristics of study participants (n = 88).

HoNOSCA scores

Overall, there were significant improvements in the HoNOSCA scores at discharge compared to those obtained at admission (Table 2). The scores varied by diagnosis and stakeholder group. Patients and clinicians conferred higher scores (indicating greater severity of dysfunction) when the diagnosis was autistic spectrum disorder (n = 26) (19.31 and 25.81 respectively) or psychotic disorders (n = 7) (19.71 and 23.71 respectively), with the highest scores from patients corresponding to adjustment disorder (n = 4) (20.5). Parents or legal guardians conferred the highest scores when the diagnoses were behavioral or emotional disorders (n = 13) (23.88 for attention deficit/hyperactivity disorders and 22.40 for other behavioral and emotional disorders). These trends remained constant through discharge, especially for psychotic disorders and behavioral and emotional disorders (Table 2).

Table 2. HoNOSCA scores at admission and discharge.

Comparing HoNOSCA scores across stakeholder groups, there were differences between clinicians and patients and/or their parents or legal guardians at admission. In general, clinicians conferred higher scores, meaning that they perceived patients' health status to be worse than reported by the patients themselves and/or their parents or legal guardians (Tables 2 and 3). Differences between groups were considered statistically significant when the p values were less than 0.05.

Table 3. Differences in HoNOSCA assessments at admission and discharge.

At discharge, scores decreased and became more homogeneous, becoming almost equal across all three stakeholder groups (Table 3 and Figure 1).

Figure 1. Plot of HoNOSCA scores at admission, discharge.

Discussion

Measuring relevant outcome variables in mental health facilitates assessment of the efficacy, strengths, and limitations of the services provided[3] In this study we measured health outcomes in day hospital patients from the perspective of patients, their parents or legal guardians, and clinicians, using HoNOSCA. Our work shows that HoNOSCA is a viable tool to achieve this goal in a systematic and reproducible way in the day hospital psychiatric hospital setting. The improvement in scores from admission to discharge is consistent with the expected improvement after therapeutic treatment. However, given the ample variability of diagnosis and the sample size calculation we implemented, this study did not have sufficient power to detect differences in HoNOSCA scores by diagnosis. We therefore report the results we obtained but make no inferences regarding this data.

At admission, there were significant differences in the assessment of mental health status by clinicians compared to patients and their parents or legal guardians. At discharge, however, the appraisals became more homogeneous (Table 3). This phenomenon speaks positively about the treatment and psychoeducational work provided by the health professionals was effective. We cannot assess whether the differences between the admission and discharge scores were relevant to patients, however, since no studies have assessed the minimal important difference (MID) for HoNOSCA. MIDs are defined as “the smallest difference that patients perceive as beneficial and that would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management” and facilitate the interpretation of patient-reported outcomes[21].

The reasons behind the differences between assessments from patients and parents or legal guardians and clinicians are unclear. On the one hand, patients may be able to better understand their condition, especially in the less acute, outpatient setting studied here, especially at admission, before clinicians have fully grasped patients’ overall condition. Over the span of the hospital stay, however, clinicians may become better informed of the patient’s complete medical history and condition[18],[19]. On the other hand, clinicians’ medical training may equip them well to provide a more accurate assessment of a patient’s status[19].

Our results are in line with those from previous studies that report improvement in HoNOSCA scores after therapeutic interactions and at discharge[18],[19],[22]. Like our study, Gowers et al. found that HoNOSCA scores from clinicians were generally higher than those from patients in the first assessment and in the early stages of the interaction, but over time and at discharge the scores became more homogenized[18]. This finding was true both for inpatient and outpatient settings[18]. Different findings emerged in a more recent study at a National Health System tier 4 adolescent inpatient unit for youth in the UK, however, where patients considered their mental health status to be worse than clinicians did, according to the HoNOSCA instrument, at both intake and discharge[19].

Strengths and limitations
The main strength of our study is the large sample size we were able to recruit. Another strength was the extensive experience the hospital personnel responsible for administering HoNOSCA had with the scale administration due to their participation in a previous translation and validation study of this tool¹¹. The main limitation of our study was that it was restricted to a single health facility in Catalonia, thus potentially compromising its external validity. In addition, this study was limited to a day hospital setting; results could be different in intensive care and other clinical settings.

Implications for practice
The Spanish and Catalan versions of HoNOSCA have proven to be useful tools for outcome measurement in children and adolescents in the day hospital setting. Use of HoNOSCA is feasible within this setting, which allows for assessment from the perspective of patients, parents or legal guardians, and clinicians. HoNOSCA can also be useful for gauging level of severity and improvement in patients with different diagnoses and for promoting evidence-based decision-making in day hospital settings.

Implications for research
The results we present here highlight differences in outcome assessment between clinicians and patients and legal guardians in a day hospital setting. It would be interesting to explore whether these differences persist in other contexts, such as full-hospitalization settings. In addition, our work is limited to a single health facility; additional research to confirm these findings across multiple facilities would be useful. Further research could also explore, in depth, the reasons for the differences in the assessments from the three different stakeholder groups reported here.

This work was not designed to draw any conclusions regarding sensitivity to change in the Spanish and Catalan versions of HoNOSCA. This is the object of another research project currently under way. Lastly, the MID for HoNOSCA could be explored using an anchor-based approach that incorporates an independent standard that both patients and clinicians can recognize and that can provide a measure of small but important differences correlated to HoNOSCA[23].

Conclusion

HoNOSCA allows for mental health outcome assessment in the day hospital setting not only from the perspective of clinicians but also from the perspective of patients’ parents or legal guardians and, even more important, the patients themselves.

Notes

Authorship contributions
AB: Study design, data collection, statistical analysis, first draft of the manuscript, approval of the final version of the manuscript. JT: Study design, data collection, approval of the final version of the manuscript. PM: Study design, statistical analysis, approval of the final version of the manuscript. HP: Study design, statistical analysis, first draft of the manuscript, approval of the final version of the manuscript.

Acknowledgments
Anna Ballesteros-Urpí is a PhD candidate in the Education Department at the University of Barcelona (UB) in Barcelona, Spain.

We want to acknowledge Eloísa Arias Núñez, Montserrat Pàmias Massana, and Joaquim Puntí Vidal from Parc Taulí Hospital Universitari for their help during data collection. We also want to thank Andreu Ferrero-Gregori and Francesc Pla Juncà for their help with the statistical analysis.

Competing interests
The authors declare that there are no conflicts of interest involving this work.

Funding sources
This study was self-funded.

Ethics approval
Approval to conduct this study was obtained from the Hospital Clinic de Barcelona Ethics Committee (# HCB/2014/0337).

Data sharing
All data from this study are available upon request from the corresponding author.

From the editors
The original version of this manuscript was submitted in Spanish. The English translation was copyedited by the journal.

Table 1. Characteristics of study participants (n = 88).

Table 2. HoNOSCA scores at admission and discharge.

Table 3. Differences in HoNOSCA assessments at admission and discharge.

Figure 1. Plot of HoNOSCA scores at admission, discharge.

Esta obra de Medwave está bajo una licencia Creative Commons Atribución-NoComercial 3.0 Unported. Esta licencia permite el uso, distribución y reproducción del artículo en cualquier medio, siempre y cuando se otorgue el crédito correspondiente al autor del artículo y al medio en que se publica, en este caso, Medwave.