Julia Pape-Neumann1, Hans-Georg Bosshardtz2, Ulrich Natke3, Horst Oertle4
1Bandesvereinigung Stotterer-Selbsthilfe e.l/., Gereonswall 112, 50670 Koln, Germany
2Faculty of Psychology, Rahr- University Bochum, Germany
3Faculty of Experimental Psychology, University Diisseldorfi Germany
4Centre for Stationary Speech Therapy, Bad Salzdetfurth, Germany
On the initiative of the German self-help organisation the program for the evaluation of stuttering therapies PEVOS (Programm zur Evaluation von Stottertherapien) was developed to evaluate different stuttering therapies over a time period of two years after therapy. The concept was designed by a group of therapists and scientists and was tested since 2001. In the test-phase data were collected from ten therapists with 100 clients. Fluency data were obtained by telephone calls. Functional outcomes and changes in attitudes and emotions were measured with questionnaires. Results of the test-phase’s evaluation including the first two assessments will be presented, organisational problems and possible solutions will be discussed.
The aims of the program were as follows:
- The usefulness of measurement-instruments integrated in this program should be evaluated.
- Short- and long-term outcomes of stuttering therapies of different therapists with varying
- therapy programs should be evaluated with the same measurement instruments.
- Stuttering therapies should be improved by feedback provided to the therapists about the results of their therapies.
- The situation Of people who stutter in Germany should be improved by offering exact and objective information about the long-term-outcomes of individual therapies.
In the beginning of the test-phase therapists registered those clients who were willing to take part in PEVOS-program, and those who refused to take part were registered anonymously. Data were collected immediately before (pre-treatment condition) and 4 to 6 weeks following therapy (post-treatment condition) and follow-ups will be carried out one and two years after the end of therapy. In intensive therapy programs the end of therapy was defined as the end of the intensive phase. Data were collected and processed independently of the therapists by employees of the German stuttering association: questionnaires were sent directly to the clients who were asked to return them directly to the association and speech samples were obtained by telephone interviews without appointments if possible. Speech samples of young children less than seven years old were recorded by the therapists.
At the end of the treatment the therapists indicated which therapy modules they had used in the particular therapy and they were also asked to specify the time over which these modules were administered. For this a list of modules (modules like “informing about stuttering” ,“relaxation”, “identification , prolongation” and “pu1lout”) was provided. Clients were categorized in two age groups: the children group consisted out of clients younger than 17 and the adult group consisted out of clients 17 years old or older. Because of different evaluation conditions for children of different age the children group was further divided into subgroups of pre-school children (younger than 7 years), elementary school children (7-10 years old), pupils (11-14 years old) and youths (15-16 years old).
Therapists and their clients will be individually informed about their results after the first follow-up. Therapists are allowed to publish the results of their therapies on a non-selective base. The self-help association guarantees that results will be published only in such a form that privacy of both therapists and clients will be protected.
Several questionnaires which had already been translated into German and partly modified and which had been used in few German evaluation studies before were combined in PEVOS. The following questionnaires were integrated in the program to evaluate the change of functional therapy outcomes, emotions and attitudes belonging to stuttering or communication in adults:
S24 (Erickson,l969; Andrews & Cutler, 1974; translated by Jehle et al., 1989), avoidance PSI- V (Woolf, 1967; translated and worked over by Frischmuth, (1995), stuttering frequency in social situations (Brutten, 1973; modified by Fiedler & Standop, 1986), and a revised version of scales for assessing preferred functional outcomes of stuttering treatment (Cooper & Cooper, 1998). The latter is a translation of questionnaires, which were developed by the Special Interest Division No. 4 of the American Speech-Language-Hearing-Association.
Oertle (1999) designed a questionnaire for children out of several items from different questionnaires to refiect attitudes, emotions and behaviour related to stuttering and speaking. The questionnaire is identical “for parents of the client and the child himself. Depending on the age of the child only the parents, parents and child or the child only have to answer the questions.
2.3 Speech samples
Speech samples were obtained from telephone interviews and consisted out of 300 to 500 syllables. Following to Yairi disfluencies were categorized as “stuttering-like” and “other disfluencies” (Yairi, 1997). "Stuttering-like” disfluencies in PEVOS are defined as repetitions of sounds and syllables, prolongations and blocks. “ Other disfluencies” are defined as interjections, revisions, unfinished words, word repetitions and phrase repetitions. The frequency of syllables with stuttering-like and other disfluencies was counted and the overall-percentage calculated. Speech rate was calculated on the base of spoken syllables which included stuttered syllables. Every spoken syllable was only counted once irrespective of how often it was repeated. Breaks between interviewer’s and clients’ utterances were cut off the speech sample whereas breaks within the clients’ utterances were not cut off. Speech naturalness was rated with the 9-point scale of speech naturalness (Martin, Haroldson & Triden, 1984) with lower scores indicating more natural speech. To evaluate interjudge reliability 30 speech samples were presented to a second clinician. The clinician was instructed to make naturalness ratings on the 9-point scale, to count “stuttering-like” disfluencies and syllables on each speech sample. The Pearson correlation coefficient for the naturalness ratings of the first and the second clinician was 0.93, 0.98 for stuttering-like disfluencies and 0.99 for syllable counts.
Nine therapists registered 100 clients for the participation in PEVOS during the test-phase. Five therapists offered ambulant therapies in their private clinics in single sessions or single sessions combined with group therapy and four therapists offered intensive group therapies, one of them in a stationary setting. Nineteen of the registered clients had therapies in private clinics and 81 took part in intensive group therapies.
Fifty-one clients were older than 16 years (mean age=28.7 years, SD=l2.4 years) and hence received questionnaires for adults. Forty-nine clients were younger than 16 years (mean age=l3.5 years, SD=3.4 years) and received questionnaires for pupils and/ or parents. Seventy additional clients refused to take part and were registered anonymously.
Ninety-seven clients returned their questionnaires to the German stuttering association and 97 speech samples were obtained before therapy. At the second assessment 4-6 weeks after the end of therapy only 82 clients returned their questionnaires and 82 speech samples were obtained whereas 89 clients had finished therapy. Eleven clients had not finished therapy yet.
Speech data immediately before and 4 to 6 weeks following therapy were based on 42 children. There was a decrease in the mean percentage of stuttered syllables between pre-treatment condition (M=ll.93%, SD=ll.06) and post-treatment conditions (M=5.00%, -SD=5.85). A T-Test was performed to assess the significance of this observed difference. The results of this test showed that there was a statistically significant probability of a difference (p<.0Ol). The effect size2 was calculated to evaluate the practical relevance of differences from pre- to post-treatment results. The effect size of changes in percentage of stuttered syllables was 0.63.
The mean speech rate from pre-treatment condition was 127.43 syllables/min. (SD=55.79) and increased to 147.98 syllables/min. (SD=53.85) in the post-treatment condition. Results of the T-Test showed that the probability of a difference was statistically significant (p=.0Ol). The effect size concerning the change of speech rate was 0.37.
The overall mean speech naturalness rating3 for all children improved from 5.02 (SD:2.l0) in the pre-treatment condition to 3.76 (SD=l.99) in the "post-treatment condition. Results of the T- Test showed that there was a statistically significant probability of improvement in the mean speech naturalness (p<.0Ol). The effect size of differences in speech naturalness was 0.60.
Figure 1. Percentage of stuttered syllables children
Figure 2. Speech rate (syllables/min.) children
Figure 3. Speech naturalness children
The evaluation of the children’s attitudes, emotions and behaviour related to stuttering and speaking was based on questionnaires from 34 children and 20 questionnaires from parents at the pre- and post-treatment condition. The sum of added means of all items for children’s questionnaires was 79.62 (SD=14.18) and 72.87 (SD=18.35) for parents’ questionnaires. In 15 cases there were questionnaires returned from both, the child’s parents and the child himself. The Pearson correlation coefficient between the children’s and the parents’ questionnaires was 0.85 (p<.001) for the pre- treatment condition and 0.88 (p<.001) for the post-treatment condition. The effect size concerning the difference from pre- to post-treatment condition was 1.70 for the children’s questionnaires and 0.95 for the parents’ questionnaires.
To evaluate if changes in stuttering frequency and in questionnaires’ scores between pre- and post-treatment condition were related to each other the Pearson correlation coefficient4 was calculated.
The Pearson correlation coefficient between mean differences in stuttering frequency and mean differences between pre- and post-treatment condition in scores of questionnaires filled in by children was 0.27 (p>.05) and 0.18 (p>.05) for differences in stuttering frequency and mean differences in questionnaires filled in by parents.
Speech data immediately before and 4 to 6 weeks following therapy were based on 43 adults. There was a decrease in the mean percentage of stuttered syllables between pre-treatment condition (M=l3.69%, SD=11.56) and post-treatment conditions (M:4.81%, SD=6.19). A T-Test was performed to assess the significance of this observed difference. The results of this test showed that the probability of a difference was statistically significant (p<.001). The effect size of changes in percentage of stuttered syllables was 0.77.
The mean speech rate from pre-treatment condition was 135.95 syllables/min. (SD=65.25) and increased to 158.98 syllables/min. (SD=56.53) in the post-treatment condition. Results of the
T-Test showed that there was a statistically significant probability of a difference in mean speech rate (p<.01). The effect size concerning the change of speech rate was 0.35.
The overall mean speech naturalness rating for all subjects was 5.77 (SD=2.56) in the pre- treatment condition and 4.65 (SD=2.32) in the post-treatment condition. Results of the T-Test showed that the probability of a difference in the mean speech naturalness rating was statistically
significant (p<.01). The effect size of differences in speech naturalness was 0.44.
Figure 4. Percentage of stuttered syllables adults
Figure 5. Speech rate adults
Figure 6. Speech naturalness adults
The evaluation of avoidance was based on PSI-avoidance questionnaires from 35 clients at the pre- and post-treatment condition. There was an improvement between the mean score between pre-treatment condition (M=l0.92, SD=3.86) and post-treatment condition (M=3.81, SD=3.55). Results of the T-Test showed that there was a statistically significant probability of a difference (p<.00l). The effect size concerning the differences in avoidance was 1.84. The Pearson correlation coefficients5 between mean differences in stuttering frequency and mean differences between pre- and post-treatment condition in PS1-avoidance scores was 0.34 (p>.05).
Attitudes and behaviour related to stuttering and speaking were evaluated by 39 S-24 questionnaires. The probability of an improvement between the pre-treatment score (M=18.l5, SD=3.99) and the post-treatment score (M=8.93, SD=5.69) was statistically significant (p<.00l) according to the T-Test. The effect size of changes in attitudes and behaviour was 2.26. The Pearson correlation coefficient between mean differences in stuttering frequency and mean differences between pre- and post-treatment condition in S-24 scores was 0.45 (p<.01).
Assessment of stuttering frequency in social situations was based on SSC questionnaires from 41 clients at the pre- and post-treatment condition. The mean score improved between pre-treatment condition (M=1.98, SD=0.46) and post-treatment condition (M=2.95, SD=0.59). According to the T-Test the probability of a difference was statistically significant (p<.00l). The effect size of changes in stuttering frequencies in social situations was 2.15. The Pearson correlation coefficient between mean differences in stuttering frequency and mean differences between pre- and post-treatment condition in SSC scores was 0.49 (p<.0l).
Preferred functional outcomes of stuttering treatment were assessed by 40 FOAI questionnaires. According to the Wilcoxon-Test there was a statistically significant probability of a difference (p<.00l) between the pre-treatment score (M=3.28, SD=0.87) and the post-treatment score (M25 .41, SD=l.03). The effect size of differences in the FOAI score was 2.46. The Pearson correlation coefficient between mean differences in stuttering frequency and mean differences between pre- and post-treatment condition in FOAI scores was 0.37 (p<.05).
Figure 7. Mean questionnaires’ scores adults
One important finding of this study is that improvements in attitudes, emotions or behaviour related to stuttering or communication or improvements in functional outcomes of therapy are more pronounced than improvements in speech fluency data. As reported before correlations of differences between pre- and post-treatment in stuttering frequency and questionnaires’ scores were quite low. While the improvements at the post-treatment condition in all questionnaires showed strong effect sizes (between 1.84 and 2.46 for adults, 0.95 for parents and 1.70 for children) the effect sizes for percentage of syllables stuttered were comparably smaller (0.77 for adults and 0.63 for children).
There are two possible reasons for therapy effect size to be smaller for syllables stuttered. Therapy can either improve speech fluency of all clients to a smaller degree than communication attitudes or it can improve speech fluency of only part of the clients whereas it influences the attitudes of all clients in a comparable way. These alternative interpretations can be identified when the standard deviations of the pre- and post-treatment measures are taken into consideration. If therapy improves the speech fluency of only some clients but leaving that of others unaffected or even making it worse then therapy would increase the individual differences between clients. If on the other hand pre- and post-treatment measures show comparable variances this can be taken as an indication that effect size differences result from stronger or weaker therapy effects. As all standard deviation values (except for Erickson S 24) are comparable for pre- and post-treatment measures it can be assumed that weaker effect sizes in speech fluency compared to effect sizes of attitudes result from weaker therapy effects on speech fluency.
The test-phase of PEVOS was carried out to eliminate mistakes and optimise the program’s concept. Several problems occurred while the first clients and therapists went through the program. For some therapists it turned out to be difficult to register their clients in time so that the pre- treatment evaluation could take place before the beginning of therapy. In some cases the course of registration could be speeded up by improved communication between the stuttering association and the therapists whereas in other cases the therapy began before a client could fill in the questionnaires or give the telephone interview. Experiences showed that single clients should be registered at least 7-10 days and therapy groups at least 14 days before beginning of treatment to enable the stuttering association to perform the pre-treatment evaluation in time.
In case the evaluation of stuttering therapies was obstructed because some of the clients did not return their questionnaires to the German stuttering association at all the questionnaires were sent out a second time. If more than 10% of one questionnaire’s items were not answered, the questionnaire was returned to the client and he was requested to fill it in completely.
Partly it was difficult to get hold of clients on the phone so appointments had to be arranged and the changed interview condition was registered.
Some clients who initially became convinced of the program’s advantages by their therapist indirectly refused to take part in the evaluation by keeping the questionnaires, leaving most of the questionnaires’ items unanswered or being not cooperative during the telephone interviews. Experiences showed that therapists must leave the decision over participation in PEVOS to the clients themselves as pressure might have negative consequences on the evaluation.
Motivation to take part in post-treatment evaluation sometimes decreased when clients were not satisfied with the results of therapy. This became clear during the telephone interviews with clients who cooperated well at the pre-treatment condition and were less cooperative at the post-treatment condition. Some of these clients expressed their frustration about the therapy in the beginning of the telephone call. In these cases the employees of the stuttering association had to stress that the program is independent of participating therapists. Additionally advantages for clients like exact and objective information about therapy results were explained to the clients to improve their motivation.
To gain the necessary size of speech samples the standard set of questions for the telephone interview had sometimes to be extended by more detailed questions about the same topic when clients’ answers were monosyllabic. Additionally the clients were instructed to answer as detailed as possible.
Speech samples of children younger than 7 years were recorded by the therapist because it was assumed that young children would not talk to strangers at the phone. The quality of the therapists’ recording usually was worse than the quality of the telephone interviews so that the therapists will be offered to borrow qualitative better recording equipment from the stuttering association in future evaluations.
Another problem of evaluation was the returning of the therapists’ final report in which the therapists were asked to indicate for every client which therapy modules they had used in the particular therapy and to specify the time over which these modules were administered. Some of the therapists forgot to send them back and thus had to be requested repeatedly to return them to the association. Additionally, some therapists did not fill in one report per client but filled in only one report for a complete course. In these cases information on the therapy of individual clients were prevented. In the beginning of the test-phase the standard form had to be supplemented by a second form which was specified for the needs of intensive courses because the standard form was too detailed.
The described problems showed that it is necessary to be in good Contact with therapists as well as with the clients. Additionally, it is indispensable to control the return of all forms and questionnaires that were sent to therapists and clients. In addition to the detailed assessment of data by telephone interviews and questionnaires and the following evaluation this means that the program is very time-consuming and therefore expensive. PEVOS has been financed by the German stuttering association during the test-phase and its’ follow-ups. Several unsuccessful attempts have been made to secure a funding for a main implementation of PEVOS. Considering these failed attempts it is required to think the concept over: it must be considered to let therapists and/or professional associations of therapists contribute to the costs. At the same time it must be considered how the program could be realized less time-consuming.
1Besides the authors the working pool consists of Reiner Nonnenberg, Patricia Sandrieser, Peter Schneider, Nicole Spindler and Haitmut Ziickner.
2The effect size was always calculated in the way that an improvement of measured units at the post- treatment condition led to a positive effect size. According to Bortz and Doring (1995) effects are strong if the effect sizes are 0.8 or bigger.
3In the employed 9-point scale score ,,1” meant highly natural-sounding speech and score ,, 9” meant highly unnatural sounding speech.
4The Pearson correlation coefficient was calculated in the way that an improvement of measured units at the post-treatment condition led to a positive coefficient.
5Calculation of Pearson correlation coefficients between differences in stuttering frequency and differences in questionnaires’ scores between pre- and post-treatment condition for adults was based on those 34 clients who had filled in all 4 questionnaires.
Andrews, G. & Cutler, J. (1974). Stuttering therapy, The relation between changes in symptom level and attitudes. Journal of Speech and Hearing Disorders, 39, 312-319.
Bortz, 1., & Dohring, N. (1995). Forschungsmethoden und Evaluation fur Sozialwissenschaftler. 2. Auflage. Berlin: Springer-Verlag. -
Brutten, E.J (1973). Behaviour assessment and the strategy of therapy. In: Y. Lebrun, and R. Hoops, (Eds.): Neurolinguistic approaches to stuttering, (pp. 82-89). The Hague: Mouton.
Cooper, E.B. & Cooper, C.S. (1998). The impact of identifying preferred treatment outcomes on conceptualizing, assessing, and treating chronic stuttering. unpublished work..
Erickson, R.L. (1969). Assessing communication attitudes among stutterers. Journal of .Speech and Hearing Research J 2, 711-724.
Fiedler, P. & Standop., R. (1986). Stottern. Atiologie, Diagnose, Behandlung. Weinheim: Psychologie-Verl.-Union.
Frischmuth, M. (1995). Untersuchung des Einflusses einer verhaltenstherapeutischen Gruppentherapie nach Van Riper auf Verschiedene Verhaltensbereiche des Stotterns - unter Beriicksichtigung der Situationsabhangigkeit von Veranderungen. Unpublished draft, UniversitÃ©it Hamburg, 1978; nach J.A. Renner: Erfolg in der Stottertherapie, Edition Marhold, Berlin.
Jehle, P., Kuehn, T. & Renner, J .A. (1989). Einstellungen Stotternder und Nicht-Stotternder zur Kommunikation: Einige Ergebnisse aus der Anwendung der Skala "S 24” von Erickson und Andrews/Cutler. Die Sprachheilarbeit 34: 121-128.
Langefeld, S., Bosshardt, H.-G., Natke, U., Oertle, H.M., & Sandrieser, P. (2001) The German program for the evaluation of stuttering therapies (PEVOS). In H. G. Bosshardt, J. S. Yaruss & H. F. M. Peters (Eds.), Fluency Disorders: Theory, Research, Treatment and Self-help. Proceedings of the Third World Congress of Fluency Disorders in Nyborg, Denmark. (pp. 359-360). Nijmegen: Nijmegen University Press.
Martin, R.R., Haroldson, S.K., & Triden, K.S. (1984). Stuttering and speech naturalness. Journal of Speech and Hearing Disorders, 49, 23-58.
Oertle, H. M. (1999). Fragebogen fiir Schiller und Schiilerinnen (FF-SS)/ Fragebogen fur Eltem (FF-E), unpublished questionnaire.
Oertle, H.M., Natke, U., Bosshardt, H.G., & Sandrieser, P. (2001). Wie wirkt Stottertherapie? PEVOS, das Programm zur Evaluation von Stottertherapien der Bundesvereinigung Stotterer- Selbsthilfe e.V.. Sprache Stimme Gehor 25, 39-41.
Woolf, G. (1967). The assessment of stuttering as struggle, avoidance and expectancy. British Journal of Disorders of Communication, 2, 158-171.
Yairi, E. (1997). Disfluency characteristics of childhood stuttering. In: R. F. Curlee and G. M. Siegel (Eds.), Nature and treatment of stuttering. (pp. 49-78) Needham Heights: Allyn & Bacon.