Publication Details
Title: Confidence-Based Scoring: A Useful Diagnostic Tool for Detection Tasks
Author: T. J. Tsai and A. Janin
Bibliographic Information: Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech 2013), Lyon, France
Date: August 2013
Research Area: Speech
Type: Article in conference proceedings
PDF: https://www.icsi.berkeley.edu/pubs/speech/ConfidenceBasedScoring13.pdf
Overview:
This paper uses an unconventional analysis as a tool to diagnose the problems with three different speech activity detection systems. The unconventional analysis is to score the frames in an audio file in order of confidence, starting with the frame that we have the most confidence in and progressing towards less and less confident frames. By keeping track of the cumulative number of errors, we can determine how the errors are distributed across the data. Using speech activity detection on highly degraded audio as a case example, we show how this simple analysis can yield useful insight into system performance. In our case example, we use the analysis to establish that (1) a small percentage of the frames account for a lion’s share of the errors, (2) three different systems perform very poorly on the same small subset of ‘hard’ data, and (3) the ‘hard’ data is primarily characterized by its proximity to speech-nonspeech boundaries. Through follow-up analyses, we show that the boundaries are ‘smoothly’ hard, and that scoring collars alone are not enough to handle the problem. Through this case example, we demonstrate the utility of confidence-based scoring as a general diagnostic tool for detection tasks on time-series data.
Acknowledgements:
This work was partially supported by funding provided to ICSI by the U.S. Defense Advanced Research Projects Agency (DARPA) under contract number D10PC20024. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of DARPA or of the U.S. Government.
Bibliographic Reference:
T. J. Tsai and A. Janin. Confidence-Based Scoring: A Useful Diagnostic Tool for Detection Tasks. Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech 2013), Lyon, France, August 2013
Author: T. J. Tsai and A. Janin
Bibliographic Information: Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech 2013), Lyon, France
Date: August 2013
Research Area: Speech
Type: Article in conference proceedings
PDF: https://www.icsi.berkeley.edu/pubs/speech/ConfidenceBasedScoring13.pdf
Overview:
This paper uses an unconventional analysis as a tool to diagnose the problems with three different speech activity detection systems. The unconventional analysis is to score the frames in an audio file in order of confidence, starting with the frame that we have the most confidence in and progressing towards less and less confident frames. By keeping track of the cumulative number of errors, we can determine how the errors are distributed across the data. Using speech activity detection on highly degraded audio as a case example, we show how this simple analysis can yield useful insight into system performance. In our case example, we use the analysis to establish that (1) a small percentage of the frames account for a lion’s share of the errors, (2) three different systems perform very poorly on the same small subset of ‘hard’ data, and (3) the ‘hard’ data is primarily characterized by its proximity to speech-nonspeech boundaries. Through follow-up analyses, we show that the boundaries are ‘smoothly’ hard, and that scoring collars alone are not enough to handle the problem. Through this case example, we demonstrate the utility of confidence-based scoring as a general diagnostic tool for detection tasks on time-series data.
Acknowledgements:
This work was partially supported by funding provided to ICSI by the U.S. Defense Advanced Research Projects Agency (DARPA) under contract number D10PC20024. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of DARPA or of the U.S. Government.
Bibliographic Reference:
T. J. Tsai and A. Janin. Confidence-Based Scoring: A Useful Diagnostic Tool for Detection Tasks. Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech 2013), Lyon, France, August 2013