The Tao of ATWV: Probing the Mysteries of Keyword Search Performance

TitleThe Tao of ATWV: Probing the Mysteries of Keyword Search Performance
Publication TypeConference Paper
Year of Publication2013
AuthorsWegmann, S., Faria A., Janin A., Riedhammer K., & Morgan N.
Other Numbers3613

In this paper we apply diagnostic analysis to gain a deeper understanding of the performance of the the keyword search system that we have developed for conversational telephone speech in the IARPA Babel program. We summarize the Babel task, its primary performance metric, ``actual term weighted value'' (ATWV), and our recognition and keyword search systems. Our analysis uses two new oracle ATWV measures, a bootstrap-based ATWV confidence interval, and includes a study of the underpinnings of the large ATWV gains due to system combination. This analysis quantifies the potential ATWV gains from improving the number of true hits and the overall quality of the detection scores in our system's posting lists. It also shows that system combination improves our systems' ATWV via a small increase in the number of true hits in the posting lists.


This work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Defense US Army Research Laboratory contract number W911NF-12-C-0014 ("Spoken WOrdsearch with Rapid Development and Frugal Invariant Subword Hierarchies - Swordfish"). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoD/ARL, or the U.S. Government

Bibliographic Notes

Proceedings of the Automatic Speech Recognition and Understanding Workshop (ASRU 2013), Olomouc, Czech Republic

Abbreviated Authors

S. Wegmann, A. Faria, A. Janin, K. Riedhammer, and N. Morgan

ICSI Research Group


ICSI Publication Type

Article in conference proceedings