Starting with a brief introduction to statistical language modeling, I will proceed with a short summary of several ongoing language modeling projects at my lab.

The bulk of my talk will describe a new kind of language model, which models whole sentences or utterances directly using the Maximum Entropy (ME) paradigm. The new model is conceptually simpler, and more naturally suited to modeling whole-sentence phenomena, than the conditional ME models proposed to date. By avoiding the chain rule, the model treats each sentence or utterance as a ``bag of features'', where features are arbitrary computable properties of the sentence. The model is unnormalizable, but this does not interfere with training (done via sampling) or with use. Using the model is computationally straightforward. The main computational cost of training the model is in generating sample sentences from a Gibbs distribution. Interestingly, this cost has different dependencies, and is potentially lower, than in the comparable conditional ME model.