Podcasting has grown large enough and fast enough to garner the attention of the larger marketing research firms, and the latest 2007 results are in.

2007 Arbitron/Edison Media Study on Podcast Consumers, 
Slide Show Summary
 (PDF)

Comscore's results of an iTunes podcast audience study, May 2007.

And, some results from last year as well as 2005, for comparison:

PEW Internet & American Life Project's
 Podcast Downloading Research Report, Nov 2006

Arbitron/Edison Media Study on Podcast Consumers, 2006.

Older Stats:
Google Answer from 2005 for podcasting stats - with lots of leads to track down new stats:
http://answers.google.com/answers/threadview?id=597303

Also from 2005:
http://www.pewinternet.org/pdfs/PIP_podcasting2005.pdf