Entropy and Information Theory
This book is an updated version of the information theory classic, first published in 1990. About one-third of the book is devoted to Shannon source and channel coding theorems; the remainder addresses sources, channels, and codes and on information and d
- PDF / 4,061,309 Bytes
- 429 Pages / 439.37 x 666.142 pts Page_size
- 88 Downloads / 254 Views
Robert M. Gray
Entropy and Information Theory Second Edition
Robert M. Gray Department of Electrical Engineering Stanford University Stanford, CA 94305-9510 USA [email protected]
e-ISBN 978-1-4419-7970-4 ISBN 978-1-4419-7969-8 DOI 10.1007/978-1-4419-7970-4 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2011920808 © Springer Science+Business Media, LLC 2011 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
to Tim, Lori, Julia, Peter, Gus, Amy, and Alice and in memory of Tino
Preface
This book is devoted to the theory of probabilistic information measures and their application to coding theorems for information sources and noisy channels, with a strong emphasis on source coding and stationary codes. The eventual goal is a general development of Shannon’s mathematical theory of communication for single user systems, but much of the space is devoted to the tools and methods required to prove the Shannon coding theorems, especially the notions of sources, channels, codes, entropy, information, and the entropy ergodic theorem. These tools form an area common to ergodic theory and information theory and comprise several quantitative notions of the information in random variables, random processes, and dynamical systems. Examples are entropy, mutual information, conditional entropy, conditional information, and relative entropy (discrimination, Kullback-Leibler information, informational divergence), along with the limiting normalized versions of these quantities such as entropy rate and information rate. In addition to information we will be concerned with the distance or distortion between the random objects, that is, the accuracy of the representation of one random object by another or the degree of mutual approximation. Much of the book is concerned with the properties of these quantities, especially the long term asymptotic behavior of average information and distortion, where both sample averages and probabilistic averages are of interest. The book has been strongly influenced by M. S. Pinsker’s classic Information and Information Stability of Random Variables and Processes and by the seminal work of A. N. Kolmogorov, I. M. Gelfand, A. M. Yaglom, and R. L. Dobrushin on information measures for abstract alpha
Data Loading...
