Information Theory
lastmod 2018-08-17created 2014-06-121 edits
Information theory is a branch of applied mathematics that quantifies data in order to store as much data as possible on a medium or communicate it over a channel. Information entropy, the unit of data, is usually expressed as the average number of bits used for storage or communication. For example, if daily weather information has an entropy of 3 bits, a day's weather can be described with about 3 bits on average.
Information entropy is a concept not very different from entropy in thermodynamics: the higher the information entropy (amount of information), the greater the uncertainty. Conversely, since the amount of information in a system is the degree of uncertainty, the information entropy (amount of information) of a system with little uncertainty can decrease. For the English alphabet, if every letter occurred with equal probability, the information entropy (amount of information) would be 4.754887 bits, but calculating with the frequency of each letter reduces it to 4.08 bits. Beyond this, there are various other reasons, such as a very high probability that a particular letter follows a given letter, so the information entropy (amount of information) can be reduced even further. For example, when q appears, the probability that a,u follows is almost 100%.
Technologies that apply the basic topics of information theory include ZIP files (lossless data compression), ~MP3 files (lossy data compression), DSL (channel coding), and MIMO (channel models). The field of information theory is connected to mathematics, statistics, computer science, physics, and electronic engineering, and it has had a major influence in many areas, such as the success of the Voyager program, the invention of the compact disc, the commercialization of mobile phones, the development of the Internet, the study of linguistics and human perception, and the understanding of black holes.