Redundancy is a property that information theory attributes to messages.
Information theory is a postulate that explains how data is processed and measured when information is transmitted . Also called the mathematical theory of communication , it was developed by biologist Warren Weaver and electrical engineer and mathematician Claude Shannon in the late 1940s .
Before moving forward, it is useful to analyze the terms that make up the concept. A theory is a set of principles or precepts that allow linking and explaining phenomena. The notion can also refer to a hypothesis or speculation. Information , meanwhile, is knowledge, news or an indication.
It can be said that information theory analyzes the functioning of communication systems , taking into account the role of the sender and the receiver. It also considers the circulation of messages through a channel.
History of information theory
The history of information theory began long before the works of Weaver and Shannon . As always happens in the field of science, these theorists relied on research and publications by other specialists to advance the construction of knowledge.
Andrei Markov is mentioned as one of the precursors thanks to his studies on probabilities and randomness at the end of the 19th century and the beginning of the 20th century . Then came the contributions of Ralph Hartley , who examined how voice is transmitted over carrier waves in the late 1920s .
Another key contribution was made by Alan Turing , who is noted as a pioneer of computing and computer science. The so-called Turing machine , a scheme that is based on data processing through the manipulation of symbols, allows the simulation of the logic of the algorithms used in the computing field.
All this background made it possible for Weaver and Shannon to develop the mathematical theory of communication. It was Shannon who, in 1948 , published an article entitled "A Mathematical Theory of Communication " in The Bell System Technical Journal .
Information theory, in this framework, emerged with the aim of finding efficient ways to take advantage of communication channels for the transmission of messages without loss of quality (that is, without interference generated by noise).
For information theory, channel coding minimizes the probability of error in communication.
Its components
Information theory recognizes various components or elements that are part of a communication process.
The message is the set of data that is transmitted. Said information is transported through communication channels (the means that make transmission possible) and is interpreted according to the rules of a certain code .
The one who sends the message, on the other hand, is the sender . Within the framework of information theory, the sender is understood as the information source. The receiver , meanwhile, is the one who receives the message.
Another central element is noise . These are the causes that lead to the message not reaching the recipient correctly, who for this reason cannot understand it. There are different types of noise, such as white noise (a random signal whose values at two different times do not maintain a statistical correlation) or Gaussian noise (which has a probability density that is identical to that of a normal distribution).
Information theory analyzes data compression to promote the transmission of information.
Features of information theory
Information theory examines the development of the information process, pointing out what is the simplest and most effective way to transmit a message. That is why it warns about noise, looks for which channels are the most suitable and analyzes the possible understanding of the data .
According to this vision, the sender and the receiver must be able to carry out both the encoding and decoding of the messages.
It should be noted that, at a conceptual level, information is made up of variables capable of adopting different values : this implies that, by observing the value of a variable, information is obtained. The idea of information entropy is associated with the quantification of the different possible values that a variable contains.
Information theory, based on mathematical probability , maintains that information must be proportional to the number of bits required for the recognition of the message. It must be taken into account that a bit is a unit that measures the amount of information and is equivalent to the choice between two alternatives that present identical probability. It must also be considered that messages can be represented, in this context, with a series of zeros ( 0 ) and ones ( 1 ).
Continuing with the analysis, it can be stated that a message has many meanings : it is the receiver who, by sharing the code with the sender, gives it meaning. Information is a bit when, in the communicative process, there are alternatives that are equally probable. As there are more alternatives, and as long as they all have the same probability, there is more information.
It is interesting to note that the principles of information theory have multiple applications. They appear in very diverse fields such as computer science (with regard to cryptography, for example) and biology (the genetic code and deoxyribonucleic acid sequencing), as well as statistics and engineering .