Posted by
Grant Holland on
Jun 06, 2011; 3:21pm
URL: http://friam.383.s1.nabble.com/Quote-of-the-week-tp6442957p6445543.html
It seems backwards to almost everybody. Me too. So much so that this
little conundrum pushed me to take a deeper look into information
theory.
The key for me was realizing that I.T. is addressing how much
information THERE IS in a "situation" (probability distribution) -
rather than how much information YOU HAVE (one has) about that
situation.
I think Owen is right: taking a look at Shannon's "The Mathematical
Theory of Communication" is good. Try to get the edition with the
Warren Weaver essay in the front - an essay
about Shannon's
paper. Weaver talks about the measure in section 2.2 (p. 9 in my
copy). He talks in terms of logs of "the number of available
choices" rather than inverses of probabilities. Weaver refers to
what is being measured as "information".
Most telling, on page 50, Shannon uses the terms "information",
"choice" and "uncertainty" in the same breath as being measured by
his entropy formula.
Another very good popular-level book is "
Decoding
Reality: The Universe as Quantum Information [2010]" by
Information Theorist Vlatko Vedral. He begins the book with this
conversation.
Grant
On 6/6/11 8:17 AM, Owen Densmore wrote:
Nick: Next you are in town, lets read the original Shannon paper together. Alas, it is a bit long, but I'm told its a Good Thing To Do.
-- Owen
On Jun 6, 2011, at 7:44 AM, Nicholas Thompson wrote:
Grant,
This seems backwards to me, but I got properly thrashed for my last few postings so I am putting my hat over the wall very carefully here.
I thought……i thought …. the information in a message was the number of bits by which the arrival of the message decreased the uncertainty of the receiver. So, let’s say you are sitting awaiting the result of a coin toss, and I am on the other end of the line flipping the coin. Before I say “heads” you have 1 bit of uncertainty; afterwards, you have none.
The reason I am particularly nervous about saying this is that it, of course, holds out the possibility of negative information. Some forms of communication, appeasement gestures in animals, for instance, have the effect of increasing the range of behaviors likely to occur in the receiver. This would seem to correspond to a negative value for the information calculation.
Nick
From: [hidden email] [[hidden email]] On Behalf Of Grant Holland
Sent: Sunday, June 05, 2011 11:07 PM
To: The Friday Morning Applied Complexity Coffee Group; Steve Smith
Subject: Re: [FRIAM] Quote of the week
Interesting note on "information" and "uncertainty"...
Information is Uncertainty. The two words are synonyms.
Shannon called it "uncertainty", contemporary Information theory calls it "information".
It is often thought that the more information there is, the less uncertainty. The opposite is the case.
In Information Theory (aka the mathematical theory of communications) , the degree of information I(E) - or uncertainty U(E) - of an event is measurable as an inverse function of its probability, as follows:
U(E) = I(E) = log( 1/Pr(E) ) = log(1) - log( Pr(E) ) = -log( Pr(E) ).
Considering I(E) as a random variable, Shannon's entropy is, in fact, the first moment (or expectation) of I(E). Shannon entropy = exp( I(E) ).
Grant
On 6/5/2011 2:20 PM, Steve Smith wrote:
"Philosophy is to physics as pornography is to sex. It's cheaper, it's easier and some people seem to prefer it."
Modern Physics is contained in Realism which is contained in Metaphysics which I contained in all of Philosophy.
I'd be tempted to counter:
"Physics is to Philosophy as the Missionary Position is to the Kama Sutra"
Physics also appeals to Phenomenology and Logic (the branch of Philosophy were Mathematics is rooted) and what we can know scientifically is constrained by Epistemology (the nature of knowledge) and phenomenology (the nature of conscious experience).
It might be fair to say that many (including many of us here) who hold Physics up in some exalted position simply dismiss or choose to ignore all the messy questions considered by *the rest of* philosophy. Even if we think we have clear/simple answers to the questions, I do not accept that the questions are not worthy of the asking.
The underlying point of the referenced podcast is, in fact, that Physics, or Science in general might be rather myopic and limited by it's own viewpoint by definition.
"The more we know, the less we understand."
Philosophy is about understanding, physics is about knowledge first and understanding only insomuch as it is a part of natural philosophy.
Or at least this is how my understanding is structured around these matters.
- Steve
On Sun, Jun 5, 2011 at 1:15 PM, Robert Holmes [hidden email] wrote:
>From the BBC's science podcast "The Infinite Monkey Cage":
"Philosophy is to physics as pornography is to sex. It's cheaper, it's easier and some people seem to prefer it."
Not to be pedantic, but I suspect that s/he has conflated "philosophy" with "new age", as much of science owes itself to philosophy.
marcos
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org
============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at
http://www.friam.org