Entropy and Fractals: Search Google and the number of links will be enormous. But, is there a connection? How are entropy and fractals related? If they are related, what are the implications of the relation? Everything in GREEN is copyrighted.
Sunday, March 27, 2005
Multifractal
Two (beautiful) interfering fractals could produce cacophony. In other contexts, separate fractals would not necessarily produce either destructive or constructive interference. Separation and isolation of each fractal could involve a kind of signal filtering.
False Fractals
Captain Ed (link) writes, "the peculiar desire in humanity to see patterns and conspiracies where none exists always made us vulnerable to Gnostic-like fables." His post is a commentary on the Church's rebuke of the "Da Vinci Code". I suggest that the conspiracy-susceptibles are seduced by "false fractals". The typical susceptibles see an apparent pattern, one for which they are on the lookout. The template for the anti-W suscepts consists of a set of "axioms": W. is dumb. He takes his marching orders from the smart, but evil people around him: Rove, Cheney, Rumsfeld. If W. clearly succeeds, there's a dark, not silver-lining to the invisible cloud that doesn't obscure the success. A false fractal could consist of a triangle embedded in a pentagon embedded in a square. See the pattern!
Happy Easter!
Monday, March 14, 2005
Equal Probability
Equal Probabilities from Shannon Entropy
Note that the probabilities are all equal to 1/n, and the Lagrange multiplier is equal to -log (n).
Friday, March 11, 2005
Jump into the middle...
Simple, start at the bottom, and work your way up.
Tuesday, March 01, 2005
And a Non-Fractal
"We think your mission is corrupt, pointless, based on lies and a waste, and we make that argument loudly and frequently. But considering the views of many of those who buy the tickets to our movies, we also feel it is important that we appear to support you anyway."
Middle East Fractality
Iraq, Lebanon, Saudi Arabia, Egypt... (link) Add in Jordan.
The question is whether it's the same signal or not -- the rise of democracy in the Arab world. Add in Afganistan, and it's the Muslim world.
Monday, February 28, 2005
Flea fractality
"So each blogger in his kind is bit by him who comes behind."
Wrote Swift:
"So, naturalists observe, a flea
Has smaller fleas that on him prey;
And these have smaller still to bite ’em;
And so proceed ad infinitum." (link)
Swift's little gem describes a fractal - progressively smaller and smaller chomping insects.
I suggest that Captain Ed's blog manifests fractal structure, as well, particularly when contrasted with a recent Jonathan Chait column he critiqued (link).
The Cap'n quotes Chait: "What's uncanny about the Bush administration is that its dissidents invariably recant, usually in zombie-like fashion." He then notes, "Wead has never been a 'dissident' in the Bush administration -- he's never even held a job with Bush, either in a campaign or in the administration."
So what does this have to do with fractals? Chait tried to interpret the Wead imbroglio as the remarkable ability of Bush & Co. to deal with its problem people. Chait created a template based on one supposed example and then applied it to a second. He thought he saw the simplest of fractals, but, the Cap'n called him on it. The structures in either case Chait cited didn't fit the defined template, but Chait applied it anyway.
Consider the recent Eason Jordan saga in which Captain Ed (with a number of other bloggers) played a prominent role (link). At first, bloggers were intrigued by the intitial account of the Davos meeting and began chasing confirmations from others present (and got them). Of particular importance were confirmations from "non-conservatives", even given that the post that started the swarm was by an essentially apolitical person. But, more importantly, was recognition of the importance of previous comments by Mr. Eason, especially in Lisbon. So self-similarity was found in multiple attestations of the Davos comments and in previous comments in foreign venues. The bloggers developed a fractal that ensnared a prominent newsman.
Symbolically Challenged
When it comes to concepts like fractals, a coffee table book might be acceptable -- pictures with an apparent, if questionable, aesthetic appeal based on some mumbo-jumbo created by somebody named Mandelbrodt. Didn't he have something to do with the IBM PC?
Finally, there's entropy. A post-impressionist or even modernist word, isn't it -- sort of a rationalization for Roaring Twenties degeneracy or Seventies consciousness-raising: "not with a bang, but a whimper."
There's been a kind of unconscious feeling among many that there's a connection between entropy and fractals. Google "entropy fractal". At the time of this post, I get 117,000 hits. Delve more deeply however into these links, and the connection (with few exceptions) is esoteric at best.
Fractals are peculiar components of natural or human-created structures, phenomena, or processes that show self-similarity at a variety of scales.
Entropy can be thought of as a measure of ignorance or, conversely, information.
In a fractal, repetition of a structural leit motif over multiple scales implies that the particular information nugget that describes the leit motif exists at each scale. In a human creation the leit motif might be constructed by algorithm or formula.
The technique of maximum entropy, popularized by physicist Edwin Jaynes (1922-1998), is a method of interpreting a phenomenon (or process) for which some mathematical structure is inferred and some "inadequate" constraints are known. By application of the technique (or principle), the most probable state of the phenomenon given the constraints may be derived.
In the case of fractals, their occurrence suggests a possible information content constraint at each scale. In a particular geographic area (e.g., Southern California) earthquake magnitudes and frequencies show fractal behavior over a broad range of earthquake energies. This implies that the processes that generate earthquakes (sudden fracturing of rock under dynamic stress) operate in a consistent manner over a wide scale. Small areas have the same kind of distribution of earthquake magnitudes and frequencies as large areas. Small earthquakes occur more frequently than large earthquakes.
It took Pastor-Satorras and Wagensburg (link) to demonstrate the mathematic connection between entropy and fractals. However, intuition can be used to understand the rationality of the connection (without equations).
Knowing something concrete about a process, having some knowledge of the likely structure of the process, the "most probable" state of the process might be inferred by applying the principle of maximum entropy. In the case of a chaotic process which demonstrates scale-invariant self-similarity we may well recognize entropy (the most probable) at work.
Saturday, February 26, 2005
Intuitive Leap
With the more recent connection of fractals and maximum entropy (link), and the existence of fractals in both the natural and human-created world, it is becoming difficult to separate thermodynamic and information entropy. But, in the world of human creativity, to speak of either fractals or entropy is to introduce an alien conceptualization.
Fractals are structures, real or imagined, in which there is scale-independent self-similarity. Look at a coastline from a distance, then up close: the structure -- the ins and outs -- may well appear to be very similar at the two scales. It's the same thing with a stock exchange index. Variability in stock price from minute to minute may appear no different day to day. The latter is very much a human-created process, even if it seems to lack any rational underpinning. The former is natural, although observation and characterization of a coastline, by definition, requires an observer and characterizer.
Fractals appear in music -- harmony itself and repetition of the musical scale are intrinsically fractal. Add in additional accompaniment to the melody and harmony, notes of varied duration, and phrasing: whether classical or popular, fractal structures emerge (link).
Fractals appear in Tolkien, both in the grand myth of creation (link) and in the saga of the Hobbits and the Ring. Spiders, big and small, appear and reappear. Villains great and petty confront heroes grand and humble.
Shakespeare's comedies incorporate rhyme, verse by verse, and revisited themes of misunderstood and misplaced love. Puns abound and are, in fact, are the most surprising of fractals, as they surface multiple meanings of contrary connotation and context. More generally, comedy can be seen as a larger scale fractal of perversely associated ideas and events.
Fractals Everywhere -- a coffee table book -- doesn't know the half of it, especially in light of the fratal-entropy insight. Entropy is a human way of understanding the deeper manner in which the complexity of the universe is organized. Given that which is known, measured, or understood, that which is not known, measurable or understood can still be assumed to behave reasonably. In a sense, maximum entropy maximizes our ignorance while preserving that which we do in fact know.
Maximum entropy constrained by our knowledge is most profoundly reasonable.
Friday, February 25, 2005
Conservative >> Fractal; Liberal >> Nonfractal
Here is the hypothesis: The discrepancy between conservative and liberal might be the difference between fractal and non-fractal (or the degree of fractality). The title is to be interpreted as Conservative implies fractality; Liberalism (contemporary) implies nonfractality.
Example: Consider a recent post by relatively conservative Ed Morrissey (Captain Ed; link), a discussion of a column by relatively liberal Jonathan Chait (link). (Important: This is an analysis of Morrissey's interpretation of Chait not of Chait's column independent of Morrissey.)
The Captain quotes Chait:
"'What's uncanny about the Bush administration is that its dissidents invariably recant, usually in zombie-like fashion.'"
And then, Cap'n notes,
"Wead has never been a 'dissident' in the Bush administration".
Wead "never even held a job with Bush", though "he certainly tried his best to get a position with him."
So where is the fractality and what is its significance?
The point is, in a fractal, there is self similarity at multiple scales within a phenomenon, process, creative product, or natural observation. In the Cap'n's (note the two apostrophes!) analysis, the "self-similarity" claimed by Chait occurs at only two levels, not the multiple levels between.
The same generalization applies to the Eason Jordan flap. First, several of those present at the meeting "heard" the same thing. Then, as the Captain and others discovered, Mr. Jordan had made similar accusations in other foreign venues. There was a consistency of inconsistency. But, the MSM refused to see the fractality. To their credit some liberal blogs acknowledged the evidence that other Jordan bloggers noted.
I assert that liberalism simulates fractality by recognizing apparent similarities at NO MORE THAN TWO LEVELS, and, such "similarities" apply only to non-liberals. The conservative looks for self-similarity (or dissimilarity) at multiple levels. What could be more reasonable? What could make more sense?
[Alternatively, liberals and conservatives see things through different fractals. But, since I can't see the liberal fractal, I'm not sure.]
Look at it in another way. Consider the apparent shortcomings of a recent President. It was asserted that his private actions, while clearly discordant with public "perceptions", were irrelevant. There need be no correlation. Missiles hitting pharmaceutical factories and camel tents have no connection with the underdesk of the Oval Office and a blue dress.
Then, there are the tapes of Mr. Wead. Curiously, there is really nothing new there (so far). What we see as the public President Bush is entirely consistent with what Mr. Wead managed to extract surrepitiously (while, simultaneously apparently attempting to acquire some advantage).
The authenticity treasured by the true conservative is a consistency in every level of life. No coincidence: this is a key component of Christianity, as derived from Judaism. That which is hidden will be revealed (therefore, make sure that that which is hidden is worthy of such revelation -- thus consistent with what is visible). And, that which was of value in prior ages does not lose value in the present. Whitewashed tombs can, conversely, conceal rot within.
Postmodernism (the implicit presumption of contemporary liberalism) declines to accept any kind of self-consistency in its art or science. Even the orthodoxies of previously contemporary liberalism, e.g., Darwinism, are not immune to attack.
Recent analysis from statistical mechanics and information theory shows that fractality -- self similarity at all relevant levels -- is the most probable stable state of a complex system (link). Coincidentally (but not really), in Moral Philosophy, the person of integrity is the one who demonstrates consistency of goodness at every level of life.
Intuitive inference: The discordance between left and right is that between the unnatural and the natural, between the fractal and the nonfractal. Catholic interpretation: God's creation is good (at every level); the corruption due to the sin of the first humans is superficial, not pervasive. If we reclaim, through the great gift of the Passion, Death, and Resurrection, what was intended in the beginning, the goodness of God's creation permeates all of creation at every level.
All of this from a single blogger's post? Hey, Cap'n Ed is Catholic, isn't he?
And Catholics believe that all that scientists authentically discovery of God's creation will only enhance the his glory and our understanding of his goodness.
(Best wishes for the recover of First Mate.)
Wednesday, January 12, 2005
Music Fractality
Tuesday, January 11, 2005
Fractal structure transmits information
Besides clouds, trees, shorelines, earthquake magnitudes, mammalian circulatory systems ...
There are human creations that are also fractal, especially music (and not just fabricated "fractal" music). A Bach fugue is fractal. The typical form a popular song (ABAA) is fractal. A phrase (A or B) may be four measures (sixteen measures comprise a chorus). Within each measure (in 4/4) there are full, half, quarter ... notes. Look at the musical patterns, and quite often there is repetition (not only A and A or B and B) of note sequences at different rhythms.
Then there's a chord. A major chord includes harmonically consistent notes, with compatible harmonic vibrations. They "fit" together.
Then, consider an arrangement. Say, Nelson Riddle of Fifties Sinatra. There's the repetition, and then background melodies that contrast with the primary melody, but a fractal content is very much there.
Is Tolkien's Creation Fractal?
Then the voices of the Ainur … began to fashion the theme of Ilúvatar to a great music; and a sound arose of endless interchanging melodies woven in heights … and the music and the echo of the music went out into the Void, and it was not void. …
But as the theme progressed, it came into the heart of Melkor to interweave matters of his own imagining that were not in accord with the theme of Iluvatar; for he sought therein to increase the power and glory of the part assigned to himself. …
Some of these thoughts he now wove into his music, and straightway discord arose about him, and … some began to attune their music to his rather than to the thought which they had at first. Then the discord of Melkor spread ever wider, and the melodies which had been heard before foundered in a sea of turbulent sound. …
Then Ilúvatar arose, and … a new theme began amid the storm, like and yet unlike to the former theme, and it gathered power and had new beauty. But the discord of Melkor rose in uproar and contended with it, and again there was a war of sound more violent than before…
Then again Ilúvatar arose, and … behold! A third theme grew amid the confusion, and it was unlike the others. … [I]t could not be quenched, and it took to itself power and profundity. And it seemed at last that there were two musics progressing at one time … and they were utterly at variance. The one was deep and wide and beautiful, but slow and blended with an immeasurable sorrow, form which its beauty chiefly came. The other had now achieved a unity of its own; but it was loud, and vain, and endlessly repeated; and it had little harmony, but rather a clamorous unison as of many trumpets braying upon a few notes. And it essayed to drown the other music by the violence of his voice, but it seemed that its most triumphant notes were taken by the other and woven into its own solemn pattern. …
… Ilúvatar spoke … ‘And thou, Melkor, shalt see that no theme may be played that hath not its uttermost source in me, nor can any alter the music in my despite.’
Tolkien, J.R.R., The Silmarillion, Christopher Tolkien, editor, Houghton Mifflin Company, Boston, 1977, p. 15-17.
So where is the fractal? In the Music: Theme, endless interchanging melodies, echo, beauty, power and profundity
And the anti-fractal? Interwoven discord, foundered melodies, turbulent sound, variance, loud, vain, endlessly repeated, little harmony, clamorous unison.
But, can't a fractal emerge from turbulence? ... triumphant notes were taken by the other and woven into its own solemn pattern.
Fractals Maximize Entropy
Shannon Entropy:
H(P) = −S(k) [p(k) ln p(k)]
Probabililities, by definition sum to unity:
S(k) = ln p(k) = 1
An element of order k, has a magnitude ℓ(k). N(k) = ℓ(k) / ε indistinguishable atoms of size ε, arranged in a certain way, comprise the element. The information needed to specify the arrangement of the N(k) atoms in the element is equivalent to selecting N(k) objects with the same probability. From information theory[Shannon, 1948] this information is ln N(k). The generating information for specifying an element of order k is ℓ(k) = ln N(k) = ln [ℓ(k)/ε]. The average information over the entire iteration process P is:
= S(k) p(k) I(k) =S(k) p(k) ln [ℓ(k)/ε]
F = −S(k) [p(k) ln p(k)] + β { −S(k) p(k) ln [ℓ(k) / ε ]}+ β′ [1 − S(k) p(k)]
∂F/∂pk = 0 = −ln p(k) − 1 − β ln [ℓ(k) / ε] − β′ = 0.
p(k) = exp {− 1 − β′ [ℓ(k) / ε] −β}
p(k) ~ n(k),
n(k) = const. × (ℓ(k) / ε) − β
“[T]he occupation numbers scale as a power of ℓ(k), which implies a self-similar behaviour” (Pastor-Satorras and Wagensberg, 1998).
Thus, Fractals are evidence for maximization of entropy (T-Rex's conclusion).
Information Entropy: A Derivation
Given a set of M symbols, x(i), they can be combined together to make up a set of messages. The probability of occurrence of any symbol, xi, is given by p[x(i)] with total probability of unity.
S { p[x(i)], i=1,M } = 1 (1)
A measure of the information gained from a message is the minimum number of yes and no questions required to identify the message. Thus if the message is a number less than 64, and each number has an equal probability of 1/64 of occurring, we would first ask if the number is less than or equal to 32. If the answer is “yes”, we could ask if it is less than or equal to 16. If “no”, we could ask it is greater than 16 (and, implicitly, less than r equal to 32), and so on. Thus the number of questions with two possible answers is six. Note that each question adds one unit of information and a convenient measure in a binary system which is additive is the logarithm to base 2 [“log2”; 1 unit = log2 (2)]. The addition of 6 such units is obviously (^ implies exponent)
6 log2 (2) = log2 (2^6) = log2 (64) = 6.
Define a quantity called self information, S, which is log of the reciprocal of the probability of occurrence.
S[x(i)] = log {1/p[x(i)]} = - log {p[x(i)]} (2)
Assigning base 2, for the example above S[x(i)] = 6.
Properties of S[x(i)] and implications for message interpretation:
a) Decreases if a symbol occurs with greater probability.
b) A rare event contains a more significant message.
c) If the p(xi) = 1, then the S[x(i)] = 0.
d) If a message consists of an infinite string of yeses, there is no information or significance to this.
e) Further, if all probabilities are bracketed, 0 <> 0.
Over a long period of time, T, occurrence of x is p(xi)T; thus the total information obtained about the system is
–p[x(1)]T log2{p[x(1)]} – p[x(2)]T log2{p[x(2)]} – ….
The average information per unit time, also called the expectation value, E, of S(x), in which x = [x(1), x(2), x(3), … , x(M)], is then
E [S(x)] = S { p[x(i)] S[x(i)], i=1, M } (3)
And this termed, by analogy with Gibbs thermodynamics, entropy.
H = – S {p[x(i)] log p[x(i)], i=1, M} (4)
In thermodynamics H (with a multiplicative constant) measures the degree of disorder in a physical process. Here it measures the uncertainty of the system in a particular state or the “ignorance”. The base of the logarithm can be any convenient number but is logically 2 if a binary arithmetic is used. In the example, yes = 1 and no = 0.
Entropy of a discrete distribution is a maximum when probabilities are all equal. Thus for a binary system that only transmits a 0 and 1, the probabilities are p and 1-p.
H = – p log2(p) – (1 – p) log2(1 – p) (5)
The entropy is a maximum when p = ½; H is, therefore, 1.
If the variance of x is a constant, s^2, then the entropy is a maximum if the probability has a normal or Gaussian distribution.
p(x) = (2ps^2) ^(-½) exp ( –x^2/2s^2) (6)
The entropy for the Gaussian case is
H = (½) ln [2p exp (s^2)] (7)
(Based on p. 146-147 of Kanasewich, E. R., 1981, Time sequence analysis in geophysics, University of Alberta Press, Edmonton, Alberta, Canada, Third Edition, 480 p.)