Improving Result Diversity using Probabilistic Latent Semantic Indexing

ABSTRACT
The diversity of a set of search results reflects that set’s coverage of multiple interpretations of the query it is based upon. Interest in search result diversity has increased as search engines struggle to return relevant information for ambiguous queries from an increasingly large data set. We develop a result diversification system which uses probabilistic latent semantic indexing to form clusters that are then used to reorder search results and increase result list diversity scores. We show that adjusting the influence of rank in the reordering algorithms improves their performance by finding a balance between the importance of rank and the importance of generated clusters. Finally, by applying the result diversification system to the known clusters used in judging, we generate reordered lists that estimate upper- bounds of the diversity scores.

Written for Advanced Information Retrieval taught by Marteen de Rijke.
Download the full paper [PDF].

Along the IJmeer

Helioid Preview

A couple months ago we opened up the Helioid development version to the public, but haven't yet announced it. There are significant performance problems stemming partly from limited server resources and partly from core algorithms being written in Ruby, which can't perform as a fast as a compiled language, such as C. There is a lot to still be done, but we believe this reflects a healthy starting point.

Please let us know if there are any suggestions, comments, criticisms, complaints, anything. The Helioid preview is accessible here: http://dev.helioid.com.

Main Loop

November 8th, 2006

Beneath these intentions, these tendencies, actions and reasons lies a complex tangle of biochemical processes, neurocomputational methods, quantum mechanical properties – or whatever the current explanatory fad happens to be. I persist in attempting to convince myself that I – my self, my conscious, my will – am in control, am authoritative, and maintain ascendancy. However the more effort I put into this belief the more obvious it becomes that it is untenable or at least paradoxical.

“But according to whose logic?” One may ask. Quickly I respond: “why to the only relevant logic, my logic.” Just as quickly it dawns upon me that I have once again fallen into the trap. For if we suppose that I am not in control of my reality and further suppose the supposition that I am in control of my reality is illogical it appears inconsistent to hold that “my logic” is the relevant logic when my logic is not “mine” at all but, from our first presumption, must be the logic of the reality imposed on me. This notion, this imposition of a reality, is this at all sensible or has there been some misstep taken in the core of my thinking to lead to such a contradictory statement? Although, perhaps it is not contradictory. Perhaps a clarification of terms is necessary. My reality is the world and the things as it and I perceive them. It is inescapable, it is ever present, and it is apparently poetic. Would reality be any less imposed upon me if I were a brain in a vat being fed sensory stimuli by some sort of machinery? It seems that it is a question of influence. How much influence can I have over the reality, how much interaction do I have with this reality?

The question then is not what is real but why is it real?

[December 5th, 2009 - I've now decided the question doesn't exist, can't be asked, or doesn't matter]

Jannette Pulido's Artwork

Above 3 photos copyright 2008 Jannette Pulido. All Rights Reserved.

African Cosmologies - Tabwa

The summer of 2006 I worked with Manuel Jordan Perez researching the cosmological systems of civilizations in Africa. I have copied below text and diagrams from research done on the Tabwa civilization. My best to Manuel who led me in this research. Below is a chart of Tabwa cosmology and cosmogony:

TABWA COSMOLOGY

Central Ideas

1. 1. Duality – most things have a positive and negative side to them these opposing sides are synthesized into a representative whole that is the manifest of the thing.


1. a. Moon


1. i. Positively may provide light and allow for safety, knowledge, and order to be perceived out of chaos and obscurity.


2. ii. Negatively may maliciously withhold its light.


3. iii. Goes through cycles but always containing within it positive and negative aspects


2. b. Chiefs


1. i. Offer knowledge and wisdom allowing people to perceive and defend themselves against danger.


2. ii. May be responsible for allowing evil to exist, example “Drunken King” Nkongolo Mwamba a tyrant and king of excess embodied by the sun.


3. iii. Must exist balancing positive and negative forces and events
1. 1. Use lunar symbolism such as the Mpande shell to represent their success at this balancing.


3. c. Spirits


1. i. Mipasi


1. 1. May be identified by an ng’anga as the source of an affliction in which case mikisi will be commissioned to represented the mipasi


2. 2. May provide guidance through dreams serving as a source of wisdom and inspiration


2. ii. Mikisi


1. 1. When ignored they gain attention through “-kisa” (the root verb) which means, “to cause suffering or bring misfortune”


2. 2. “-kisa” also refers to the power that knowledge and the recognition of knowledge lends one.


3. 3. These figures therefore represent the duality of the spirits effecting them.


3. iii. Mutumbi


1. 1. Nearly blinded by daylight so rarely seen and apparently “invisible”.


2. 2. Yet lots of tracks seen so must exist.


3. 3. Similar to moon it this “invisible” but existent fashion.


4. iv. Twins


1. 1. Take lunar names and are covered in ashes hiding their differences from each other when the new moon has risen.


2. 2. Celebrated and given gifts in honor of their positive side.


3. 3. Beaten with a switch and feared because of their negative side: the possibility that they could use their powers for evil.

2. 2. Mulalambo – “endless lines of symmetry” serve to create divisions between the parts of things yet at the same time unite and offer reconciliation between the divided parts of these things. An “endless line” is a division that exists yet cannot be felt.



1. a. Mutumbi’s Tunnels


1. i. Mutumbi followed his dogs down a “seemingly endless tunnel” to the find the supreme deity, Leza, in the city of light.


2. b. Body Midline
1. i. Defines the symmetry and opposition existent in the human condition.
2. ii. There exist half-beings that represent cases of unresolved litigation in need of arbitration and the deployment of social justice.
3. iii. Myth of great leader associated with hunting whose body split along the midline and is composed of half beeswax and half flesh.


4. iv. This recognition of division symbolizes the unification of the body
3. c. Natural Formations
1. i. Mwila divide
2. ii. Lake Tanganyika
3. iii. Milkyway

Duality and Symmetry

Everyone and everything has power. It appears as though they are both made up of precisely the same parts just being viewed from different perspectives.

Below is a chart of Tabwa cosmology as based on opposition:

Below is a chart of the Balamwezi cosmology integrated into a Balamwezi pattern:

Below is a chart of the Balamwezi cosmology:

Structure (or lack there of)

Structure (or lack there of)

September 5th, 2008

Throughout the slowly decaying cortices of our humble: understanding abounds, knowledge surmises, meaning exists.

Shadows, claimed. Nothing more, still! We create? Still, levied over restlessness, challenged to remove, to “recompose” the, an original. An? True? Still must this not hear left false to the failure of transcription. For, it does did not rise there. It was, were it lay, stayed it mopped up without clarity, cleanliness. That, it has gone for to, realism. There is none, still. The access, entrance, possibility, destroyed through pliability. Left still back once, through creatibility. Back once more, through possibility. There back will always be somewhere more to go, right there forward, still rift the similarity, identically. Then still! Of what interest have we more to? For truly, grammatically, it was erased best, the end will be something only to be spoken of for when or if or past it has arisen then more, back will be for a turning point in the foundation. The expressability through it’s destruction can reach its. So then is attempted, a limitation on the limits? Placed by, the, there is one place that it can cannot. Arisen limits, imposed, created, irrelevant and still irrelevant. Usage lost to users importantly the unavoidable inversion.


October 25th, 2008

Then the discord, concealed so effectively tightly, would see light only through its inherent obscurity.

Is there a deeper truth to this? Is there some “fundamental framework” upon which we are building, out of which our lives – our minds – are created? Suppose the existence of a framework is as chimerical as a blue sky. Between the grimaces of oppressive skyscrapers our blue sky is not more than light shifting spectrum as it moves through humidity in our atmosphere, is not more than water modifying the wavelengths of photons. Ad infinitum we learn our precious sky is farther and farther from our comprehension. And soon that slips away as well, our comprehension, which we had been able to use so fluently to instruct us on our lack of comprehension, fails us. And finally a statement we could comprehend, that we cannot comprehend our lack of comprehension, is lost forever. But the framework, despite our tranquilization by the recursive, does it, hiding from the unlikely afflatus, remain? Would a proof, or a consensus by our world’s comparative literature departments, as to the frameworks impossibility have any effect on its existence or nonexistence?

Would the existence of this superstructure impose rules upon what can be built and the manner in which it can be built? Supposing that the answer to this question is yes, or at the least “maybe”, then there would exist an incentive to search for this truth or these truths, to discover the nature of this framework and investigate its properties so that we may better understand ourselves and our world. I inherently assume that the nature is hidden, that it must be “discovered”. (I additionally assume, or at the least heavily imply, the standard western distinction between being and existence.)

Would a simple existentialist standpoint solve all our problems? It could claim that the truths that need to be “discovered” and the “framework” underlying our world are inventions we have imposed upon the world and upon ourselves. The “deeper truth”, if I dare keep to this terminology, is that there is no deeper truth (or fundamental truth, or ultimate truth). The world is at it appears, the answers we search and yearn for are (1) non-existent or (2) answered by this statement. The second option seems entirely unsatisfying and is problematic in its circularity and that it is facially self-eliminative, yet deeply not. The first option begs the question, if the deeper truth’s non-existence answers the question of what is the deeper truth then the deeper truth does indeed have an existence, that of non-existence. And, quite directly, we are led to the transconsistent or: the same circularity of the second option.

[original with typos removed]