Bayesian Empire

New domain: orchycore.com

2007-04-27T22:06:00.000-07:00

I've now given this blog its very own domain name: orchycore.com and have switched to s9y software, because we've found it easy to include blogs in our other sites.

No more posts here! Off to orchycore.com thanks everyone.

Links from books the ultimate authoritative source?

2007-01-29T19:09:00.000-08:00

It just occurred to me as I was doing a vanity search that lead to Google Books: wouldn't books be the ultimate authoritative source of links for a search engine?

Books sometimes have full URLs in their footnotes like the example above. They're not links you can click on, but a search engine (like Google) that indexes books can certainly read them.

There are any number of ways of spamming websites that have arbitrary been deemed authoritative, like with form spam. But nobody, to my knowledge, has thought of negotiating links on actual tree flesh in order to get into Google? It would be a very pure source of quality links.

Federal govt copyright legislation.

2007-01-07T18:57:00.000-08:00

The Australian Internet Industry Association has published a risk analysis for different kinds of entities for the federal government's new copyright legislation. Scary stuff. Here it is for small businesses.

Pandora and art

2006-12-23T19:12:00.000-08:00

I don't remember the last time I spent 4 hours on one site. But Pandora's got me. I'm gonna blog about it again.

I really love how Pandora is telling me why I like the songs I do. Every time I click "Yes, I like it" I'm adding to my list of songs I like and refining the criteria for future songs. With each rating, it gets more articulate about my tastes, far surpassing what I could have come up with myself. I'm learning about my own preference for modal harmonies, slow moving bass lines and highly synthetic sonorities.

Pandora still keeps most of its smarts to itself, which is understandable if they want to be your radio station. But take communication a little further and you a fantastic way to teach people about art and culture: telling the story through your own tastes. Imagine a service that highlights elements that your favourite paintings have in common. It would expose you to new works and when your tastes develop and change, contrast what you liked before to what you like now, explaining why and how. It would congratulate you on your growing sophistication and challenge you with art it knows you'd like if it you'd just give it a chance. It would also connect you also with similar people going on a very similar artistic journey.

Opening Pandora's Box

2006-12-23T16:09:00.000-08:00

I was referred to Pandora today. it plays you a personal radio station where you can specify seed songs and artists and plays similar ones based on hundreds of attributes discovered in the Music Genome Project.

I just set up a channel based on The Orb. It's currently playing Moog Apella by 16B, all new to me but sounds great. I sent the channel to a fellow Orb fan. He said he really liked the track playing now, but it was different.

Was thinking it would be great if you could share channels on an ongoing basis. With such a wide library of tracks available, you'd think it could resolve disputes on what music to play in office environments. One person would play the music through their speakers (others could do if they wanted the sound closer to them). But everyone could rate the tracks that were coming through and put in a set number of request tracks for others to rate. The anonymity of the rating system would make it fair too; a secret ballot.

They should have a feature "combine channels" where different users select channels they want to hear and submit them. Then the system merges those to create something new, common and live. Would be great for being in the same space as remote workers.

A nice way of sharing for now is just to get visitors to enter their favourite tracks now into the same channel. I went round to this friend's place yesterday and we each took turns to name favourite songs. And yeah, a lot of the time the new songs played were to everyone's liking.

Next time I have a nice lady over, I'm going to ask her what her 3 favourite romantic songs are. First she'll be happy I have the song, then be amazed how close my taste is with other songs. It's this kind of technological edge that will ensure us geeks make a real impact on the genepool of generations to come.

My tip to Alexadex users. Buy Pandora and hold!

technorati tags: data mining streaming radio music mining pandora music sharing

Tis the season to be networking

2006-12-11T14:46:00.000-08:00

While I'm blogging, I'd like to thank everyone involved with STIRR. Creative party games made for a good vibe in the room and it was a good focused crowd. Our glorious team, team 3 won with our ShoeWave.com business; a peer to peer sock sharing service where you send in an odd sock with a dollar, and receive 2 matching socks back in the mail. Genius :).

Also great fun was Clickaholics. It was a younger crowd this time, not coming on the tail end of a big expensive conference. Free alcohol didn't last long but who cares? People did!

Met fun, interesting and smart people, all of whom will be great to see again.

Baby loves to ChaCha?

2006-12-11T14:41:00.000-08:00

Just tried out ChaCha, a new stab at the old humans-search-for-you concept. The conversation was very slow. Each response took 1-3 minutes and overall it was about half an hour. I guess if you wanted to use them effectively you could open up 10 windows and ask 10 questions. I hope they have good protection against bots running thousands of queries at once.

Status: Looking for a guide ...
Status: Connected to guide: ErinL
ErinL: Welcome to ChaCha!
ErinL: hello!
You: Hi!
ErinL: Welcome to ChaCha! Please wait a moment while I search for your results.
You: are you there?
ErinL: yes I'm still searching. i'm just getting the news show.
You: it's an australian show
You: a comedy
You: different the us one
ErinL: I've found it. bare with me.
ErinL: this is the correct show, right?
You: Yep
You: I'll give you a clue
You: http://www.google.com/search?q=%22rob+sitch+played%22+frontline&hl=en&sourceid=gd&rls=GGLD,GGLD:2006-41,GGLD:en
ErinL: here is the wikipedia site. he played Mike Moore.
You: Thanks..Does that url come up now?
ErinL: Are these results sufficient? Is there anything else I can find for you?
You: Yep what's the most popular site published by Rapid Intelligence?
ErinL: ok bare with me as I find that answer as well.
ErinL: Is this what you are looking for?
You: ..
ErinL: hmmm?
You: is somehting supposed tocome up in the guide results now?
You: I don't see a url
ErinL: yes. you should have 3.
ErinL: www.rapint.com
You: nothing showed up..I'm using firefox
You: one more question.. Who is the opposition leader in Australia?
ErinL: oooh. I don't think it's compatible with fire fox. This is the 2nd time I've had this happen.
ErinL: if I give you the www site can you find it from there? I'm not familiar with firefox.
You: yep no prob
ErinL: www.en.wikipedia.org/wiki/Frontline_%Australian_TV_series%29
ErinL: www.rapint.com for the rapid intelligence question
ErinL: I hope this helps you. If not let me know. Is there anything else I can do?
You: Yep Who is the opposition leader in Australia?
ErinL: http://en.wikipedia.org/wiki/List_of_Australian_Opposition_Leaders
ErinL: It's Kevin Rudd.
You: Thank you
You: One more quesiton: where are you based?

[many minutes pass]

ErinL: Thanks for searching ChaCha! Have a nice day. thank you.
ErinL: Please RATE ME. Thanks for using ChaCha.
Status: Session ended.
Status: Looking for a guide ...
Status: Connected to guide: AmandaG
AmandaG: Welcome to ChaCha!
AmandaG: Chacha is based in Indiana.
You: Where are you personally right now? (which city)

[10 minutes pass. Things start getting really slow here]

You: are you there?
You: Are you checking to see if you're allowed to answer this or?
You: Hello?
You: Thanks, I'm done.
Status: Session ended.
Status: Looking for a guide ...
Status: Connected to guide: Steven C
Steven C: Welcome to ChaCha!
Steven C: hi
You: Hi Steven
Steven C: hi
You: where are you?
Steven C: in USA
You: Ok thanks

Impressive they could actually field queries despite a recent spike in traffic. But I can't say I think it's a great business. Perhaps if you publish the chats, you've got yourself an easy way to generate content for AdSense. Perhaps ones that both chatters agree is worth publishing. Or it could work doing verticals. I'm sure there's room on the net for a few Indian mesothelioma experts giving you advice then sending you to affiliated sites.

Plagiarism on Wikipedia

2006-11-05T15:43:00.000-08:00

Wikipedia-Watch.org owner, Daniel Brandt has published a report showing widespread plagiarism in Wikipedia.

A more sophisticated methodology could be used here. One idea that has just hit me would be using archive.org and the Wikipedia history function to discover phrases that appeared on other sites first. This could be all automated.

But such activity should be applauded. Whatever people may speculate about Daniel Brandt's motives, the same thing could have been done by a Wikipedian with the same positive effect: as an open system that assimilates criticism, it will learn and evolve from negative press out there. And as a fast moving encyclopedia, it can react quickly to address the problem.

Google inadvertently invents an acronym generator

2006-10-05T20:56:00.000-07:00

Hats off to Google Labs for creating the first search engine to allow regular expressions, even if it's just for their new code search engine.

This is something I've been waiting for for ages. Why? acronym discovery!

This regex "\s+I\w+\s+P\w+\s+O\w+\s+D\w+" means find sequences of words that match I*, P* O* D*. We get from this:

 indexes pairs of digits
 invalid path or domain
 important property of DOM

\sL\w+\s+O\w+\s+V\w+\s+E\w+ searches for love and gets "lock on VLDB entry"

\sI\w+\s+B\w+\s+M\w+ finds i.b.m. and gets :

is being moved

image being manipulated

s.h.i.t.

 signal handler is trashed

systems hide it there

r.s.s.

 random string sent

 related structures stored

report some statistics

Why the big sunnies?

2006-10-04T18:52:00.000-07:00

Aidan B over at Yahoo asks in his blog why people are still wearing those ridiculously large sunnies.

I woulda commented on his blog but Yahoo 360 wants me to sign up, so I'll stick it here.

My theory (and it's probably not very original) is this: fashions take influences from new developments in society. In the 70s it was all about colour because colour TV just came out. Now in the naughties we have a massive increase in the consumption of porn (and I've read experts are having to reevaluate their estimates on how popular fetishes are). So these porn sunnies are worn to reflect society's growing tolerance towards porn.

By wearing them, girls are unconsciously sending a signal that they are sexually comfortable. (Even though of course it's mostly teenage girls who wear them, probably the least sexually beings in our society.)

Broadband in Gas

2006-09-21T19:55:00.000-07:00

A company called NetherComm want to do Broadband-in-gas. Internet through your gas pipe. I thought it was a joke but apparently this kind of thing is possible.

I'm gonna patent Broadband-from-beer-tap. They can cache the internet in kegs under the pub.

FactBites on the up

2006-09-17T03:14:00.000-07:00

Our FactBites site is enjoying growing traffic, and I've just read a great review on the blog of Matthew Hurst, Director of Research at Intelliseek:

The results that Factbites generates are still extracts from web pages, but they are of far higher quality than the summaries that main stream web search engines provide. By focusing the technology on passages that make statements about the topic being searched for, the user experiences a far richer response. It should be noted that this type of interface has the potential to change the way in which searches are performed as the search engine is now doing more of the heavy lifting and is not trying to guess the intention of the user.

The site also has an In The News feature now, linking to the recent topics.

Migrants: Let em know they're in for.

2006-09-15T19:54:00.000-07:00

Some thoughts on Australian immigration, but would probably apply for any country.

Australian Prime Minister, John Howard is floating ideas around at the moment about making people sit a test in order to become Australian citizens. This was a response to Opposition Leader, Kim Beazley's proposal that new arrivals to Australia pledge their allegiance to Australian values.

But why wouldn't applicants just see these things as just another bureaucratic hurdle? And I can't say I'm entirely comfortable with parliament legislating what our values are (even with the help of polling and the mass media) and having the bureaucracy word it for our benefit.

What about something less divisive that's in the interests of all parties? Just let them know what they're in for. Give them information they really want, before they make that fateful trip to the other side of the world. Ask them what they imagine their life will be like and tell them how realistic it is.

It benefits noone when Australian life turns out to be letdown. It costs our social security system and criminal justice system when disappointment turns to sloth and/or resentment. And for the immigrants themselves, even when they have no choice but to leave their own country, perhaps another is more suited to them.

Having gone to a very mixed school, I got some insight into the migrant experience. Very often their expectations about the country are different to what they get. It's amazing how little knowledge their parents had before taking the plunge.

Migrants should be told the following things:

Whether your professional qualifications will be accepted.
If those skills won't be accepted, how hard you will have to work to attain an average standard of living.
Even once you've achieved that you'll have to pay high levels of tax, and how strictly enforced.
Australians will feel that your children have a right to marry anyone they see fit, including people of other races.
That your children may adopt western values more readily than you might think, and may not care for you in your old age.
Australians generally believe that religion is a personal thing and don't like seeing it (mis)used in the public sphere.
You may not feel so welcome when you return to homeland years later. They may view you as betraying the motherland, or not suffering along with your people, yet seeking to benefit when times are good again.

This is just a few pitfalls I came up with off the top of my head. They could have different ones for each ethnic group, and they could be based it on migrant surveys.

Imagine if all this was delivered in a documentary style video starring their fellow countrymen who'd make the move and in their own language.

Oh Tube, Mirror My World

2006-09-13T02:29:00.000-07:00

I've been following the lonelygirl15 controversy like everyone. It's taken a lot of older people aback how popular this bedroom broadcasting medium has become. In fact, she is just one of many "vloggers" that have attracted audiences that some TV stations would envy. Yet most of what she says is actually pretty pedestrian and boring. What's going on? Don't people have better things to do on the net? Isn't supposed to be approaching sum of all knowledge?

When television first came out, they dreamed it would enlighten the masses, bringing far off civilisations and education into the living room. What ended up happening was families gathered round to watch sitcoms which were fictionalised mirrors of themselves - other families in domestic environments.

The internet was originally going to be the great educator, empowering everyone with all the world's knowledge. Now it's turning into a mirror too with kids watching other kids sitting in their bedrooms in front of their computer. And in the case of LonelyGirl, it's fictionalised too.

FactBites getting fatter

2006-08-28T22:04:00.000-07:00

I've just updated a new index for our site FactBites. It now fields 2 million keywords. More features to come.

Can you handle the truth?

2006-08-24T16:39:00.000-07:00

Or "If I say something weird, will you leave me and never come back?"

A big issue with running stat sites is providing relevant, useful info that users understand while not cutting down what the users see until they can't question you.

On NationMaster and StateMaster, there are thousands of stats, so you're always confronted with contradictions. Figures seemingly don't make sense when you put them altogether. Our competitors like the CIA World Factbook have so few stats that inconsistencies can't arise. It's the easy option.

Our correlations feature wasn't as much of a hit as I'd hoped, because people looked at the data and said "wa? Murder rate correlates to gun ownership that makes sense. But look it correlates even more strongly to orange juice consumption!"

In a world without sites like NationMaster, we just leave it to experts to select which stats are most relevant. Of course an expert is by definition someone more knowledgable of the domain, so they will be able to digest the info more readily. But when more statistically significant figures are lying around and are not used, everyone should have access to them so they can ask why.

Now with SEO Sleuth, I chose to show every search going to every site. Now, any webmaster can tell you that people come to your site looking for pretty different things to what the site offers. And looking at the terms as a whole may give you a distorted view of what the site's about (but perhaps a good view of what parts of the site are of interest to searchers). But yeah, we're left with the same problem; people give a quick "that doesn't look right" and leave the site. One thing I considered was linking to the actual searches to prove it, but I didn't want to be republishing such sensitive data. So once again, the quandry.

G'day World

2006-08-23T13:38:00.000-07:00

Back in February when I was in San Fran, an American friend of mine told me about this show. And it's only last weekend round to giving it a listen.

I've been listening to podcasts for a while; a fair few IT Conversations some of which can be good, but I often feel a little hyped out. I sometimes got this queasy feeling that it's all not real.

So it's really refreshing to discover Cameron Reilly and Ben Barren from G'day World. Industry news from that certain intangible Australian sensibility; the cynicism, the good ole Aussie accent. Keep it up guys.

Speaking of saying hello to the world, my Technorati Profile.

What turns me on

2006-08-21T03:42:00.000-07:00

Blogging as a medium is supposed to allow people to be a little more personal. While I don't feel much need to talk much about my private life, I would really like to talk about why I do what I do.

I've been programming since I was 5 and doing generating stats for communities since I was 13 (back in the days of BBS's). Since then I've worked for a publisher, a web developer and an internet marketing agency. Now almost 30, and 4 years into running my own business, the same thing has always driven me.

For me, I have an introverted motivation and an extraverted one.

On the internal side of things, it's all about that eureka moment when something new arises out of nothing. When the machine gives you back something more intelligent and sophisticated than the code you wrote to discover it.

For the outer world, it's about knowledge; helping more people see the facts for themselves, without relying on any elite to digest them first.

NationMaster and StateMaster are referenced thousands of times on the web. For whenever you're in a discussion or debate, you know there's one site you can go to compare countries/states on just about anything. Of course you can generally use stats to support either side of an argument, but it does increase the quality of the debate and encourages people to find common ground in some area approaching reality.

Politics is very polarised these days, particularly in the US and the mass media landscape are more bombarded by sophisticated PR and marketing than ever. Our society in general is growing exponentially more complex. The reality of looking at a stat site like NationMaster is that you're going to be confronted with figures that don't fit nicely with your belief system, or even rival belief systems. Stats aren't perfect, but they bring people closer to the complexity of reality. That's something I really hope we can get on top of in this coming information age.

Climbing over the Great Firewall

2006-08-21T03:13:00.000-07:00

Got this email recently about our Qwika product which includes Wikipedia mirrors. This is the sort of mail that makes it all worthwhile:

I am in china .maybe you know ,the wikipedia was blocked in china . I visited your web site and i find that i can read the articles from wikipedia by clicking the "cached "in the searching results .it is wonderful !now i can read the full text but no images . i very appreciate your work !

luck and happy !

It's also a good argument for free content being available through many sources, which is the spirit of the GFDL anyway.

Announcing SEO Sleuth

2006-08-20T02:42:00.000-07:00

Tuesday 8th August was an adrenaline pumping day for me. Sitting on the ferry I was going through newly downloaded files on my laptop's desktop and came across these large gzips a mate had sent me that morning. I was thinking it was going to be that Google N-gram data. But browsing through it, it was clear these were searches, millions of them! I let out an involuntary audible expletive and then day I wondered round the city thinking up things to do with the data.

The most obvious site, it seemed, would be one that allows ordinary people to read this goldmine of personal information, instead of just people who knew how to read large files. (I was surprised by a number of news stories on the subject where the journalist disclaimed that they hadn't seen the data themselves). But I thought that was a bit unethical. As it turns out, there's no shortage of such sites out there now with dontdelete.com and aolsearchdatabase.com among the more popular.

I also started to see more research-oriented sites crop up that looked at it completely in aggregate - what proportion of people click which ranked result for all terms, how long is the tail and such.

With my background in search, what was really interesting for me was the possibility of generating search engine reports for any site. Except for the rare occasion when a log reports are made publically available, we really don't know what people search for to find our competitors' sites.

And the results are certainly interesting. For example, The Open Directory Project's incoming keywords are predominently adult. The top term is actually "adult" itself.

Next thing I did was a transpose: for given keyword, it will tell you which sites get the traffic and where they rank. Obviously simply performing a search and seeing what ranks best is the traditional way of doing this, but very often lower ranking sites get more traffic. Knowing both where the results rank and how much traffic they got, and being able to look at the original search (just click on the keyword title) you can then use the tool to work out which sites had the best titles and descriptions (or perhaps brand power).

Anyway check it out. The site as had some success with mentions on Threadwatch and the New Scientist blog.

Every move you make..

2006-07-13T16:38:00.000-07:00

I just saw ClickTale in delicious. It will allow webmasters to view movies of their users' behaviour on their site, right down to mouse movements.

There's a lot of potential here. It could be a real help in isolating bugs. When a user writes a complaint email, match his IP/cookie to the profile and just watch exactly what he did. Then even those myriad "i went to ur website and it doesnt work" would be become useful.

And mouse tracking alone, wow. I can imagine in the not too distant future, I let a shaky old person take the keyboard and when I get it back, the ads on all my favourite sites are related to Parkinson's Disease. When they see the mouse get up to speed again the ads may ask me to write a testimonial for their miracle cure.