Some good ideas and nice observations here, thanks. This post along with a number of others recently (by, eg, John Borthwick and Adam Greenfield) seem to be tending on a convergent course with some things I have been thinking lately; helpful and encouraging.
In your last couple of paragraphs you go toward an idea I like: 'emergent search' by association or connotation; the kind of lateral jump we make during conversation from one thread to another, bound by metaphor as much as anything... I frequently get the feeling of this at work reading twitter.
Concerning your remarks about fragmentation and complexity ("Since web 1.0...), the feeling that it is making life more complicated is a product of the extent to which we don't quite yet understand how the parts of the emerging system fit together. The incredible speed of popular adoption of social/communication technology tells us that more problems are being solved than created, I think. The fragmentation is necessary to discover the underlying potential for unity -- as you say, necessary for the next phase.
trevoro
· 10 months ago
Pubsub did this back in 2005 but they were either too early or couldn't execute, and ever since we've been looking for a real-time notification and search system. I think one of the problems with delivering truly relevant information is that a lot of the content that's published doesn't come with relevant semantic information at the source, and I think that's preventing us from delivering really good results that can scale. Sure we can put information through semantic analysis machines, and crunch away at everything but at the volumes required to impact peoples lives... it just won't work. That being said, finding out about when things happen in real time is a good first step, and that's what summize does so well. I really liked your post and I'd love to continue this discussion.
-Trevor
gerry campbell
· 10 months ago
Trevor - thanks for the comment... ping me on twitter and we can talk directly. Would love to have a conversation about your thoughts/experience...
G
masapola
· 10 months ago
Gerry,
You're wrong here...there is only one point and purpose...revenue generation and revenue growth, growth, growth.
gerry campbell
· 10 months ago
Interesting point, but experience tells me otherwise.
Trees don't grow to the sky. Revenue generation/growth is an endless cycle of innovation, discontinuity and cannibalism.
It's the single-variable management mindset (e.g. revenue-only) that leads to strategically soulless companies - those are the companies that lose touch with their customers and open the door wide for the competition.
Kevin Marshall
· 10 months ago
fascinating stuff - myself and a few other developer friends have been thinking a lot about this sort of thing as well lately...
I think one of the important things that you touch on here is that to make search more relevant, it needs to know more/do more with context...and that's sort of the angle we're hacking the most on now...
It really feels like it's at least the 'next' step to me...if you think about it, you are already spending a lot of time and energy revealing your interests and experience (via Twitter and blogs for example)...on a basic level, if you could take your recent tweet history and apply it as a lens on any searches you do...it seems that would provide you with some REALLY interesting and relevant (to you) results...and as a side effect, it would also help train you to share (ie. tweet and blog) more about the things you really do care about (so you could get better and better use from your searching systems)...
It doesn't go all the way to taking us from here to where we need to be...but it seems like a good next step (and aside from the issues of scaling, it seems like the easiest next step too)...anyway - anyone interested in chatting more about this approach can ping me on Twitter @falicon ... I've love to hear more people openly discussing different approaches and ideas!
A couple of quick responses: - Agreed: Twitter is by no means the only source of realtime-ness. No way. - Can Google (or Yahoo, or...)build a product that addresses this? Yes. No doubt. - Should we sit and wait until they do? Nope. - Don't miss my point that expressed information, vs published, benefits from a social graph for prioritization/interpretation
David Semeria
· 10 months ago
"Realtime search, using social inference for discovery, ranking and prioritization."
It's an interesting idea. Unfortunately, I disagree with both parts. I think real time is way overdone. I'd much rather wait for the signal to noise ratio to straighten itself out, and then find out what really happened.
The friends issue is trickier. By definition they are a subset of all the people you have met in the real world. But meeting people in the real world is limited by many things such a location, social status, occupation etc.The online world is more efficient in this respect. There is more scope for meeting people whose interests more closely match your own. And, so, when it comes to certain conversations, I'd much rather interact with my online contacts, rather than my 50 or so 'real' friends on FB.
gerry campbell
· 10 months ago
David - I think you missed the point that the online social graph is a critical ingredient... the physical world is only an illustration...
And getting the signal to noise relationship solved is the crux of the issue. In the simple query-response world we would simply call it "relevance" or "precision at X."
David Semeria
· 10 months ago
OK Gerry, point taken wrt social graph.
My own approach is slightly different. I explicitly assign authority to people based on subject. So, for example, I give Tim O'Reilly and Scoble high marks for 'tech', and zero marks for 'soccer', whereas I give my friend Boz top marks for humour, because he is a really funny guy, but zero marks for everything else (because he ain't the sharpest knife in the drawer)
I then break down a given stream (eg Twitter) into subject-specific virtual feeds which I then filter and throttle on the above and other criteria.
The basic point is: Tim O'Reilly is in my tech stream (with high relevance) even though our social graphs don't overlap.
Greg Sterling
· 10 months ago
Gerry. Nice article. Would love to catch up.
pangaro
· 10 months ago
in response to invitation to share ideas: social networks help search because they represent a valuable conversation involving language and values shared by an individual.
another opportunity (as yet unrealized) is to understand and design for human conversations, not in the sense of talking or "natural-language processing" but in the sense of supporting an individual to use the sole means of understanding and generating knowledge: the internal conversation we all use to grok our world.
our interfaces are still designed for "users", that is, humans who click buttons and receive information from machines. if instead we designed for "participants in conversation" in a strict sense---the development of points-of-view that hold beliefs that are subject to challenge and evolution---we might improve more than just search (at best we'd remove the false choices between searching and browsing, consuming and generating content).
since "conversation works", that is, we are fined-tuned by evolution to converse effectively, it seems to me that the adoption of conversation as the means to design interfaces has a long runway and much to contribute to interface design. for further development of the approach, see http://pangaro.com/search/
Great post. Check out OneRiot.com. It's a real-time search engine, collecting data from people who opt-in to share their user activity. Our panel is now 2million strong and the results come from a fresh, real-time indexing of the web as it changes. It's early days, but we love where it's going.
Kimbal - thanks for the comment. I have had a look... interesting approach. Sort of reminds me of what Seth Goldstein was doing with AttentionTrust before it evolved into SocialMedia. Sort of... but his was about profiles/privacy and I see you're building an index.
BTW, I feel like we should meet - I heard a lot about you from Jay, Abdur and Greg over the summer...
I will ping you some time soon to talk.
Hank Lambert
· 10 months ago
Jerry - right on! But, in the end, it's pretty simple: search doesn't work because it's a paradigm that we avoid in our lives (looking for lost keys, needle-in-a-haystack, etc). When we pick up the newspaper we know that there is a limited data set available to review - our digital lives should be no different, but personalized by our preferences and desires. Serve me the information that I want (whether I know I want it or not) and you have a user-for-life.
So, we'd like to introduce you to our solution: Kayanta. (http://www.kayanta.com) We'd love to give you a demo of the site/app and get your thoughts. We're ready for our Alpha and are about to start building our Alpha audience.
You can reach me at hank@kayanta.com. Looking forward to the discussion.
gerry campbell
· 10 months ago
Hank - would be happy to talk. I am crushed over the next week, but ping me on Twitter and we can set up a chance to talk.
Evil
· 10 months ago
Nice read. I've been thinking about this problem in a similar way. I think of these modes and contexts as analogous to the soft partitions we setup in our daily lives - professional, personal, familial, etc. If social tools such as Gmail, FaceBook, Flickr, MySpace et al. actually connect their graphs and data together then we can analyze our behaviors and automatically discern what our mode. Even cooler things happen with photo & video data. Think about searching across your entire graph for photos/scenes from a party you attended the previous night. You didn't post any but at least 10 other people did, some in your graph, maybe some just outside. You need only click on two or three related images and the system should be able to do the rest - determine by the EXIFs GeoCode and Date/Time what cluster you are requesting and farming your graph to figure out who else has data you need based on your mode. Think Pandora for searching via your graph - "Nope, Close but not it, Yes, that's it". In a few clicks the system can figure out the most logical metric for your responses and fetch data that will seem most natural to you.
Kingsley Idehen
· 10 months ago
Please read my post about SDQ (Serendipitous Discovery Quotient). Basically, search has long been broken, it is autistic to the consequences of human cognition (we don't do singularity, everything has many facets).
Gerry, Long time no see, and it's great to read your post. I agree that traditional search engines are oriented around authoritative content (where lots of links from authorities matter). This makes them less well-suited to ranking real-time content. I also have thought about the theme of fragmentation, though I think it applies to the real-world as well as to the web (you have soccer buddies but don't much discuss technology with them). (There's actually an inverse fragmentation happening with social networks, where you get to have just one persona to share with all your friends, instead of different personas you get to present in context of real conversations).
It sure seems like knowledge of social graph and context should make search better, but real progress would come from examples of actual broken-ness. Hence: What are 10 actual queries, or tasks, for which search is broken today?
John Furrier
· 10 months ago
Just found this post after friending you on twitter. Totally right on. Search is so broken and I mean so 1999-2001.
If you look at user behavior in the terms of "discovery and navigation" across today's apps then the notion of just an index seems silly
John Furrier
· 10 months ago
I would add for more on how f'd up search is my personal blog has a ton of opinion posts on it re: the future of search www.furrier.org
Alex Bunardzic
· 9 months ago
Good writeup, Gerry. If you'd like to learn more about what I deem to be the future of the web, and how will we make it serve humans (instead, as you've pointed, us serving it), please read this: http://tinyurl.com/cqnylv
Reed M Meseck
· 9 months ago
One of the huge emerging problems of the web is that as information moves faster and faster, it becomes stale quicker and quicker. Stale data isn't just less useful than than fresh data - it is potentially incorrect or misleading.
Take the example of real estate listings in this down market. The prices are moving faster than the search index can reflect so 9 out of 10 search results on the page often have the wrong price.
So how useful is search if what is returned is not "correct" ? The list price was correct at some point in time but is no longer valid. The heart of the problem is that search today lacks temporal semantics - i.e. the ability to represent the same piece of information along a timeline with different values. Making the problem worse is that current search engines have no way of reflecting temporal semantics and seeing the history of values - the changes along the timeline - might be of great interest. Stream search is a completely untapped facet of search.
There are many aspects of search that remain unexplored. Search communities are a natural social pattern. Personalized search contexts are another opportunity - why after all these years should each search be completely divorced from the context of your prior search request ?
Search is not getting better - it is getting worse. In part because information becomes stale more quickly and in part because we have not expanded on the search paradigm. Google dominates the space and is remarkably not very innovative when it comes to advancing Search. Everything else but Search.
street wear
· 6 months ago
Nice blog there.. a very interesting post.. I enjoy reading your post..
All Comments
In your last couple of paragraphs you go toward an idea I like: 'emergent search' by association or connotation; the kind of lateral jump we make during conversation from one thread to another, bound by metaphor as much as anything... I frequently get the feeling of this at work reading twitter.
Concerning your remarks about fragmentation and complexity ("Since web 1.0...), the feeling that it is making life more complicated is a product of the extent to which we don't quite yet understand how the parts of the emerging system fit together. The incredible speed of popular adoption of social/communication technology tells us that more problems are being solved than created, I think. The fragmentation is necessary to discover the underlying potential for unity -- as you say, necessary for the next phase.
-Trevor
G
You're wrong here...there is only one point and purpose...revenue generation and revenue growth, growth, growth.
Trees don't grow to the sky. Revenue generation/growth is an endless cycle of innovation, discontinuity and cannibalism.
It's the single-variable management mindset (e.g. revenue-only) that leads to strategically soulless companies - those are the companies that lose touch with their customers and open the door wide for the competition.
I think one of the important things that you touch on here is that to make search more relevant, it needs to know more/do more with context...and that's sort of the angle we're hacking the most on now...
It really feels like it's at least the 'next' step to me...if you think about it, you are already spending a lot of time and energy revealing your interests and experience (via Twitter and blogs for example)...on a basic level, if you could take your recent tweet history and apply it as a lens on any searches you do...it seems that would provide you with some REALLY interesting and relevant (to you) results...and as a side effect, it would also help train you to share (ie. tweet and blog) more about the things you really do care about (so you could get better and better use from your searching systems)...
It doesn't go all the way to taking us from here to where we need to be...but it seems like a good next step (and aside from the issues of scaling, it seems like the easiest next step too)...anyway - anyone interested in chatting more about this approach can ping me on Twitter @falicon ... I've love to hear more people openly discussing different approaches and ideas!
Great post. I disagree with some points though and posted more thoughts here: http://blog.betterlabs.net/2009/02/09/why-googl...
A couple of quick responses:
- Agreed: Twitter is by no means the only source of realtime-ness. No way.
- Can Google (or Yahoo, or...)build a product that addresses this? Yes. No doubt.
- Should we sit and wait until they do? Nope.
- Don't miss my point that expressed information, vs published, benefits from a social graph for prioritization/interpretation
It's an interesting idea. Unfortunately, I disagree with both parts. I think real time is way overdone. I'd much rather wait for the signal to noise ratio to straighten itself out, and then find out what really happened.
The friends issue is trickier. By definition they are a subset of all the people you have met in the real world. But meeting people in the real world is limited by many things such a location, social status, occupation etc.The online world is more efficient in this respect. There is more scope for meeting people whose interests more closely match your own. And, so, when it comes to certain conversations, I'd much rather interact with my online contacts, rather than my 50 or so 'real' friends on FB.
And getting the signal to noise relationship solved is the crux of the issue. In the simple query-response world we would simply call it "relevance" or "precision at X."
My own approach is slightly different. I explicitly assign authority to people based on subject. So, for example, I give Tim O'Reilly and Scoble high marks for 'tech', and zero marks for 'soccer', whereas I give my friend Boz top marks for humour, because he is a really funny guy, but zero marks for everything else (because he ain't the sharpest knife in the drawer)
I then break down a given stream (eg Twitter) into subject-specific virtual feeds which I then filter and throttle on the above and other criteria.
The basic point is: Tim O'Reilly is in my tech stream (with high relevance) even though our social graphs don't overlap.
individual.
another opportunity (as yet unrealized) is to understand and design for human conversations, not in the sense of talking or "natural-language processing" but in the sense of supporting an individual to use the sole means of understanding and generating knowledge: the internal conversation we all use to grok our world.
our interfaces are still designed for "users", that is, humans who click buttons and receive information from machines. if instead we designed for "participants in conversation" in a strict sense---the development of points-of-view that hold beliefs that are subject to challenge and evolution---we might improve more than just search (at best we'd remove the false choices between searching and browsing, consuming and generating content).
since "conversation works", that is, we are fined-tuned by evolution to converse effectively, it seems to me that the adoption of conversation as the means to design interfaces has a long runway and much to contribute to interface design. for further development of the approach, see http://pangaro.com/search/
Great post. Check out OneRiot.com. It's a real-time search engine, collecting data from people who opt-in to share their user activity. Our panel is now 2million strong and the results come from a fresh, real-time indexing of the web as it changes. It's early days, but we love where it's going.
Hope you like it.
Kimbal
CEO, OneRiot.com
BTW, I feel like we should meet - I heard a lot about you from Jay, Abdur and Greg over the summer...
I will ping you some time soon to talk.
So, we'd like to introduce you to our solution: Kayanta. (http://www.kayanta.com) We'd love to give you a demo of the site/app and get your thoughts. We're ready for our Alpha and are about to start building our Alpha audience.
You can reach me at hank@kayanta.com. Looking forward to the discussion.
Links:
1. http://bit.ly/3jZTWP - Post about SDQ with a BCG matrix
2. http://www.openlinksw.com/weblog/public/search.... - collection of post related to SDQ
Kingsley
Long time no see, and it's great to read your post. I agree that traditional search engines are oriented around authoritative content (where lots of links from authorities matter). This makes them less well-suited to ranking real-time content. I also have thought about the theme of fragmentation, though I think it applies to the real-world as well as to the web (you have soccer buddies but don't much discuss technology with them). (There's actually an inverse fragmentation happening with social networks, where you get to have just one persona to share with all your friends, instead of different personas you get to present in context of real conversations).
It sure seems like knowledge of social graph and context should make search better, but real progress would come from examples of actual broken-ness. Hence: What are 10 actual queries, or tasks, for which search is broken today?
If you look at user behavior in the terms of "discovery and navigation" across today's apps then the notion of just an index seems silly
Take the example of real estate listings in this down market. The prices are moving faster than the search index can reflect so 9 out of 10 search results on the page often have the wrong price.
So how useful is search if what is returned is not "correct" ? The list price was correct at some point in time but is no longer valid. The heart of the problem is that search today lacks temporal semantics - i.e. the ability to represent the same piece of information along a timeline with different values. Making the problem worse is that current search engines have no way of reflecting temporal semantics and seeing the history of values - the changes along the timeline - might be of great interest. Stream search is a completely untapped facet of search.
There are many aspects of search that remain unexplored. Search communities are a natural social pattern. Personalized search contexts are another opportunity - why after all these years should each search be completely divorced from the context of your prior search request ?
Search is not getting better - it is getting worse. In part because information becomes stale more quickly and in part because we have not expanded on the search paradigm. Google dominates the space and is remarkably not very innovative when it comes to advancing Search. Everything else but Search.