I’ve posted some pictures to Flickr, but I’ve never really paid much attention to the “interestingness” rankings the site uses.
Interestingness and clustering were first used in August of last year, as announced by Stewart Butterfield on the Yahoo Search Blog and the Flickr blog.
Blog posts about Flickr’s interestingness, and a February Flickr forum post on changes to the interestingness rankings, show a lot of interest in the “secret sauce” on how photos are determined to be interesting. A couple of patent applications were published by Yahoo this week that delve into interestingness rankings, clustering of pictures, and metadata associated with Flickr images.
Before jumping into those, I found some other blog posts that shared some thoughts about interestingness:
- Anil Dash on the economics of interestingness (October 25, 2005)
- Jeff Jarvis writes on Interestingness, and wonders whether it could be useful for an overall search algorithm, and could work across a distributed network like the web. A couple of commentors provide great responses, referencing the importance of community sizes, the effectiveness of a site like Digg, using an interestingness ranking for mp3s, and how Del.icio.us determines interestingness. (October 31st, 2005)
- Harold Davis writes about Interestingness Is to Flickr As PageRank Is to Google (October 31st, 2005)
- Wesley Hein takes a closer look at the ranking method in Deconstructing Flickr’s “Interestingness!†(May 12, 2006)
- Chuqui writes about a need for a personalization of the ranking system in O’Reilly Radar > Flickr and Interestingness. We see a nod to personalization in the documents that came out this week. (August 19, 2006)
- Peter Forret considers ways to create an PageInterest ranking for Google and Yahoo in “Interestingness†for Google web search.
The Interestingness Algorithm
Interestingness ranking of media objects
Invented by Daniel S. Butterfield, Caterina Fake, Callum James Henderson-Begg, Serguei Mourachov
Assigned to Yahoo! Inc.
US Patent Application 20060242139
Published October 26, 2006
Filed: February 8, 2006
Abstract
Media objects, such as images or soundtracks, may be ranked according to a new class of metrics known as “interestingness.” These rankings may be based at least in part on the quantity of user-entered metadata concerning the media object, the number of users who have assigned metadata to the media object, access patterns related to the media object, and/or a lapse of time related to the media object.
When reading these patent applications, skip past the “claims” section to the summary descriptions, and the detailed description. The detailed description sections of both documents are the same.
Interestingness rank is based in part on:
- The quantity of user-entered metadata concerning the media object,
- The number of users who have assigned metadata to the media object,
- An access pattern related to the media object,
- A lapse of time related to the media object, and/or;
- On the relevance of metadata to the media object.
- Whether the media contains undesirable content such as obscene imagery or promotions of a competitor’s product.
It may also be “personalized” to a user. The rank that someone may see for a photo could be based upon their identity and their relationship with the person who posted the image. It could, for example, look at the number of media objects tagged or designated as favorites in common by the person posting the image, and the person who might be interested in it. If there is a location associated with the picture or the residence of the person who uploaded the photo, that may also be a factor considered.
The patent filings are for “media objects” and could involve videos and audio as well as photos, but this first patent notes that “much of the functionality of the invention may be observed at www.flickr.com, which is incorporated by reference herein in its entirety.”
Relatedness of metadata may be considered, and could be used for clustering:
In another embodiment, the statistics engine may determine the “relatedness” of metadata, i.e., a co-occurrence measure of the frequency with which a particular metadatum (e.g., tag) (or term within a metadatum (e.g., within a comment)) is assigned to a media object along with at least one other particular metadatum (or term within a metadatum). In one embodiment, the co-occurrence measure may determine the frequency of co-occurrence of metadata of the same type. For example, out of all 100 images tagged with the word “Italy,” 50 of those images may also be tagged with “Rome,” 25 tagged with “Venice,” 10 with “Florence,” and 2 with “Sienna.” The co-occurrence index would respectively be 50 for “Italy-Rome,” 25 for “Italy-Venice,” 10 for “Italy-Florence,” and 2 for “Italy-Sienna.” In summary, include location as subset of tags, Tag MD can include location.
The patent filing also includes a paragraph about how ads based upon context might be determined for a page, using a relatedness metric.
The Metadata Patent Application
Media object metadata association and ranking
Invented by Daniel Stewart Butterfield, Eric Costello, Caterina Fake, Callum James Henderson-Begg, Serguei Mourachov, and Joshua Eli Schachter
Assigned to Yahoo
US Patent Application 20060242178
Published October 26, 2006
Filed: February 8, 2006
Abstract
Metadata may be associated with media objects by providing media objects for display, and accepting input concerning the media objects, where the input may include at least two different types of metadata. For example, metadata may be in the form of tags, comments, annotations or favorites. The media objects may be searched according to metadata, and ranked in a variety of ways.
Since the two documents are related, much of the content from the first one is duplicated in this second document.
It does go into more detail about the metadata associated with images, videos, and audio in its “summary of the invention” section.
Metadata means things like:
- tags,
- comments,
- annotations,
- descriptions,
- additions to favorites (“favoriting”) or playlists (“playlisting”), and;
- location information.
Conclusion
I hadn’t really looked at the “interestingness” rankings and metadata of Flickr before, but they do seem to provide a nice framework for indexing non-textual content. They could also be useful on a site like Digg. Would the personalization described here work on Digg? Does it work well on Flickr?
There are some issues about the practice of tagging that may limit some of its effectiveness. One of the main ones I see is that if other peoples’ tags can be seen when someone tags a picture, they may be influenced in their selection of tags for a photo by those who have tagged it before them and not use the words that they would have chosen in the absence of those tags.
[Disclaimer, I am the Evangelist and CEO of Zooomr]
This sucks. Yahoo should not be able to “own” either their user’s data nor how other non Yahoo communities choose to organize, categorize or showcase their own community of users’ data.
This patent would seem to me to say that a whole host of media sites from Digg to Reddit to Zooomr would be violating Yahoo’s exclusive right to showcase user content based on activity.
Highlighting the best media in any community based on activity should not “belong” to Yahoo. Each community ought to feel free to sort and rank their community’s data anyway that they want without having to worry about a lawsuit from Yahoo.
http://thomashawk.com/2006/10/should-yahoo-own-social-search-and.html
Hi Thomas,
Really like your comments on Jeff’s post.
At this point, these are still just patent applications, and haven’t been granted, and are still under review and challengeable.
Yahoo only had a certain amount of time to patent this process, or they couldn’t protect it. Probably something that they would want to do when competiting with the Google’s and Microsoft’s of the world. Unfortunately, it could impact upon the many other sites that are exploring this area.
One important question is if there is existing prior art where someone has been doing something very similar already. I think that’s a possibility.
How harmful might this be to you? When it comes to search, there are a lot of things that Google, Yahoo, and Microsoft are doing that end up with similar results, yet use different processes, Many of those are patent protected, yet the way they are described are different enough from each other that they aren’t stepping on each other’s patents.
For instance, the idea of using links to determine some type of link popularity or materiality is common in some ways amongst each. Likewise a relevance measure based upon words found upon a page – all of them use it, yet in different ways. User behavior is also likely being used by all three, yet in different manners.
I like the Tabblo statement, which describes some of the potential flaws involved in a process like this, and the idea of the development of a different approach that is more meaningful. I think that was a great response to the publications of these patents.
I don’t think the people at Zooomr or Digg or Reddit should treat this as an impediment, but rather as an opportunity to understand, and to differentiate themselves in a meaningful manner.
Do your own thing, do it well, and do it differently. The benefit that you have now is that Yahoo’s cards are on the table and they’re showing.
Thanks for the response William. And I’m glad to hear that this patent hasn’t actually been granted yet. I thought it had. Do you know what the process is for it to go through next and do you know if there is a formal way to express an objection to the patent?
At Zooomr we are developing our own algorithm for both search and photos to highlight and showcase based on our own community’s input. Most significantly our algorithm will involve a combination of our user’s activity (especially when user’s favorite photos) as well as subjective based Zooomr driven editorial standards and input.
This concept is not something new though. In fact I wrote an article entitled “Better Than Flickr… Better Than Picasa.. It Just Doesn’t Exist Yet” http://thomashawk.com/2005/01/better-than-flickr-better-than-picasa.html where I described a system for both search and rank based on user generated activity.
“The problem with Google Image Search is in the quality of the searches. Many searches are somewhat meaningless because there are limited tools to refine search. Type in “Ween” and you get 15,200 images. Some of these images are obviously better than others. Refine the search down to only “large’ images and you still get 673 images. Now Ween the band is no Britney Spears but if I’m a Ween fan, and I am, what I need is some kind of ranking system. This ranking system should have three components: rank by photo views, rank by user votes, rank by Photohawk Editors. The idea here is that there are initially three different ways to judge the subjectivity of a photo.”
I wrote that post in January of 2005, well before Flickr came out with interestingness.
For one company to try to own the idea of ranking of both highligted photos or search priority based on user input is just wrong in my opinion. Each community should feel free to create their own systems and algorithms based on their community’s activity without Yahoo owning the concept of a social network.
We are definitely doing our own thing and our algorithm will in fact be very different than Yahoo’s. But it will very much be based on the activity of our community and if by doing this we run afoul of a Yahoo! patent then this sucks.
I’d love to understand this patent more and what it’s implications are because there are many other social networks that use user generated activity to rank and organize their media.
I appreciate your digging out this patent information and sharing it with us.
Hi Honor,
It will be interesting to see if the patent applications do get past the patent examiner, and are found to be non-obvious. For Yahoo to even begin to try to claim some trademark protection, they had to file within a certain amount of time of starting to use the process described.
As I mentioned above, it’s not necessarily a bad thing for people involved in this field to see how Yahoo may be ranking images, or at least how they say they are in a patent application.
This is looks like a desperate attempt from a company that no longer is seen as innovative. As they lag behind Google, and web 2.0 startups, the only way for Yahoo to protect itself is to make such attempts at filing invalid patents. A requirement for a patent to be acceptable is that it is non-obvious. Maybe to whoever read the patent application, this was a non-obvious invention, but to most of us Silicon Valley and working on Internet products for many years, this is certainly obvious. I find it really desperate that such a large company as Yahoo to be wasting such time filing invalid patents instead of trying to innovate like Google.
Maybe “Interestingness” could have been filed as a Trademark, and I think that might have been ok, but “Interestingness” as a trademark is just totally lame.
You’re welcome, Thomas.
There may be ways to present challenges to the patent application. It’s not something that I’ve gone through, and patent law is complex enough that they are something you probably want to talk to an attorney about.
I did see this, looking around, and it seems like it presents some options:
Challenging Patents – Short of Litigation. One of the methods described that may (or may not) be available to you looks like it needs to be filed within 2 months of the publication of the patent application. So if you want to go through some type of challenge process, you may need to act quickly.
I’d hate to see these patent applications acting to interfere with much of the innovation I see happening with companies like yours. Chances are that if what you are developing is significantly different, then these patent applications shouldn’t impede your efforts. But it’s worth exploring that more fully.
I ended up here, at this site, when I started to ponder about encouraging google to patent concepts. Is all the patenting you are talking about just about this whole “interestingness” issue? Is there an international internet law that has to be followed? Are their actual courts that decide cyber-rights and cyber-wrongs? Whatever the case… could I patent a product that has never been imagined before but simply looks real enough in an image?
(Previous comment)
1:35 AM
Tuesday, January 16th, 2007
Thomas Michael Keirnan,
Santa Cruz,
Bolivia,
South America
(Moderate this last one)
Some of the greatest inventions come from ideas and suggestions in comic books. Just like no one right now has the slightest clue as to making time travel possible; a thousand years ago no one would have imagined having a conversation with someone halfway across the earth.
So, doesn’t the stranger’s design on the napkin get part of the credit earned by the engineer’s blue prints?
The Intellectual Property Patent Law in reference to ideas states:
A patent cannot be obtained on a mere idea or suggestion. Methods of doing business and printed matter cannot be patented.
Who created this Intellectual Property Patent Law? The U.S. Government? Well, what about the Bolivian Government, Does the United States suddenly rule intellectual property internationally?
Well, my idea would be to legalize the patenting of international functional concepts, ideas, and suggestions for future inventions, here, in Bolivia, under the laws of my revolutionary government.
Simply said, the earliest proven creator is the owner of the concept.
Thomas Michael Keirnan, Patenting Concepts Concept – 1/16/07 – the first patented concept to turn into a business.
2:23 AM
Tuesday, January 16th, 2007
Thomas Michael Keirnan,
Bolivian Government
Santa Cruz,
Bolivia,
South America
Hi Thomas,
Thank you for stopping by, and for asking some interesting questions.
In this post, I’ve written about one patent application that has been applied for, but not yet granted. It may never be granted, but that depends upon a patent examiner making a decision as to whether it is something that can be patented.
Very generally, something must be new, useful, and non-obvious. The US Patent Office decides What Can Be Patented.
This post is about one patent application, but I’ve written about many, from Google, Yahoo, Microsoft, and other companies. If I refer to something as a “patent application,” that often means that it hasn’t been granted yet. If I refer to it as a granted patent, or just a patent, that means that it has.
Something filed to be protected as a patent has to describe an actual process, and not just a general idea. As the patent office states, in my link above:
There is a World Intellectual Property Organization (WIPO), and there is a method to apply for a patent that will be granted protection of the laws across many countries, the Patent Cooperation Treaty. I’ve read that the Bolivian Senate has ratified that Treaty in July of 2003, but that no action has been taken to enforce it there.
I’d love to see Bolivia be involved with other countries in the formation of International Law involving intellectual property, and I appreciate your posting and asking about the law.
It is possible to file a patent for something that doesn’t exist yet, but that could be developed. Patent protection only lasts for a limited amount of time, so filing a patent for something that couldn’t be developed yet, with existing technology is a gamble. In the US, as I noted, the invention described would have to be new, nonobvious, and useful. Assuming that you have something that fits all of those, an ideal might possibly be patentable.
Interestingness huh? I suspect based on the information above it would be pretty easy to game the system like Digg for example
Hi Brisbane,
I’m not so sure. I expect that there have been a lot of lessons learned by the team that put together the interestingness algorithm in the face of actual practice and implementation
William, thanks for this article.
Do you know if the patent has been granted? Google Patent Search still shows it as an application. I don’t know much about the patent granting process, so I’m sorry if the answer is obvious.
Hi tayfun,
The Flickr/Yahoo patent on interestingness has not been granted, and there’s some question about whether or not it will ever be granted. A final rejection notice was sent out by the US patent office in January 2009, based upon the exisitence of some previously filed applications that cover similar territory. Here are two of the applications that were cited in the final rejection notice:
There is room for some actions to be taken after a final rejection, and it appears that some of those are being pursued by Yahoo.
It is possible to file a patent for something that doesn’t exist yet, but that could be developed. Patent protection only lasts for a limited amount of time, so filing a patent for something that couldn’t be developed yet, with existing technology is a gamble. In the US, as I noted, the invention described would have to be new, nonobvious, and useful. Assuming that you have something that fits all of those, an ideal might possibly be patentable.
Are you referring to the one year ‘grace period’ before filing for a full patent?
Hi Phillip,
No, I was responding to a question above about the basic tenets of applying for a patent, meeting the requirements of presenting an idea in the form of an actual process that could be invented that was “new, nonobvious, and useful.” One doesn’t need a real working prototype in front of them to file for a patent. But, as I noted in that comment: