Creation and Ranking of Image Annotation Suggestions

Sharing is caring!

The ability to annotate pictures, songs, video, and web pages is growing as site owners attempt to find new ways to build communities and attract visitors to their sites.

Enabling people to add their thoughts and opinions, their words to a page means that a site can grow and evolve with the community that it was built for and that it becomes more interesting and valuable to other visitors.

Some very popular sites that allow for annotations include Flickr, Delicious, StumbleUpon, and YouTube, and the original intent behind Google may have been to build an annotation system. Knowing what other people are saying about a site or a photo or a video can be pretty interesting.

Several questions can be raised about annotations:

  • How do you encourage people to leave an annotation of their own?
  • Will offering annotation suggestions help?
  • If so, how do you rank the suggestions that you provide?
  • Might it make a difference based upon the kind of devices that people are using, such as a desktop computer, or a handheld camera phone?
  • Can the personal history of past annotations help with future annotation suggestions?
  • Can you build community with a site offering annotations, and can community input and input of all members of a site help offer annotation suggestions?
  • What role might annotations play in how search engines return which results to searchers?

A patent application from Yahoo explores suggestions for annotations, offering some examples from ZoneTag and Flickr, though some of the concepts behind the annotation suggestions discussed could easily be added to places like Delicious.

And don’t discount the possibility that annotations and user reactions could find their way into search results.

Annotations as Meta Data for Media

One area of growth on the web is the number of photographs appearing online, with a fast increase in the number of devices like digital cameras and camera-phones and a decrease in the cost of storing media.

While such growth is happening, the creation and capture of semantic metadata about photos and other media content, that is relevant to the content or context of that media can be difficult to capture.

It can be almost get people to go through a large collection of images, videos, songs, and web pages all at once and add information about those objects. It’s likely much better to allow them to do that as they upload images, bookmark pages, view videos, and comment upon the objects that they see.

Making it easier for people to tag and annotate those, and offering intelligent suggestions for them to do so may be helpful.

I’ve written a couple of posts before on Yahoo’s efforts to help people tag images – Yahoo’s Studies on Mobile Camera Phones and Geo-Referencing Images and Exploring Yahoo’s Photosharing Applications. The papers and pages linked to those posts provide some background on what Yahoo has been doing with annotations of media objects.

The patent filing adds more information on a specific aspect of annotations – suggestions for annotations that might be based upon many factors, including an individual’s location, their history of providing annotations, as well as annotations from a social network that they may be part of, or from members of the general public.

That information can be helpful to potential future viewers or visitors, in finding media objects that they might want to see.

Context-Based Community-Driven Suggestions for Media Annotation
Invented by Mor Naaman, Marc E. Davis, Shane P. Ahern, Simon P. King, Rahul Nair, and Jeannie Hui-I Yang
Assigned to Yahoo
US Patent Application 20080195657
Published August 14, 2008
Filed February 8, 2007


Disclosed are apparatus and methods for facilitating the annotation of media objects by a user. Mechanisms present a user with an easily usable set of annotation suggestions that are most likely to be relevant to the particular user and/or media context.

In general, existing annotations are analyzed to determine a set of suggested annotations. Annotation suggestions for a particular user are based on an analysis of the relevance, to the particular user, of existing annotations of one or more media objects so that the most likely relevant annotations are presented as suggested annotations.

In particular embodiments, this analysis depends on whether the existing annotations were created and/or selected by the particular user, a member of the particular user’s social network, or members of the general public.

Facilitating Media Annotation

The idea behind this patent application is to make it easier for people to leave annotations for some kind of media such as photos. by providing suggestions based upon the context of the objects and the social community of a user, and to include a way to rank those annotation suggestions in a way that might be relevant to someone who might want to use the suggestions.

In Zonetag, these annotation suggestions might be based upon the location of someone uploading an image using a camera phone, or from previous annotations from others who have taken and uploaded an image at the same location.

The length of time that has passed since that previous annotation might be meaningful, as well as if the annotation was left by someone related in a shared social network.

Sources of Annotations

Suggestions for annotations may be gathered from a number of sources, such as:

  • Information on a media organization system,
  • A personal media collection source,
  • A web-based media collection source,
  • A blog associated with a photo,
  • A caption of a photo,
  • Text on a web page that is near a photo,
  • A tag associated with a photo,
  • Other sources.

Further Reading and Activities

Install Zonetag on compatible Motorola or Nokia phones

Install the location-based photo browser Zurfer on compatible phones – when you visit a place, you can see images and annotations from others who visited that location.

Explore Yahoo’s new location data application, Fire Eagle, and learn more about how it can be used to share information about your location on different social networks at the Blog on Fire pages. A Fire Eagle API may help developers create applications that can use the location data shared by Fire Eagle users.

Sharing is caring!

8 thoughts on “Creation and Ranking of Image Annotation Suggestions”

  1. Once again, thought provoking post! I find it so interesting that because there is no “standardized web platform” that the search engines must jump through a series of far-fetched hoops to assign value to a website. Its hard to believe that as advanced as we are today nobody has said… lets get together than create a platform where we can easily identify what is text or what is a product/event/service, etc. if the data was stored in logical format (such as in a database or in structured XML) that the relevance would be inherent in the data (ie. location/date/industry, etc).

    Instead we have a multitude of diluted tools and floating in the sea which is web 2.0 that further fragments the web instead of bringing it together and making both search engines and end-users lives easier. As a SEM enthusiast I find it fun being the kid who walks around the messy room picking up different building blocks off the floor and figuring out how I can piece them all together to create a web castle, but I cant expect my clients to have the time to investigate them all.

    Anyways, thats just my typical ranting about how the internet is broken 🙂 Have a great week Bill, thanks again for the post!

  2. Annotations as Meta-Data sounds quite interesting. Annotations on your described gallery situation could be done with ajax scripts , which is realy easy for the end-user at all. But it could be misused with wrong ones , but that can happen with nearly anthing …

    Great Blog
    Regards !

  3. Hi Rob,

    Appreciate the Rant. As much as I like standards, I think that there’s a balancing act that needs to be done with innovation, too. I like a number of things that Yahoo is working on in this area, including their ZoneTag work. I just wish that it were compatible with a wider range of phones.

    And we do have search engines attempting to develop some standardized platforms, like Google’s Android, which may just go beyond mobile devices.

    Part of the fun for me too, is exploring all of the different approaches and technologies coming out, and trying to make sense of them.


    Hi div^

    Thanks for visiting and commenting. Search engines have been using link text pointing to pages as if it were meta data for a number of years. I like that annotation, from visitors to pages instead of publishers, can be used in the same way.

    Ajax scripts might work as long as search engine spiders can read the annotations that users might leave.

    Good to meet you.

  4. Very interesting read in the morning 🙂 Rather than putting “bulky text” in the page, this could be a nice way.

    As with anything else, i am sure spam would also creep in 😉 but without that, i am sure there wouldnt be no fun 😀

  5. Hi Praveen,

    Thank you. Your point about the possibility of spam is a very good one. There may be some ways to try to keep spam out of annotations. This paper points out user identification management and allowing other users of a system to rate previous annotations:

    Towards Trust for Semantic Web Annotations (pdf)

    Other methods might be helpful for identifying and filtering spam in annotations may be used, too. Those might include blacklists for certain words, identifying annotations that may appear automated in nature, and other approaches.

    There are also some interesting papers coming out of the Adversarial Information Retrieval on the Web (AIRWeb) workshops, such as the following:

    The Anti-Social Tagger – Detecting Spam in Social Bookmarking Systems (pdf)

    Combating Spam in Tagging Systems (pdf)

  6. I seem to remember that there used to be some sort of web browser plugin that could let you draw/write directly on top of a web page. Other people using that plugin could then see what you had created. Can’t remember what it’s called. It was described as a big web grafitti mess but something like that (maybe you only see what a select community has written) could be useful.

  7. Hi Scott,

    There have been a number of services on the web that would enable people to make annotations for pages. I remember a few in the late 90s.

    Of those, I think that Third Voice was the one that you are referring to:

    But the seemingly innocuous “sticky notes” gained enemies quicker than users. Launching a grassroots campaign called Say No to TV, some 400 independent Web hosts banded together to gag Third Voice, which they likened to “Web graffiti.”

    There are a number of more modern approaches to annotations for web pages that seem to be thriving, such as Stumbleupon, Delicious, Digg, and many others.

Comments are closed.