Search engines use programs to crawl the web, and identify new pages and newly updated pages to include in their index. These are often referred to as robots, or crawlers, or spiders. But there are other ways that the search engine gets information about pages that it might include in search results.
A whitepaper from Google, Sitemaps: Above and Beyond the Crawl of Duty (pdf), examines the effectiveness of XML sitemaps, which Google announced as an experiment called Google Sitemaps in 2005. The experiment seems to have been a success.
XML sitemaps are a way for web site owners to help the search engine index pages on their web sites, through the use of an xml Sitemap. Yahoo and Microsoft joined Google in adding support for XML sitemaps not long after, and a set of pages explaining the sitemaps protocol was launched.
The paper tells us that approximately 35 million websites publish XML sitemaps, as of October 2008, providing data for several billion URLs. While XML sitemaps have been adopted by a large number of sites, we haven’t had much information from any of the search engines on how helpful those sitemaps have been, how they might be used together with web crawling programs, and if they make a difference in how many pages get indexed, and how quickly.
Many tasks are trivial for humans but continue to challenge even the most sophisticated computer programs. Traditional computational approaches to solving such problems focus on improving artificial intelligence algorithms. Here, we advocate a different approach: the constructive channeling of human brainpower through computer games. Toward this goal, we present general design principles for the development and evaluation of a class of games we call “games with a purpose,” or GWAPs, in which people, as a side effect of playing, perform tasks computers are unable to perform.
– Designing Games With A Purpose
A paper from Yahoo researchers, Thumbs-Up: A Game for Playing to Rank Search Results, describes a game that they developed and tested internally at Yahoo to allow participants to compete against each other in ranking how relevant pages are for specific search queries.
Is there any value in using keywords in the URLs of web pages? Would a search engine look at keywords that you might include in the addresses of your pages, and associate those keywords with the content of your pages in the search engine’s index?
If so, how would a search engine go about looking at the web addresses indicated in the URLs to your pages, and break them down into meaningful parts to identify keywords?
Breaking URLs down into parts may also play a role in how the pages of a web site might be crawled by a search engine.
A newly published Yahoo patent application gives us some ideas on how it might extract keywords from the URLs of pages, and rank them, as well as using information uncovered in the process to determine which pages to crawl first from a web site.
When a search engine indexes pages and other documents on the web, hoping to provide meaningful and relevant results to searchers, it doesn’t just rely upon the content found on web pages, but also considers the quality and quantity of of links pointing to those pages.
A search engine like Google might determine that a page is relevant to a specific query based upon the content found on that page, and the anchor text found in links pointing to the page.
It might also look at what it considers “relationships” between pages by looking at how pages are linked to each other. PageRank is one method of viewing those links that Google states that it uses, and assigning a measure of importance to pages that are linked to from other pages. This measure, or rank might be simplified as a probability that someone might arrive at a certain page if they are arbitrarily and randomly clicking on links on pages that they’ve surfed.
Why does Google customize some search results based upon a previous query that you’ve performed? Is there a special relationship between those query terms, and if so, how did Google define that relationship?
Imagine searching for “luxury car” at Google, and then performing another search for “infiniti.” On the second search, you find a page in the search results that looks like it will provide you with information that you are looking for, and you select a page.
Now imagine that a number of other people perform the same series of searches and select the same page.
It’s possible that Google might start considering the search for “luxury car” and the search for “infiniti” to be related queries. It’s also possible that the page selected in the second search for “infiniti” might start ranking more highly for the query “luxury car.”
What concepts does your website cover?
A search engine might look at phrases that you use on your pages to get an idea of the concepts covered by your site.
The search engine might try to decide that certain phrases you use are the “top phrases” that describe topics or concepts about your site.
But what if the search engine is wrong?
What if those top phrases don’t reflect the content of your site accurately? What if some other phrases more meaningfully indicate what your site is about?
If a search engine assigned phrases to your site which might affect the way that your pages are being presented to searchers in responses to queries at the search engine, would you want the search engine to give you the chance to make changes to those phrases that they think your site is about?
New Google Phrase-Based Indexing Patent Filing
What’s the difference between icecream and ice cream, or paperclips and paper clips? How about sandpaper and sand paper, or thumbtack and thumb tack? A compound word comes about when two words can be joined together to form a new word.
In my examples, the meanings of each pair of two words joined together is the same as those two words as phrases.
When someone searches for icecream at Google, should they see the same search results for ice cream or icecream, given that the words mean the same thing? If a page about paperclips shows adsense advertisements, should the ads be for paper clips and paperclips? If an adwords advertiser runs adwords advertisements to show during search results, should their ads run both when someone searches for sandpaper and sand paper?
A few years back, I wrote about a patent filing that explored ways that Google might handle compound and hyphenated words and other spellings of words, in my post SEO and Compound Words, Inflections, and Alternative Spellings. That patent filing gave us a pretty high level overview of how it might treat compounds, but didn’t delve too deeply into the details.
Last week, another patent filing from Google came out on compound words, focusing primarily on describing how it might be used for search advertisements that it shows on web pages and during search results.
A woman says to a man, “It’s cold in here.” The man puts his arms around her and holds her. He could have turned up a thermostat, or brought her a sweater, or asked her if she wanted to go somewhere warmer. But imagine that the man and woman are in a relationship and have had that conversation a number of times in the past, and the intention behind the words was easily understood by both.
We can sometimes understand the intent behind certain words even though the words don’t actually match up well with the intentions of the people who voice them, though the intentions behind words can be difficult things to understand. Sometimes nonverbal communication that accompanies words can be helpful in interpreting them, though humans aren’t necessarily that good at reading nonverbal communication either.
Sometimes past experience can be informative in understanding what certain words might mean, like the man and woman in my example above.
If human beings can grow easily confused about the intentions behind words, how well can a computer understand the intent behind a handful or less words in a query at a search engine?