There are a lot of pages on the Web that conventional search engines can’t find, crawl, index, and show to searchers. The University of California (UC), funded partially by the US Government, has been working to change that.
When you search the Web at Google or Yahoo or Bing, you really aren’t searching the Web, but rather the indices that those search engines have created of the Web. To some degree, it’s like searching on a map of a place instead of the place itself. The map is only as good as the people mapping it.
Map makers have consistently worked to develop new ways to get more information about the areas that they survey. For example, a New Deal program in the 1930s under the Agricultural Adjustment Administration led to the creation (pdf) of a $ 3,000,000 map. Continue reading Mapping the Deep Web
Paul Boag wrote a post at his site Boagworld asking a number of questions about SEO. I started writing a comment at his blog, but it quickly grew to become longer than his post and the questions and comments that he had about SEO, so I decided to post my response here.
In Paul’s post, Why I don’t get SEO, he came up with five reasons why he had doubts about SEO. My response doesn’t address his concerns in the order that he asked them, and it touches upon some of the comments written by others as well. If you have questions or concerns about SEO that aren’t addressed in this response, please feel free to ask them in the comments below.
What is Good SEO?
Good SEO is not “cheating the system,” or “manipulating search results.” Good SEO is part of a marketing plan that makes it more likely that the good content you create will be found by people who might be interested in what your web site has to offer.
Continue reading Good SEO
When someone types “George Washington” into a search box, they are probably more interested in the Revolutionary War general and President than some random George in Washington. A search for “Washington Hotels” is more likely looking for lodging in Washington than hotels named Washington. Searches for places with signs that say “Washington Slept Here” are probably not about hotels (and those searchers probably have too much time on their hands).
When words used in search queries can have more than one meaning, a search engine may provide better search results to searchers if the search engines can calculate a probability of the most likely meaning of that word. That’s the focus of a patent granted to Yahoo this past week:
Continue reading Playing the Odds: How Probable Meanings May Influence Search Engine Rankings
Three patents granted today to Google, Microsoft, and Yahoo all describe how each of the search engines might take a close look at page addresses, or URLs on dynamic web sites.
I wrote about the patent from Microsoft back when it had just been published as a pending patent application, in Microsoft Creating Rules for Canonical URLs. It appears that the patent examiner who reviewed the patent saw my blog post, because it is referred to in the patent within the “other references” section (Slawski, “Microsoft Creating Rules for Canonical URLs,” Sep. 29th, 2006, pp. 1-5. cited by examiner.). I don’t know if it is the first blog post to be cited as a reference in a granted patent (probably not), but it’s the first of my posts to be listed in one.
All three patents take a close look at the structures of URLs on dynamic web pages, which can often include large amounts of information within those URLs. For example, here’s a link to a page about a pair of jeans:
Continue reading Google, Microsoft, and Yahoo Patents on Interpreting Dynamic Page URLs
A misconception about web pages that lingered for a long time on the Web was that most people who visit your web site will enter your site on your home page. Another one is that the meta description you choose for a page will usually be what a search engines shows as the snippet, or summary, for your page in search results.
Search engines have made it a lot easier for a visitor to enter your site at pages other than your home page. And the summary, or description snippet, that those search engines provide about pages listed in search results are more likely to be taken from text on your page that matches the query terms used to find your page, especially if your meta description doesn’t include that text.
You’ve created a web page, carefully chosen a title for that page that carefully describes the contents of that page, and uses a keyword phrase that you hope your audience will use to try to find the page. You created a meta description for the page that is persuasive, engaging, and (you hope) likely to convince visitors to click on the link to your page when they see it in search results.
How likely is it that a search engine will show your page title and your meta description when your page does show up in search results?
Continue reading Why a Search Engine Might Choose Something Other Than Meta Descriptions for Page Summaries in Search Results
Sometimes when you search at one of the major search engines, you’ll see an extra set of links showing up under one of the listings in those search results. Referred to as either quicklinks or site links, most often those will show up for the listing at the top of the search results like in the following image:
Sometimes, those extra links will also appear for pages listed a little futher down in search results as well. A lot of questions have been raised about how those search engines decide which pages to show as quicklinks or site links, and the reasons why. I’ve written a number of posts about whitepapers and patent filings from the search engines that have provided some clues to answer those questions, and there’s a list of links to those posts at the bottom of this post.
But, another mystery surrounds those quicklinks or site links, which is how search engines might decide upon the text used in those links. I haven’t seen an answer from any of the search engines previously. At least, until now.
Continue reading How a Search Engine Might Choose Text for Quicklinks or Site links
One of the words that often appears when someone describes how search engines work is relevance. A search engine attempts to show searchers web pages and other results that might be relevant to the words that they used when they perform a search. Yet, there are a number of different ways that you can define relevance.
For instance, Rutger’s professor Tefko Saracevic, who has been studying the concept of relevance for years, explores different thoughts and literature on the topic to describe a number of ways to define relevance in a 2006 paper on Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance*.
Relevance could be considered a way of finding documents that contain words someone might search for, or documents that are related to concepts involved in those query terms. Relevance could be determined by looking at a relationship between a searcher and the search terms they use, while considering their past browsing and searching history, and possibly the searches of people who might socially related to them, or who share some common interests with them.
Relevance could also be determined by a problem or task that a searcher is faced with when performing a search.
Continue reading How a Search Engine Might Weigh Pages with Relevant Annotations Higher in Search Results
When you perform a search at a search engine, you usually see a list of links to web pages in response to your search.
Over the past few years, search engine have started showing a mix of other types of results, including images, links to related news stories or blog posts, videos, book and music search results, listings of reviews, maps and business location information, related search queries and query suggestions, stock charts, weather forecasts, and other non-web page listings.
This richer mixture of choices presented by search engines in response to searchers’ queries provides an often colorful and often useful set of options to someone searching for information or to fulfill some kind of task.
The query suggestions and refinements that searchers are offered are intended to help searchers with suggestions of other searches that might yield them more information. The mix of non-web page results are often referred to by search engines as blended or universal search results.
Continue reading The Evolution of Search Results into Query Portals