This is the second part of a series on Google Custom Search Engines.
Why spend so much time looking at Google Custom Search Engines? Here are a few reasons which I’ve written about in previous posts:
- Google Subscribed Links, which can be created in Google Custom Search Engines, sometimes appear in Google’s Web Search even if you don’t subscribe to those links.
- Google’s patent describing their Trust Rank approach explores how the kind of labels used as annotations by trusted sources (such as some Custom Search Engine builders) might influence web search results.
- Another patent application from Google explains how labels, which can be created in Google Custom Search Engines, might affect the classification of Web pages by Google, and help to define query refinements that appear above Web search results, as does an additional granted Google Patent describing how Google might be Filtering search results using annotations.
In the first part of this series of posts on Google Custom Search Engines and, SEO and Assumptions behind Web Searches, I mentioned that often the assumptions uncovered in a patent, about the Web, about search, and about searchers, were one of the key takeaways one could receive from reading a patent.
Another very useful approach to understanding the context of a patent and the research behind it involves learning more about the inventor of a patent, and learning what else he or she may have been involved in.
The inventor behind the patent Aggregating Context Data for Programmable Search Engines is Ramanathan V. Guha. While at Google, he started Google Custom Search Engines, Google’s Search Based Keyword Tool, and Google SMS Channels.
In May of last year the Google Webmaster Central blog published a post entitled Introducing Rich Snippets, which described how web publishers could add special markup formats (microformats and RDFa) to their webpages that might be shown as snippets in search results.
These enhanced snippets would enable you to possibly have things in your snippets such as starred ratings, an indication of how many reviews a page might contain, and more, based upon meta data that you include in the HTML code used to present that information on a page. The authors of the post were Kavi Goel, Ramanathan V. Guha, and Othar Hansson.
It’s not surprising that Ramanathan V. Guha was included as one of the authors. He was involved in the development of an earlier version of one of those XML formats, RDF, while working with Netscape Corporation.
Some of the other places he worked at before joining Google in 2005 includes Apple and IBM, and he was a co-founder of Epinions. At Apple, he developed the Meta Content Framework (MCF) format.
At Netscape, in addition to RDF, he developed Netscape’s smart browsing feature and the earliest version of RSS, to display feeds on Netscape homepages.
If you want to learn more about the things he worked upon in the past, he has a copy of his resume on his website, and he was interviewed by Marc Andreessen in 1999 on the Netscape blog, available through the internet archive at Innovators of the Net: Ramanathan V. Guha and RDF.
He’s also listed as the inventor or co-inventor behind the patents I wrote about in my list at the start of this post.
Guha’s previous work with specialized XML formats such as RDF and RSS is essential to the way that an advanced Custom Search Engine might work.
Google Custom Search Engines not only allows you to customize a search engine by deciding which sites to include in search results, but it also provides a way to customize how search results are presented to searchers, allowing you to change around the way pages might be ranked, enabling you to emphasize some pages over others and include special information at the tops of search results, and to offer things like query refinements above page results listings.
Some of this can be done using a control panel when you set up a custom search, or you can create XML files that define how your custom search engine may work.
XML and Google’s Custom Search Engines
If you create Google Custom Search Engines, you have a few different choices on how to set it to work.
You could just add one or more whole websites or pages or parts of websites to be included as sources for the search engine. If you do that, the order that pages are shown in search results are based upon the ranking algorithm that Google uses in its web search.
While you can set up a fairly simple custom search engine (see the Getting Started Guide) if you want, you can also do things like add keywords to your search engine that might boost certain pages on search results if they contain a searcher’s query terms and the keywords that you select.
But Google’s custom search also provides other ways to change how your search results appear.
For example, it allows people to set up context files to describe the structure of custom search engines and define how they behave.
You can set up Subscribed Links, which enable you to set up snippets for specific queries that you define either through a control panel or your context file.
You can also create promotions for specific queries, which might move up a specified page in search results that you want to draw attention to, useful for “making announcements and promoting products, services, events, and content that you want your users to discover.”
Another important aspect of Google Custom Search Engines is the ability to create query refinements for your search results that appear above those results.
You can create refinement labels for sites. If someone performs a fairly broad search, labels associated with sites in the search results may appear above the results that can help searchers drill down to pages about specific topics.
For example, someone searches for “diabetes.” Query refinements would show up above search results if sites listed on a search for “diabetes,” were labeled with terms such as “treatment,” “Symptoms,” “Diagnosis,” “Causes.” Those refinements would match the labels used.
For a more detailed look at how you can make changes to search results in a Custom Search Engine, the Google Custom Search Developer’s Guide is essential reading.
In my next post on Google Custom Search Engines, I’ll look more deeply into how aggregated information from multiple Custom Search Engines might be used to influence what you see when you perform a search on Google’s main web search, as described in the most recently granted patent on Google Custom Search Engines.
That Google researcher Ramanathan Guha sure seems like a genius of its own.
I’ll dig up some of his info, I’ve read some of his stuff including his paper http://bakara.eng.tau.ac.il/~semcomm/GKRT.pdf he really seems like an innovator no matter what he does.
But I’m not sure sure about this patent, does this mean using an internal search engine (Google custom search engine?) will help Google Web Search?
Hi Jeremy,
Interesting list of authors on that document you link to. All of them seem to have moved from IBM to do something with one or more of the major commercial search engines.
The patent describes a number of ways that people creating and using Google CSE might potentially influence the search results we see in Google’s main Web search. I’ll go into that in more detail on the next post in this series.
Here is one potential way that the Google Method could be applied to custom search.
Let’s say you have 100 Websites in a given vertical that all define CSEs. The each site included in these CSE databases gets a value assigned to it. Now, if any of the sites included in the databases also have CSEs, they can then pass their CSE value to the sites they include in their CSEs.
Although this system is not as closed as a typical PageRank-like scenario, in a small enough vertical or a large enough selection of Websites with CSEs you might begin to see a citation-based probability distribution develop.
As I have noted many times through the years, citation-based analysis is extremely flawed (according to the Math Union) for a number of reasons, but Google seems to continue using it in a variety of algorithms.
Hi Michael,
As much as Google might like popularity to be an indication of quality, it’s still more of an indiction of popularity and may often keep the best answers to queries from being identified and ranking high enough to be seen by searchers. It may be better in some ways that just providing a list of pages that all contain the keywords in a query that might also look at the frequency of usage of those keywords and how close those words appear together on a page, but a citation-based approach is still limited.
There may be some value in aggregating information, such as labels used, from multiple CSES on similar topics, but there are probably some limitations as well.
It’s not surprising to see that Google’s customization/personalization efforts are beginning to influence search result generation in general. I suppose it’s another method of sifting through the chaff and improving on search result relevancy. Since that’s what your next post is about, I’ll stay tuned.
Hi Tony,
Got a little sidetracked with Google’s Quick Scroll and its unannounced addition to the Google Toolbar. I’ll be back on track to write more about Google Custom Search Engines soon, but I may include some other posts before I finish up with the series.
Hmmm…To tell you the truth, I didn’t know any of this and it will prove very useful. On my site, I literally just plugged in the custom search code, and left it at that. Due to me not knowing about these features (Or just being really lazy) I have not taken advantage of everything that the custom search field has to offer. Especially “subscribed links” and “promotions”, which will come in handy in utilizing to get my product in front of even more consumers.
Kudos to another great post!
Hi Mike,
Thank you. I do think there’s a lot of value in learning more about custom search engines and their context files.
I’ve been wondering if Google’s new Brand, Store, and Type query refinements might be related to the process described in this patent.
See: http://googleblog.blogspot.com/2010/10/this-week-in-search-102910.html
There’s a possibility that it is.
Well, doesn’t that beat all. Bill, I haven’t stopped by here in a couple of weeks. I check in today and you do a series of blog posts on something I just posted about myself, GCSE’s. personally, I think they are great. The function that they include in which you list out specific pages to include is a great way to help them index things their crawler may have missed.
I think GCSE’s will continually play a larger and larger role in the future.
That being said, if somebody could create a GCSE that actually became a trusted and highly used niche search engine, as you pointed out they could definitely sell promoted search results quite easily. If an offer was good enough, they could even sell competitor exclusions. Now there’s a war for you.
Will it ever happen? Probably. Will it happen any time soon? Probably not.
Mark
Hi Mark,
Nice article. It is an interesting coincidence that we both covered Google’s CSE.
I think the potential is there for Google to start looking at external sources like the information found in context files from CSEs to show query refinements, like Google’s fairly new ones covering brands, sites, and places when it seems that a product search is happening.
I also think CSEs are a decent option for use as site searches, and potentially valuable resources as trusted domain specific searches.
Maybe not tomorrow, but not too far off in the future.
I would assume that google would maybe utilise custom search when relevant. Who knows? Like Bill says “Maybe not tomorrow, but not too far off in the future”
Hi Yoba,
Google Custom Search is available for webmasters to use for their own sites, or to create custom search engines that may involve a number of sites, and there are a lot of people using it.
It is possible that Google is aggregating that information from context files associated with custom search engines created by a lot of people as a signal for when to use certain query suggestions, and in other ways as well. It’s just hard for us to confirm whether or not they are doing that.
The thing I hate most about it is the fact that it makes it hard to keep track of your own ranking without using something like SEOmoz. On top of that, the SERPS will look different for everyone, which skews everything even more. It can be pretty frustrating.
Wow, I didn’t even know Google Custom Search exsisted, I know it sounds silly, but I didn’t. Thanks for the info, I’m going to read up on it a little more on the actual definition, etc. Thanks!
Hi Amber,
You’re welcome. It’s definitely worth trying building one or more Google Custom Search Engines, including trying out some of the more advanced stuff like the context files.
Hi,
I have got a query regarding google custom search engine.
I am using google custom search engine in my website.
I have disallowed google web search to crawl on some of my private pages through robots.txt.
Will it also effect the google custom search engine I have used in my website?
for example i have the below code in my robots.txt page
User-agent: *
Disallow: /users
Allow: /
I hope this code will disallow the google web search to show the pages under users/. But Will it also disallow the google custom search engine to list out the results under users page.