Good questions to ask when it comes to duplicate content.

There are a number of different ways that search engines respond to duplicate content. When it’s on the same site, they may try to identify one version that is the canonical one – or the best one to show in search results. A problem is that those duplications may mean that the PageRank or link equity to pages may not be as strong as it should be.

Search engines will look for duplicate content during the crawling of web pages, during indexing, and when they display search results. What they look for may vary at the different stages, and how they respond may as well. For instance, if a search engine comes across a mirror of a site, with the same content and link structure (the URLs may vary, but the way pages are connected together may be the same), a search engine may stop visiting and crawling one version. When search engines see a lot of duplicate pages on the same site (or pages that are very similar), the crawling programs may not index the pages of that site very deeply unless it has a lot of links pointing to it and/or a lot of PageRank or link equity. When smaller parts amounts of content are duplicated, and those parts might appear in the snippets for the pages as they would appear in search results, some of those results might be filtered from showing in search results.

Another issue is that while content may be duplicated on pages, it isn’t always an exact duplicate. There’s often different content headings, footers, sidebars and even some changes to the content in the main content areas of pages. “Near Duplicate” content can be a little harder to identify than exact copies. That doesn’t mean that search engines can’t identify it, and the best result for a search engine may not always be to remove versions from their index.

When content is duplicated on other sites, it isn’t always clear to search engines which version is the original and which might be a copy. Some duplication of content is “legitimate” in the eyes of search engines, such as the syndication of content or the use of wire stories by news sources. Others may not be as legitimate. Search engines will try to show pages in the search engines from the most authoritative page, or the one that has the highest pagerank, but that doesn’t mean that they will necessarily remove content from the Web that is duplicate. They will often try to not show duplicates in responses to queries, so that searchers don’t see the same content in every search result.

Sometimes you have to take things in your own hands when someone duplicates your content without permission or a license, such as filing a DMCA complaint with the search engines or a web host, or contacting the host about an authorized use policy infraction.

It’s possible that I introduced that element of concern into this post when I wrote it. I’m all for the search engines trying to find the right URL for a page when there is more than one possible option, but I’m a firm believer that if a developer or designer can make it so that a search engine doesn’t have to make a choice like that, then it’s better for everyone – One URL per page is ideal.

I was surprised that all three were granted on the same day, since they have so many similarities. At this point, I’m not sure if the problem might be that these patents could be used to exclude others from doing what they describe, or that anyone could patent very similar processes if they wanted to if they were able to describe in the processes differently enough.