When Google’s “Panda” Algorithm Collides with Duplicate Content

If you’re a Webmaster, it’s likely you’ve heard about the Google latest search algorithm — “Panda” — and all the benefits and implications of this update. Today, we wanted highlight what happens when Google Panda collides online with duplicate content. There have been plenty of opinions written about Google Panda and duplicate content, but we want to provide some background and examples to help you better understand how Panda and duplicate content might affect you.

What is Duplicate Content?

Duplicate content is a term used in the field of search engine optimization to describe content that appears on more than one web page, within the same web site. When multiple pages within a web site contain essentially the same content, search engines such as Google can penalize/not display that site in any relevant search results.

Should you be Concerned?

When Google released Panda, there was a significant outcry from legitimate business and publishers who were either downgraded overnight in their search engine page rank or dropped all together. For many of the businesses, the Panda algorithm reduced SEO rank and decreased visitors, site revenue and online market awareness. Some websites even experienced damage to their brand, as their customers and prospects questioned whether they were still in business.

We’ve spoken with Cult of Mac, Digital Trends and several Fortune 1000 businesses, and they’ve all said the same thing: They were penalized and downgraded as a result of the Panda release as a result of unauthorized duplication of their content. They had done everything to comply with Google in optimizing their SEO configurations, but the third-party websites scraping and duplicating their content (outside of their control) caused their page ranks to fall.

Google’s Official Stance on Duplicate Content:

“We do a good job of choosing a version of the content to show in our search results.”

“In rare situations, our algorithm may select a URL from an external site that is hosting your content without your permission. If you believe that another site is duplicating your content in violation of copyright law, you may contact the site’s host to request removal. In addition, you can request that Google remove the infringing page from our search results by filing a request under the Digital Millennium Copyright Act.”

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66359

Where is This “External” Duplicate Content Coming From?

Sometimes, it’s not clear how third-party sites obtain copies of legitimate work. Typically, they either steal it by manually or automatically scraping the content. The scraped content is then republished onto their sites, providing no credit or link to the original work.

What does that look like? It’s not difficult to find examples, but I tracked one down that seemed particularly ironic. Here’s an original article by PC World on Google’s War Against Scraper Sites:

Here’s a duplicate copy of the same story that doesn’t give any credit to the original PC World article:

It’s clear that we’re not looking at a coincidence here. The title, article content and images are all identical. The scraping site didn’t even attempt to mask their plagiarism with synonym changes. Why would they do that? Just take a look at the ads on the scraper site … They want to profit from the keywords and traffic driven by PC World’s content.

What Can You Do About It?

• Listen to Google
  Google provides a list of tips for using rel=”nofollow” and canonicalization to ensure they are able to identify you as the original author of content and avoid penalizing or downgrading your business’s search ranking results.
• Learn About DMCA and Use It
  If your content has already been duplicated by unauthorized publishers, you should learn more about the Digital Millennium Copyright Act (DMCA) and how it can help get help remove your content from infringing websites. Two helpful resources to start learning the law and your rights are Google’s official DMCA policy page and the United States Copyright Office.
• Be Proactive About Stopping Scrapers
  We believe the best solution is to implement practices and or services to proactively prevent people or web scrapers from harvesting or scraping your content in the first place. Although web scrapers can be difficult to detect, there are tactics and/or services that can be implemented to limit certain behaviors on your website(s). Some of the quickest ways to make strides in the right direction are to implement rate limiting rules, to block traffic from blacklisted IP addresses and to use Captcha to help reduce automated web scrapers.

While none of these tactics are fool-proof ways to completely prevent your content from being duplicated, the more barriers to entry you have, the more difficult it will be for web scrapers to repeatedly duplicate your content. Distil built an enterprise-ready platform to monitor and prevent site scraping, so if you want some help in the protecting your content, try our our service. Whatever route you take, the key is to make sure that whatever tactics or services you implement, you don’t forget about your legitimate traffic … You don’t want to throw out the baby with the bathwater. Be proactive, but make sure you keep your priorities on the user-experience and quality of your site(s).

-Sean Harmer, Distil

Based on OpenStack Object Storage (codenamed Swift) — open-source software that allows the creation of redundant, scalable object storage on clusters of standardized servers — SoftLayer Object Storage provides customers with new opportunities to leverage cost-effective cloud-based storage and to simultaneously realize significant capex-related cost savings.

OpenStack has been phenomenally successful thanks to a global software community comprised of developers and other technologists that has built and tweaked a standards-based, massively scalable open-source platform for public and private cloud computing. The simple goal of the OpenStack project is to deliver code that enables any organization to create and offer feature-rich cloud computing services from industry-standard hardware. The overarching OpenStack technology consists of several interrelated project components: One for compute, one for an image service, one for object storage, and a few more projects in development.

SoftLayer Object Storage
Like the OpenStack Swift system on which it is based, SoftLayer Object Storage is not a file system or real-time data-storage system, rather it’s a long-term storage system for a more permanent type of static data that can be retrieved, leveraged and updated when necessary. Typical applications for this type of storage can involve virtual machine images, photo storage, email storage and backup archiving.

One of the primary benefits of Object Storage is the role that it can play in automating and streamlining data storage in cloud computing environments. SoftLayer Object Storage offers rich metadata features and search capability that can be leveraged to automate the way unstructured data gets accessed. In this way, SoftLayer Object Storage will provide organizations with new capabilities for improving overall data management and storage efficiency.

File Storage v. Object Storage
To better understand the difference between file storage and object storage, let’s look at how file storage and object storage differ when it comes to metadata and search for a simple photo image. When a digital camera or camera-enabled phone snaps a photo, it embeds a series of metadata values in the image. If you save the image in a standard image file format, you can search for it by standard file properties like name, date and size. If you save the same image with additional metadata as an object, you can set object metadata values for the image (after reading them from the image file). This detail provides granular search capability based on the metadata keys and values, in addition to the standard object properties. Here is a sample comparison of an image’s metadata value in both systems:

Using the rich metadata and search capability enabled by object storage, you would be able to search for all images with a dimension of 2240×1680 or a resolution of 72×72 in a quick/automated fashion. The object storage system “understands” more about what is being stored because it is able to differentiate files based on characteristics that you define.

What Makes SoftLayer Object Storage Different?
SoftLayer Object Storage features several unique features and ways for SoftLayer customers to upload, access and manage data:

• Search — Quickly access information through user-defined metadata key-value pairs, file name or unique identifier
• CDN — Serve your content globally over our high-performance content delivery network
• Private Network — Free, secure private network traffic between all data centers and storage cluster nodes
• API — Access to a full-feature OpenStack-compatible API with additional support for CDN and search integration
• Portal — Web application integrated into the SoftLayer portal
• Mobile — iPhone and Android mobile apps, with Windows Phone app coming soon
• Language Bindings — Feature-complete bindings for Java, PHP, Python and Ruby*

*Language bindings, documentation, and guides are available on SLDN.

We think SoftLayer Object Storage will be attractive to a broad range of current and prospective customers, from web-centric businesses dependent on file sharing and content distribution to legal/medical/financial-services companies which possess large volumes of data that must be stored securely while remaining readily accessible.

SoftLayer Object Storage significantly extends our cloud-services portfolio while substantially enriching the storage capabilities that we bring to our customers. What are you waiting for? Go order yourself some object storage @ $0.12/GB!

-Marc

On the end-user side, I just read an article about how public DNS providers like OpenDNS and Google are breaking the internet. OK, maybe not breaking the internet, but the public DNS providers are confusing CDN location-based algorithms. The article is here: http://www.sajalkayan.com/in-a-cdnd-world-opendns-is-the-enemy.html and I recommend strongly that both content providers and content consumers read it.

The summary is that some CDN algorithms use the ip address (and location) of the DNS server making the request and if that DNS server is nowhere near the end-user on the internet, the end-user will get served content from farther away and will get that content slower than desired. The conclusion is that an end-user should always use a DNS server located as close as possible network-wise, usually that ends up being a DNS server of the network provider.

That is good advice for the end-user, but what about the content provider? If you flip this around and come at DNS from the content provider’s point of view who doesn’t use CDN, you want to make sure that when a DNS request is made, that your authoritative DNS server gets the ip address as fast and reliably as possible back to the end-user.

SoftLayer has built out authoritative DNS farms in all our Datacenters and Network POPs and anycasted the ip addresses for the name servers. What that means is that SoftLayer customers – who get to use our DNS for free – can have their authoritative domain services hosted at all 10 points in North America and through the routing optimization inherent in the internet, the name to number conversion for those domains will happen as close as possible to the end-user and the results will be delivered as quickly as possible.

One very important goal of every content provider is to get the end-user the best experience as possible. Understanding how the internet works from the end-user and well as the server-side is critical. It doesn’t matter how good your content or app is if the end-user has a poor experience.

-@nday91

A recent article talks about the impact of latency on web site viewers. It sounds like common sense that the slower a site is, the more viewers you lose, but what is amazing is that even a latency measured in milliseconds can cost a web site viewers.

The article focuses mainly on application specific solutions to latency, and briefly mentions how to deliver static content like images, videos, documents, etc. There are a couple ways to solve the static content delivery problem such as making your web server as efficient as you can. But that can only help so much. Physics – the speed of light – starts to be your enemy. If you are truly worried about shaving milliseconds off your content delivery time, you have to get your content closer to your viewers.

You can do this yourself by getting servers in datacenters in multiple sites in different geographic locations. This isn’t the easiest solution for everyone but does have its advantages such as keeping you in absolute control of your content. The much easier option is to use a CDN (Content Delivery Network).

CDNs are getting more popular and the price is dropping rapidly. Akamai isn’t the only game in town anymore and you don’t have to pay dollars per GB of traffic or sign a contract with a large commit for a multi-year time frame. CDN traffic costs can be very competitive costing only a few pennies more per Gb compared with traffic costs from a shared or dedicated server. Plus, CDNs optimize their servers for delivering content quickly.

Just to throw some math into the discussion let’s see how long it would take an electron to go from New York to San Francisco (4,125,910 meters / 299,792,458 meters per second = 13.7 milliseconds). 13.7 millisconds one way, now double that for the request to go there and the response to return. Now we are up to 27.4 milliseconds. And that is assuming a straight shot with no routers slowing things down. Let’s look at Melbourne to London. (16,891,360 meters / 299,792,458 meters per second = 56.3 milliseconds). Now double that, throw in some router overhead and you can see that the delays are starting to be noticeable.

The moral of the story is that for most everybody, distributing static content geographically using a CDN is the right thing to do. That problem has been solved. The harder problem is how to get your application running as efficiently as possible. I’ll leave that topic for another time.

-@nday91