HubPages is an open publishing platform, meaning that we allow Hubs on a wide range of topics to be published. People new to the site can also enjoy the gratification of being able to share their Hubs with others and receive feedback almost immediately after publishing.
For the past few years, the HubPages staff (yep, all of us) has been rating a sampling of Hubs every month in order to get a sense of quality changes over time. Using a specially-designed admin-only Hopper that’s similar to the one available to Hubbers, we rate Hubs on a scale of 1-10 (from abysmal to mind-blowing; I’ve shared some of our internal guidelines at the bottom of this post). The Hubs are a mixture of old and new Hubs, to weed out any rating drift (i.e. any tendency to rate Hubs more critically or more generously over time). The rating systems gives us the opportunity to benchmark our efforts to raise the quality bar and see how Hubs and traffic respond.
The graph below is a trendline showing the ratings of Hubs that received traffic during the months on the x-axis, weights them with the traffic they receive, and averages them (i.e. rated Hubs with more traffic the month will influence the average more than those with less traffic during that month):
Quality was relatively high during the first two months, when HubPages was in private beta and only open to family and friends of the founders and a select group of first users. Quality then dipped when HubPages was open to the public, and when we got a lot of adult content and adult traffic. When we took down adult content in July 2007, it accounted for 10% of our Hubs but about a third of our traffic!
We also really didn’t begin to moderate until 2007, and didn’t have a scalable way of doing so until later in that year. A lot of Hubs that were lower in quality than we wanted were published, and got traffic, until we started enforcing higher standards. Standards have continued to rise, and traffic to higher-quality Hubs has also risen remarkably: the traffic-weighted average rating is almost double that of the trough in 2006-2007, and even 30% more than when we were in private beta.
Another interesting graph looks at the quality ratings of Hubs published by month of publication. The trendline in green shows the rating of Hubs that were ever published in the month on the x-axis; the blue trendline shows only the rating of Hubs that are still published. The difference is attributed to Hubs that were unpublished, primarily by our moderation team:
We can glean some interesting insights from this:
- Average Hub quality has never been higher, even better than when HubPages was in private beta, limited to people we trusted.
- The moderation team has really improved the quality of Hubs still published. Maddie Ruud came on as our first full-time moderator in early 2008, and has been “kicking ass and taking usernames” ever since. Norah Casey joined us in early 2010 with moderation experience from working at Justin.tv. She has recently been managing our team of moderators, which tripled in size as we enforced tougher standards this year in response to Panda.
- The gap between what’s ever been published and what’s still published has narrowed, primarily due to the fact that increasingly sophisticated publishing technology can prevent publishing of some types of Hubs that don’t meet our standards. Our engineering team continues to work on things that prevent Hubs that don’t meet our minimum quality standards from even being published.
As HubPages recovers from Panda, this is the first time that we’ve seen a significant shift in traffic patterns towards higher-quality. August 2011 was the first month where the average weighted Hub rating for those rated Hubs receiving traffic was above 7. We think that in the future, effort, authenticity and quality will continue to be rewarded when it comes to useful, interesting content on HubPages.
Sample of ratings guidelines:
3 – Mediocre article; cursory treatment of the topic or “low quality” topic (obviously commercial, etc.)
5 – Average article, with stilted language, grammatically-poor or misspelled words; has potential, with a serious rewrite and/or edit by the author
7 – Great article, although additional subject treatment and media could have been added to make it superb
9 – A perfectly-written, comprehensive article, but could use more links or media (images, videos, maps, etc)