feed hat blog

I wrote a post yesterday about the dangers of allowing (some) custom shortened URLs. Really though, I only talked about one particular name on one particular shortener, so I’ll try to make this a little more general. Incidentally, bit.ly appear to have fixed their robots.txt as of a couple of hours ago – they obviously thought it was a problem!

There were a couple of reasons I was pretty sure about Google following the redirect on robots.txt – the main one being Google’s own Webmaster Tools! If you believe their own tool, Google were quite happily following the redirect and trying to parse a blog as a robots.txt. Depending on the random chance I mentioned yesterday, this sometimes came up with the “other” robots.txt.

This brings me neatly onto the next potential problem – verifying your URL shortener in Google Webmaster Tools or Yahoo Site Explorer.

Webmaster Tools

One of the metrics Google Webmaster Tools uses to decide if you have control over a particular site is for you to place an empty file with a specified name on your site. What happens if I try to add bit.ly to my Webmaster Tools account?

It asks me to create http://bit.ly/google4516834ef16f6fa21.html (or thereabouts)

Now, bit.ly won’t let me create a custom name with a dot in, but until this morning they would silently strip them out.

So, one could just make http://bit.ly/google4516834ef16f6fa21html instead (note the lack of a dot). When Google come along and try to verify the existence of the file, they will do so with the dot in there; bit.ly silently dropped it, redirected to somewhere – anywhere, it doesn’t matter – and the account is marked as verified. Considering there are some fairly odd looking removal requests in there, I don’t think I’m the first person to think of it.

When checking of the specially named file, Google also look for a file they are not expecting to find. If it doesn’t return an error code, they reject the verification. This is due to many misconfigured sites (including URL shorteners!) that fail to return a 404 error code for paths that do not exist. This happens to stop the procedure here from working on several other shorteners that I tried, but luckily for the purposes of this article, Yahoo Site Explorer isn’t so picky.

Site Explorer

Considering bit.ly seem to have fixed their problems, let’s pick on a different target. How about tr.im? They’re another big player for several reasons, the biggest of which is that now they’re open source there are dozens of clones out there. This works on all of them that I tried.

To verify a URL for Site Explorer, you can create a file in a similar manner to the Google one above.

For tr.im, Yahoo asked me to create: http://tr.im/y_key_a41e58c3dc1e462a.html (or again, close enough!)

They don’t allow dots in their short names either, nor is the box big enough to enter the whole string. Unfortunately, they will ignore any extension added to the URL, so creating a short name of y_key_a41e58c3dc1e462a is all that is required for the requested link to work.

Rather than relying on your 404 reporting to be working, Yahoo require that a particular string appears in the content of the file they request; that way they can be sure it’s not an error page and you have control over the content. In my case this looked like fc140e122451a103. As you will probably have guessed by now, they will follow a redirect before looking for that string. Make the destination URL for your short link point to a page containing that value (like, say, this one!) and Yahoo will verify the site as your own.

Summary

No, your URL shortener is not going to get hacked, the world is not going to end, and these particular issues are easy to avoid. Having said that, please think carefully about how your own URLs can be used against you.

Is there any good reason for Google and Yahoo to follow redirects when verifying ownership of a site? Removing this would seem to be a good idea to me. I suspect following redirects on robots.txt is a more complicated issue but perhaps some restrictions could be considered there. Search engines currently have an uneasy relationship with URL shorteners and I think that will only continue.

Thanks for all the comments.

James

Read the original here:
Don’t Make The Same Mistakes As Bit.ly and Tr.im

I wrote a post yesterday about the dangers of allowing (some) custom shortened URLs. Really though, I only talked about one particular name on one particular shortener, so I’ll try to make this a little more general. Incidentally, bit.ly appear to have fixed their robots.txt as of a couple of hours ago – they obviously thought it was a problem!

There were a couple of reasons I was pretty sure about Google following the redirect on robots.txt – the main one being Google’s own Webmaster Tools! If you believe their own tool, Google were quite happily following the redirect and trying to parse a blog as a robots.txt. Depending on the random chance I mentioned yesterday, this sometimes came up with the “other” robots.txt.

This brings me neatly onto the next potential problem – verifying your URL shortener in Google Webmaster Tools or Yahoo Site Explorer.

Webmaster Tools

One of the metrics Google Webmaster Tools uses to decide if you have control over a particular site is for you to place an empty file with a specified name on your site. What happens if I try to add bit.ly to my Webmaster Tools account?

It asks me to create http://bit.ly/google4516834ef16f6fa21.html (or thereabouts)

Now, bit.ly won’t let me create a custom name with a dot in, but until this morning they would silently strip them out.

So, one could just make http://bit.ly/google4516834ef16f6fa21html instead (note the lack of a dot). When Google come along and try to verify the existence of the file, they will do so with the dot in there; bit.ly silently dropped it, redirected to somewhere – anywhere, it doesn’t matter – and the account is marked as verified. Considering there are some fairly odd looking removal requests in there, I don’t think I’m the first person to think of it.

When checking of the specially named file, Google also look for a file they are not expecting to find. If it doesn’t return an error code, they reject the verification. This is due to many misconfigured sites (including URL shorteners!) that fail to return a 404 error code for paths that do not exist. This happens to stop the procedure here from working on several other shorteners that I tried, but luckily for the purposes of this article, Yahoo Site Explorer isn’t so picky.

Site Explorer

Considering bit.ly seem to have fixed their problems, let’s pick on a different target. How about tr.im? They’re another big player for several reasons, the biggest of which is that now they’re open source there are dozens of clones out there. This works on all of them that I tried.

To verify a URL for Site Explorer, you can create a file in a similar manner to the Google one above.

For tr.im, Yahoo asked me to create: http://tr.im/y_key_a41e58c3dc1e462a.html (or again, close enough!)

They don’t allow dots in their short names either, nor is the box big enough to enter the whole string. Unfortunately, they will ignore any extension added to the URL, so creating a short name of y_key_a41e58c3dc1e462a is all that is required for the requested link to work.

Rather than relying on your 404 reporting to be working, Yahoo require that a particular string appears in the content of the file they request; that way they can be sure it’s not an error page and you have control over the content. In my case this looked like fc140e122451a103. As you will probably have guessed by now, they will follow a redirect before looking for that string. Make the destination URL for your short link point to a page containing that value (like, say, this one!) and Yahoo will verify the site as your own.

Summary

No, your URL shortener is not going to get hacked, the world is not going to end, and these particular issues are easy to avoid. Having said that, please think carefully about how your own URLs can be used against you.

Is there any good reason for Google and Yahoo to follow redirects when verifying ownership of a site? Removing this would seem to be a good idea to me. I suspect following redirects on robots.txt is a more complicated issue but perhaps some restrictions could be considered there. Search engines currently have an uneasy relationship with URL shorteners and I think that will only continue.

Thanks for all the comments.

James

Original post:
Don’t Make The Same Mistakes As Bit.ly and Tr.im

Photo by Lee Carson via Creative Commons

Who hasn’t made time for their TGIF chair dance? It’s okay, I’m not judging. It’s just that I know how hard you work, and sometimes you gotta make some time for yourself, you know? On three, let’s tap our toes, wiggle our hips, point our fingers in the air and create a little doo-wop moment. One, two, three… Happy dance!

Ahh. That’s better. Now we can take a look at some of this week’s neat news.

The SEO Newsletter comes out Monday and if you’re not already subscribed, there’s no time like the present. See the form in the blog’s right-hand nav that says “Subscribe to our SEO Newsletter”? Yeah, it’s that easy. If you’re the type that needs convincing, here’s a quick teaser: wonder no more what Matt Cutts really said regarding PR sculpting with nofollow.

Tonight at midnight Eastern, Facebook will put vanity URLs up for grabs. Profiles and pages are both eligible for custom addresses, although every page or profile is only allowed one each and there are restrictions on length and character usage. The social media wizards at 10e20 have put together a helpful preparation guide that includes eligibility requirements. According to Mashable’s sources, fan pages with less than a thousand followers can claim a custom URL after June 28. [Face it, it's not like you're doing anything else on date night, right? --Susan]

Photo by Srini G via Creative Commons

It’s always impressive when a company captures a bit of viral video magic, considering how rare this hallowed rite of online marketing truly is. Carl’s Jr.’s latest campaign has amassed more than 3 million video views. Of course it helps that the video for the mushroom burger features such a fungi. (Ah, I crack myself up!) [Rimshot --Susan] If you’re looking to get into the online video space, check out ReelSEO’s list of tools that let you to upload videos to multiple video sharing sites. Both paid and free tools make the list, so whatever your budget, it’s worth checking out.

The release of the iPhone 3GS was announced this week. Gadget lovers are understandably excited, but the shiny new toy could have marketers smiling, too. AdAge has listed features of the new smartphone, explaining how marketers can leverage each technology. If you’re more of an RIM fan, you can snag a free BlackBerry before the deal ends Sunday. This is one seriously awesome offer. [Having bought my 8900 when it came out four months ago, I now know how iPhone owners feel. And I don't like it. -- Susan]

FYI: I have not been bribed to write about the above products. If I had, the FTC would come after me, and I’m not looking to get on any federal agencies’ bad side. If you do find yourself in dire straits with the authorities, take a deep breath. I hear it’s pretty easy to get a pass from the Commander in Chief. Tellingly, comic book writers have translated the President’s inner superhero into pen and ink, and Comicbook.com thinks a few other celebs might have what it takes as well.

Photo by chrs_snll via Creative Commons

Unfortunately, even the POTUS can’t save us from an otherworldly threat. In a billion years, when Mars or Venus decide to stop by for a visit and inadvertently smash Earth to smithereens, I have a feeling I might not make it. One teen got a small-scale preview of the action when he was hit by a pea-sized meteorite. It left him with a scar and the absolute coolest story ever.

In the Googleverse, updates and releases are raining down faster than comets. Last week we read Search Engine Watch’s super-sized list of Google updates, and this week we get a list of nine more. Links to more info on reconsideration request updates, the new mobile iGoogle, and the updated Webmaster Tools interface — they’re all there.

Before we part ways, I just want to cheer us all on as we enter a new age. An age where “Web 2.0″ is the one millionth word to enter the English language. Does anyone else see good times on the horizon?

Things I learned from Boing Boing this week:

• Is your goat bored? Unmotivated? Has the old sparkle left his eyes? Build him a goat tower!
• Next time don’t skip lunch. Instead, warm up some beans with your computer. Appetizing. [Yum yum fail. --Susan]
• As soon as some evidence of extraterrestrial life surfaces, the kids go and burn it.
• Sure the bird can dance, but can it head bang with a gusto that would put Judas Priest to shame? Uh… that’s a yes!

Read the original here:
Friday Recap - Shoot for the Stars Edition

Photo by Pink Sherbet Photography via Creative Commons

Google’s getting pats on the back for a new tool that lets webmasters tell the search engine when a site’s been moved to a new domain. According to Google’s Webmasters/Site owners Help documentation, if you use the change of address tool, the Google index will be updated to reflect your new domain for 180 days. Before those 180 days are up, Googlebot will have come by to crawl and index pages at the new domain and — voila! — your new site will be forever present in the only search engine that matters. [Yay! --Susan]

It’s great to hear that at least one task is being simplified for overloaded SEOs, but Google’s convenient shortcut may have some unintended consequences. First of all, Google’s not the only search engine. Submitting your new domain through Google’s handy new tool is not going to affect the index of engines like Yahoo, Bing and Ask. (Rocket scientist here, I know.)

And second, while Google may be the big guy on campus today, that may not be the case tomorrow. Taking shortcuts can have a nasty habit of coming back to bite you. Just because Google’s created an easy domain-change-submission process, that shouldn’t be an excuse to ignore your responsibilities in optimizing a site for all the engines. After all, we’re talking about search engine optimization and not Google optimization. Although, GO would be a pretty catchy title…

Anyway, if you catch my drift and you want to make sure your new site’s sitting pretty with more than just Big G, check out this simple instructional on what to do after you’ve moved your site.

What to Do After Moving a Site

We recommend that all Web sites include an XML Sitemap. XML Sitemaps are used by search engines to find all the pages on a site. More information on XML Sitemaps guidelines and what XML Sitemaps are used for can be found in the SEO Newsletter archives. XML Sitemaps can be generated at http://www.xml-sitemaps.com/.

With your new site chilling until the search engines stop by, you’ll want to submit the new domain’s XML Sitemap to each engine according to its specific procedures. Now that I’ve explained what to do, here’s the how.

See the rest here: 
Don’t Let Shortcuts Create Bad Habits