There are plenty of bots out there and, as a result, some conventions have arisen.  Well-behaved bots identify themselves with a unique user-agent.  They also follow the robots.txt conventions, which allow webmasters to control how their sites are crawled.

Here at Live Search, our crawlers are identified by the user-agent ‘MSNBot’.  This may seem a little non-intuitive, but many webmasters depend on this, and so we chosen not to change it.  In order to make things a little more transparent, we also identify our different types of crawlers.  The complete list is as follows: 

                MSNBot                                        Main web crawler (www.live.com)

                MSNBot-Media                               Images & all other media (images.live.com)

                MSNBot-NewsBlogs                         News and blogs (search.live.com/news)

                MSNBot-Products                           Products & shopping (products.live.com)

                MSNBot-Academic                          Academic search (academic.live.com)

But what about crawlers that aren’t so well-behaved?  After all, anyone could call themselves ‘MSNBot’, and proceed to be as rude and aggressive as they like.  Fortunately, there is a way you can catch these impersonators. Here is how it works: 

1. When you get a page view request, it specifies a user-agent and an IP address.  As I described above, all requests from Live Search use a user agent starting with the word ‘MSNBot’.
2. If you see the MSNBot user-agent, it’s time to check the identity of the bot.  Starting with the IP address (i.e. 207.46.98.149), you can use reverse DNS lookup to find out the registered name of the machine.
3. Once you have the host name (in this case, livebot-207-46-98-149.search.live.com), you can check that it really is coming from Live Search.  The name of all live search crawlers will end with ‘search.live.com’.  If the name doesn’t end with ‘search.live.com’, you know it’s not really our crawler.
4. Finally, you need to verify that the name is accurate.  In order to do this, you can use Forward DNS to see the IP address associated with the host name.  This should match the IP address you used in Step 2 – if it doesn’t, it means the name was fake.

By verifying the crawler’s identity, you can catch masquerading crawlers.  When you do catch one, you can simply return an HTTP Error, thus blocking them from seeing your content.  

We are constantly looking for your feedback to help improve our engine – please send it our way using this link.

Brent Hands, Program Manager, Live Search

What if mapping the world was more like a video game?

What if you could fly like Superman and figure out where to meet your friends for a good meal and which routes to avoid to beat traffic?

With Live Search maps you can find yellow pages and white pages information, get live traffic conditions and view stunning 3D and Birds Eye …just as if you were gaming in Second Life, but with real-world information embedded in real-world cities where advertising really matters?

Check out our beta for Virtual Earth™  3D , a new online mapping feature launched this week by Live Search.  You can see terrain information for all over the world, and explore these U.S. urban areas with textured buildings (with many more to come in the next month!):  San Francisco, San Jose, Seattle, Boston, Philadelphia, Los Angeles, Las Vegas, Detroit, Phoenix, Houston, Baltimore, Atlanta, Denver, Dallas and Fort Worth.  There is a virtual tour that you can review before diving into the goodness.

While many folks will opt to use their keyboards, you can even use an Xbox controller to navigate the three-dimensional world for greater deftness and speed.

In addition, developers can use the Virtual Earth 3D application programming interface to build these search capabilities into their own applications and Web sites. This and other APIs for Live Search are open to developers, with the option to acquire additional support and other benefits through a service-level agreement with Microsoft. Developers can find more information about the Virtual Earth API at http://dev.live.com/virtualearth and in the Virtual Earth SDK http://dev.live.com/virtualearth/sdk/.  Also, contact maplic@microsoft.com for details about additional support.

Some folks in the blogosphere noticed and we appreciate it: Liveside.net ,  TechCrunch, Brady Forrest , Robert Scoble, Search Engine Journal to name a few.  Keep talking about it and also, let us know how to improve. As with any beta product we’d love to have feedback.  You can let us know on the Virtual Earth blog at http://virtualearth.spaces.live.com/, where product updates are posted continually, or use the handy feedback link at the bottom of the application.

--The Spaceland team

Today, we are excited to announce that Google, Microsoft and Yahoo! are coming together in support of the SiteMaps protocol.  The goal of this effort is to improve search results for customers around the world.  This protocol enables site owners everywhere to tell search engines about the content on their site instead of having to rely  solely on crawl algorithms to find it.

So, why are we excited to work on this?  Because by agreeing on a standard, we can provide site owners with one simple way to share information with every search engine.  You just publish a sitemap, and every engine is instantly able to read and use the data to more effectively index your site.  Since this is a free, widely supported protocol, our hope is that this will foster an even broader community of developers building support for it.

We are 100% behind this protocol - this kind of collaboration will help improve the search experience for all of our customers, and we are working hard to release full support in 2007.  We are starting to alpha test with internal partners such as MSDN and Microsoft Support now.  Like all teams at Microsoft, we like to dogfood our work internally to ensure that it is working properly before it is publicly released.  Watch this space for an update as soon as we’re done.

Interested in the gritty details?  Read more about the Sitemaps protocol at the official website: http://www.sitemaps.org.  If you have any comments, please let us know by leaving a comment.

Ken Moss

General Manager, Live Search

So, now that you’ve (a) provisioned a Live Search Box for your site, and (b) updated it with a cool look and feel, you are finally ready to build the search query of your dreams.

Back in the old days, search boxes generally gave you two options: (1) search the whole Web or (2) search only your site. With the advanced query parameters of Live Search, you now have complete control to search the subset of the Web you define, and get the results that you are looking for.
Let’s take a look at several scenarios (posted here on MSDN)

• Searching a predefined set of URLs. For example, if you would like to constrain the search to your Web site, or a specific group of Web sites.
• Using a predefined search macro. For example, if you would like to use one of the macros from the Live Gallery to power your search engine. Each of these macros is predefined to search a specific topic, such as health care, or something less academic, such as Britney Spears.
• Searching your Web site’s neighborhood. A new feature that lets you search a Web site and all the sites to which that Web site links. This could be really useful in a blog, where the author is recommending additional sites or news articles. If you really wanted to, you could include the sites that link to your Web site, too.
• Advanced search parameters. You can combine a few additional query parameters with the aforementioned scenarios to look for things such as specific file formats, RSS feeds, or results by language or country/region.

Try it out, let us know what you think.

-- Nathan Buggia, Live Search Marketer

We’ve just posted an article on dev.live.com that looks at a few simple hacks you can do with the Live Search Box, to change its look and feel. You can use these hacks to make the search box fit better into the design of your site. List of hacks:

• Applying the Windows Live styling to the basic search box
• Adding default text to the search box
• Auto-selecting the search box when the page loads
• Ensuring compatibility with ActiveX and other smart objects

But this is really only the beginning of what you can do. As you play around with the search box on your site, let us know what you come up with, and maybe we'll promote it on this blog. And if there is a hack you'd like to do but just don't know how let us know.

Small print - before you go hack-wild, please read the Terms of Use to ensure responsible hacking.

-- Nathan Buggia, Live Search Marketer

(And, yes, it supports Firefox too!)

Today, we’re proud to announce the launch of the Live Search Box, to bring the power of search to your Web site or blog through a cool widget. Check it out:

When the user enters a query, the search box dynamically builds a floating <DIV> on your page to display the search results. You can customize the query in the first tab to search your site, your macro or anything else, while the second tab will return general web search results. The floating <DIV> will position itself appropriately, whether you decide to place the box on the left, right, top, or bottom of your Web site.

The search box also comes in a pure-HTML flavor:

The HTML version is a simple <Form> element that you can paste into your site and which will redirect search results back to the http://www.live.com/ page. You can customize this search box to search either a specific Web site or the entire Web.
 
Sound like fun? Check it out on search.live.com/siteowner, and let us know what you think. And keep your eyes open for a few posts on hacking the search box in the next couple of days. Note this feature is currently only officially supported for the US market.

Happy Halloween,
-- Nathan Buggia, Live Search Marketing