The experiments

I am going to run an experiment testing sitemap.xml and hanging pages. This will not be a flawless test but it will help me prove my assumptions.

I have been telling clients that if you add a url to an XML sitemap that it will be discovered. Why do I say this? Often times due to different varying factors search engines have a hard time finding pages inside of dynamic content pages. This test is not going to be about accessibility or how content is delivered. The SEO Experiement is only about sitemap files.

The test is going to be limited to hard coding the url into an xml sitemap and see if it can be found. My assumption will not be a solution to linking or standard url structure.

We already know that Google, Yahoo, and MSN will index urls similar to http://www.domain.com/?=123. We also know that pages with session id’s are usually not indexed because of the session variables. What I am trying to prove is that a url listed in an XML sitemap will be found by Google, Yahoo, and MSN without back links and only submitted through the different search engine webmaster tools .

How I am going to do this test is by creating a unique page that is not linked by any other pages and add it to its very own xml sitemap to be discovered. Once I created the pages I will add the sitemap to the webmaster tools section of each search engine. I am going to create another page and add it to a urllist.txt (an old protocol used by Yahoo. In the xml Sitemap I am going to put in 1 page only that is free floating and see how long it takes to get indexed by each search engine. In the urllist.txt file I am going to add another page and see if it is indexed.

Assumptions/problems/and forecast – Hypothesis

The file will be picked up in Yahoo and cached much fast than Google with xml sitemap file. The urllist.txt file will probably not be found on MSN which will be one problem. The other problem with MSN is they don’t yet have a way to add multiple sitemap files through their portal, even though they read robots.txt. The files from the urllist.txt will be picked up first by Yahoo and Google will follow.

Scope

2 html pages

1 sitemap.xml page

1 urllist.txt page

Result

To be continued

Because I don’t want to spoil the experiment I won’t give the specifics locations of the files or exactly how I am going to do it. The experiment could take 1 day or 2 weeks. For now we will see. Stay tuned for the results and summary. More experiments are coming.

Over the past few weeks increased additional security has been added. An Secure Socket Layer (SSL) to secure and encrypt data via the internet has been installed.

I have also done a lot of server work. Including .htaccess like the previous article. I have also been doing some design work for Bagley School. What started out as fixing broken image tags turned into a design and upgrade. You can see the work in progress here. Suggestions are welcome. I have taken this on for free using open source and currently available resources.

If you have been following this blog you know that I do this for entertainment and hold down a regular day job… Well, over the past few weeks I have been undergoing career changes. I am currently between jobs and actively seeking full-time employment. Being between jobs theoretically leaves more time for blogging. I have found that assumption to be the opposite. If you know of job opportunities please let me know. Don’t hesitate to share my online portfolio which includes my resume. portfolio.chris-estes.com Comments and corrections are greatly appreciated.

In other news you will notice some changes to Search Engine Optmization by Chris. In the month of January a new brighter more colorful design has been added. Also, Yahoo Publisher Network ads sponsored ads have been added. A different placement of ad’s has been put in to offer you a a much better range of services. As a supporter of Search Engine College I recomnd taking a look into one of their classes. All thought a class will help you get up to speed on SEO basices but n0thing is going to be as sufficient as an experienced professional. As always I have some affiliate marketers online that would like to put ads on the blog. So stay tunned to some deals I am tyring to work out for you the readers Discount programs and discounted SEO and other equipment.

You don’t want to miss out on some of the exclusive deals I am working out with companies like, Apple, Itunes, and dell stay tuned by subscribing to my feed.

.htaccess File and commentary

There are several standard components but most of these I am using to try and prevent access. I also use the robots.txt standard to block Search Engine’s from indexing my portfolio.

Sample .htaccess file

The”order allow, deny” section with all of the IP addresses are listed to prevent spam and to deny total access from those IP addresses. Over time I have assembled these IP addresses from list, recommendations, and spam received. If you see that I need to add or delete an IP address drop me a message.

order allow,deny
deny from 68.239.83.76
deny from 66.117.254.
deny from 66.249.64.
deny from 63.148.99.
deny from 65.118.41.
deny from 63.148.99.
deny from 65.118.41.
deny from 128.121.217.
deny from 207.87.178.
deny from 63.100.73.
deny from 63.100.163.
deny from 207.87.178.68
deny from 209.49.118.24
deny from 38.100.
deny from 38.105.83.
deny from 38.112.21.
deny from 38.118.42.
deny from 63.146.13.
deny from 65.213.208.
deny from 65.222.176.
deny from 65.222.185.
deny from 151.173.0.
deny from 66.194.6.
deny from 208.80.192.
deny from 204.15.64.
deny from 208.17.184.
deny from 69.36.144.
deny from 82.80.248.
deny from 62.0.8.
deny from 206.28.72.
deny from 217.169.46.
allow from all

The “DirectoryIndex” section is meant to show which file should be called up as the root of the directory. You can edit this line to specify any file to be called as the root file for any directory.

DirectoryIndex index.shtml

The Mod Rewrite in this section is to rewrite file extension. Sample usages would be changing scripting languages from html to php.

RewriteEngine on
RewriteBase /
RewriteRule ^(.*)\.html$ $1.shtml [R=permanent]

The following is an interesting rewrite tool. This section blocks all of the listed agents to be redirected to a specified URL. If the agent is registers on this list I choose a blank domain for them to be redirected to. Agents, robots, Search Engines, or anything that isn’t human is supposed to register itself as an agent. In the data transmitted when they access a website includes part or all of the user agent names.

Most of this is based on information from forums and other websites. This is a pretty complete list feel free to use this list in your website.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon
RewriteRule /*$ http://www.domain.com [L,R]

This rewrite is being used to prevent hot linking. Hot Linking or using images from other websites is very common. Since the images are hosted remotely away from the website someone else is using your website bandwidth and resources.

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://portfolio.domain.com/.*$ [NC]
RewriteRule .*.(gif|GIF|jpg|JPG|png|PNG|doc|DOC|pdf|PDF|txt|TXT|pl|PL|ico|ICO|css|CSS|js|JS)$ – [F]

Edited for security for my online portfolio.