Here are a few mistakes why search engine robots away at our web:
Robots.txt damaged or contains the wrong wrong parameters
If google robots read the contents of your robots.txt, then google will ignore your web pages.
You can learn the technique in writing robots in http://www.robotstxt.org/
You have a web page code is too long
If your web page contains code javascript, CSS, Html and more that are too many and long, it will make you hard in the web access (slow access).
If that happens then the search engine robots will find it hard to parse your web content.
Web menu navigation you have a problem
If you create a menu using javascript, DHTML or AJAX, then the menu can not be in the index by search engine robots.
because the search engine robots can only index the links that use HTML.
URL of your website are too many variables
If your url too many variables, then google will ignore your web pages.
Examples of URLs that are too many variables:
http://domain.com/page.php?page=1&id=2&mirror=3&direct=index.html
Official Statement from google search engines:
"Google indexes dynamically generated webpages, including. Asp pages,. Php pages, and pages with question marks in their URLs. However, these pages can cause problems for our crawler and may be ignored. "
Using session IDs in your web
Search engine google will not index URLs that contain a session ID for causing duplicate content.
The use of cookies is better than using session IDs
