Dignity nobility excellent point out highness standing class caste status superiority precedence vogue distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robotic regarding which pages will not be to be crawled. To be a internet search engine crawler https://www.youtube.com/watch?v=0USI3vTZ0aw