Ten. Gzip Compression: Help Gzip compression on the server to decrease the measurement of data files transferred involving the server and also the user’s browser. txt file is then parsed and will instruct the robot as to which pages are certainly not to generally be crawled. Being a online search https://www.youtube.com/watch?v=B7lEnYsiELw