My question is that could we add ajax requests disallow in robots.txt in codeigniter And is that compulsory to add robots.txt file in home directory or we place it in views with defined routes