I'm currently working on an application built with Express. I have tried to implement robots.txt.
This is what I have right now:
app.use(
'/robots.txt',
function (req, res) {
res.type('text/plain');
res.send("User-agent: *\nAllow: /\nDisallow: /search")
);
});
While I am able to see the results in the website, google search is giving me the warning:
Indexed, though blocked by robots.txt
Am I doing something wrong or is there a better way to implement the robots.txt file?