Different robots.txt per environment

[hiramhibbard]

Is there a way to either ignore the robots.txt file in specific environments, or include different robots.txt files for each environment?

Here's the scenario: I have a production branch, a dev branch, and a staging branch. For production I obviously want to allow crawlers via Allow: /. However in dev and staging environments, I want to block crawlers.

Due to the deployment strategy I'm forced to adhere to, I'm unable to just edit the file after deploying it and then .gitignore it like I normally would. Can I somehow configure this file in create-react-app per environment, or using environment variables?

[heyimalex]

To the best of my knowledge, cra doesn't give you any way to do this.

You can write a script that does it, and then change your build line to execute it after, ie react-scripts build && node deleteRobotsIfNotProd.js.

[hiramhibbard]

Yeah I considered that, unfortunately the build happens automatically during deployment without the ability to modify it.

[heyimalex]

🤷‍♂️

[miraage]

@hiramhibbard nobody forces you to run npm run build on the CI. You can create your own script, which would generate the build and also do something with public/robots.txt.

[hiramhibbard]

Actually, as I said previously, I don't have access to the build/deployment process. Please see the rest of the discussion above as this has already been suggested.

For now I'm using a meta tag with an environment variable, and that will suffice (although not the ideal solution).