Commit Graph

3 Commits

Author SHA1 Message Date
Alejandro Celaya
83e8801827 Move env var default values to EnvVars enum 2024-10-14 09:41:22 +02:00
Alejandro Celaya
4b52c92e97 Add option to customize user agents in robots.txt 2024-07-05 08:54:54 +02:00
Alejandro Celaya
163244f40f Add option to allow all URLs to be crawlable via robots.txt 2024-04-22 09:16:44 +02:00