o ưi@s(ddlmZmZddlmZddZdS))OptionalUnion) str_to_boolcCspddlm}m}m}|dd}d}t|tr|}n t|tr#t|}|dur6|dur4t d|j j dSdS)z Returns True if the robots.txt file should block web crawlers Controlled by ```yaml general_settings: block_robots: true ``` r)CommonProxyErrorsgeneral_settings premium_user block_robotsFNTz0Blocking web crawlers is an enterprise feature. ) Zlitellm.proxy.proxy_serverrrrget isinstanceboolstrr ValueErrorZnot_premium_uservalue)rrrZ _block_robotsrrU/home/app/Keep/.python/lib/python3.10/site-packages/litellm_enterprise/proxy/utils.py_should_block_robotss    rN)typingrrZlitellm.secret_managers.mainrrrrrrs