### Description

https://github.com/apache/airflow/pull/17946 implements a `/robots.txt` endpoint to block search engines crawling Airflow - in the cases where it is (accidentally) exposed to the public Internet.

If we record any GET requests to that end-point we'd have a strong warning flag that the deployment is exposed, and could issue a warning in the UI, or even enable some kill-switch on the deployment. 

Some deployments are likely intentionally available and rely on auth mechanisms on the `login` endpoint, so there should be a config option to suppress the warnings.

An alternative approach would be to monitor for requests from specific user-agents used by crawlers for the same reasons

### Use case/motivation

People who accidentally expose airflow have a slightly higher chance of realising they've done so and tighten their security.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

tests/www/views/test_views_base.py
],
    ids=[
        "userstatschertview-admin",
        "userstatschertview-viewer",
        "actions-admin",
        "actions-viewer",
        "resources-admin",
        "resources-viewer",
        "permissions-admin",
        "permissions-viewer",
        "resetpassword-admin",
        "resetpassword-viewer",
        "users-admin",
        "users-viewer",
    ],
)
def test_views_get(request, url, client, content):
    resp = request.getfixturevalue(client).get(url, follow_redirects=True)
    check_content_in_response(content, resp)
def _check_task_stats_json(resp):
    return set(list(resp.json.items())[0][1][0].keys()) == {'state', 'count'}
@pytest.mark.parametrize(
    "url, check_response",
    [
        ("blocked", None),
        ("dag_stats", None),
        ("task_stats", _check_task_stats_json),
    ],
)
def test_views_post(admin_client, url, check_response):