### Description

https://github.com/apache/airflow/pull/17946 implements a `/robots.txt` endpoint to block search engines crawling Airflow - in the cases where it is (accidentally) exposed to the public Internet.

If we record any GET requests to that end-point we'd have a strong warning flag that the deployment is exposed, and could issue a warning in the UI, or even enable some kill-switch on the deployment. 

Some deployments are likely intentionally available and rely on auth mechanisms on the `login` endpoint, so there should be a config option to suppress the warnings.

An alternative approach would be to monitor for requests from specific user-agents used by crawlers for the same reasons

### Use case/motivation

People who accidentally expose airflow have a slightly higher chance of realising they've done so and tighten their security.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

airflow/www/views.py
)
            .one_or_none()
        )
        if ti is None:
            ti_attrs: Optional[List[Tuple[str, Any]]] = None
        else:
            ti.refresh_from_task(task)
            
            with warnings.catch_warnings():
                warnings.simplefilter("ignore", DeprecationWarning)
                all_ti_attrs = ((name, getattr(ti, name)) for name in dir(ti) if not name.startswith("_"))
            ti_attrs = sorted((name, attr) for name, attr in all_ti_attrs if not callable(attr))
        attr_renderers = wwwutils.get_attr_renderer()
        task_attrs = [
            (attr_name, attr)
            for attr_name, attr in (
                (attr_name, getattr(task, attr_name))
                for attr_name in dir(task)
                if not attr_name.startswith("_") and attr_name not in attr_renderers
            )
            if not callable(attr)
        ]
        
        special_attrs_rendered = {
            attr_name: renderer(getattr(task, attr_name))
            for attr_name, renderer in attr_renderers.items()
            if hasattr(task, attr_name)
        }
        no_failed_deps_result = [
            (