### Description

https://github.com/apache/airflow/pull/17946 implements a `/robots.txt` endpoint to block search engines crawling Airflow - in the cases where it is (accidentally) exposed to the public Internet.

If we record any GET requests to that end-point we'd have a strong warning flag that the deployment is exposed, and could issue a warning in the UI, or even enable some kill-switch on the deployment. 

Some deployments are likely intentionally available and rely on auth mechanisms on the `login` endpoint, so there should be a config option to suppress the warnings.

An alternative approach would be to monitor for requests from specific user-agents used by crawlers for the same reasons

### Use case/motivation

People who accidentally expose airflow have a slightly higher chance of realising they've done so and tighten their security.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

airflow/www/views.py
"""Filter using DagIDs"""
    def apply(self, query, func):
        filter_dag_ids = current_app.appbuilder.sm.get_editable_dag_ids(g.user)
        return query.filter(self.model.dag_id.in_(filter_dag_ids))
class AirflowModelView(ModelView):
    """Airflow Mode View."""
    list_widget = AirflowModelListWidget
    page_size = PAGE_SIZE
    CustomSQLAInterface = wwwutils.CustomSQLAInterface
class AirflowPrivilegeVerifierModelView(AirflowModelView):
    """
    This ModelView prevents ability to pass primary keys of objects relating to DAGs you shouldn't be able to
    edit. This only holds for the add, update and delete operations.
    You will still need to use the `action_has_dag_edit_access()` for actions.
    """
    @staticmethod
    def validate_dag_edit_access(item: Union[DagRun, TaskInstance]):
        """Validates whether the user has 'can_edit' access for this specific DAG."""
        if not current_app.appbuilder.sm.can_edit_dag(item.dag_id):
            raise AirflowException(f"Access denied for dag_id {item.dag_id}")
    def pre_add(self, item: Union[DagRun, TaskInstance]):
        self.validate_dag_edit_access(item)
    def pre_update(self, item: Union[DagRun, TaskInstance]):
        self.validate_dag_edit_access(item)
    def pre_delete(self, item: Union[DagRun, TaskInstance]):
        self.validate_dag_edit_access(item)
    def post_add_redirect(self):  
        return redirect(self.get_default_url())
    def post_edit_redirect(self):