### Description

https://github.com/apache/airflow/pull/17946 implements a `/robots.txt` endpoint to block search engines crawling Airflow - in the cases where it is (accidentally) exposed to the public Internet.

If we record any GET requests to that end-point we'd have a strong warning flag that the deployment is exposed, and could issue a warning in the UI, or even enable some kill-switch on the deployment. 

Some deployments are likely intentionally available and rely on auth mechanisms on the `login` endpoint, so there should be a config option to suppress the warnings.

An alternative approach would be to monitor for requests from specific user-agents used by crawlers for the same reasons

### Use case/motivation

People who accidentally expose airflow have a slightly higher chance of realising they've done so and tighten their security.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

airflow/www/views.py
.all()
        )
        dag_states = [
            {
                
                
                'date': dr.date if isinstance(dr.date, str) else dr.date.isoformat(),
                'state': dr.state,
                'count': dr.count,
            }
            for dr in dag_states
        ]
        data = {
            'dag_states': dag_states,
            'start_date': (dag.start_date or DateTime.utcnow()).date().isoformat(),
            'end_date': (dag.end_date or DateTime.utcnow()).date().isoformat(),
        }
        doc_md = wwwutils.wrapped_markdown(getattr(dag, 'doc_md', None))
        
        data = htmlsafe_json_dumps(data, separators=(',', ':'))
        return self.render_template(
            'airflow/calendar.html',
            dag=dag,
            doc_md=doc_md,
            data=data,
            root=root,
            dag_model=dag_model,
        )
    @expose('/graph')
    @auth.has_access(