### Description

https://github.com/apache/airflow/pull/17946 implements a `/robots.txt` endpoint to block search engines crawling Airflow - in the cases where it is (accidentally) exposed to the public Internet.

If we record any GET requests to that end-point we'd have a strong warning flag that the deployment is exposed, and could issue a warning in the UI, or even enable some kill-switch on the deployment. 

Some deployments are likely intentionally available and rely on auth mechanisms on the `login` endpoint, so there should be a config option to suppress the warnings.

An alternative approach would be to monitor for requests from specific user-agents used by crawlers for the same reasons

### Use case/motivation

People who accidentally expose airflow have a slightly higher chance of realising they've done so and tighten their security.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

airflow/www/views.py
"""Process form data."""
        conn_type = form.data['conn_type']
        conn_id = form.data["conn_id"]
        extra = {
            key: form.data[key]
            for key in self.extra_fields
            if key in form.data and key.startswith(f"extra__{conn_type}__")
        }
        
        
        extra_conn_params = form.data.get("extra")
        if extra_conn_params:
            try:
                extra.update(json.loads(extra_conn_params))
            except (JSONDecodeError, TypeError):
                flash(
                    Markup(
                        "<p>The <em>Extra</em> connection field contained an invalid value for Conn ID: "
                        f"<q>{conn_id}</q>.</p>"
                        "<p>If connection parameters need to be added to <em>Extra</em>, "
                        "please make sure they are in the form of a single, valid JSON object.</p><br>"
                        "The following <em>Extra</em> parameters were <b>not</b> added to the connection:<br>"
                        f"{extra_conn_params}",
                    ),
                    category="error",
                )
        if extra.keys():
            form.extra.data = json.dumps(extra)
    def prefill_form(self, form, pk):