### Apache Airflow version

2.5.2

### What happened

Not sure if this is a feature not a bug, but I can use KubernetesPodOperator fine without setting a kubernetes_conn_id.
For example:
```
start = KubernetesPodOperator(
        namespace="mynamespace",
        cluster_context="mycontext",
        security_context={ 'runAsUser': 1000 },
        name="hello",
        image="busybox",
        image_pull_secrets=[k8s.V1LocalObjectReference('prodregistry')],
        cmds=["sh", "-cx"],
        arguments=["echo Start"],
        task_id="Start",
        in_cluster=False,
        is_delete_operator_pod=True,
        config_file="/home/airflow/.kube/config",
    )
```

But if I add deferrable=True to this it won't work. It seems to require an explicit kubernetes_conn_id (which we don't configure).

Is not possible to the deferrable version to work as the non deferrable one?

### What you think should happen instead

I hoped that kpo deferrable would work the same as non deferrable.

### How to reproduce

Use KPO with deferrable=True but no kubernetes_conn_id setting

### Operating System

Debian 11

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

actual_pod = self.api_client.sanitize_for_serialization(k.pod)
        expected_dict = {
            "apiVersion": "v1",
            "kind": "Pod",
            "metadata": {
                "annotations": {},
                "labels": {
                    "test_label": test_label,
                    "airflow_kpo_in_cluster": "False",
                    "dag_id": "dag",
                    "run_id": "manual__2016-01-01T0100000100-da4d1ce7b",
                    "kubernetes_pod_operator": "True",
                    "task_id": mock.ANY,
                    "try_number": "1",
                },
                "name": "memory-demo",
                "namespace": "mem-example",
            },
            "spec": {
                "affinity": {},
                "containers": [
                    {
                        "args": ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"],
                        "command": ["stress"],
                        "env": [],
                        "envFrom": [],
                        "image": "ghcr.io/apache/airflow-stress:1.0.4-2021.07.04",
                        "name": "base",
                        "ports": [],
                        "resources": {"limits": {"memory": "200Mi"}, "requests": {"memory": "100Mi"}},