### Apache Airflow version

main (development)

### What happened

_list_pods function uses kube list_namespaced_pod and list_pod_for_all_namespaces kube functions. Right now, these Kube functions will get the entire pod spec though we are interested in the pod metadata alone. This _list_pods is refered in clear_not_launched_queued_tasks. try_adopt_task_instances and _adopt_completed_pods functions.

When we run the airflow at large scale (with worker pods of more than > 500). The _list_pods function takes a significant amount of time (upto 15 - 30 seconds with 500 worker pods) due to unnecessary data transfer (V1PodList up to a few 10 MBs) and JSON deserialization overhead. This is blocking us from scaling the airflow to run at large scale

### What you think should happen instead

Request the Pod metadata instead of entire Pod payload. It will help to reduce significant network data transfer and JSON deserialization overhead.

### How to reproduce

I have reproduced the performance issue while running 500 concurrent jobs. Monitor  kubernetes_executor.clear_not_launched_queued_tasks.duration and kubernetes_executor.adopt_task_instances.duration metrics.

### Operating System

CentOS 6

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes

### Deployment

Other Docker-based deployment

### Deployment details

Terraform based airflow deployment

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

for key, timestamp in list(self.last_handled.items()):
                if time.time() - timestamp > allowed_age:
                    del self.last_handled[key]
            if not queued_tis:
                return
            
            kwargs = {"label_selector": f"airflow-worker={self._make_safe_label_value(str(self.job_id))}"}
            if self.kube_config.kube_client_request_args:
                kwargs.update(self.kube_config.kube_client_request_args)
            pod_list = self._list_pods(kwargs)
            
            label_search_set = set()
            for pod in pod_list:
                dag_id = pod.metadata.labels.get("dag_id", None)
                task_id = pod.metadata.labels.get("task_id", None)
                airflow_worker = pod.metadata.labels.get("airflow-worker", None)
                map_index = pod.metadata.labels.get("map_index", None)
                run_id = pod.metadata.labels.get("run_id", None)
                execution_date = pod.metadata.labels.get("execution_date", None)
                if dag_id is None or task_id is None or airflow_worker is None:
                    continue
                label_search_base_str = f"dag_id={dag_id},task_id={task_id},airflow-worker={airflow_worker}"
                if map_index is not None:
                    label_search_base_str += f",map_index={map_index}"
                if run_id is not None:
                    label_search_str = f"{label_search_base_str},run_id={run_id}"
                    label_search_set.add(label_search_str)
                if execution_date is not None:
                    label_search_str = f"{label_search_base_str},execution_date={execution_date}"
                    label_search_set.add(label_search_str)