### Apache Airflow version

2.4.1

### What happened

When there are large number of dag files, lets say 10K and each takes sometime to parse,  dag_parser doesnt pick any newly created files till all 10k files are finished parsing

`if not self._file_path_queue:
                self.emit_metrics()
                self.prepare_file_path_queue()`

Above logic only adds new files to queue when queue is empty

### What you think should happen instead

Every loop of dag processor should pick new files and add into file for parsing queue

### How to reproduce

_No response_

### Operating System

All

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""
    Given a list of DAG definition files, this kicks off several processors
    in parallel to process them and put the results to a multiprocessing.Queue
    for DagFileProcessorAgent to harvest. The parallelism is limited and as the
    processors finish, more are launched. The files are processed over and
    over again, but no more often than the specified interval.
    :param dag_directory: Directory where DAG definitions are kept. All
        files in file_paths should be under this directory
    :param max_runs: The number of times to parse and schedule each file. -1
        for unlimited.
    :param processor_timeout: How long to wait before timing out a DAG file processor
    :param signal_conn: connection to communicate signal with processor agent.
    :param dag_ids: if specified, only schedule tasks with these DAG IDs
    :param pickle_dags: whether to pickle DAGs.
    :param async_mode: whether to start the manager in async mode
    """
    def __init__(
        self,
        dag_directory: os.PathLike,
        max_runs: int,
        processor_timeout: timedelta,
        dag_ids: list[str] | None,
        pickle_dags: bool,
        signal_conn: MultiprocessingConnection | None = None,
        async_mode: bool = True,
    ):
        super().__init__()
        self._file_paths: list[str] = []
        self._file_path_queue: list[str] = []
        self._max_runs = max_runs