### Apache Airflow version

2.4.1

### What happened

When there are large number of dag files, lets say 10K and each takes sometime to parse,  dag_parser doesnt pick any newly created files till all 10k files are finished parsing

`if not self._file_path_queue:
                self.emit_metrics()
                self.prepare_file_path_queue()`

Above logic only adds new files to queue when queue is empty

### What you think should happen instead

Every loop of dag processor should pick new files and add into file for parsing queue

### How to reproduce

_No response_

### Operating System

All

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

#
#
#
from __future__ import annotations
import logging
import multiprocessing
import os
import pathlib
import random
import socket
import sys
import threading
from datetime import datetime, timedelta
from logging.config import dictConfig
from tempfile import TemporaryDirectory
from textwrap import dedent
from unittest import mock
from unittest.mock import MagicMock, PropertyMock
import pytest
from freezegun import freeze_time
from sqlalchemy import func
from airflow.callbacks.callback_requests import CallbackRequest, DagCallbackRequest, SlaCallbackRequest
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
from airflow.configuration import conf
from airflow.dag_processing.manager import (
    DagFileProcessorAgent,