**Apache Airflow version**: 2.1.0

**What happened**:

We use a python script that reads JSON configs from db to dynamically generate DAGs. Sometimes, when JSON Configs is updated, we expect DAGs to be removed in Airflow.  This doesn't always happen. From my observation, one of the following happens:
1) DAG is removed because the python script no longer generates it.
2) DAG still exists but run history is empty. When trigger the DAG, first task is stuck in `queued` status indefinitely.

**What you expected to happen**:
I expect the right behavior should be 
>1) DAG is removed because the python script no longer generates it.

**How to reproduce it**:
* Create a python script in DAG folder that dynamically generates multiple DAGs
* Execute these dynamically generated DAGs a few times
* Use Airflow Variable to toggle (reduce) the # of DAGs generated
* Examine if number of dynamically generated DAGs in the web UI


airflow/models/dag.py
more information about the function signature and parameters that are
        passed to the callback.
    :type sla_miss_callback: callable
    :param default_view: Specify DAG default view (tree, graph, duration,
                                                   gantt, landing_times), default tree
    :type default_view: str
    :param orientation: Specify DAG orientation in graph view (LR, TB, RL, BT), default LR
    :type orientation: str
    :param catchup: Perform scheduler catchup (or only run latest)? Defaults to True
    :type catchup: bool
    :param on_failure_callback: A function to be called when a DagRun of this dag fails.
        A context dictionary is passed as a single parameter to this function.
    :type on_failure_callback: callable
    :param on_success_callback: Much like the ``on_failure_callback`` except
        that it is executed when the dag succeeds.
    :type on_success_callback: callable
    :param access_control: Specify optional DAG-level actions, e.g.,
        "{'role1': {'can_read'}, 'role2': {'can_read', 'can_edit'}}"
    :type access_control: dict
    :param is_paused_upon_creation: Specifies if the dag is paused when created for the first time.
        If the dag exists already, this flag will be ignored. If this optional parameter
        is not specified, the global config setting will be used.
    :type is_paused_upon_creation: bool or None
    :param jinja_environment_kwargs: additional configuration options to be passed to Jinja
        ``Environment`` for template rendering
        **Example**: to avoid Jinja from removing a trailing newline from template strings ::
            DAG(dag_id='my-dag',
                jinja_environment_kwargs={
                    'keep_trailing_newline': True,
                    # some other jinja2 Environment options here