### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

Greetings!
Recently I’ve faced a problem. It seems that assigning object, which can’t be serialized to JSON, to the dag_run.conf dict cause critical errors with UI.
After executing code example in "How to reproduce": 
Grid representation of the DAG breaks with following result:
<img width="800" alt="Pasted Graphic" src="https://github.com/apache/airflow/assets/79107237/872ccde5-500f-4484-a36c-dce6b7112286">
Browse -> DAG Runs also becomes unavailable.
<img width="800" alt="Pasted Graphic 1" src="https://github.com/apache/airflow/assets/79107237/b4e3df0c-5324-41dd-96f3-032e706ab7a9">
Dag itself continues to work correctly, this affects only UI graph and dagrun/list/
I suggest to use custom dictionary with restriction on setting non json values.

### What you think should happen instead

Raise an error

### How to reproduce

Execute following task. 
Composer 2 version 2.4.2
Airflow version 2.5.3
```
@task 
def test_task(**context):
    context['dag_run'].conf["test"] = np.int64(1234)
```

### Operating System

Ubuntu 20.04.6 LTS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Variable.set(key="arg1", value=[1, 2])
    with dag:
        task_2.expand(arg2=task_1()).operator
    
    
    serialized_dag = SerializedDAG.from_dict(SerializedDAG.to_dict(dag))
    dr.dag = serialized_dag
    
    ti = dr.get_task_instance(task_id="task_1")
    task1 = dag.get_task("task_1")
    ti.refresh_from_task(task1)
    ti.run()
    
    dr.task_instance_scheduling_decisions()
    tis = dr.get_task_instances()
    indices = [(ti.map_index, ti.state) for ti in tis if ti.map_index >= 0]
    assert sorted(indices) == [
        (0, State.NONE),
        (1, State.NONE),
        (2, TaskInstanceState.REMOVED),
    ]
def test_mapped_literal_faulty_state_in_db(dag_maker, session):
    """
    This test tries to recreate a faulty state in the database and checks if we can recover from it.
    The state that happens is that there exists mapped task instances and the unmapped task instance.
    So we have instances with map_index [-1, 0, 1]. The -1 task instances should be removed in this case.
    """
    with dag_maker(session=session) as dag:
        @task
        def task_1():