### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

Greetings!
Recently I’ve faced a problem. It seems that assigning object, which can’t be serialized to JSON, to the dag_run.conf dict cause critical errors with UI.
After executing code example in "How to reproduce": 
Grid representation of the DAG breaks with following result:
<img width="800" alt="Pasted Graphic" src="https://github.com/apache/airflow/assets/79107237/872ccde5-500f-4484-a36c-dce6b7112286">
Browse -> DAG Runs also becomes unavailable.
<img width="800" alt="Pasted Graphic 1" src="https://github.com/apache/airflow/assets/79107237/b4e3df0c-5324-41dd-96f3-032e706ab7a9">
Dag itself continues to work correctly, this affects only UI graph and dagrun/list/
I suggest to use custom dictionary with restriction on setting non json values.

### What you think should happen instead

Raise an error

### How to reproduce

Execute following task. 
Composer 2 version 2.4.2
Airflow version 2.5.3
```
@task 
def test_task(**context):
    context['dag_run'].conf["test"] = np.int64(1234)
```

### Operating System

Ubuntu 20.04.6 LTS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

self, dag: DAG, ti_mutation_hook, *, session: Session
    ) -> set[str]:
        """
        Check for removed tasks/restored/missing tasks.
        :param dag: DAG object corresponding to the dagrun
        :param ti_mutation_hook: task_instance_mutation_hook function
        :param session: Sqlalchemy ORM Session
        :return: Task IDs in the DAG run
        """
        tis = self.get_task_instances(session=session)
        
        task_ids = set()
        for ti in tis:
            ti_mutation_hook(ti)
            task_ids.add(ti.task_id)
            try:
                task = dag.get_task(ti.task_id)
                should_restore_task = (task is not None) and ti.state == TaskInstanceState.REMOVED
                if should_restore_task:
                    self.log.info("Restoring task '%s' which was previously removed from DAG '%s'", ti, dag)
                    Stats.incr(f"task_restored_to_dag.{dag.dag_id}", tags=self.stats_tags)
                    
                    Stats.incr("task_restored_to_dag", tags={**self.stats_tags, "dag_id": dag.dag_id})
                    ti.state = None
            except AirflowException:
                if ti.state == TaskInstanceState.REMOVED:
                    pass  
                elif self.state != DagRunState.RUNNING and not dag.partial: