### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

When running a DAG that is structured like:
```
@dag{dag_id="my_dag", default_args={"retries":0"}}
def dag():
    op = MyOperator.partial(task_id="my_task", retries=3).expand(...)
```

The following test fails:
```
    def test_retries(self) -> None:
        dag_bag = DagBag(dag_folder=DAG_FOLDER, include_examples=False)
        dag = dag_bag.dags["my_dag"]
        for task in dag.tasks:
            if "my_task" in task.task_id:
                self.assertEqual(3, task.retries) # fails - this is 0
```

When printing out `task.partial_kwargs`, and looking at how the default args and partial args are merged, it seems like the default args are always taking precedence, even though in the `partial` global function, the `retries` do get set later on with the task-level parameter value. This doesn't seem to be respected though.

### What you think should happen instead

_No response_

### How to reproduce

If you run my above unit test for a test DAG, on version 2.4.3, it should show up as a test failure.

### Operating System

OS Ventura

### Versions of Apache Airflow Providers

_No response_

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

kwargs[arg] = default_args[arg]
            missing_args = non_optional_args - set(kwargs)
            if len(missing_args) == 1:
                raise AirflowException(f"missing keyword argument {missing_args.pop()!r}")
            elif missing_args:
                display = ", ".join(repr(a) for a in sorted(missing_args))
                raise AirflowException(f"missing keyword arguments {display}")
            if merged_params:
                kwargs["params"] = merged_params
            hook = getattr(self, "_hook_apply_defaults", None)
            if hook:
                args, kwargs = hook(**kwargs, default_args=default_args)
                default_args = kwargs.pop("default_args", {})
            if not hasattr(self, "_BaseOperator__init_kwargs"):
                self._BaseOperator__init_kwargs = {}
            self._BaseOperator__from_mapped = instantiated_from_mapped
            result = func(self, **kwargs, default_args=default_args)
            
            self._BaseOperator__init_kwargs.update(kwargs)  
            
            
            if not instantiated_from_mapped and func == self.__init__.__wrapped__:  
                self.set_xcomargs_dependencies()
                
                self._BaseOperator__instantiated = True
            return result
        apply_defaults.__non_optional_args = non_optional_args  
        apply_defaults.__param_names = set(non_variadic_params)  
        return cast(T, apply_defaults)
    def __new__(cls, name, bases, namespace, **kwargs):