### Apache Airflow version

main (development)

### What happened

Task failing with error "missing upstream values" when running a dag with dynamic task group mapping
error -
```
[2022-11-23, 15:50:07 UTC] {abstractoperator.py:456} ERROR - Cannot expand <Task(_PythonDecoratedOperator): increment_and_verify.hello_there> for run manual__2022-11-23T15:49:59.771915+00:00; missing upstream values: ['x']
```

### What you think should happen instead

It should succeed 

### How to reproduce

Run the below dag code 
```
from airflow import DAG
from airflow.models.taskinstance import TaskInstance
from airflow.operators.dummy import DummyOperator
from datetime import datetime, timedelta
from airflow.decorators import task, task_group
with DAG(
    dag_id="taskmap_taskgroup",
    tags=["AIP_42"],
    start_date=datetime(1970, 1, 1),
    schedule_interval=None,
) as dag:

    @task
    def onetwothree():
        return [1,2,3]

    @task
    def hello_there(arg):
        print(arg)

    @task_group
    def increment_and_verify(x):
        hello_there(x)

    increment_and_verify.expand(x=onetwothree())>>DummyOperator(task_id="done")
```

### Operating System

mac os

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

"""Get a flat list of relatives, either upstream or downstream."""
        dag = self.get_dag()
        if not dag:
            return set()
        return [dag.task_dict[task_id] for task_id in self.get_flat_relative_ids(upstream)]
    def _iter_all_mapped_downstreams(self) -> Iterator[MappedOperator]:
        """Return mapped nodes that are direct dependencies of the current task.
        For now, this walks the entire DAG to find mapped nodes that has this
        current task as an upstream. We cannot use ``downstream_list`` since it
        only contains operators, not task groups. In the future, we should
        provide a way to record an DAG node's all downstream nodes instead.
        Note that this does not guarantee the returned tasks actually use the
        current task for task mapping, but only checks those task are mapped
        operators, and are downstreams of the current task.
        To get a list of tasks that uses the current task for task mapping, use
        :meth:`iter_mapped_dependants` instead.
        """
        from airflow.models.mappedoperator import MappedOperator
        from airflow.utils.task_group import TaskGroup
        def _walk_group(group: TaskGroup) -> Iterable[tuple[str, DAGNode]]: