### Summary

When using `set_fact` in combination with `delegate_to`, `delegate_facts` and `when` it seems to actually be only run on the first host: 

```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] }}"
      # note: the following will STILL fail!
      #loop: "{{ groups['all'] | difference(["localhost"]) }}"
```

The problem only arises when setting the variable that is actually used in the `when` clause. If I had to guess, I assume the `when` is checked in the wrong loop iteration. Setting a fact that is not used in `when` will work fine.

#20508 seems to be related in some regard but was fixed some time ago.

### Issue Type

Bug Report

### Component Name

set_fact

### Ansible Version

```console
$ ansible --version
ansible [core 2.11.5]
  config file = /Users/hojerst/.ansible.cfg
  configured module search path = ['/Users/hojerst/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/hojerst/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/hojerst/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/hojerst/.pyenv/versions/3.9.4/bin/ansible
  python version = 3.9.4 (default, Apr 11 2021, 17:03:42) [Clang 12.0.0 (clang-1200.0.32.29)]
  jinja version = 2.11.3
  libyaml = True
```


### Configuration

```console
$ ansible-config dump --only-changed

DEFAULT_VAULT_PASSWORD_FILE(/Users/hojerst/.ansible.cfg) = /Users/hojerst/.ansible/vault-pass%
```


### OS / Environment

MacOS 12.0 Beta (21A5506j)

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] | difference(['localhost']) }}"

    - debug:
        var: test
```

```sh
ansible-playbook playbook.yml -i host1,host2,host3
```


### Expected Results

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
ok: [localhost -> host2] => (item=host2)
ok: [localhost -> host3] => (item=host3)

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


### Actual Results

```console

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
skipping: [localhost] => (item=host2)
skipping: [localhost] => (item=host3)

TASK [debug] ******************************************************************************************************************
ok: [localhost] => {
    "test": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
'''
        Reads the connection property for the host, and returns the
        correct connection object from the list of connection plugins
        '''
        
        if cvars.get('ansible_connection') is not None:
            self._play_context.connection = templar.template(cvars['ansible_connection'])
        else:
            self._play_context.connection = self._task.connection
        
        
        
        connection_name = self._play_context.connection
        
        conn_type = connection_name
        connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
            conn_type,
            self._play_context,
            self._new_stdin,
            task_uuid=self._task._uuid,
            ansible_playbook_pid=to_text(os.getppid())
        )
        if not connection:
            raise AnsibleError("the connection plugin '%s' was not found" % conn_type)
        
        if cvars.get('ansible_become') is not None:
            become = boolean(templar.template(cvars['ansible_become']))
        else:
            become = self._task.become
        if become: