### Summary

When using `set_fact` in combination with `delegate_to`, `delegate_facts` and `when` it seems to actually be only run on the first host: 

```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] }}"
      # note: the following will STILL fail!
      #loop: "{{ groups['all'] | difference(["localhost"]) }}"
```

The problem only arises when setting the variable that is actually used in the `when` clause. If I had to guess, I assume the `when` is checked in the wrong loop iteration. Setting a fact that is not used in `when` will work fine.

#20508 seems to be related in some regard but was fixed some time ago.

### Issue Type

Bug Report

### Component Name

set_fact

### Ansible Version

```console
$ ansible --version
ansible [core 2.11.5]
  config file = /Users/hojerst/.ansible.cfg
  configured module search path = ['/Users/hojerst/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/hojerst/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/hojerst/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/hojerst/.pyenv/versions/3.9.4/bin/ansible
  python version = 3.9.4 (default, Apr 11 2021, 17:03:42) [Clang 12.0.0 (clang-1200.0.32.29)]
  jinja version = 2.11.3
  libyaml = True
```


### Configuration

```console
$ ansible-config dump --only-changed

DEFAULT_VAULT_PASSWORD_FILE(/Users/hojerst/.ansible.cfg) = /Users/hojerst/.ansible/vault-pass%
```


### OS / Environment

MacOS 12.0 Beta (21A5506j)

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- hosts: localhost
  gather_facts: no
  connection: local
  tasks:
    - set_fact:
        test: 123
      delegate_to: "{{ item }}"
      delegate_facts: true
      when: test is not defined
      loop: "{{ groups['all'] | difference(['localhost']) }}"

    - debug:
        var: test
```

```sh
ansible-playbook playbook.yml -i host1,host2,host3
```


### Expected Results

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
ok: [localhost -> host2] => (item=host2)
ok: [localhost -> host3] => (item=host3)

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


### Actual Results

```console

PLAY [localhost] **************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************
ok: [localhost -> host1] => (item=host1)
skipping: [localhost] => (item=host2)
skipping: [localhost] => (item=host3)

TASK [debug] ******************************************************************************************************************
ok: [localhost] => {
    "test": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
finally:
        termios.tcsetattr(master, termios.TCSANOW, old)
    os.close(master)
    if p.returncode == 0:
        result = json.loads(to_text(stdout, errors='surrogate_then_replace'))
    else:
        try:
            result = json.loads(to_text(stderr, errors='surrogate_then_replace'))
        except getattr(json.decoder, 'JSONDecodeError', ValueError):
            
            result = {'error': to_text(stderr, errors='surrogate_then_replace')}
    if 'messages' in result:
        for level, message in result['messages']:
            if level == 'log':
                display.display(message, log_only=True)
            elif level in ('debug', 'v', 'vv', 'vvv', 'vvvv', 'vvvvv', 'vvvvvv'):
                getattr(display, level)(message, host=play_context.remote_addr)
            else:
                if hasattr(display, level):
                    getattr(display, level)(message)
                else:
                    display.vvvv(message, host=play_context.remote_addr)
    if 'error' in result:
        if play_context.verbosity > 2:
            if result.get('exception'):
                msg = "The full traceback is:\n" + result['exception']
                display.display(msg, color=C.COLOR_ERROR)
        raise AnsibleError(result['error'])
    return result['socket_path']