### Summary

Using the user module to enforce group membership.  When I add a user to a group by name I get `changed: true` on the first run, and `changed: false` subsequently.  But when I use a group number instead of name, I get `changed: true` every time.

### Issue Type

Bug Report

### Component Name

user

### Ansible Version

```console
ansible [core 2.12.4]
  config file = /home/gareth/src/ansible-test/ansible.cfg
  configured module search path = ['/home/gareth/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/gareth/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]
  jinja version = 3.0.3
  libyaml = True
```


### Configuration

```console
DEFAULT_ROLES_PATH(/home/gareth/src/ansible-test/ansible.cfg) = ['/home/gareth/src/ansible-test>
DEFAULT_TIMEOUT(/home/gareth/src/ansible-test/ansible.cfg) = 30

BECOME:
======

CACHE:
=====

CALLBACK:
========

CLICONF:
=======

CONNECTION:
==========

ssh:
___
control_path(/home/gareth/src/ansible-test/ansible.cfg) = /tmp/ansible-ssh-%%h-%%p-%%r
pipelining(/home/gareth/src/ansible-test/ansible.cfg) = True
timeout(/home/gareth/src/ansible-test/ansible.cfg) = 30

HTTPAPI:
=======

INVENTORY:
=========

LOOKUP:
======

NETCONF:
=======

SHELL:
=====

VARS:
====
```


### OS / Environment

Ubuntu 22.10

### Steps to Reproduce

```
- name: Add user to group by name
  user:
    name: testuser
    groups: testgroup
    append: yes

- name: Add user to group by number
  user:
    name: testuser
    groups: 1001
    append: yes
  register: why_broken
- debug: 
    msg: "{{ why_broken }}"
```


### Expected Results

When run the first time, I'd expect to see:
```
TASK [test : Add user to group by name] ***************************************************************
changed: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
changed: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************
ok: [ANSIBLETEST1] => {
    "msg": {
        "append": true,
        "changed": true,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```

When run twice, I'd expect to see:
```
TASK [test : Add user to group by name] ***************************************************************
ok: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
ok: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************

ok: [AK-TEST-01] => {
    "msg": {
        "append": true,
        "changed": false,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```

### Actual Results

```console
When run the second time, I actually see:

TASK [test : Add user to group by name] ***************************************************************
ok: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
changed: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************

ok: [AK-TEST-01] => {
    "msg": {
        "append": true,
        "changed": true,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
'''Add or remove SELF.NAME to or from GROUP depending on ACTION.
        ACTION can be 'add' or 'remove' otherwise 'remove' is assumed. '''
        rc = 0
        out = ''
        err = ''
        changed = False
        current = set(self._list_user_groups())
        if self.groups is not None:
            target = set(self.groups.split(','))
        else:
            target = set([])
        if self.append is False:
            for remove in current - target:
                (_rc, _out, _err) = self.__modify_group(remove, 'delete')
                rc += rc
                out += _out
                err += _err
                changed = True
        for add in target - current:
            (_rc, _out, _err) = self.__modify_group(add, 'add')
            rc += _rc
            out += _out
            err += _err
            changed = True
        return (rc, out, err, changed)
    def _update_system_user(self):