### Summary

Using the user module to enforce group membership.  When I add a user to a group by name I get `changed: true` on the first run, and `changed: false` subsequently.  But when I use a group number instead of name, I get `changed: true` every time.

### Issue Type

Bug Report

### Component Name

user

### Ansible Version

```console
ansible [core 2.12.4]
  config file = /home/gareth/src/ansible-test/ansible.cfg
  configured module search path = ['/home/gareth/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/gareth/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]
  jinja version = 3.0.3
  libyaml = True
```


### Configuration

```console
DEFAULT_ROLES_PATH(/home/gareth/src/ansible-test/ansible.cfg) = ['/home/gareth/src/ansible-test>
DEFAULT_TIMEOUT(/home/gareth/src/ansible-test/ansible.cfg) = 30

BECOME:
======

CACHE:
=====

CALLBACK:
========

CLICONF:
=======

CONNECTION:
==========

ssh:
___
control_path(/home/gareth/src/ansible-test/ansible.cfg) = /tmp/ansible-ssh-%%h-%%p-%%r
pipelining(/home/gareth/src/ansible-test/ansible.cfg) = True
timeout(/home/gareth/src/ansible-test/ansible.cfg) = 30

HTTPAPI:
=======

INVENTORY:
=========

LOOKUP:
======

NETCONF:
=======

SHELL:
=====

VARS:
====
```


### OS / Environment

Ubuntu 22.10

### Steps to Reproduce

```
- name: Add user to group by name
  user:
    name: testuser
    groups: testgroup
    append: yes

- name: Add user to group by number
  user:
    name: testuser
    groups: 1001
    append: yes
  register: why_broken
- debug: 
    msg: "{{ why_broken }}"
```


### Expected Results

When run the first time, I'd expect to see:
```
TASK [test : Add user to group by name] ***************************************************************
changed: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
changed: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************
ok: [ANSIBLETEST1] => {
    "msg": {
        "append": true,
        "changed": true,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```

When run twice, I'd expect to see:
```
TASK [test : Add user to group by name] ***************************************************************
ok: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
ok: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************

ok: [AK-TEST-01] => {
    "msg": {
        "append": true,
        "changed": false,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```

### Actual Results

```console
When run the second time, I actually see:

TASK [test : Add user to group by name] ***************************************************************
ok: [ANSIBLETEST1]

TASK [test : Add user to group by number] ********************************************************
changed: [ANSIBLETEST1]

TASK [test : debug] ***************************************************************************************

ok: [AK-TEST-01] => {
    "msg": {
        "append": true,
        "changed": true,
        "comment": "",
        "failed": false,
        "group": 1000,
        "groups": "1001",
        "home": "/home/testuser",
        "move_home": false,
        "name": "testuser",
        "shell": "/bin/sh",
        "state": "present",
        "uid": 1000
    }
}
```
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
cmd.append('UMASK=' + self.umask)
        else:
            cmd.append('-M')
        if self.system:
            cmd.append('-r')
        cmd.append(self.name)
        (rc, out, err) = self.execute_command(cmd)
        if not self.local or rc != 0:
            return (rc, out, err)
        if self.expires is not None:
            if self.expires < time.gmtime(0):
                lexpires = -1
            else:
                
                lexpires = int(math.floor(self.module.params['expires'])) // 86400
            (rc, _out, _err) = self.execute_command([lchage_cmd, '-E', to_native(lexpires), self.name])
            out += _out
            err += _err
            if rc != 0:
                return (rc, out, err)
        if self.groups is None or len(self.groups) == 0:
            return (rc, out, err)
        for add_group in groups:
            (rc, _out, _err) = self.execute_command([lgroupmod_cmd, '-M', self.name, add_group])
            out += _out
            err += _err
            if rc != 0:
                return (rc, out, err)
        return (rc, out, err)
    def _check_usermod_append(self):