### Description

The Elasticsearch APM agent accepts labels, which are then attached to the trace events generated by Elasticsearch. These labels are useful for searching especially in large clusters.

The current format of the labels the agent accepts is like this:

```
"deployment_id=456,deployment_name=APM Tracing,organization_id=123"
```

Essentially, the different labels are `,` separated, while the label key/value pair is `=` separated.

The Elasticsearch configuration provides a field to configure the labels `tracing.apm.agent.global_labels`, which passed down whatever is set directly to the APM agent.

Given that the APM agent parses the incoming labels string and splits based on `,` and `=`, we rely on the supplier of the configuration to properly sanitize the incoming values. While this works, there are few challenges:
    - Elasticsearch isn't in control of the validation of the labels.
    - The values for all of the labels may not always be in control of the configuration provider.
    - It's not the business of the configuration provider to understand the intricacies of how the labels are communicated between Elasticsearch and the Java APM agent.

To mitigate this issue, we should make `tracing.apm.agent.global_labels` be an affix (suffix) setting and allow the configuration to be in the following form:

```
tracing.apm.agent.global_labels.deployment_id: "456"
tracing.apm.agent.global_labels.deployment_name: "APM Tracing"
tracing.apm.agent.global_labels.organization_id: "123"
```

Once all the settings are supplied, we'll sanitize them for `,` and `=` and supply the global labels in the form the APM agent expects them to be in.
distribution/tools/server-cli/src/test/java/org/elasticsearch/server/cli/APMJvmOptionsTests.java
}
    public void testFindJar() throws IOException {
        assertNotNull(agentPath);
        Path anotherPath = Files.createDirectories(installDir.resolve("another"));
        Path apmPathDir = anotherPath.resolve("modules").resolve("apm");
        Files.createDirectories(apmPathDir);
        assertTrue(
            expectThrows(UserException.class, () -> APMJvmOptions.findAgentJar(anotherPath.toAbsolutePath().toString())).getMessage()
                .contains("Installation is corrupt")
        );
    }
    public void testFileDeleteWorks() throws IOException {
        var tempFile = APMJvmOptions.createTemporaryPropertiesFile(agentPath.getParent());
        var commandLineOption = APMJvmOptions.agentCommandLineOption(agentPath, tempFile);
        var jvmInfo = mock(JvmInfo.class);
        doReturn(new String[] { commandLineOption }).when(jvmInfo).getInputArguments();
        assertTrue(Files.exists(tempFile));
        Node.deleteTemporaryApmConfig(jvmInfo, (e, p) -> fail("Shouldn't hit an exception"));
        assertFalse(Files.exists(tempFile));
    }
    private Path makeFakeAgentJar() throws IOException {
        Path tempFile = createTempFile();
        Path apmPathDir = tempFile.getParent().resolve("modules").resolve("apm");
        Files.createDirectories(apmPathDir);
        Path apmAgentFile = apmPathDir.resolve("elastic-apm-agent-0.0.0.jar");
        Files.move(tempFile, apmAgentFile);
        return tempFile.getParent();
    }
}