Attach (recommended) or Link to PDF file here: [test.pdf](https://github.com/mozilla/pdf.js/files/13053891/test.pdf)

Configuration:
- Web browser and its version: Firefox 118.0.2
- Operating system and its version: Windows 10 Pro 10.0.19045
- PDF.js version: 3.10.86
- Is a browser extension: No

Steps to reproduce the problem:
1. Open the PDF in Firefox.

What is the expected behavior? (add screenshot)
When I open the PDF in Google Chrome or Adobe Acrobat, it looks like this:
![after](https://github.com/mozilla/pdf.js/assets/138867750/6a76ca25-f82a-4041-9d03-2edb36cecfa6)

What went wrong? (add screenshot)
Instead, it looks like this:
![before](https://github.com/mozilla/pdf.js/assets/138867750/4b7789df-56e2-49f4-b0b9-82c22ca6bcaa)

const length = maskData.length;
  for (let i = 3; i < length; i += 4) {
    const y =
      maskData[i - 3] * 77 + 
      maskData[i - 2] * 152 + 
      maskData[i - 1] * 28; 
    layerData[i] = transferMap
      ? (layerData[i] * transferMap[y >> 8]) >> 8
      : (layerData[i] * y) >> 16;
  }
}
function genericComposeSMask(
  maskCtx,
  layerCtx,
  width,
  height,
  subtype,
  backdrop,
  transferMap,
  layerOffsetX,
  layerOffsetY,
  maskOffsetX,
  maskOffsetY
) {
  const hasBackdrop = !!backdrop;