Minimum, reproducible example

I have an SVG grid pattern looking like this

// positioned at 0, 0 (relative to the browser's viewport) and takes 100% of the screen's width and height

      <svg width="100%" height="100%">
        <defs>
          <pattern
            id="pattern"
            patternUnits="userSpaceOnUse"
            x={0}
            y={0}
            width={50}
            height={50}
          >
            <rect x={0} y={0} height={5} width={5} fill={"blue"} />
          </pattern>
        </defs>

        <rect width="100%" height="100%" x={0} y={0} fill={"url(#pattern)"} />
      </svg>

This is my buggy zoom implementation. The logic behind it is described here, in another stackoverflow question

    let scale = 1;
    const transform = {
      x: 0,
      y: 0,
    };
    const scaleFactor = 0.1;
    svg.onwheel = (event) => {
      // Normalize mouse wheel movement to +1 or -1 to avoid unusual jumps.
      const wheel = event.deltaY < 0 ? 1 : -1;

      const newScale = scale + wheel * scaleFactor;

      const scaleChange = newScale - scale;
      const offsetX = event.clientX * scaleChange;
      const offsetY = event.clientY * scaleChange;

      transform.x -= offsetX;
      transform.y -= offsetY;

      pattern.setAttribute("x", transform.x.toString());
      pattern.setAttribute("y", transform.y.toString());

      pattern.setAttribute("width", (50 * newScale).toString());
      pattern.setAttribute("height", (50 * newScale).toString());

      scale = newScale;
    };

It only works correctly when I zoom up and down at the same point (without moving my mouse). When I zoom up at say mouse position (10, 10) and zoom down at mouse position (20, 20) it looks off and I can't seem to figure out why

Here is a code sandbox with the minimum, reproducible example