I am trying to decode the h264 stream from ws-scrcpy frame by frame to apply some computer vision operations. (ScrCpy is basically a mobile device application that broadcasts the device screen via h264. WS-ScrCpy is a web application that is able to receive and display the stream in a browser).

Since I need to do the computer vision operations in RGB space and on a server (via nodejs), I need to convert frames first from YUV to RGB format. Since JS-only implementations are slow, I am trying to get this done with OpenCV.

Unfortunately, the decoded frames look like this:

Here is what I do in more detail:

I use Broadway Acv decoder to decode data (NALUs?- received via websocket) into frames. Then I use OpenCV to convert YUV frames to RGB.

import Avc from './vendor/Broadway/Decoder'
//...
this.avc.onPictureDecoded = (buffer: Uint8Array, width: number, height: number) => {
  const mYUV = new cv.Mat(h + h / 2, w, cv.CV_8UC1, buffer)
  const mRgb = mYUV.cvtColor(cv.COLOR_YUV420p2BGR)
  cv.imwrite('./del/test.png', mRgb)
}

The fact that the image size of the resulting Mat object is correct, tells me that I am doing something right. But as you can see there also is obviously something wrong.

Any idea what the problem might be?