there is a need to combine Unity3d and OpenCV (preferably with NumPy, since functions will also be used from there, which means +Python is also needed). The main idea is to pass images from camera from Unity to OpenCV for processing and then sending actions back to Unity.

I've tried many variants, made by taking picture bytes stream from RenderTexture (Unity) and sending it to python server on websockets. Everything works, and the transfer speed is acceptable, but during the "photographing" of the texture it takes a lot of resources and the main thread of Unity hangs... so I get 5 fps, which is not good. I want to rewrite it the right way: either integrate openCV into unity, or pull texture from video memory (but that's horrible C++ and openGL, I want to get python, because I need numPy).

I tried streaming with FMETP STREAM (it has not started front end, which means it works, but image is not displayed in browser), then put in IronPython openCV and use them in C# (I can not install opencv in any way), tried to put modules for C#: OpenCVSharp (seems to be ok with this package), NumSharp (lacks some methods to be used in the project), and socket image transfer method (works, but awfully long and lags). .. in each variant tried and studied almost everything thoroughly.

Actually, I wanted to ask for advice, which integration is better to choose, how to do it better?

Code I use to send image bytes to python server: https://pastebin.com/pRyutRcj

    WebSocket ws;

    void Start () {

        ws = WebSocketFactory.CreateInstance("ws://localhost:12345");

        // ...

        // Connect to the server
        ws.Connect();

        InvokeRepeating("Capture", 1f, 0.2f);  //1s delay, repeat every 1s

    }

    public void Capture()
    {
        RenderTexture activeRenderTexture = RenderTexture.active;
        RenderTexture.active = Camera.targetTexture;

        Camera.Render();

        Texture2D image = new Texture2D(Camera.targetTexture.width, Camera.targetTexture.height);
        image.ReadPixels(new Rect(0, 0, Camera.targetTexture.width, Camera.targetTexture.height), 0, 0);
        image.Apply();
        RenderTexture.active = activeRenderTexture;

        byte[] bytes = image.EncodeToPNG();
        Destroy(image);


        ws.Send(bytes);
    }