Load the hook as a module. Then mock the module:

jest.mock('module_name', () => ({
    useClientRect: () => [300, 200, jest.fn()]
}));

mock should be called on top of the file outside test fn. Therefore we are going to have only one array as the mocked value.

If you want to mock the hook with different values in different tests:

import * as hooks from 'module_name';

it('a test', () => {
    jest.spyOn(hooks, 'useClientRect').mockImplementation(() => ([100, 200, jest.fn()]));
    //rest of the test
});

Well, this is quiet tricky and sometimes developers get confused by the library but once you get used to it, it becomes a piece of cake. I faced similar issue a few hours back and I'm sharing my solution for you to derive your solution easily.

My custom Hook:

  import { useEffect, useState } from "react";
    import { getFileData } from "../../API/gistsAPIs";
    
    export const useFilesData = (fileUrl: string) => {
      const [fileData, setFileData] = useState<string>("");
      const [loading, setLoading] = useState<boolean>(false);
      useEffect(() => {
        setLoading(true);
        getFileData(fileUrl).then((fileContent) => {
          setFileData(fileContent);
          setLoading(false);
        });
      }, [fileUrl]);
    
      return { fileData, loading };
    };

My mock code: Please include this mock in the test file out side of your test function. Note: Be careful about the return object of mock, it should match with the expected response

const mockResponse = {
  fileData: "This is a mocked file",
  loading: false,
};
jest.mock("../fileView", () => {
  return {
    useFilesData: () => {
      return {
        fileData: "This is a mocked file",
        loading: false,
      };
    },
  };
});

complete test file would be:

import { render, screen, waitFor } from "@testing-library/react";
import "@testing-library/jest-dom/extend-expect";
import FileViewer from "../FileViewer";

const mockResponse = {
  fileData: "This is a mocked file",
  loading: false,
};
jest.mock("../fileView", () => {
  return {
    useFilesData: () => {
      return {
        fileData: "This is a mocked file",
        loading: false,
      };
    },
  };
});

describe("File Viewer", () => {
  it("display the file heading", async () => {
    render(<FileViewer fileUrl="" filename="regex-tutorial.md" className="" />);
    const paragraphEl = await screen.findByRole("fileHeadingDiplay");
    expect(paragraphEl).toHaveTextContent("regex-tutorial.md");
  });
}

Cheers!! and if this is helpful please be kind to the other developers and give it a thumbs up.

Adding on to this answer for typescript users encountering the TS2339: Property 'mockReturnValue' does not exist on type error message. There is now a jest.MockedFunction you can call to mock with Type defs (which is a port of the ts-jest/utils mocked function).

import useClientRect from './path/to/useClientRect';

jest.mock('./path/to/useClientRect');

const mockUseClientRect = useClientRect as jest.MockedFunction<typeof useClientRect>

describe("useClientRect", () => {
  it("mocks the hook's return value", () => {
    mockUseClientRect.mockReturnValue([300, 200, () => {}]);
    // ... do stuff
  });

  it("mocks the hook's implementation", () => {
    mockUseClientRect.mockImplementation(() => [300, 200, () => {}]);
    // ... do stuff
  });
});