The best way would be to use the Transfer event since that's what for. You can query all transfer events, and filter by ID.

Yet you can too create your custom function(id, address) which will be called on each nft transfer, there you can push to an array in a mapping id -> address[] where the array represents the addresses of the peeps who owned that NFT.

And then create another function(id) to return that array depending on the id sent to the function.

Well, after a few hours more research, I found some solutions and wanted to share those.

There are some ways to do this:

1. If you are working on a specific contract and this contract extends ERC721Enumerable, your case study is a little bit easier. Because you can find the token count using totalSupply() so you can skip the first part of the code below.
2. If you want to use a 3rd party tool, you may want to check out Moralis. It returns the owner count with a list of addresses and extra data. See the already exist answer
3. If you want to write the code manually, here is my code:

async testGetOwnerCount({ }, payload) {
    // Connect the contract 
    let nftContract = new ethers.Contract(nftContractAddress, ERC721.abi, signer);

    // First find the mint count which is equal to the token count
    const transferFilter = nftContract.filters.Transfer("0x0000000000000000000000000000000000000000", null, null)
    const tokens = await nftContract.queryFilter(transferFilter)
    const tokenCount = tokens.length;

    // Iterate over tokens and store owner addresses in an array
    let owners = []
    for (let i = 0; i < tokenCount; i++) {
      // First, find the all transfers of the token 
      // from null` to `null` so we get all the transfers of `tokenId` 
      const transferFilter = nftContract.filters.Transfer(null, null, parseInt(tokens[i].args.tokenId))
      const tokenTransfers = await nftContract.queryFilter(transferFilter)

      // `args.to` of the last element gives the current owner of issued token
      let lastTransfer = tokenTransfers[tokenTransfers.length - 1]
      let currentOwner = lastTransfer.args.to

      // If the address has already found before, don't add it...
      if (!owners.includes(currentOwner)) {
        owners.push(currentOwner)
      }
    }
}

There might be better ways to achieve this but this was the best I could find since I want to handle all the contracts not only those are ERC721Enumerable and don't want to use another tool.