I managed to do this by filtering all the "transfer" events using getPastEvents here. The to, from and tokenId are indexed on chain. So we can apply those filters. Had to somehow take care of the infura limits by recursively calling the function.