I'm new to API world and just want to ask... To check whether an API is working or not, do I just have to do GET request on its baseUrl and make sure it gets status 200 (OK) or do I have to test all its endpoints and also make sure that all of them get status 200 (OK)?
It depends on what you mean by "working".
Testing just the base URL will tell you if it works in the sense that "You installed it, and started it, and the whole thing hasn't fallen over". (Assuming that the API isn't running of a bunch of microservices which might have individually failed).
Testing every endpoint, and making sure that the data returned is the data expected (so just looking for 200 OK is inadequate) will tell you if it works in the sense that "Everything is implemented correctly and the API does what it is supposed to do" (for a value of "correct" based on the quality of the tests).