mapbox-gl-js version: Latest
browser: All Browsers
Code Sandbox Link - https://codesandbox.io/embed/flamboyant-smoke-0v546?codemirror=1&fontsize=14&hidenavigation=1&theme=dark
Radius Screen Shot - https://prnt.sc/1yf0a0x
I'm not exactly sure why you expected a "perfect radius" (I assume you mean a perfect circle) here. The distance from the equator to the north pole is only 10,000km. So a circle with a radius of 11,000km, even if it was situated on the equator, would extend past both the north and south poles. Your circle would actually be getting smaller again (because the furthest north and south points would be approaching each other on the far side of the planet). A similar thing is happening east and west: the eastern and western points are starting to get closer to each other on the far side.
As for the specific issues you're seeing, the Web Mercator projection (which Mapbox uses) is only defined to around 85 degrees north and south, so your circle would be out of bounds for a Mapbox map. That may be the cause of it.
TL,DR: The Earth is not big enough for what you're trying to do.