In fact here your code seems to work fine.

Reading the Date documentation :

JavaScript Date objects represent a single moment in time in a platform-independent format. Date objects contain a Number that represents milliseconds since 1 January 1970 UTC.

So when you're doing new Date(1653991056493) it's 1653991056493ms after Jan 1st 1970 which is 19143.4 days.

If you want the ms between a date and the current date, you can just substract the current date with the timestamp

new Date() - 1653991056493

function msToTime(ms) {
  let seconds = (ms / 1000).toFixed(1);
  let minutes = (ms / (1000 * 60)).toFixed(1);
  let hours = (ms / (1000 * 60 * 60)).toFixed(1);
  let days = (ms / (1000 * 60 * 60 * 24)).toFixed(1);
  if (seconds < 60) return seconds + " Sec";
  else if (minutes < 60) return minutes + " Min";
  else if (hours < 24) return hours + " Hrs";
  else return days + " Days"
}
console.log(msToTime(new Date() - 1653991056493))

I interpretted the question slightly differently to the accepted answer and am posting this as it might help people seeking to do what I though was being asked:

namely to reduce an elapsed period of milliseconds to either (rounded) days OR (rounded) hours OR (rounded) minutes OR (rounded) seconds - dependent on which fits the scale of the elapsed duration (as one might want to do where, for example, a page is to report "comment made 2 days ago" or "comment made 10 seconds ago" etc. - just like SO does when reporting when answers or comments were made.

As with the accepted answer, the elapsed time has to first be calculated by subtracting the passed ms value from a new date value (and, since units smaller than seconds will never be needed, the elapsed value converted to seconds by dividing by 1000):

const now = new Date();
const secondsPast = Math.round((now-pastMs)/1000);

This value is then filtered down a series of if checks, each containing a conditional return statement if the relevant time unit has been reached. Thus, if the 'scale' is seconds (i.e the elapsed duration is less than a minute), the function returns the seconds value and exits immeadiately:

if (secondsPast<60) {return `${secondsPast} seconds`} // terminates here if true;

If the seconds value is greater than 60, minutes are checked and a return made if they are less than sixty. The process repeats until larger values are eventually returned as days if no other unit was appropriate. Note the use of Math.floor to only return whole numbers for the relevant unit.

(this is, I think, what the original question was trying to achieve).

function elapsedTime(pastMs) {
  const now = new Date();
  const secondsPast = Math.round((now-pastMs)/1000);
  
    if (secondsPast<60) {return `${secondsPast} seconds`} // terminates here if true;
    
  const minutesPast = Math.floor(secondsPast/60);
    if (minutesPast<60) {return `${minutesPast} minutes`} // terminates here if true;
    
  const hoursPast = Math.floor(minutesPast/60);
    if (hoursPast<24) {return `${hoursPast} hours`} // terminates here if true;
    
  return `${Math.floor(hoursPast/24)} days`;
  
} // end function elapsedTime;
    
console.log(elapsedTime(1653991056493))