When num < 5 it returns yes.

If the first argument doesn't trigger that, they that yes will be returned to:

return [...ABC(rest)];

Where it will be spread and put into an array.

When you make a recursive call, you have [...ABC(rest)] which first runs the function, then spreads its result into an array. This is clearer if we add an intermediate variable:

let recursive_result = ABC(rest);
return [...recursive_result];

What you intended to do was first spread rest into the arguments to the function, and then return the result directly:

let recursive_result = ABC(...rest);
return recursive_result;

Fixing that, we find another problem, which is that once the array is empty, the function will be called with no arguments, and num will be undefined, which is still not "less than 5". That means the if statement will never be reached, and we'll keep recursing forever (until your JS engine gives up and raises an error).

Detecting undefined explicitly, we get the result you expected:

function ABC(num, ...rest) {
  if (num < 5 || typeof num == 'undefined' ) {
    return 'yes';
  } else {
    return ABC(...rest);
  }
};
console.log(ABC(8, 5, 6, 7));

If you add a few console.log it becomes clearer :

function ABC(num, ...rest) {
    console.log(`num=`, num, ", rest=", rest)
  if (num < 5) {
      console.log(`returning 'yes'`)
    return 'yes';
  } else {
      console.log(`returning [...ABC(${rest})]`)
    return [...ABC(rest)];
  }
};
console.log(ABC(8, 5, 6, 7));

• Run 1: num=8 , rest=[5, 6, 7]

So it goes into the else and returns [...ABC([5, 6, 7])]. There is only one argument, an array. So at run 2:

• Run 2 : num=[5, 6, 7] , rest= []

[5, 6, 7] < 5 ? False. So it goes in the else again. It returns [...ABC([])].

• Run 3 : num=[] , rest= []

[] < 5 ? True, oddly enough. So ABC([]) returns 'yes'.

But then this 'yes' is being returned into the previous function call, that is, [...ABC([])], which becomes [...'yes'], which is ['y', 'e', 's'].