Are you trying to say that a and b are supposed to be smaller than bestof.value?

Unfortunately that is not how the syntax works, the && seperates statements, so basically you are saying while a is true and b is smaler than...

What you need is this:

if (checkBox.checked == true){
    while(a < bestof.value && b < bestof.value){
    myFunction();
};

As you realized correctly, your code only stops as soon a and b are above the value, since it checks if a exists and b is over the value, so basically your trigger is when b surpasses the value.

Another example:

let a = 1
let b
if (a) {
  console.log("a exists")
}
if (b) {
  console.log("b exists")
}

As you can see "b exists" is not being printed, and this is basically what you ask ur while loop before the &&, if a exists ...

Mistakes you made:

1. while condition for "I want loop to stop once one of a or b become greater than my game limit" is same as "run loop till both a and b is less then limit":

while(a < bestof.value && b < bestof.value) { ... }

2. It's unnecessary to convert/compare if condition to boolean on your own, JS will do it automatically, so this is enough:

if (checkBox.checked) { ... }

3. You missed '}'. Always compare count of open and close brackets if your IDE/editor don't do it.

if (checkBox.checked == true){
    while(a && b < bestof.value) {
        myFunction();
//   ↑ here you forget to close while body
};

Also: You always can stop any loop with break keyword:

white(condition) {
  if (needToStop) { break; }
}

Conclusion: your code should look like this:

if (checkBox.checked) {
    while(a < bestof.value && b < bestof.value) {
        myFunction();
    }
};