(V8 developer here.)

V8 constructs a different stackframe for each kind of code object. It means V8 should know the kind of the callee before it executes.

Stack frames are set up by whoever needs them, so callers don't have to inspect callees. (They could, if they had to.)

If the machine code is run directly on the processor, nothing can help V8 determine the kind of callee.

V8 could emit machine code that checks the kind of callee. But to repeat myself: knowing the optimization status of a callee isn't necessary.

Does V8 monitor the execution of (optimized) machine code?

No, it doesn't.

how does V8 check the invalidation of code dependency

Dependencies are installed on something that can change (e.g. maps, protectors). When that change happens, registered code dependencies will be deoptimized (i.e. marked for "lazy deoptimization" when they have activations on the stack, or simply thrown away otherwise).

[how does V8] detect whether the callee is compiled or not?

It doesn't need to. When a function isn't compiled yet, its "code" will be a stub that triggers compilation.

To look at the concrete example you gave in the comments:

var b = false;

function change_o() {
  // as long as b is false, this won't be reassigned ...
  if (b) o = { y : 1, x : 0};
}

var o = { x : 1 };

function f() {
  change_o();
  // ... and thus o.x can be compiled down
  return o.x;
}

// f and change_o are compiled somewhen 
f(); f(); f(); // ...

// a precondition changes
b = true;

// during execution of change_o o is reassigned and changes its shape, accessing o.x will now need a different offset and thus f (which is currently on the stack) becomes invalid
f();

Thus at the moment the assignment o = happens, the runtime jumps into some deoptimization logic, which has to invalidate all the code based on the false assumption. Invalidating calls from non-compiled code to compiled code is easy, simply remove the reference to the compiled code from some kind of registry and nobobdy will call it anymore. Invalidating future calls from compiled functions to the invalidated compiled function is harder, though if the call uses some indirection, this indirection can be rewritten to redirect all future calls. This leaves two kind of invalidation problems:

• the currently evaluated function needs to be invalidated (-> eager deoptimization), not that problematic either as it's the function that bailed out to the deoptimization routine
• a function that called the evaluated function needs to be invalidated (-> lazy deoptimization), this is problematic, as they might be somewhere in the stack, and continuing execution would somewhen resume to them (e.g. f in the example above)

However in the later case, there is a way to find and replace those functions: Every time one function calls to another, it pushes the return address onto the stack (see e.g. x86 stack usage), and when the called function executes the ret instruction, it pops the address und jumps there. Thus by scanning the stack for return addresses residing inside invalidated functions, and replacing those by another address, when the inner function like change_o returns to f, it doesn't actually jump back to the compiled f code but into a handler, which can then make the necessary adjustments.

So V8 does not "monitor" the execution, it actively modifies it when necessary by rewriting the stack.

A detailed walkthrough can be found here.