For how parameterised grammar productions work in general, see JavaScript grammar notation.

If I am not wrong I think that the annotation [UnicodeMode] refers to the Unicode flag of the regexp.

Yes indeed, it is used in ParsePattern which is called both for regex literals and from the RegExp constructor, and the UnicodeMode grammar flag is set when the /u regex flag is given.

Can someone guide me into what [N] means?

This flag is used to change the meaning of named backreferences. If the regex contains at least one named group (?<name>…), then \k<name> notation will refer to such groups (and a group with the given name must exist, according to early error static semantics). But if the regex does not use (?<…>…) anywhere, then \k will just be treated as an IdentityEscape (i.e. matching a literal k in the input text).

What about [Sep]?

That's just a reference to the DecimalDigits production from the normal number literal grammar. It prevents the use of a NumericLiteralSeparator in the number, essentially making it a production matching only integer literals.