| has special meaning in regular expressions, it's used to specify alternatives. You need to escape it to match it literally.

There's no need to use lookbehinds when you're capturing the part after that. The purpose of lookarounds is to keep them out of the matched string, but if you're only interested in the capture groups this is irrelevant.

This regexp should work for you:

Group:\s(\w+) \|SubGroup:\s(\w*)

DEMO

If by "word" you're happy with the definition of \w (which is [A-Za-z0-9_]; more below), you can do it like this:

const rex = /Group:\s*(\w+).*?SubGroup:\s*(\w+)/;

Add the i flag if you want to allow Group and SubGroup to be in lower case.

That:

1. Looks for Group:
2. \s* - allows for optional whitespace after it
3. (\w+) captures all "word" characters that follow that
4. .*? - looks for optional anything (the ? is important to make it non-greedy)
5. Looks for SubGroup:
6. \s* optional whitespace again
7. (\w+) captures all "word" chars after that

Live Example:

const str = "Group: i_am_group |SubGroup: i_am_sub_group";
const rex = /Group:\s*(\w+).*?SubGroup:\s*(\w+)/;
console.log(str.match(rex));

If you want a different definition for "word" character than \w, use [something_here]+ instead of \w+, where within the [ and ] you list the characters / character ranges you want to consider "word" characters.

For instance, in English, we usually don't consider _ as part of a word (though your examples use it, so I'll leave it in), but we often consider - to be part of a word. We also frequently allow letters borrowed from other languages like é and ñ, so you might want those in the character class. You might go further and (in ES2015+ environments) use Unicode's definition of a "letter", which is written \p{Letter} (and requires the u flag on the expression):

const rex = /Group:\s*([-\p{Letter}0-9_]+).*?SubGroup:\s*([-\p{Letter}0-9_]+)/u;

(The - at the very beginning is treated literally, not as an indicator of a range.)

Live Example:

const str = "Group: i_am_group |SubGroup: i_am_sub_group";
const rex = /Group:\s*([-\p{Letter}0-9_]+).*?SubGroup:\s*([-\p{Letter}0-9_]+)/u;
console.log(str.match(rex));