Do you really need a regular expression?

First off, if you split by |, each odd entry in the result is a list of extensions. You can then split that again by ; to get the extensions, which you can then flatten into a single sequence and trim each element of the starting *. Finally, get the distinct set of that and put that into an array.

This can all be accomplished with Split and Linq:

var extensions = filter.Split('|', StringSplitOptions.RemoveEmptyEntries)
                       .Where((x, i) => i % 2 != 0)
                       .SelectMany(x => x.Split(';', StringSplitOptions.RemoveEmptyEntries))
                       .Select(x => x.TrimStart('*'))
                       .Distinct()
                       .ToArray();

Removing empty entries from the split ensures that if you end with a separator it just gets ignored.

See it in action on .NET Fiddle.

Your example strings seem to consist of an optional leading part between parenthesis followed by a pipe | and then the extensions.

If you want to match the format of the data, you can use a capture group with the same name, and the Group.Captures Property.

\b[fF]iles\b[^*()]*(?:\([^()]*\))?\|\*(?<ext>\.[\w*]+)(?:[,;]\*(?<ext>\.[\w*]+))*

In parts, the pattern matches:

• \b[fF]iles\b Match either files or Files
• [^*()]* Optionally match any char except * ( )
• (?:\([^()]*\))? Optionally match a part between parenthesis
• \|\* Match |*
• (?<ext>\.[\w*]+) Named group ext, match . and 1+ times a word char or *
• (?: Non capture group
  • [,;]\* Match either , or ; followed by *
  • (?<ext>\.[\w*]+) Same pattern for group ext
• )* Close the non capture group and optionally repeat it to get all the extensions

See a C# regex 101 demo and a C# demo.

For example

string pattern = @"\b[fF]iles\b[^*()]*(?:\([^()]*\))?\|\*(?<ext>\.[\w*]+)(?:[,;]\*(?<ext>\.[\w*]+))*";    
string input = @"txt files `(*.txt)|*.txt|All files (*.*)|*.*`
Image Files`|*.jpg;*.jpeg;*.png;`
Excel Files `(*.xls, *.xlsx)|*.xls;*.xlsx|CSV Files (*.csv)|*.csv`";

var strings = Regex.Matches(input, pattern)

.SelectMany(m => m.Groups["ext"].Captures.Select(c => c.Value))
.ToArray()
.Distinct();

foreach (var s in strings)
{
    Console.WriteLine(s);
}

Output

.txt
.*
.jpg
.jpeg
.png
.xls
.xlsx
.csv

Another shorter pattern not related to the data format, matching a dot and either 3 or 4 word characters or an asterix and asserting a dot to the left:

 (?<=\*)\.(?:\w{3,4}\b|\*)

See another regex demo.