Not when using all as it does not expose the "iteration variable". You will need an explicit, nested loop. You also forgot the f'' prefix to denote an f-string:

for l1 in [['a']]:
    for l2 in l1:
        for e in l2:
            assert isinstance(e, int), f"Values of the dictionnary 
aren't lists of integers. Assert found in  '{l2}'"

AssertionError: Values of the dictionnary aren't lists of integers. Assert found in  'a'

You can use an assignment expression to capture the "witness" to all returning False. To do this, we'll do away with some generators. We'll use map to apply isinstance to each value of l2, which lets us capture the current value of l2 in a variable that will persist for use in the assert message. We'll also use itertools.chain to eliminate the need for the name l1.

from itertools import chain


def is_int(x):
    return isinstance(x, int)


sequence = {
    'key1': [[1, 2, 3], [4, 5, 6]],
    'key2': [[7, 8, 9], ['a', 11, 12]]
}

assert all(all(map(is_int, (witness := l2)))                
           for l2 in chain(*sequence.values())), \
       f"Values of the dictionary aren't lists of integers. Assert found in '{witness}'"

With a little modification, you can iterate over sequence.items() so that you can also capture the key whose list contains the offending l2. I leave that as an exercise for the reader.

I would unpack all your e's and then do an assertion:

# Assuming l1 is a list of lists
l1 = [[1,2,3], [4,5,6], ["7", 8, 9]]


# TO TEST SUBLISTS
l2 = [x for x in l1]
for sublist in l2:
    assert all([isinstance(e, int) for x in sublist]), 
        f'Values isn\'t a list of integers: {sublist}'


# TO TEST FOR INDIVIDUAL VALUES
all_e = [y for x in l1 for y in x]

# To raise an error on each value:
for e in all_e:
    assert(isinstance(e, int)), f'Value isn\'t an int: {e}'

# To operate in one line
assert all([isinstance(e, int) for x in all_e]), 
    f'Values aren\'t a list of integers: {all_e}'

Could remember it in a variable that does leak out simply by inserting if [witness := l2]:

sequence = {                             # thanks chepner
    'key1': [[1, 2, 3], [4, 5, 6]],
    'key2': [[7, 8, 9], ['a', 11, 12]]
}

assert all(isinstance(e, int)
           for l1 in sequence.values()
           for l2 in l1 if [witness := l2]
           for e in l2), f"Values of the dictionnary aren't lists of integers. Assert found in  '{witness}'"

Output:

Traceback (most recent call last):
  File ".code.tio", line 6, in <module>
    assert all(isinstance(e, int)
AssertionError: Values of the dictionnary aren't lists of integers. Assert found in  '['a', 11, 12]'

Try it online!

Could also use (witness := l2) here instead of [witness := l2], but the latter is applicable in general.