As an FYI, the textpattern plugin in TinyMCE gives you a way to insert lists using (by default) markup style text:

https://www.tiny.cloud/docs/plugins/opensource/textpattern/

The list specific options are:

// The following text patterns require the `lists` plugin
{start: '1. ', cmd: 'InsertOrderedList'},
{start: '* ', cmd: 'InsertUnorderedList'},
{start: '- ', cmd: 'InsertUnorderedList' }

This requires no coding to implement and allows you to determine the text that can start a list. The example above allows a list to start with an asterisk or dash but you can customize that list to your needs.

I'm glad you asked! I went through a few iterations on this solution before finding one that seemed to work pretty well.

First, I defined both the trigger character for Autobullets (hyphen), and the HTML containers within which I'd like Autobullet to be triggered (in this case, paragraphs, divs, table cells, and spans):

const AUTOBULLET_TRIGGER_CHARACTER = "-";
const AUTOBULLET_VALID_CONTAINERS = [
  "HTMLTableCellElement",
  "HTMLParagraphElement",
  "HTMLDivElement",
  "HTMLElement",
];

Then, I added this event handler in the editor setup, with comments:

        editor.on("KeyUp", function (e) {
          var sel = editor.selection.getSel();
          var caretPos = sel.anchorOffset;
          var txtData = sel.anchorNode.textContent;
          
          // Char code 160 is a non breaking space
          const lineMatch = `${AUTOBULLET_TRIGGER_CHARACTER}${String.fromCharCode(
            160
          )}`;

          if (e.key === " " && caretPos === 2 && txtData === lineMatch) {
            if (
              AUTOBULLET_VALID_CONTAINERS.includes(
                sel.focusNode.parentElement.constructor.name
              )
            ) {
              // Create an unordered list
              editor.execCommand("InsertUnorderedList", false);

              // Delete the last two characters from the cursor position,
              // which we know are "- " since that's what triggered the autobullet
              //
              // Modified from https://stackoverflow.com/a/43798749
              const edRange = editor.selection.getRng();
              const edNode = edRange.commonAncestorContainer;
              let range = document.createRange(); // create a new range object for the deletion
              range.selectNodeContents(edNode);
              range.setStart(edNode, edRange.endOffset - 2); // current caret pos - 3
              range.setEnd(edNode, edRange.endOffset); // current caret pos
              range.deleteContents();
            }
          }
        });

As you can see, the handler detects whether or not on keyUp the user has triggered the character and space. Then, we insert the unordered list (critically, we use the InsertUnorderedList command, as inserting a raw ul and li seems to break the list functionality) and remove the hyphen/space triggers.