A simple return statement won't stop the collector. It just makes the rest of the code inside that function not to run. But, if there is a new incoming message, the callback function gets executed again.

However, you can use the Collector#stop() method that stops the collector and emits the end event. You can also add a reason the collector is ending.

Check out the code below:

const filter = (m) => m.author.id === message.author.id;

const collector = message.channel.createMessageCollector({
  filter,
  max: 5,
  time: 1000 * 20,
});

collector.on('collect', (collected) => {
  if (collected.content.toLowerCase() === 'quit') {
    // collector stops and emits the end event
    collector.stop('user cancelled');
    // although the collector stopped this line is still executed
    return message.reply('Bye!');
  }

  // this line only runs if the above if statement is false
  message.reply(`You said _"${collected.content}"_`);
});

// listening for the end event
collector.on('end', (collected, reason) => {
  // reason is the one you passed above with the stop() method
  message.reply(`I'm no longer collecting messages. Reason: ${reason}`);
});