For a few million rows, there is essentially no chance that MD5(URL) will have an accidental collision. That's a 32 hex digits (CHAR(32) CHARACTER SET ascii). Or better would be UNHEX(...) and put it in BINARY(16).

Then add a UNIQUE index on that column.

What version are you using? Some newer version of MariaDB has something like that builtin.

The are two ways to add index on long string:

1. add index for crc32(urls),just add a column named crc32_urls and create a index on this.It is possible some urls have same result by crc32(urls),so every time you search you should run like this:

SELECT * FROM table WHERE crc32_urls = xxx AND urls = xxx

2. use prefix index.For assuming how much prefix's length should be.You can caculate by

SELECT COUNT(DISTINCT urls)/COUNT(*) FROM table

compared with

SELECT COUNT(DISTINCT left(urls, x))/COUNT(*) FROM table

the x is length you specified.When two number is close,choose min length.