you can play with popstate event browser fired when browser history change

window.addEventListener('popstate', function(event) {
  alert('you click on back or forward button');
}

The popstate event is fired when the current history changes :

• when buttons back/forward are click by user
• when methods history.back(), history.forward(), history.go() are called.

Try looking through these answers here.

So, in short, and for people that aren't necessarily using an in-page back button or an array to store the history:

document.onmouseover = function() {
    //User's mouse is inside the page.
    window.innerDocClick = true;
}

document.onmouseleave = function() {
    //User's mouse has left the page.
    window.innerDocClick = false;
}

window.onhashchange = function() {
    if (window.innerDocClick) {
        //Your own in-page mechanism triggered the hash change
    } else {
        //Browser back button was clicked
    }
}

And there you have it. a simple, three-part way to detect back button usage vs in-page elements with regards to hash navigation.

Your alert would go into the else section of the window.onhashchange handler.