I have a candy-crush like game and use this four listeners to swap items

canvas.addEventListener('mousemove', onMouseMove);
canvas.addEventListener('mousedown', onMouseDown); // also tried pointerup and pointerdown
canvas.addEventListener('mouseup', onMouseUp);
canvas.addEventListener('mouseout', onMouseOut);

It works perfectly on desktop, but sometimes second click (mousedown) (on second tile is not being fired at all). I mean listener function is not being called. I have

cursor: pointer

in my CSS and I don't know what to do.

Note: If you click next time somewhere on canvas, event is being triggered as if it was clicked in the old spot. So if I what to change places of tiles x: 0, y: 0 and x: 0, y: 1:

1. I select first tile (x: 0, y: 0)
2. I try to select second tile (x:0, y:1). Event is not being triggered
3. If I click on any random tile those two will swap places somehow

Help please! Thanks!