The shaking is caused by your element shrinking and therefore your cursor not hovering anymore (and then re-growing as you aren't hovering, causing your cursor to change again)

Instead of changing the size of your marker, why not use a pseudo element, then your hand won't "shake" as it will always hover the marker (but the pseudo element will change size instead).

.marker {
  width: 20px;
  height: 20px;
  display: inline-flex;
  justify-content: center;
  align-items: center;
}

.marker:hover {
  cursor: pointer;
}

.marker:after {
  content: '';
  display: block;
  transition: all 0.5s ease;
  border: 2px solid #fff;
  background-color: #61ba9e;
  border-radius: 50%;
  width: 20px;
  height: 20px;
  box-sizing:border-box;
}

.marker:hover:after {
  height: 12.5px;
  width: 12.5px;
  box-shadow: 0 0 0 4px rgb(97 186 158 / 60%);
}

<div class="marker"></div>

It's shaking because your object is originally 20x20 pixels in size. Once it your mouse :hover's it shrinks in size.

Since the cursor is not perfectly centered on the object it reaches a point where the mouse is out of the bounding box of your element, which restores it's original width... which then makes it bigger and the mouse it's again inside the bounding box...

You see where this is going right? We have an infinite loop of shrinking and growing the element's bounding box.

A trick for getting around this may be to provide a wrapper that stays in size 20x20 and have the marker respond to the wrapper being :hover'ed