In the following code, I set up margins so that it would not be able to pan outside of the svg boundaries as follows:

const map = SVG('#map')
        .size(500, 500)
        // @ts-ignore: next-line
        .panZoom({
          zoomMin: 1,
          margins: {
            top: 500,
            left: 500,
            bottom: 500,
            right: 500,
          },
        })
        .zoom(1);

I thought that I could adjust adjust the margins in the zoom callback by multiplying the values with the zoom level, but that does not work and causes the panning to stop working:

map.on('zoom', event => {
 const { level } = event.detail
 map.panZoom({
  margins: {
    top: 500 * level,
    left: 500 * level,
    bottom: 500 * level,
    right: 500 * level
   }
 })
})