Most of the set up was fine, just changed a few things to get it working:

func maskPreview()
{
    // No change from your code
    let path = UIBezierPath(arcCenter: CGPoint(x: view.frame.size.width/2,
                                               y: view.frame.size.height/2),
                            radius: view.frame.size.width/2.5,
                            startAngle: CGFloat(0).toRadians(),
                            endAngle: CGFloat(360).toRadians(),
                            clockwise: false)
    
    // Removed this, not sure it was needed
    // Add it back if needed
    //path.reversing()
    
    // No change
    let shapeLayer = CAShapeLayer()
    shapeLayer.path = path.cgPath
    
    // This should not be transparent since we need
    // the camera preview to be seen through here
    shapeLayer.fillColor = UIColor.white.cgColor
    
    // No change
    shapeLayer.strokeColor = UIColor.blue.cgColor
    shapeLayer.lineWidth = 5
    
    // No change
    previewView.layer.mask = shapeLayer
    
    // You do not need to add the shape layer as a sublayer
    // after defining it as your mask
    // previewView.layer.addSublayer(shapeLayer)
    
    // No change
    previewView.clipsToBounds = true
}

End result

Update to discuss about the mask layers color

The documentation for mask defines this behavior that the alpha channel determines what content, if any will be allowed to show.

mask

An optional view whose alpha channel is used to mask a view’s content.

Discussion

The view’s alpha channel determines how much of the view’s content and background shows through. Fully or partially opaque pixels allow the underlying content to show through but fully transparent pixels block that content.

The fill color is not the circle color as you rightly pointed out. You could change this and it would not matter as long as it is not clear color. It determines whether the mask renders the underlying view or not.

To actually add some color to this, I believe you cannot do it directly through the mask layer and you would need an intermediary layer.