I recently solved a similar issue by using an autoreleasepool

Try wrapping the area where sampleBuffer is used in an autoreleasepool. Something like this:

var outputSamples = [Float]()

assetReader.startReading()
while assetReader.status == .reading {
    let trackOutput = assetReader.outputs.first!

    autoreleasepool {
        if let sampleBuffer = trackOutput.copyNextSampleBuffer(),
            let blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer) {
            let blockBufferLength = CMBlockBufferGetDataLength(blockBuffer)
            let sampleLength = CMSampleBufferGetNumSamples(sampleBuffer) * channelCount(from: assetReader)
            var data = Data(capacity: blockBufferLength)
            data.withUnsafeMutableBytes { (blockSamples: UnsafeMutablePointer<Int16>) in
                CMBlockBufferCopyDataBytes(blockBuffer, atOffset: 0, dataLength: blockBufferLength, destination: blockSamples)
                CMSampleBufferInvalidate(sampleBuffer)

                let processedSamples = process(blockSamples,
                                               ofLength: sampleLength,
                                               from: assetReader,
                                               downsampledTo: targetSampleCount)
                outputSamples += processedSamples
            }
        }
    }
}
var paddedSamples = [Float](repeating: silenceDbThreshold, count: targetSampleCount)
paddedSamples.replaceSubrange(0..<min(targetSampleCount, outputSamples.count), with: outputSamples)

If I understand correctly, once it moves out of the scope of autoreleasepool, the sampleBuffer will be released

This is not really a solution, because it seems that releasing memory manually is impossible and using while loop in conjunction with assetReader results in memory not being released when unsafe mutable bytes are read.

The problem was solved by a workaround: converting the audio file into CAF format before exposing it to the while loop.

Downside: it takes a hot second, the longer the audio file - the more time it takes.

Upside: it only used minuscule amount of memory, which was the problem in the first place.

Inspired by: https://stackoverflow.com/users/2907715/carpsen90 answer in Extract meter levels from audio file