As long as a single sample can fit into GPU memory, you do not have to reduce the effective batch size: you can do gradient accumulation. Instead of updating the weights after every iteration (based on gradients computed from a too-small mini-batch) you can accumulate the gradients for several mini-batches and only when seeing enough examples, only then updating the weights.
This is nicely explained in this video.

Effectively, your training code would look something like this. Suppose your large batch size is large_batch, but can only fit small_batch into GPU memory, such that large_batch = small_batch * k. Then you want to update the weights every k iterations:

train_data = DataLoader(train_set, batch_size=small_batch, ...)

opt.zero_grad()  # this signifies the start of a large_batch
for i, (x, y) in train_data:
  pred = model(x)
  loss = criterion(pred, y)
  loss.backward()  # gradeints computed for small_batch
  if (i+1) % k == 0 or (i+1) == len(train_data):
    opt.step()  # update the weights only after accumulating k small batches
    opt.zero_grad()  # reset gradients for accumulation for the next large_batch

Shai's answer is suitable, but I want to offer another solution. Recently, I've been observing awesome results from Nvidia AMP - Automatic Mixed Precision, which is a nice combination of the advantages of fp16 vs fp32. A positive side effect is that it significantly speeds up training as well.

It's only a single line of code in tensorflow: opt = tf.train.experimental.enable_mixed_precision_graph_rewrite(opt)

More details here

You can also stack AMP with Shai's solution.