Visual example

Consider a full recangle (initially the whole picture) and take away one excluded box. You will get 2x2x2x2=16 possible rectangular subdivisions, for example this one.

  ┌────────────────────────┐
  │                        │
  │                        │
  ├───────┬───────┬────────┤
  │       │  exc  │        │
  │       │ lude  │        │
  │       ├───────┴────────┤
  │       │                │
  │       │                │
  └───────┴────────────────┘

For each box in the subdivision, take away the next excluded box. Do this N times, and take the biggest box of the final step.

Assumption: you want the largest box from your array that complies with your rules, and it is not the largest NEW bounding box that complies.

This is pseudo code, you still have to fill in blanks

int largestBoxIndex = -1;
int largestBoxArea = -1;

for (i=0; i<allBoxes[].length; i++)
{
    box CurrentBox =  allBoxes[i];
    bool isComply = false;
    for (j=0; j<allBoxes[].length; j++)
    {
        isComply = false;

        if(i==j) break;

        ComparedBox = allBoxes[j]

        if (isIntersected(CurrentBox, ComparedBox)) break;

        if (isInside(CurrentBox, ComparedBox)) break;
        
        isComply = true;    
    }

    if(isComply)
        if(Area(allBoxes[i]) > largestBoxArea)
        {
            largestBoxArea = Area(allBoxes[i]):
            largestBoxIndex =i;
        }

}

if(largestBoxIndex != -1)
    largestBoxIndex;//this is the largest box

A simple mathematical solution to the problem

Suppose you are given 5 rectangles as shown below:

rects = [[100, 100, 200, 200],
         [200, 200, 200, 200],
         [200, 500, 200, 200],
         [350,  50, 150, 200],
         [500, 400, 200, 300]]

Note that the format of these rectangles is: [x, y, width, height] Where, (x, y) is the coordinate of the top left corner of the rectangle, and width & height are the width and height of the rectangle respectively. You will have to covert your coordinates in this format first.

3 out of these 5 are intersecting.

Now what we will do is iterate over these rectangles one by one, and for each rectangle, find the intersection of this rectangle with the other rectangles one by one. If any rectangle is found to be intersecting with any of the other rectangles, then we'll set the flag value for the two rectangles as 0. If a rectangle is found not to be intersecting with any other rectangle, then its flag value will be set to 1. (Default flag value is -1). Finally, we'll find the rectangle of the greatest area among the rectangles with flag value 1.

Let's see the code for finding the intersection area of the two rectangles:

# Rect : [x, y, w, h]
def Intersection(Rect1, Rect2):
    x = max(Rect1[0], Rect2[0])
    y = max(Rect1[1], Rect2[1])
    w = min(Rect1[0] + Rect1[2], Rect2[0] + Rect2[2]) - x
    h = min(Rect1[1] + Rect1[3], Rect2[1] + Rect2[3]) - y
    
    if w < 0 or h < 0: 
        return None
    return [x, y, w, h]

This function will return None if there is no intersecting area between these rectangles or it will return the coordinates of the intersection rectangle(Ignore this value for the current problem. This might be helpful in other problems).

Now, let's have a look at the algorithm.

n = len(rects)

# -1 : Not determined
#  0 : Intersects with some
#  1 : No intersection
flag = [-1]*n

for i in range(n):
    if flag[i] == 0:
        continue

    isIntersecting = False
    for j in range(n):
        if i == j or flag[j] == 1:
            continue
        
        Int_Rect = Intersection(rects[i], rects[j])
        if Int_Rect is not None:
            isIntersecting = True
            flag[j] = 0
            flag[i] = 0
            break
    
    if isIntersecting == False:
        flag[i] = 1

# Finding the maximum area rectangle without any intersection.
maxRect = None
maxArea = -1

for i in range(n):
    if flag[i] == 1:
        if rects[i][2] * rects[i][3] > maxArea:
            maxRect = rects[i]
            maxArea = rects[i][2] * rects[i][3]

print(maxRect)

Note: Add the "excluded areas" rectangle coordinates to the rects list and assign their flag value as 0 to avoid them from getting selected as the maximum area rectangle.

This solution does not involve any images so it will be the fastest algorithm unless it is optimized.

You can use the cv2.boundingRect() method to get the x, y, w, h of each bounding box, and with the x, y, w, h of each bounding box, you can use the condition x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1 to check if any two bounding boxes intersect or are within each others:

import cv2
import numpy as np

def intersect(b1, b2):
    x1, y1, w1, h1 = b1
    x2, y2, w2, h2 = b2
    return x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1

# Here I am generating a random array of 10 boxes in the format [[x,y], [x,y], [x,y], [x,y]]
np.random.seed(55)
boxes = np.random.randint(10, 150, (10, 4, 2)) + np.random.randint(0, 300, (10, 1, 2))

bounds = [cv2.boundingRect(box) for box in boxes]
valids = [b1 for b1 in bounds if not any(intersect(b1, b2) for b2 in bounds if b1 != b2)]
if valids:
    x, y, w, h = max(valids, key=lambda b: b[2] * b[3])
    print(f"x: {x} y: {y} w: {w} h: {h}")
else:
    print("All boxes intersect.")

Output:

x: 75 y: 251 w: 62 h: 115

For visualization:

import cv2
import numpy as np

def intersect(b1, b2):
    x1, y1, w1, h1 = b1
    x2, y2, w2, h2 = b2
    return x2 + w2 > x1 > x2 - w1 and y2 + h2 > y1 > y2 - h1

np.random.seed(55)
boxes = np.random.randint(10, 150, (10, 4, 2)) + np.random.randint(0, 300, (10, 1, 2))

bounds = [cv2.boundingRect(box) for box in boxes]
valids = [b1 for b1 in bounds if not any(intersect(b1, b2) for b2 in bounds if b1 != b2)]

img = np.zeros((500, 500), "uint8")
for x, y, w, h in bounds:
    cv2.rectangle(img, (x, y), (x + w, y + h), 255, 1)

if valids:
    x, y, w, h = max(valids, key=lambda b: b[2] * b[3])
    cv2.rectangle(img, (x, y), (x + w, y + h), 128, -1)

cv2.imshow("IMAGE", img)
cv2.waitKey(0)

Output:

Find the biggest square in numpy array

Maybe this would help? If you know the size of the whole area you can calculate the biggest box within numpy array. If you set all your given boxes to 1 and your whole area to 0 you need to find the largest area that is unique and not 1.

In other words:

1. rectangles that share points are excluded
2. of the remaining rectangles, take the largest

No need for contours, centroids, bounding boxes, masking or redrawing pixels!

As stated before, in the provided case, the rectangles coordinates contain duplicates. Here, we use a single class to store the outer limits of the rectangle. The Separating Axis theorem from this answer by @samgak is used in an intersects() method.

from __future__ import annotations # optional
from dataclasses import dataclass # optional ?

@dataclass
class Rectangle:
    left: int
    top: int
    right: int
    bottom: int
    def __repr__(self):
        """String representation of the rectangle's coordinates."""
        return f"⟔ {self.left},{self.top} ⟓ {self.right},{self.bottom}"
    def intersects(self, other: Rectangle):
        """Whether this Rectangle shares points with another Rectangle."""
        h = self.right < other.left or self.left > other.right
        v = self.bottom < other.top or self.top > other.bottom
        return not h or not v
    def size(self):
        """An indicator of the Rectangle's size, equal to half the perimeter."""
        return self.right - self.left + self.bottom - self.top

main = Rectangle(100, 100, 325, 325)

others = {
    0: Rectangle(100, 100, 400, 400),
    1: Rectangle(200, 200, 300, 300),
    2: Rectangle(200, 300, 300, 500),
    3: Rectangle(300, 300, 500, 500),
    4: Rectangle(500, 500, 600, 600),
    5: Rectangle(350, 350, 600, 600),
}

for i, r in others.items():
    print(i, main.intersects(r), r.size())

Simply put, h is True if the other rectangle is completely to the left or to the right; v is True if it's at the top or the bottom. The intersects() method returns True if the rectangles share points (even so much as a corner).

Output:

0 True 600
1 True 200
2 True 300
3 True 400
4 False 500
5 False 200

It is then trivial to find the largest:

valid = {r.size():i for i, r in others.items() if not main.intersects(r)}
print('Largest:', valid[max(valid)], 'with size', max(valid))

Output:

Largest: 4 with size 500

This answer assumes left < right and top < bottom for all rectangles.

The following function turns the provided rectangle coordinates to the kind used by the Rectangle class above. This assumes that the order is [[l, t], [r, t], [r, b], [l, b]] (a path).

def trim(coordinates):
    """Remove redundant coordinates in a path describing a rectangle."""
    return coordinates[0][0], coordinates[1][1], coordinates[2][0], coordinates[3][1]

Finally, we want to do this for all rectangles, not just a "main" one. We can simply have each rectangle be the main one in turns. Use itertools.combinations() on an iterable such as a list:

itertools.combinations(rectangles, 2)

This will ensure that we don't compare two rectangles more than one time.

Here's a O(n^2) solution. find_maxbox takes array of rectangles and convert them into Box objects and then compare each pair of boxes to eliminate invalid rectangles. This solution assumes that the rectangles' sides are parallel to X-Y axes.

class Box():
    def __init__(self, coordinates):
        self.coordinates = tuple(sorted(coordinates))
        self.original = coordinates
        self.height = abs(self.coordinates[0][1] - self.coordinates[3][1])
        self.width = abs(self.coordinates[0][0] - self.coordinates[3][0])
        self.excluded = False

    def __eq__(self, b2):
        return self.coordinates == b2.coordinates

    def get_area(self):
        return self.height * self.width

    def bounding_box(self, b2):
        maxX, maxY = map(max, zip(*self.coordinates, *b2.coordinates))
        minX, minY = map(min, zip(*self.coordinates, *b2.coordinates))
        return Box([(minX, minY), (maxX, minY), (minX, maxY), (maxX, maxY)])

    def intersects(self, b2):
        box = self.bounding_box(b2)
        if box.height < self.height + b2.height and box.width < self.width + b2.width:
            return True
        else: return False

    def encloses(self, b2):
        return self == self.bounding_box(b2)

    def exclude(self):
        self.excluded = True

    def is_excluded(self):
        return self.excluded

    def __str__(self):
        return str(self.original)

    def __repr__(self):
        return str(self.original)

# Pass array of rectangles as argument.
def find_maxbox(boxes):
    boxes = sorted(map(Box, boxes), key=Box.get_area, reverse=True)
    _boxes = []
    _boxes.append((boxes[0], boxes[0]))
    for b1 in boxes[1:]:
        b2, bb2 = _boxes[-1]
        bbox = b1.bounding_box(bb2)
        if not b1.intersects(bb2):
            _boxes.append((b1, bbox))
            continue
        for (b2, bb2) in reversed(_boxes):
            if not b1.intersects(bb2):
                break
            if b1.intersects(b2):
                if b2.encloses(b1):
                    b1.exclude()
                    break
                b1.exclude()
                b2.exclude()
        _boxes.append((b1, bbox))

    for box in boxes:
        if box.is_excluded():
            continue
        else: return box.original

    return None