I need to draw the line with several additional elements on top of it (like text and start-end points). The easiest way to do it is to use the Fabric.JS grouping that allows moving and other manipulations. There is a good rotating line approach - https://stackoverflow.com/a/53424718/8467261 allowing to build a line with two points set and limiting the selection to the area around that line if it's rotated (not a usual big rectangle around the diagonal line). But adding such a line to a group with several additional elements expands the selection area to fit all the elements in it.

Is there a way to create a group of several elements and limit the selection area to the selection are of only one of them? Or is there some other approach, like making the 'main' element and other moving with it?