I'm trying to create a mosaic of multiple images, to resemble the pattern but be as small as possible (in total bytes). I thought that would be a good task for a neural network.

I imagined that based on the inputs (information about the brightness, average color of the image and the target image), the neural network will go through the whole image set (always 4 images from the mosaic that are closest to each other) and decide whether to exchange some of them.

So I have a cursor that moves gradually through the lines of the mosaic:

12---->  -12--->  --12-->  ...
34       |34      | 34
|        |        |
V        V        V

And in some cases, the neural network will tell me that it is necessary to switch, for example, 1 with 2 or so on. I can go through with this network for 10 full mosaic scans and then I can evaluate the resulting image (so I can't evaluate such a network right after the calculation, but only after several cycles of calculations) The result is two numbers - the first one I want to maximize (similarity) and the second to minimize (size ratio).

I tried to create a neural network that would have the required number of inputs and outputs. The function that compares the source image vs mosaic is given in advance (for me it is a black box and returns a value between 0-1, which is some degree of similarity). The size of the entire mosaic will be determined by creating the image (save into .png with full compress) and can be compared to size of all pieces (should always be smaller, so I can convert even this to value between 0-1). And now I have a problem how to train such a network.

I use tensorflow in node.js. Pseudo part of the code:

import * as tf from '@tensorflow/tfjs-node';

const m = tf.sequential();
m.add(tf.layers.dense({
  inputShape: [4 * 4],
  units: 17,
  activation: 'relu',
}));
...
m.add(tf.layers.dense({
  name: 'output',
  units: 6,
  activation: 'sigmoid',
  dtype: 'int32',
}));
...
m.compile({
  optimizer: 'sgd',
  loss: minimizeEntropy,
});
...
const md = prepareMosaicData();
...
for(move cursor in rows na columns) {
  const actions = m.predict(md);
  if (action[0] > 0.5) { switch(0, 1); }
  if (action[1] > 0.5) { switch(0, 2); }
  ...
}
...
savePng();
...
results = evaluatePng();
...

I don't know what I could do now. I don't know if it is possible to use GAN, because the discriminator is not a neural network, but only two functions and how I could do it. And how to write a training function that tries to maximize and minimize two "external" results.