nginx-proxy should be good enough, provided it supports a docker swarm attached network (issue 520, still pending)

You could also consider vfarcic/docker-flow-proxy, an easy way to reconfigure proxy every time a new service is deployed, or when a service is scaled.

See "Integrating Proxy With Docker Swarm (Tour Around Docker 1.12 Series)" for a concrete example.

I can think of 2 options for this:

First:

Run both apps and nginx in the same docker user-defined network. In order to do this you need to create a user-defined network first:

docker network create appnet

Then change your docker-files to use the appnet network as external (external networks wont create network and use existing)

Second:

Create 2 user-defined networks, named appnet1 and appnet2, attach a.com to appnet1 and b.com to appnet2 marked as external network in docker-compose. Then bring up the nginx and attach it to both appnet1 and appnet2.

When I docker run the nginx container I'm getting: host not found in upstream "MY_CONTAINER_HOST_NAME", which suggests my app hostname isn't resolving from within the nginx container.

Please note: User defined networks will provide internal DNS so you can call containers on the same network by container name.

External network example:

If you want your containers to join a pre-existing network, use the external option:

networks:
  default:
    external:
      name: my-pre-existing-network

Instead of attempting to create a network called [projectname]_default, Compose will look for a network called my-pre-existing-network and connect your app’s containers to it