Try this way:

.update({
            shares: (doc.data().shares - amount) >= 0 ? (doc.data().shares - amount) : 0
       })

Updated after discussion in the comments.

There is an important aspect to consider with your business logic: Do you need to execute an atomic operation on several documents? Example: You are subtracting the value of amount to the value of shares, since it is a selling operation but I guess that somewhere else (in another document) you are also adding some value, for example in the bank account of the seller.

In such a case you should use a Transaction: "if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction". You need to include in the Transaction all the documents that need to be locked while the operation is ongoing (i.e. all the docs that are involved in the operation, the ones on which you subtract and the ones on which you add).

However, since you want to update several documents, returned by a query, you cannot use a Transaction of one of the mobile/web SDKs (e.g. iOS, Android, Web/JavaScript), because the mobile/web SDKs use optimistic concurrency controls to resolve data contention.

What you can do is to use one of the Admin SDKs, like the Node.js one, since it uses pessimistic concurrency controls and therefore offers the possibility to run a Transaction on a query (see that you can pass a query to the get() method). So you could do that in a Callable Cloud Function.

Here is an example of a Transaction that will atomically update all the docs on which you substract. Since you didn't share the entire business logic (we don't know which are the docs that you need to update by adding a value) it's a bit difficult to go deeper in the example.

exports.updateTickers = functions.https.onCall((data, context) => {

    // get the value of the filter (i.e. props.name) via the data Object
    const filter = data.filter;
    const amount = data.amount;

    const db = admin.firestore();
    return db.runTransaction(transaction => {
        let queryRef = db.collection("Stocks").where("ticker", "==", filter);
        return transaction.get(queryRef)
            .then((querySnapshot) => {
                querySnapshot.forEach((doc) => {
                    const currentValue = doc.get('amount');
                    if (currentValue - amount > 0) (
                        transaction.update(doc.ref, { likes: currentValue - amount })
                    )
                });
            });
    })
        .then(() => {
            return { result: "Amounts update successful" }
        })

});

The only sure-fire way to prevent the shares count from going below 0 is to enforce that in security rules:

allow write: if request.resource.data.shares >= 0;

Any other method can be bypassed by a malicious user who uses your project configuration data with their code.

With this in place, the simplest (and fastest) way to subtract the amount from the shares count is with an atomic increment operation:

db.collection("myStocks")
  .doc(doc.id)
  .update({
      shares: firebase.firestore.FieldValue.increment(-1 * amount)
  })