To put it in simple terms IAM role is a document, definition of who (your app, AWS service etc) can use what (list of API calls) under which conditions (list of service specific conditions, optional). Roles are managed by IAM service.

STS is AWS service which is used for getting temporary credentials. If you want to assume role, you request these credentials via STS service. If your app has permissions to assume role, IAM service will grant you permissions (list of API calls) which this role allows and STS service will return you your credentials.

Now, "when to use which" is not an entirely correct question, given the explanation above. But I understand where you got the idea, and better question would be in my opinion: "When to use STS service to assume role and when don't".

Answer to that would be, as a rule of thumb, if your app is running on AWS (EC2, Elastic Beanstalk, Lambda etc) use roles directly, i.e. attach role to an instance. If your app uses AWS SDK for whatever language, it will pick it up seamlessly.

Exception to this would be if you want to perform some actions in completely different account. Then you need to use STS service directly to assume role in different account.

If your payload is running outside of AWS, use STS.