There's an extension provided for the same and you can use it as below:

@ExtendWith(OutputCaptureExtension.class)
public class MyTestClass {

    @Test
    void myTestCase(CapturedOutput capturedOutput) {
        assertTrue(capturedOutput.getOut().contains("expected string"));
        assertTrue(capturedOutput.getErr().contains("expected string"));
    }
}

We stumbled upon the same issue during our JUnit5 migration. After some research, I found a technical solution but it didn't seem like someone made a testing library out of it just yet. So that's what I did. It's published to Maven Central so you can use it right away:

https://github.com/netmikey/logunit

You can use it as follows:

public class MyModuleTest {

    @RegisterExtension
    LogCapturer logs = LogCapturer.create().captureForType(MyModule.class);

    @Test
    public void testLogging() {
        // ... do the testing ...

        // Run assertions on the logged messages
        logs.assertContains("Some message");
    }
}

(see the project's README for more examples)

The Migration Tip from the JUnit5 documentation clearly states that -

@Rule and @ClassRule no longer exist; superseded by @ExtendWith; see the following section for partial rule support.

For the purpose of using the existing @Rule support from JUnit 4, there is though a way suggested for method or class level annotations.

As in JUnit 4, Rule-annotated fields as well as methods are supported. By using these class-level extensions on a test class such Rule implementations in legacy codebases can be left unchanged including the JUnit 4 rule import statements.

This limited form of Rule support can be switched on by the class-level annotation org.junit.jupiter.migrationsupport.rules.EnableRuleMigrationSupport

A better option would still be redesigning your test suite to use the Extension Model from JUnit5 if you're using it.