Can you try below code

SeleniumLibrary.Execute Javascript   alert('OK')

The alert should be handled with the Handle Alert keyword as you seem to have done already. For returning to earlier browser instance Selenium provides two entities which can be used. You may want to see the Browsers and Windows section of the SeleniumLibrary documentation.

Basic use case is that when opening browser, you'll give it an alias and then use Switch Browser to get back to that browser instance. Within that browser you may have multiple pages open, they can be accessed with normal locators by using Switch Window keyword. For example I would commonly use the window URL as that tends to have at least some unique part in it. If none of the URL, title, or name are unique, by using keyword Get Window Handles returns a list of windows within that browser. In your case, expecting no other windows are opened during the test, the main browser window can be found with argument MAIN.

You may consider this incomplete example and fill in your specifics which you did not provide in the question.

Test 1
    Open Browser    ${your URL}    alias=browser1
    # Do whatever triggers the alert
    Handle Alert
    Switch Window    MAIN