setCartItems sets the value of cartItems for the next render, so on the initial render it's [] during the second useEffect

You can fix this by storing a ref (which doesn't rerender on state change) for whether it's the first render or not.

import React, { useState, useRef } from "react";

// ...

// in component

const initialRender = useRef(true);

useEffect(() => {
    if (JSON.parse(localStorage.getItem("cartItems"))) {
        const storedCartItems = JSON.parse(localStorage.getItem("cartItems"));
        setCartItems([...cartItems, ...storedCartItems]);
    }
}, []);

useEffect(() => {
    if (initialRender.current) {
        initialRender.current = false;
        return;
    }
    window.localStorage.setItem("cartItems", JSON.stringify(cartItems));
}, [cartItems]);

thanks to jamesmosier for his answer on gitHub

The issue you are seeing is because localStorage (aka window.localStorage) is not defined on the server side. Next server renders your components, so when that happens and it tried to access localStorage it can't find it. You'll have to wait until the browser renders it in order to use localStorage.

full answer link here

EDIT

you can try this :

useEffect(() => {
    if (typeof window !== "undefined") {
      const storedCartItems = JSON.parse(localStorage.getItem("cartItems"));
      if(storedCartItems !== null) {
        setCartItems([...cartItems, ...storedCartItems]);
      }
    }
  }, []);

  useEffect(() => {
    if (typeof window !== "undefined") {
     window.localStorage.setItem("cartItems", JSON.stringify(cartItems));
    } 
  }, [cartItems]);

React never updates state immediately It's an asynchronous process, for example, if you console.log(stateValue) just after the setState() method you'll get prevState value in a log. You can read more about it here.

That is exactly happening here, you have called setState method inside the first useEffect, state has not updated yet by react and we're trying to update localStorage with the latest state value(for now it's [] since react has not updated the state yet). that's why the localStorage value holds an empty array.

For your case, you can skip the first execution of 2nd useEffect as @Samathingamajig's mentioned in his answer.

PS: Thanks @Samathingamajig for pointing out the silly mistake, I don't know how missed that. LOL