I am using Next.js's Static HTML Export for my site which have 10 million static pages but I am running into ram issues when building the app.
Is it even possible to export it in parts like 100k pages on first build then 100k on second build and so on?
I do not want to use Incremental Static Regeneration or getServerSideProps to cut costs.
This site is using MongoDB only have two pages home page and posts page:
index.js
[postPage].js
In home page I used this code:
export async function getStaticProps() {
const { db } = await connectToDatabase();
const postsFeed = await db
.collection("myCollection")
.aggregate([{ $sample: { size: 100 } }])
.toArray();
return {
props: {
postsFeed: JSON.parse(JSON.stringify(postsFeed)),
},
};
}
In posts page I used this code:
export async function getStaticPaths() {
const { db } = await connectToDatabase();
const posts = await db
.collection("myCollection")
.find({})
.toArray();
const paths = posts.map((data) => {
return {
params: {
postPage: data.slug.toString(),
}
}
})
return {
paths,
fallback: 'blocking'
}
}
export async function getStaticProps(context) {
const postSlug = context.params.postPage;
const { db } = await connectToDatabase();
const posts = await db
.collection("myCollection")
.find({ slug: { $eq: postsSlug } })
.toArray();
const postsFeed = await db
.collection("myCollection")
.aggregate([{ $sample: { size: 100 } }])
.toArray();
return {
props: {
posts: JSON.parse(JSON.stringify(posts)),
postsFeed: JSON.parse(JSON.stringify(postsFeed)),
},
};
}
Doesn't seem to be a built-in option to process batches of static pages https://github.com/vercel/next.js/discussions/14929
I can only think of dividing the work using a bash script where you set an env variable and use it in the code where you're fetching the data to generate the paths, then run the build command as many times as parts you need to split the data, in each iteration move the generated files to another directory that will be your consolidated output.
COUNTER=1
PARTS=100 # change it to control number of parts
while [ $COUNTER -lt $PARTS ]; do
let COUNTER=COUNTER+1
CURRENT=$COUNTER PARTS=$PARTS next build
# move generated files to another directory
done
in your get getStaticPaths
export async function getStaticPaths() {
const currentPercentage = process.env.CURRENT/process.env.PARTS
// logic to fetch the corresponding current percentage of the data
// 1% when there are 100 parts, 0.5% when 200 parts, etc.
}
Be aware that if the data changes very often, you'll see incorrect results, like repeated pages or skipped ones, since each pagination will occur at different moments when running the script. I believe you could create an auxiliary node (or another language) script to better handle that quantity of records, maybe in a streamlined way, and generate JSON files for each chunk of data to use them in getStaticPaths instead of fetching them directly from the DB.