How to Generate Sitemap in a MERN Application ?

Generate Sitemap in a MERN Application refers to creating a file that lists the pages, videos, and other files on your website, as well as their relationships. Search engines (such as Google, Duckduckgo, Bing, and others) use this file to help them crawl your site more efficiently.

Prerequisites:

• MERN Stack
• NPM & Node.js
• Sitemap

We can generate the sitemaps in the MERN application for these cases:

Steps to Generate Static Sitemap in MERN Application :

If the website you are working on has a fixed number of URLs, for example, your portfolio. You can generate a sitemap.xml file with the help of any online tool and place the folder in the src folder.

Step 1: Visit https://www.xml-sitemaps.com/ and generate the sitemap.xml file.

Step 2: Move the file into the public folder of your react app.

Project Structure:

Step 3: Verify the changes by visiting https://ahampriyanshu.com/sitemap.xml.

Step 4: Finally, Add the Sitemap to your robots.txt file

User-agent: *
Allow: /
Sitemap: https://baseurl/sitemap.xml

Steps to Generate Dynamic Sitemap :

So far we have discussed creating a sitemap with static URLs. But what if the number of URLs and the content of existing URLs change from time to time. Suppose we are creating a GFG clone. So, our sitemap should contain URLs of all the articles and the important pages. 
For this, we will send a sitemap file from our backend by first looping through all the required records from our database. After this, we will manually add the URLs of other important pages like about, contact, etc.

Step 1: For demonstration purposes, We are considering a basic project with three files.

Javascript

// Filename - App.js

const express = require("express"),
    mongoose = require("mongoose"),
    todoRouter = require("./routers/todoRouter"),
    app = express();
app.use("/", todoRouter);

const port = 3000,
    db = 'mongodb://localhost/todo';

mongoose
    .connect(db)
    .then(conn => {
        console.log(`${db} connected`);
    });

app.listen(port, () => console.log(
    `Server listening on ${port}`));

Javascript

// Filename - model.js

const mongoose = require("mongoose"),
    todoSchema = new mongoose.Schema(
        {
            title: { type: String, unique: true },
        },
        { timestamps: true }
    );

module.exports = mongoose.model("Todo", todoSchema);

Javascript

// Filename - todoRouter.js

const express = require("express"),
    router = express.Router();

/* Todo Controller functions */
module.exports = router;

Step 2: Install the’ sitemap’ package to stream the sitemap buffer and write its data.

npm i sitemap

Step 3: Create the sitemapRouter.js file in your router’s directory. At this point, your folder structure will look like this.

The updated dependencies in package.json file will look like.

"dependencies": {
        "express": "^4.18.2",
        "mongoose": "^8.0.0",
        "sitemap": "^7.1.1"
}

Example: This example implements sitemap with rewuired haders and including required URLs.

Javascript

// Filename - sitemapRouter.js

const express = require("express"),
    { SitemapStream, streamToPromise } = require('sitemap'),
    Todo = require("../models/todoModel"),
    date = new Date().toISOString(),
    zlib = require("zlib"),
    router = express.Router();

let sitemap;

router.get('/', async function (req, res) {
    res.header('Content-Type', 'application/xml');
    res.header('Content-Encoding', 'gzip');

    // If we have a cached entry send it
    if (sitemap) return res.send(sitemap)

    try {

        // Fetching todo records and mapping
        // it the desired URL pattern
        const data = await Todo.find(),
            todos = data.map(({ title }) => `/todo/${title}`),

            // Base url of our site
            smStream = new SitemapStream({
                hostname: 'https://demosite.com/' }),
            pipeline = smStream.pipe(zlib.createGzip());

        // Write todo URL to the stream
        todos.forEach(
            item => smStream.write({
                url: item, lastmod: date,
                changefreq: 'daily', priority: 0.7
            }));

        // Manually add all the other important URLs
        smStream.write({
            url: '/about', lastmod: date,
            changefreq: 'monthly', priority: 0.9
        })
        smStream.write({
            url: '/contact', lastmod: date,
            changefreq: 'monthly', priority: 0.9
        })

        // Cache the response
        streamToPromise(pipeline).then(sm => sitemap = sm);
        smStream.end()

        // Stream write the response
        pipeline.pipe(res).on('error', e => { throw e });
    } catch (err) {
        console.error(err)
        res.status(500).end()
    }
});

module.exports = router;

Output: Verify the changes in output by visiting {basename}/sitemap.xml.

Add the Sitemap to your robots.txt file.

User-agent: *
Allow: /
Sitemap: https://baseurl/sitemap.xml