Effective data management is crucial for MongoDB-dependent applications. As databases grow, the need to transfer collections between databases arises. This can be for creating development or testing environments, migrating data, or archiving historical information. MongoDB provides built-in capabilities and tools to simplify this process, offering flexibility and ease of use.
In This article, We will explore the different ways to copy collections in MongoDB, providing insights into when and how to use each method. It discusses the need for copying databases or collections, highlighting scenarios such as creating test environments, data migration, backups, and security considerations.
What Is the Need for Copying a Database or Collection?
When working with MongoDB environments, there might be many reasons to copy a database or collection:
- Test Environments: Commonly, another version of a production database is created to assist in testing. This helps developers and testers to apply changes without any impact on the live data in the original. In this case, if they simply copy it entirely, they would have a fully replicated environment for executing tests on, writing queries, and ensuring that new features or bug fixes don’t interfere with real user data.
- Data Migration: One of the critical steps involved when transferring data from one database system to another involves copying the entire database or specific collections.
- Backups: The full backups of databases are important sometimes we may need to duplicate particular collections for backup purposes. For instance, historical data retention or creating backups for frequently altered collections can be critical reasons for copying individual ones instead.
- Security and Permissions: There may be some instances where duplicating a collection makes sense because we only want subsets of data which are accessible by certain users. It’s possible creating an extra customer collection that has certain fields that customer service representative can access only as an example.
Methods of Copying Collections in MongoDB
To copy collections in MongoDB is to make a replication of an existing collection, including the data and schema, into another database. There are several situations where this feature comes handy like creating development environment, migration of data or historical information archiving. MongoDB has two main ways of copying collections that have their own strengths and areas of application:
1. db.cloneCollection()
This function is a part of MongoDB and it performs better when it comes to copying collections within the same instance of MongoDB. It copies the source collection into a target database while duplicating everything including data and schema.
It is a built-in function that is recommended for copying same instance’s collections within the same MongoDB. It provides an effective and efficient process.
Here is the syntax:
use source_db // Switch to the source database containing the collection to copy
db.source_collection.cloneCollection("target_collection", target_db) // Clone to target collection in target_db
Explanation:
- use source_db: This command selects the active database to be one containing the collection you want to clone (source collection).
- db.source_collection.cloneCollection(): This starts the cloning.
- source_collection: Replace it with your original collection name.
- “target_collection”: Replace it with what you want to call your new collection in destination DB.
- target_db: An optional parameter. If not provided, then it will create copied collection within current DB (having used use source_db).
2. Using mongodump and mongorestore
This option offers much more freedom since you can move collections among different instances of MongoDB or servers even. The first thing that happens here is that a compressed archive of the source collection is created then it is restored into the target database with the desired collection name
This method provides more flexibility, allowing you to copy collections between different MongoDB instances or servers. It involves creating a backup of the source collection and then restoring it into the target database. Here’s is the syntax:
mongodump -d source_db -c source_collection -o dump
mongorestore -d target_db dump/source_collection --collection target_collection
Explanation:
- mongodump: This command creates a compressed archive of the specified collection.
- -d source_db: Replace this with the name of the database containing the source collection.
- -c source_collection: Replace this with the actual name of the collection you want to copy.
- -o dump: This specifies the output directory where the archive will be stored. You can choose a different directory name if desired.
- mongorestore: This command restores the previously created archive into the target database.
- -d target_db: Replace this with the name of the database where you want to create the copied collection.
- dump/source_collection: This points to the location of the archive created by mongodump.
- –collection target_collection: This defines the name for the copied collection in the target database.
We’ll explore the advantages and considerations for each method, along with alternative options using visual tools, in the following sections of this article.
Example
The collection called “orders” holds customer order information for a database whose name is “e–commerce”. The reason behind creating another copy of this collection is to perform analysis on it later in a different database named “analytics”. We will see both db.cloneCollection() (Same Instance) and mongodump & mongorestore (Different Servers).
Method 1: Using db.cloneCollection() (Same Instance)
If we are currently connected to our MongoDB instance via the mongo shell, then switch to the e-commerce database:
use e-commerceRun the db.cloneCollection() command that copies the orders collection into the analytics database as order_analysis:
db.orders.cloneCollection("order_analysis", analytics)Expected Output:
{ "name" : "orders", "newName" : "order_analysis" }This shows that the orders has been copied successfully as order_analysis into analytics.
Method 2: Mongodump and Mongorestore (Different Servers)
For example, on the source server (e-commerce database):
Then we have to create a dump of “orders” collection:
Bash
mongodump -d e-commerce -c orders -o dump
The command will generate a compressed archive called “dump” in the current directory containing the information from the “orders” collection.
On the target server (analytics database)
Once we are connected to destination server, navigate to the location where we would like to keep dump file if it is not same as source server.
To restore this dump and thus create order_history collections inside analytic database. use below code:
Bash
mongorestore -d analytics dump/orders --collection order_history
This command restores archive from directory named ‘dump’ and creates new collection named ‘order_history’ under analytical database.
Choosing The Right Method
In our case this could be done by simply using db.cloneCollection() when both e-commerce and analytic databases are located in one MongoDB instance. However, mongodump and mongorestore are essential when transferring collection data across network between two separate servers.
Conclusion
Overall, this article has explained the various methods available for copying collections in MongoDB, offering valuable insights into their application and suitability. It has explained the importance of copying databases or collections in scenarios such as creating test environments, data migration, backups, and ensuring security. By providing detailed explanations of two main methods such as db.cloneCollection() and mongodump/mongorestore about how to efficiently manage their MongoDB data.