Getting Started
Connect to MongoDB
mongosh is the modern MongoDB Shell (replacing the legacy mongo shell). A connection string follows the URI format mongodb://[user:pass@]host[:port]/[db][?options]. The /admin database is the default auth database. You can connect to a specific database directly by appending it to the URI.
# connect to a local server on default port
mongosh
# connect to a specific host and port
mongosh "mongodb://localhost:27017"
# connect with authentication
mongosh "mongodb://user:pass@localhost:27017/admin"
# connect to a specific database directly
mongosh "mongodb://localhost:27017/mydb"
# show connection info
db.getMongo().getDBNames()Database Basics
MongoDB creates databases and collections lazily — they only appear after you insert the first document. 'use mydb' switches context but does not create the database until data is written. show dbs does not list empty databases. dropDatabase permanently deletes all collections and their data.
# show all databases
show dbs
# switch to (or create) a database
use mydb
# show the current database
db
# create is implicit: inserting into a
# non-existent collection creates both the
# collection and the database
# drop the current database
db.dropDatabase()Collections
Collections are analogous to SQL tables and hold documents. They are created automatically on first insert, but createCollection lets you set options like capped (fixed-size collections that overwrite old documents — useful for logs). stats() shows storage size, document count, and index details.
# list collections in current database
show collections
db.getCollectionNames()
# create a collection explicitly with options
db.createCollection("users", {
capped: true,
size: 5242880,
max: 5000
})
# rename a collection
db.users.renameCollection("accounts")
# drop a collection
db.users.drop()
# get collection stats
db.users.stats()BSON Data Types
BSON extends JSON with additional types essential for applications: ObjectId (12-byte unique IDs), ISODate, NumberDecimal (exact decimal for money — avoids float rounding), NumberLong (64-bit ints), BinData (binary), and more. Using NumberDecimal for currency avoids classic floating-point errors.
# MongoDB stores BSON (binary JSON) with rich types
{
_id: ObjectId("65a1b2c3d4e5f6a7b8c9d0e1"),
name: "Alice", // String
age: 30, // Int32
score: NumberLong(9007199254740992), // Int64
price: NumberDecimal("19.99"), // Decimal128 (exact money)
active: true, // Boolean
birthday: ISODate("1995-08-15"), // Date
tags: ["red", "blue"], // Array
meta: { views: 100 }, // Embedded document
data: BinData(0, "aGVsbG8="), // Binary
id: UUID("...") // UUID
}Shell Helpers & Help
mongosh supports full JavaScript, so you can write loops, functions, and variables in the shell. .pretty() formats output for readability (default in mongosh). load() runs a .js file in the shell context — useful for repetitive admin scripts. db.collection.help() lists all available methods.
# get help on database methods
db.help()
# get help on collection methods
db.users.help()
# list common shell commands
help
# pretty-print query results
db.users.find().pretty()
# print JSON in a compact form
db.users.find().toArray()
# execute a JavaScript file
mongosh --file script.js
load("script.js")CRUD Operations
Insert Documents
MongoDB auto-generates an _id (ObjectId) if you don't provide one. The _id must be unique within a collection — duplicate _id raises a write error. ordered:false lets insertMany continue inserting remaining documents after an error, improving throughput for bulk loads.
# insert a single document
db.users.insertOne({
name: "Alice",
email: "[email protected]",
age: 30
})
# insert multiple documents
db.users.insertMany([
{ name: "Bob", age: 25 },
{ name: "Carol", age: 28 }
])
# insert with a custom _id
db.users.insertOne({ _id: 1, name: "Dave" })
# ordered insert (stops on error) vs unordered
db.users.insertMany(docs, { ordered: false })Find / Query Documents
find() returns a cursor (lazily evaluated); findOne() returns a single document or null. Projection controls which fields are returned — you cannot mix inclusion and exclusion (except _id). Use 1 to include, 0 to exclude. countDocuments is accurate; the older count() is deprecated for filtered counts.
# find all documents
db.users.find()
# find with a filter
db.users.find({ age: 30 })
db.users.find({ "address.city": "NYC" })
# find one matching document
db.users.findOne({ name: "Alice" })
# projection: include/exclude fields
db.users.find({}, { name: 1, age: 1, _id: 0 })
db.users.find({}, { email: 0 })
# count matching documents
db.users.countDocuments({ age: { $gte: 18 } })Update Documents
updateOne/updateMany use update operators ($set, $inc, etc.) to modify specific fields. replaceOne replaces the whole document except _id. upsert:true creates a new document when no match exists — useful for 'create or update' patterns. Always use $set to update fields; a bare object replaces the document (legacy behavior).
# update one document
db.users.updateOne(
{ name: "Alice" },
{ $set: { age: 31, status: "active" } }
)
# update many documents
db.users.updateMany(
{ status: "pending" },
{ $set: { status: "active" } }
)
# replace an entire document (keeps _id)
db.users.replaceOne(
{ name: "Alice" },
{ name: "Alice", age: 32, city: "NYC" }
)
# upsert: insert if no match found
db.users.updateOne(
{ name: "Eve" },
{ $set: { age: 22 } },
{ upsert: true }
)Update Operators
Update operators modify fields atomically at the document level. $inc is concurrency-safe (no read-modify-write race). $min/$max only update if the new value is smaller/larger. $currentDate is handy for 'last modified' timestamps. Multiple operators can be combined in one update.
# $set and $unset
db.users.updateOne({n:"A"}, { $set: {age:30}, $unset: {temp:""} })
# $inc increments a numeric field
db.users.updateOne({n:"A"}, { $inc: {views: 1} })
db.users.updateOne({n:"A"}, { $inc: {balance: -50} })
# $rename a field
db.users.updateOne({n:"A"}, { $rename: {n: "name"} })
# $min / $max keep the smaller/larger value
db.users.updateOne({n:"A"}, { $min: {lowScore: 80} })
# $mul multiply the field value
db.users.updateOne({n:"A"}, { $mul: {price: 1.1} })
# $currentDate sets to current date
db.users.updateOne({n:"A"}, { $currentDate: {updatedAt: true} })Delete Documents
deleteOne removes the first match; deleteMany removes all matches. deleteMany({}) empties the collection but keeps indexes (faster than drop+recreate if you want to preserve index definitions). findOneAndDelete returns the deleted document atomically. Deletions are irreversible — always filter carefully.
# delete one matching document
db.users.deleteOne({ name: "Alice" })
# delete all matching documents
db.users.deleteMany({ status: "inactive" })
# delete all documents in a collection
db.users.deleteMany({})
# findAndDelete: return the deleted document
db.users.findOneAndDelete({ name: "Bob" })
# remove is deprecated, prefer deleteOne/deleteMany
db.users.remove({ name: "Bob" }, { justOne: true })Array Update Operators
$push appends (allows duplicates); $addToSet deduplicates. $pull removes elements matching a condition. The positional operator ($) refers to the first array element matched by the query — essential for updating specific array elements without knowing their index. $[] updates all array elements.
# $push append to an array
db.users.updateOne({n:"A"}, { $push: {tags: "new"} })
# $push with $each for multiple values
db.users.updateOne({n:"A"}, { $push: {tags: { $each: ["x","y"] } } })
# $addToSet adds only if not already present
db.users.updateOne({n:"A"}, { $addToSet: {tags: "x"} })
# $pull remove all matching array elements
db.users.updateOne({n:"A"}, { $pull: {tags: "x"} })
# $pop remove first (-1) or last (1) element
db.users.updateOne({n:"A"}, { $pop: {tags: 1} })
# update an array element by position
db.users.updateOne({}, { $set: {"tags.0": "first"} })
db.users.updateOne({"tags":"old"}, { $set: {"tags.$":"new"} })Querying Documents
Comparison Operators
Comparison operators are the foundation of queries. $in is far more efficient than multiple $or conditions on the same field. You can combine multiple operators on one field (e.g. $gte and $lte for a range). $exists:true finds documents with the field, $exists:false finds those without it.
# $eq equal (same as direct value)
db.products.find({ price: { $eq: 100 } })
# $ne not equal
db.products.find({ price: { $ne: 100 } })
# $gt, $gte, $lt, $lte
db.products.find({ price: { $gte: 50, $lte: 200 } })
# $in matches any value in an array
db.products.find({ category: { $in: ["books", "music"] } })
# $nin not in array
db.products.find({ category: { $nin: ["books"] } })
# $exists check field presence
db.products.find({ discount: { $exists: true } })Logical Operators
Implicit AND (comma in a filter) is the most common and efficient. Use explicit $and only when you need multiple conditions on the same field (implicit AND on the same field overrides earlier conditions). $or uses separate index scans and merges results — ensure fields in $or are indexed for performance.
# $and (explicit, useful for same-field conditions)
db.users.find({ $and: [
{ age: { $gte: 18 } },
{ age: { $lte: 65 } }
]})
# $or matches if ANY condition is true
db.users.find({ $or: [
{ status: "active" },
{ vip: true }
]})
# $nor matches if NO condition is true
db.users.find({ $nor: [{ status: "active" }]})
# $not inverts a condition
db.users.find({ age: { $not: { $gt: 18 } } })
# implicit AND (comma-separated)
db.users.find({ age: { $gte: 18 }, status: "active" })Element & Evaluation Operators
$type filters by BSON type (useful for mixed-type fields). $regex supports options like 'i' (case-insensitive) — but regex without a prefix anchor cannot use indexes efficiently. $expr enables comparisons between document fields, which normal operators can't do. Use $expr sparingly as it can bypass indexes.
# $type filter by BSON data type
db.users.find({ age: { $type: "int" } })
db.users.find({ age: { $type: ["int", "double"] } })
# $regex regular expression match
db.users.find({ name: { $regex: "^Al", $options: "i" } })
# $mod modulo operation
db.users.find({ age: { $mod: [2, 0] } }) # even ages
# $expr compare fields within a document
db.orders.find({ $expr: { $gt: ["$total", "$budget"] } })
# $jsonSchema validate against a schema
db.users.find({ $jsonSchema: {
required: ["name", "email"]
}})Array Query Operators
$all finds arrays containing all specified values (order-independent). $elemMatch is essential when an array element must satisfy multiple conditions simultaneously (a plain query would match elements individually). $size matches exact lengths only — there's no $size with a range; pre-compute and store the length as a separate field instead.
# $all: array contains ALL specified values
db.posts.find({ tags: { $all: ["mongo", "db"] } })
# $elemMatch: at least one element matches all conditions
db.students.find({ scores: { $elemMatch: {
subject: "math", score: { $gte: 90 }
}}})
# $size: array of exact length
db.users.find({ tags: { $size: 3 } })
# match if array contains the value
db.posts.find({ tags: "mongo" })
# match a specific array index
db.users.find({ "scores.0": 95 }) # first score is 95Cursor Methods & Sorting
skip+limit implements pagination but skip becomes slow for large offsets (it scans all skipped docs). For deep pagination, use keyset pagination: query with a filter on the sort field greater than the last seen value. sort() can use an index to avoid in-memory sorting — always index your sort fields for large collections.
# limit and skip (pagination)
db.users.find().limit(10)
db.users.find().skip(20).limit(10) # page 3
# sort: 1 ascending, -1 descending
db.users.find().sort({ age: 1, name: -1 })
# count results
db.users.find({active:true}).count()
# iterate the cursor in the shell
db.users.find().forEach(doc => print(doc.name))
# get a specific field's distinct values
db.users.distinct("city")Indexing
Create Indexes
Indexes dramatically speed up queries but slow down writes and consume disk. Single-field indexes support queries on that field in either sort direction. Compound indexes follow the ESR (Equality, Sort, Range) rule for optimal ordering. TTL indexes automatically delete documents after a duration — perfect for sessions and logs.
# create a single-field index
db.users.createIndex({ email: 1 }) # ascending
db.users.createIndex({ age: -1 }) # descending
# create a unique index
db.users.createIndex({ email: 1 }, { unique: true })
# create a compound index
db.users.createIndex({ lastName: 1, firstName: 1 })
# create a text index
db.posts.createIndex({ title: "text", body: "text" })
# create a TTL (time-to-live) index
db.sessions.createIndex({ createdAt: 1 },
{ expireAfterSeconds: 3600 })
# name an index explicitly
db.users.createIndex({ email: 1 }, { name: "email_idx" })View & Manage Indexes
getIndexes lists all indexes with their keys and options. hideIndex/unhideIndex lets you test the impact of removing an index without actually dropping it — the query planner ignores hidden indexes. If performance stays the same, it's safe to drop. Always run explain() before and after index changes.
# list all indexes on a collection
db.users.getIndexes()
# total index size in bytes
db.users.totalIndexSize()
# drop an index by name
db.users.dropIndex("email_1")
# drop all indexes except _id
db.users.dropIndexes()
# hide an index (test impact before dropping)
db.users.hideIndex("email_1")
db.users.unhideIndex("email_1")
# check if queries use indexes
db.users.find({email:"[email protected]"}).explain("executionStats")Special Index Types
2dsphere indexes power geo queries ($near, $geoWithin). Hashed indexes support hash-based sharding for even data distribution. Wildcard indexes ($**) cover unpredictable/variable field names in documents — useful for polymorphic data. Partial indexes only index matching documents, saving space when you only query a subset.
# geospatial 2dsphere index (GeoJSON points)
db.places.createIndex({ location: "2dsphere" })
db.places.find({
location: { $near: {
$geometry: { type: "Point", coordinates: [-73.99, 40.73] },
$maxDistance: 1000
}}
})
# hashed index (for sharding)
db.users.createIndex({ user_id: "hashed" })
# wildcard index (index arbitrary field names)
db.products.createIndex({ "$**": 1 })
# partial index (only matching documents)
db.users.createIndex({ email: 1 },
{ partialFilterExpression: { active: true } })Explain & Query Plans
explain() is the primary performance tool. COLLSCAN (collection scan) means no index was used — a red flag for large collections. IXSCAN means an index was used. totalDocsExamined >> nReturned indicates a poor index (examining many docs to return few). hint() forces a specific index for testing the planner's choice.
# explain a query (basic)
db.users.find({age:30}).explain()
# explain with execution stats (timing)
db.users.find({age:30}).explain("executionStats")
# key fields in the output:
# winningPlan.stage -> COLLSCAN (bad) or IXSCAN (good)
# totalDocsExamined -> should be close to nReturned
# executionTimeMillis -> query duration
# indexUsed -> which index was chosen
# force a specific index (testing)
db.users.find({age:30}).hint("age_1")
db.users.find({age:30}).hint({age:1})Indexing Best Practices
The ESR (Equality, Sort, Range) rule is the single most important compound-index design principle: put equality-filtered fields first, then sort fields, then range fields. A covered query (all fields are in the index) never fetches the document — the fastest possible query. Avoid indexes you don't use; each one adds write overhead.
# ESR rule: Equality, Sort, Range for compound indexes
# equality fields first, then sort, then range
db.orders.createIndex({
status: 1, # Equality (filter)
date: 1, # Sort (order)
amount: 1 # Range (comparison)
})
# avoid over-indexing: each index slows writes
db.orders.getIndexes().length
# covered query: index satisfies query without fetching doc
db.users.createIndex({ name: 1, email: 1 })
db.users.find({ name: "A" }, { _id: 0, name: 1, email: 1 })
# background index build (deprecated in 4.2+, builds don't block)
db.users.createIndex({ name: 1 }) # non-blocking by defaultAggregation Framework
Aggregation Pipeline Basics
The aggregation pipeline processes documents through stages in sequence. $match early in the pipeline reduces the working set (and can use indexes). $group is like SQL GROUP BY. $count is a shortcut for $group + $project. Order matters: filter first, then group, then sort, then limit.
# a pipeline is an array of stages
db.orders.aggregate([
{ $match: { status: "shipped" } },
{ $group: { _id: "$customerId", total: { $sum: "$amount" } } },
{ $sort: { total: -1 } },
{ $limit: 10 }
])
# each stage transforms documents and passes them
# to the next stage
# count documents
db.orders.aggregate([
{ $match: { status: "shipped" } },
{ $count: "shippedCount" }
])$match, $project & $limit
Always put $match as early as possible — it reduces documents for all subsequent stages and can use indexes. $project reshapes documents and computes new fields using expressions. Renaming or computing fields in $project is common before $group. $unset is shorthand for excluding fields.
# $match filters documents (like find)
{ $match: { status: "active", age: { $gte: 18 } } }
# $project reshapes documents (1 include, 0 exclude)
{ $project: { name: 1, _id: 0, "address.city": 1 } }
# $project with computed fields
{ $project: {
fullName: { $concat: ["$first", " ", "$last"] },
ageNextYear: { $add: ["$age", 1] }
}}
# $limit and $skip
{ $limit: 10 }
{ $skip: 20 }
# $unset removes fields (inverse of $project)
{ $unset: "tempField" }$group & Accumulators
$group is the core aggregation stage. _id specifies the grouping key (use null to aggregate all documents). $push builds an array of values per group; $addToSet deduplicates. $first/$last refer to document order in the group — use $sort before $group if order matters. Memory limit is 100MB per stage (use allowDiskUse for large datasets).
# $group: group by a field and aggregate
{ $group: {
_id: "$category",
count: { $sum: 1 },
totalRevenue: { $sum: "$amount" },
avgPrice: { $avg: "$price" },
minPrice: { $min: "$price" },
maxPrice: { $max: "$price" },
products: { $push: "$name" }, # array of values
firstOrder: { $first: "$date" },
lastOrder: { $last: "$date" }
}}
# group by multiple fields
{ $group: { _id: { category: "$cat", status: "$status" } } }
# group ALL documents together (_id: null)
{ $group: { _id: null, total: { $sum: "$amount" } } }$lookup (Joins)
$lookup performs a left outer join — like SQL LEFT JOIN. The joined data is placed in an array field named by 'as'. The pipeline form lets you filter and transform joined documents. $unwind deconstructs array fields (one output document per array element) — often used after $lookup to flatten results. Joins are less efficient than embedded data in MongoDB.
# $lookup: join another collection (left outer join)
{ $lookup: {
from: "orders",
localField: "_id",
foreignField: "customerId",
as: "orders"
}}
# with pipeline (filtered/transformed join)
{ $lookup: {
from: "orders",
let: { custId: "$_id" },
pipeline: [
{ $match: { $expr: { $eq: ["$customerId", "$$custId"] } } },
{ $project: { amount: 1, _id: 0 } }
],
as: "recentOrders"
}}
# $unwind: flatten an array field into multiple docs
{ $unwind: "$orders" }$unwind, $sort & $facet
$facet runs multiple sub-pipelines on the same input — perfect for dashboards needing several aggregations (top N, counts, breakdowns) in one query. $bucket auto-categorizes values into ranges. $unwind with preserveNullAndEmptyArrays keeps documents whose array is empty or missing — useful to avoid data loss in pipelines.
# $unwind with preserveNullAndEmptyArrays
{ $unwind: {
path: "$tags",
preserveNullAndEmptyArrays: true
}}
# $sort within pipeline
{ $sort: { total: -1 } }
# $facet: run multiple pipelines in parallel
{ $facet: {
"topCustomers": [{ $sort: {total:-1} }, { $limit: 5 }],
"byCategory": [{ $group: { _id: "$cat" } }],
"totalCount": [{ $count: "n" }]
}}
# $bucket: group into ranges automatically
{ $bucket: {
groupBy: "$price",
boundaries: [0, 50, 100, 500],
default: "other",
output: { count: { $sum: 1 } }
}}Text Search & Geospatial
Full-Text Search
A text index enables full-text search via $text. Words are tokenized and stemmed. The search is an OR by default; phrases need escaped quotes; a minus prefix excludes. The $meta:'textScore' returns relevance; sort by it for best matches first. Only one text index per collection. For advanced search, use Atlas Search (Lucene-based).
# create a text index on multiple fields
db.posts.createIndex({
title: "text",
body: "text",
tags: "text"
})
# search for words (OR by default)
db.posts.find({ $text: { $search: "mongodb tutorial" } })
# exact phrase (escape quotes)
db.posts.find({ $text: { $search: "\"mongo db\"" } })
# exclude a word (minus)
db.posts.find({ $text: { $search: "mongodb -sql" } })
# sort by relevance score
db.posts.find(
{ $text: { $search: "mongodb" } },
{ score: { $meta: "textScore" } }
).sort({ score: { $meta: "textScore" } })Geospatial 2dsphere
GeoJSON coordinates are [longitude, latitude] (x, y order — a common gotcha). A 2dsphere index supports $near (sorted by distance), $geoWithin (inside a shape), and $geoIntersects. $maxDistance is in meters. Distance calculations use a sphere model. For 2D planar data (rare), use a '2d' index instead.
# store points as GeoJSON
db.places.insertOne({
name: "Park",
location: {
type: "Point",
coordinates: [-73.99, 40.73] # [lng, lat]
}
})
# create a 2dsphere index
db.places.createIndex({ location: "2dsphere" })
# find nearby places within a distance (meters)
db.places.find({
location: { $near: {
$geometry: { type: "Point", coordinates: [-73.99, 40.73] },
$maxDistance: 1000
}}
})
# find places within a polygon
db.places.find({
location: { $geoWithin: {
$geometry: { type: "Polygon", coordinates: [[[-73,40],[-74,40],[-74,41],[-73,41],[-73,40]]] }
}}
})Geospatial Distance & Operators
$nearSphere uses accurate spherical geometry (better for large distances). The $geoNear aggregation stage is the only way to return the computed distance in results — distanceField stores it in meters (spherical) or radians. $centerSphere uses radians for radius (radians = distance_km / 6371).
# $nearSphere: accurate spherical distance
db.places.find({
location: { $nearSphere: {
$geometry: { type: "Point", coordinates: [-73.99, 40.73] },
$minDistance: 100,
$maxDistance: 5000
}}
})
# $geoNear aggregation stage (includes distance)
db.places.aggregate([{
$geoNear: {
near: { type: "Point", coordinates: [-73.99, 40.73] },
distanceField: "dist",
maxDistance: 2000,
spherical: true
}
}])
# $center / $centerSphere for simple radius
db.places.find({
location: { $geoWithin: { $centerSphere: [[-73.99,40.73], 0.01] } }
})Regex & Pattern Matching
Prefix-anchored regexes (/^Ali/) can use an index efficiently; unanchored patterns (/ali/) cannot and scan the whole collection. For case-insensitive search on large collections, prefer a text index or Atlas Search over regex. $regexMatch (aggregation) lets you use regex results in computed fields and conditions.
# case-insensitive regex
db.users.find({ name: { $regex: /alice/i } })
# prefix-anchored regex CAN use an index
db.users.find({ name: { $regex: /^Ali/ } })
# wildcard match
db.users.find({ email: { $regex: /@example\.com$/ } })
# $regex with options string
db.users.find({ name: { $regex: "ali", $options: "ix" } })
# use $expr with $regexMatch for conditional logic
db.users.aggregate([{
$project: {
name: 1,
isAlice: { $regexMatch: { input: "$name", regex: /^ali/i } }
}
}])Wildcard & Flexible Queries
$where runs JavaScript per document — extremely slow and should be avoided; $expr usually replaces it. $elemMatch ensures one array element matches ALL conditions (a plain comma query would match different elements). $expr enables field-to-field comparisons impossible with normal operators — but it may not use indexes efficiently.
# $jsonSchema validation as a query filter
db.users.find({ $jsonSchema: {
bsonType: "object",
required: ["name"],
properties: { age: { bsonType: "int", minimum: 0 } }
}})
# $where (JavaScript) — slow, avoids indexes
db.users.find({ $where: "this.a + this.b > 10" })
# $mod modulo operator
db.users.find({ age: { $mod: [10, 0] } })
# $expr: compare two fields
db.orders.find({ $expr: { $lt: ["$paid", "$total"] } })
# $elemMatch with multiple conditions on array elements
db.scores.find({ results: {
$elemMatch: { $gte: 80, $lt: 90 }
}})Transactions
ACID Transactions
Multi-document transactions provide ACID guarantees across collections (requires a replica set or sharded cluster). Pass the session to every operation in the transaction. commitTransaction makes changes permanent and visible; abortTransaction rolls back all changes. Keep transactions short — they hold locks and can affect performance.
# multi-document transaction (replica set required)
const session = db.getMongo().startSession()
session.startTransaction()
try {
db.accounts.updateOne(
{ name: "Alice" }, { $inc: { balance: -100 } }, { session }
)
db.accounts.updateOne(
{ name: "Bob" }, { $inc: { balance: 100 } }, { session }
)
session.commitTransaction()
} catch (e) {
session.abortTransaction()
print("Transaction aborted: " + e)
} finally {
session.endSession()
}Transaction Read/Write Concern
Read concern controls how 'fresh' and consistent reads are. 'snapshot' gives a transactionally-consistent view across shards — ideal for reporting. Write concern controls durability: w:majority ensures data is replicated before ack (survives a primary failover); j:true waits for the journal flush. Higher concerns are safer but slower.
# set transaction-level read/write concern
session.startTransaction({
readConcern: { level: "snapshot" },
writeConcern: { w: "majority", j: true, wtimeout: 5000 }
})
# read concern levels:
# "local" - sees latest local data (default)
# "majority" - committed, replicated to majority
# "snapshot" - consistent snapshot across shards
# write concern levels:
# { w: 1 } - ack from primary (default)
# { w: "majority" } - ack from majority of members
# { j: true } - wait for journal writeRetry Logic & Errors
MongoDB marks retryable errors with errorLabels: TransientTransactionError (retry the whole transaction) and UnknownCommitResult (retry commit only). Production drivers offer built-in retry. Always design transactions to be idempotent where possible. Avoid long-running transactions that hold locks and block other operations.
# transactions can fail with TransientTransactionError
# retry the whole transaction on these errors
function runTransaction(txnFunc) {
while (true) {
const session = db.getMongo().startSession()
try {
session.startTransaction()
txnFunc(session)
session.commitTransaction()
return
} catch (e) {
session.abortTransaction()
if (e.errorLabels &&
e.errorLabels.includes("TransientTransactionError")) {
continue // retry
}
throw e // non-retryable error
} finally {
session.endSession()
}
}
}Atomic Single-Document Ops
Single-document operations are atomic by default — no transaction needed. findOneAndUpdate/findOneAndReplace/findOneAndDelete return the document atomically (great for generating sequential IDs). For many workloads, embedding related data in one document and using atomic operators ($inc, $push) avoids the need for multi-document transactions entirely.
# single-document operations are ALWAYS atomic
db.counters.updateOne(
{ _id: "pageViews" },
{ $inc: { count: 1 } }
)
# findAndModify: atomically find & update, return the doc
db.sequences.findOneAndUpdate(
{ _id: "orderId" },
{ $inc: { seq: 1 } },
{ returnNewDocument: true }
)
# atomic upsert (create-or-increment)
db.counters.updateOne(
{ _id: "visits", date: "2025-01-01" },
{ $inc: { count: 1 } },
{ upsert: true }
)
# use $expr for conditional atomic updates
db.products.updateMany(
{ $expr: { $lt: ["$stock", "$reorderLevel"] } },
{ $set: { needsRestock: true } }
)Two-Phase Commit (Pattern)
The two-phase commit pattern simulates distributed transactions without native multi-doc transactions. It tracks a 'pending' state so a recovery process can resume or roll back if a crash occurs mid-transfer. Use native transactions instead when available — this pattern is complex and error-prone, kept mainly for educational and legacy purposes.
# pattern for atomic transfers without transactions
# (pre-4.0 compatibility, or cross-system transfers)
# 1. create a pending transaction document
db.transactions.insertOne({
_id: "t1", from: "Alice", to: "Bob", amount: 100, state: "pending"
})
# 2. debit source
db.accounts.updateOne(
{ name: "Alice", pendingTransactions: { $ne: "t1" } },
{ $inc: { balance: -100 }, $push: { pendingTransactions: "t1" } }
)
# 3. credit destination
db.accounts.updateOne(
{ name: "Bob", pendingTransactions: { $ne: "t1" } },
{ $inc: { balance: 100 }, $push: { pendingTransactions: "t1" } }
)
# 4. mark transaction done
db.transactions.updateOne({ _id: "t1" }, { $set: { state: "done" } })
# 5. clean up pending references
db.accounts.updateMany({},
{ $pull: { pendingTransactions: "t1" } })Data Modeling
Embedding vs Referencing
The cardinal decision in MongoDB modeling: embed for data accessed together and rarely changing; reference for large, shared, or independently-updated data. Embed when the 'has-a' relationship is exclusive (a user's addresses). Reference when data is reused (a product in many orders) or grows unbounded (a user's millions of log entries).
# EMBED: store related data in one document
db.users.insertOne({
name: "Alice",
addresses: [
{ city: "NYC", zip: "10001" },
{ city: "LA", zip: "90001" }
]
})
# Pros: single read, atomic update
# Cons: document size limit (16MB), data duplication
# REFERENCE: store IDs, join with $lookup
db.users.insertOne({ name: "Alice" })
db.orders.insertOne({
customerId: ObjectId("..."),
amount: 100
})
# Pros: no duplication, independent updates
# Cons: needs joins, more queriesOne-to-Many Relationships
For 'few' relationships, embed. For 'many', reference from the child (parent ID on each child). For 'very many' (millions), use the bucket pattern: group child documents into parent-owned buckets (e.g., by month) to balance document size and query speed. This avoids both unbounded arrays and excessive child docs.
# FEW: embed (e.g., a few addresses per user)
db.users.insertOne({
name: "Alice",
addresses: [{ city: "NYC" }, { city: "LA" }]
})
# MANY: reference with parent pointer
db.posts.insertOne({ title: "Hello", authorId: ObjectId("...") })
# VERY MANY: reference from parent (bucket pattern)
db.users.updateOne(
{ _id: userId },
{ $push: { postIds: postId } } # beware 16MB limit
)
# BUCKET pattern for time series (thousands of items)
db.buckets.insertOne({
ownerId: userId,
date: "2025-01",
measurements: [ {t: 1, v: 30}, {t: 2, v: 31} ]
})Many-to-Many Relationships
For many-to-many, pick the approach by query patterns and data size. Two-sided arrays are fast but require dual maintenance. A junction collection (mapping table) is the most flexible and scales best — it can store relationship metadata (enrollment date, grade) and avoids bloating the parent documents.
# option 1: arrays of ObjectIds on both sides
db.students.insertOne({ name: "Alice", courseIds: [c1, c2] })
db.courses.insertOne({ name: "Math", studentIds: [s1, s2] })
# option 2: one side references (smaller side)
db.students.insertOne({ name: "Alice", courseIds: [c1, c2] })
# option 3: mapping/junction collection
db.enrollments.insertOne({
studentId: s1,
courseId: c1,
enrolledAt: ISODate(),
grade: "A"
})
# query: find a student's courses
db.enrollments.find({ studentId: s1 })Schema Design Patterns
The Attribute Pattern handles entities with varying attributes (products with different specs) — avoids sparse fields and enables dynamic attributes. The Polymorphic Pattern stores related-but-different shapes in one collection using a type discriminator. The Outlier Pattern handles exceptional documents (a celebrity with millions of followers) by referencing instead of embedding.
# ATTRIBUTE PATTERN: flexible attributes (e.g., products)
db.products.insertOne({
name: "Laptop",
attributes: [
{ k: "cpu", v: "i7" },
{ k: "ram", v: 16 }
]
})
# query: db.products.find({ "attributes.k": "cpu", "attributes.v": "i7" })
# POLYMORPHIC PATTERN: one collection, type discriminator
db.events.insertMany([
{ type: "click", url: "/home", userId: 1 },
{ type: "purchase", orderId: 99, total: 50 }
])
# OUTLIER PATTERN: cap large lists
db.users.updateOne({n:"A"}, { $set: { reviews: "see reviews collection" } })Schema Validation
Schema validation enforces document structure at write time — MongoDB's answer to fixed schemas. validationLevel 'strict' validates all writes; 'moderate' only validates inserts and full updates. validationAction 'error' rejects invalid docs; 'warn' logs but accepts them. Validation complements (not replaces) application-level validation.
# add a JSON schema validator to a collection
db.createCollection("users", {
validator: {
$jsonSchema: {
bsonType: "object",
required: ["name", "email"],
properties: {
name: { bsonType: "string", maxLength: 100 },
email: { bsonType: "string", pattern: "^\S+@\S+$" },
age: { bsonType: "int", minimum: 0, maximum: 150 },
role: { enum: ["admin", "user", "guest"] }
}
}
},
validationLevel: "strict",
validationAction: "error"
})
# add validation to an existing collection
db.runCommand({ collMod: "users", validator: { ... } })
# check existing validation rules
db.getCollectionInfos({ name: "users" })Replica Sets
Replica Set Basics
A replica set provides high availability and read scaling. Only one member is primary (accepts writes); others replicate asynchronously. If the primary fails, an election promotes a secondary automatically. An odd number of voting members (3 minimum) avoids election ties. rs.status() shows each member's health, state, and replication lag.
# a replica set = 1 primary + N secondaries
# all writes go to the primary; secondaries replicate
# initiate a replica set
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "host1:27017" },
{ _id: 1, host: "host2:27017" },
{ _id: 2, host: "host3:27017" }
]
})
# check replica set status
rs.status()
rs.conf()
# step down the primary (force an election)
rs.stepDown(60)
# add a new member
rs.add("host4:27017")
rs.remove("host4:27017")Read Preference
Read preference balances consistency vs load. primary guarantees you always read the latest writes. secondary/secondaryPreferred offload reads from the primary but data may be stale (replication lag, typically milliseconds). nearest minimizes latency — ideal for geographically distributed apps. Analytical/reporting workloads should use secondary reads.
# control where reads are routed
# primary (default) - always read from primary (strongest consistency)
db.users.find().readPref("primary")
# primaryPreferred - primary if available, else secondary
db.users.find().readPref("primaryPreferred")
# secondary - only secondaries (may be stale)
db.users.find().readPref("secondary")
# secondaryPreferred - prefer secondaries to offload primary
db.users.find().readPref("secondaryPreferred")
# nearest - lowest latency member
db.users.find().readPref("nearest")
# set in connection string
mongodb://host1:27017/mydb?replicaSet=rs0&readPreference=secondaryWrite Concern
Write concern trades speed for safety. w:1 acks as soon as the primary writes (fast but risks data loss if the primary crashes before replication). w:majority ensures the write is replicated — survives a single node failure (recommended for important data). j:true waits for an on-disk journal flush. Set a sensible default at the cluster level.
# control how durable a write must be before ack
# w: 1 - primary acks (default, fastest, least safe)
db.users.insertOne(doc, { writeConcern: { w: 1 } })
# w: "majority" - majority of members confirmed
db.users.insertOne(doc, { writeConcern: { w: "majority" } })
# j: true - wait for journal flush (survives crash)
db.users.insertOne(doc, { writeConcern: { w: 1, j: true } })
# wtimeout - max wait time in ms
db.users.insertOne(doc, {
writeConcern: { w: "majority", wtimeout: 5000 }
})
# set default write concern for the cluster
db.adminCommand({
setDefaultRWConcern: { defaultWriteConcern: { w: "majority" } }
})Arbiters & Election
Arbiters provide a vote to reach a majority without storing data — useful for two-node deployments, but a third data node is preferable. Priority 0 members never become primary (good for dedicated analytics or DR nodes). Hidden members are excluded from read preference 'nearest' and are ideal for backups. Delayed members lag behind, enabling recovery from accidental deletes.
# an arbiter votes in elections but holds no data
rs.addArb("host5:27017")
# useful to break ties in even-member sets (e.g., 2 data + 1 arbiter)
# but prefer 3 data members for durability
# force a member to never become primary (priority 0)
rs.conf().members[1].priority = 0
rs.reconfig(rs.conf())
# hidden member: not visible to clients, used for analytics/backup
rs.conf().members[2].hidden = true
rs.conf().members[2].priority = 0
rs.reconfig(rs.conf())
# delayed member: lags behind (insurance against human errors)
rs.conf().members[3].secondaryDelaySecs = 3600
rs.conf().members[3].priority = 0Oplog & Replication Internals
The oplog (operation log) is a capped collection recording every write — secondaries apply these entries in order to replicate. If a secondary falls behind beyond the oplog window, it needs a full resync. printReplicationInfo shows the oplog's time span; printSecondaryReplicationInfo shows each secondary's lag. Larger oplogs accommodate longer downtimes.
# the oplog is a capped collection of all writes
use local
db.oplog.rs.stats()
db.oplog.rs.find().sort({ $natural: -1 }).limit(5)
# each oplog entry is an idempotent operation
# { ts: ..., op: "i", ns: "mydb.users", o: { ... } }
# check replication lag
rs.printReplicationInfo() # primary's oplog window
rs.printSecondaryReplicationInfo() # each secondary's lag
# resync a stale secondary (full re-sync)
rs.syncFrom("host2:27017")
# change the oplog size (MongoDB 4.4+)
db.adminCommand({ replSetResizeOplog: 1, size: 16384 }) # MBSecurity & Authentication
Authentication & Users
Authentication verifies identity. Create the admin user before enabling auth, or you'll be locked out. Built-in roles range from read-only to full admin. Use passwordPrompt() in the shell to avoid plaintext passwords in history. Each user is scoped to a database but can have roles on other databases. Enable TLS in production (net.tls.mode).
# enable auth: start mongod with --auth
# or set security.authorization: "enabled" in config
# create the first admin user (before enabling auth)
use admin
db.createUser({
user: "admin",
pwd: passwordPrompt(), // prompts securely
roles: [ { role: "userAdminAnyDatabase", db: "admin" },
{ role: "readWriteAnyDatabase", db: "admin" } ]
})
# create a user for a specific database
use mydb
db.createUser({
user: "appUser",
pwd: passwordPrompt(),
roles: [ { role: "readWrite", db: "mydb" } ]
})
# authenticate as a user
db.auth("appUser", "password")Built-in Roles & Custom Roles
Built-in roles cover common cases; custom roles enable least-privilege. Privileges pair a resource (database/collection) with allowed actions (find, insert, update, etc.). Grant the minimum needed — a reporting user only needs 'find' on specific collections. Custom roles are stored in the admin database and can be reused across users.
# built-in roles:
# read, readWrite, dbAdmin, dbOwner, userAdmin (per database)
# readAnyDatabase, readWriteAnyDatabase, (all databases)
# userAdminAnyDatabase, dbAdminAnyDatabase
# root (superuser), clusterAdmin, backup, restore
# create a custom role with fine-grained privileges
db.createRole({
role: "analyst",
privileges: [
{ resource: { db: "mydb", collection: "reports" },
actions: ["find"] },
{ resource: { db: "mydb", collection: "" },
actions: ["listCollections"] }
],
roles: []
})
# grant a role to a user
db.grantRolesToUser("appUser", [{ role: "analyst", db: "mydb" }])
db.revokeRolesFromUser("appUser", [{ role: "analyst", db: "mydb" }])TLS/SSL Encryption
TLS encrypts data in transit — mandatory for production. requireTLS rejects non-TLS connections. For internal PKI, use x.509 certificate authentication (no passwords — the cert IS the credential). The $external database stores x.509 users. Always use a real CA (or internal CA) in production; self-signed certs are for testing only.
# enable TLS in mongod.conf
net:
tls:
mode: requireTLS
certificateKeyFile: /etc/mongodb/server.pem
CAFile: /etc/mongodb/ca.pem
# connect with TLS and verify the server cert
mongosh "mongodb://host:27017/mydb" \
--tls --tlsCAFile /etc/mongodb/ca.pem
# connect with a client certificate (x.509 auth)
mongosh "mongodb://host:27017/mydb" \
--tls --tlsCertificateKeyFile client.pem \
--tlsCAFile ca.pem \
--authenticationMechanism MONGODB-X509 \
--authenticationDatabase '$external'
# generate a self-signed cert for testing
openssl req -x509 -newkey rsa:4096 -nodes \
-keyout server.key -out server.crt -days 365Network & Auditing
bindIp limits which network interfaces MongoDB listens on — never expose MongoDB directly to the internet (bindIp: 0.0.0.0 + no auth has caused countless breaches). Auditing (Enterprise only) logs security-relevant operations for compliance. The community edition can use OS-level tools (auditd) or app-level logging as alternatives.
# restrict network interface in mongod.conf
net:
bindIp: 127.0.0.1,10.0.0.5 # never 0.0.0.0 in production
port: 27017
# enable auditing (enterprise feature)
auditLog:
destination: file
format: JSON
path: /var/log/mongodb/audit.log
filter: '{ atype: { $in: ["authenticate","createCollection","dropDatabase"] } }'
# view current authentication mechanisms
db.runCommand({ getCmdLineOpts: 1 })
# enable FIPS mode (government compliance)
setParameter:
fipsMode: trueField-Level Encryption (CSFLE)
CSFLE encrypts sensitive fields on the client so the server, backups, and logs never contain plaintext — strongest data protection. Deterministic encryption supports equality queries; random encryption is more secure but not queryable. Auto-encryption (via a driver schema) is transparent to app code. The encryption keys are managed in a KMS (AWS KMS, local key, etc.).
# Client-Side Field Level Encryption encrypts fields
# before sending to the server — server never sees plaintext
# define a data key in the key vault
const clientEncryption = db.getMongo().getClientEncryption()
const keyId = clientEncryption.createDataKey(
"local", { keyAltNames: ["mainKey"] }
)
# encrypt a field explicitly
const ssn = clientEncryption.encrypt(
keyId, "123-45-6789", "deterministic"
)
# store encrypted, query with encrypted value
db.patients.insertOne({ name: "Alice", ssn: ssn })
db.patients.find({
ssn: clientEncryption.encrypt(keyId, "123-45-6789", "deterministic")
})
# automatic CSFLE uses a schema to encrypt per a JSON schema
# { encryptMetadata: { keyId: [keyId], algorithm: "AEAD_AES_256_CBC..." } }Backup & Restore
mongodump & mongorestore
mongodump creates a logical backup (BSON files) — portable but slow for large datasets and not point-in-time consistent across collections without --oplog. --drop drops existing collections before restore. --archive + --gzip produce a single compressed file. For large production backups, prefer filesystem snapshots or Percona Backup instead of mongodump.
# dump an entire database
mongodump --db=mydb --out=/backup/$(date +%F)
# dump a single collection
mongodump --db=mydb --collection=users --out=/backup
# dump with a query filter
mongodump --db=mydb --collection=logs \
--query='{"date":{"$gte":{"$date":"2025-01-01T00:00:00Z"}}}'
# compress the output archive
mongodump --archive=/backup/full.gz --gzip
# restore from a dump
mongorestore --db=mydb --drop /backup/2025-01-01/mydb
mongorestore --archive=/backup/full.gz --gzipmongoexport & mongoimport
mongoexport/mongoimport work with JSON/CSV — for data exchange with other systems, not for backups (they lose BSON types like ObjectId and Date precision). For backups use mongodump. CSV exports are great for spreadsheets and BI tools. Always specify --fields for predictable CSV column order. JSON exports are limited to one document per line by default (--jsonArray for a single array).
# export to JSON (human-readable)
mongoexport --db=mydb --collection=users \
--out=users.json
# export to CSV
mongoexport --db=mydb --collection=users \
--type=csv --fields=name,email,age --out=users.csv
# export with a query filter
mongoexport --db=mydb --collection=users \
--query='{"status":"active"}' --out=active.json
# import from JSON
mongoimport --db=mydb --collection=users --file=users.json
# import from CSV with header row
mongoimport --db=mydb --collection=users \
--type=csv --headerline --file=users.csvFilesystem Snapshots
Filesystem snapshots are the fastest backup method for large datasets — they capture the entire data directory consistently. fsyncLock briefly blocks writes and flushes to disk so the snapshot is consistent. Snapshots are near-instant regardless of data size. Restore is as simple as copying files back — much faster than mongorestore. Works best on LVM, ZFS, or cloud block storage (EBS snapshots).
# snapshot the dbPath volume for a consistent backup
# 1. flush writes to disk and lock (brief)
db.fsyncLock()
# 2. take a filesystem snapshot (LVM, ZFS, EBS, etc.)
lvcreate --snapshot --size 1G --name mongosnap /dev/vg/data
# 3. unlock the database
db.fsyncUnlock()
# 4. mount the snapshot and copy or back it up
mount /dev/vg/mongosnap /mnt/snap
rsync -a /mnt/snap/ /backup/
# restore: stop mongod, replace dbPath, start mongod
# (fastest restore method for large datasets)Point-in-Time Recovery (oplog)
Point-in-time recovery (PITR) reconstructs the database state at a specific moment — essential after accidental deletes or bad migrations. Take regular base backups with --oplog, then replay the oplog up to the desired timestamp (--oplogLimit stops just before a bad operation). PITR requires a replica set (the oplog only exists on replica set members).
# 1. take a base backup (snapshot or mongodump --oplog)
mongodump --archive=/backup/base.gz --oplog --gzip
# 2. note the time of the accident
# 3. replay the oplog up to just before the error
# dump the oplog from a secondary
mongodump --db=local --collection=oplog.rs \
--query='{"ts":{"$gt":{"$timestamp":{"t":1700000000,"i":1}}}}' \
--out=/oplog_dump
# restore the base backup, then replay oplog
mongorestore --archive=/backup/base.gz --oplogReplay \
--oplogLimit=1700000123:1 # stop before this timestamp
# for a running replica set, use mongorestore --oplogReplay
# with a BSON oplog fileAtlas Backup & Cloud Options
MongoDB Atlas offers managed continuous backups with point-in-time recovery — simplest for cloud deployments. Percona Backup for MongoDB (PBM) is the leading open-source backup tool for self-hosted clusters, supporting physical backups and PITR across replica sets and sharded clusters. Always test restore procedures — a backup you can't restore is not a backup.
# MongoDB Atlas: continuous backups with PITR
# configured via UI or Atlas CLI:
atlas backups policies create [clusterId] ...
# on-demand snapshot
atlas backups snapshots create [clusterId] \
--description "pre-migration" \
--retentionDays 7
# restore an Atlas snapshot
atlas backups restores start [clusterId] \
--snapshotId [snapId] \
--deliveryType automated \
--targetClusterName [target]
# Percona Backup for MongoDB (PBM) — open source
pbm backup --type=physical
pbm restore [snapshotName]
pbm pitr restore --time "2025-01-15T14:30:00"Performance & Optimization
Profiling Slow Queries
The database profiler logs operations slower than slowms (default 100ms) into system.profile (a capped collection). Level 2 logs everything — useful for debugging but impacts performance. Look at millis (duration), nreturned (docs returned), and docsExamined to find inefficient queries. Disable profiling when done.
# enable the database profiler
# levels: 0=off, 1=slow ops only, 2=all ops
db.setProfilingLevel(1, { slowms: 100 })
# view profiled operations
db.system.profile.find().sort({ ts: -1 }).limit(5)
# find the slowest operations
db.system.profile.find({ millis: { $gt: 1000 } })
.sort({ millis: -1 })
# profile a specific collection
db.system.profile.find({ ns: "mydb.users" })
# get current profiling level
db.getProfilingStatus()
# clear the profile collection
db.system.profile.drop()explain() Deep Dive
explain() is your primary tuning tool. COLLSCAN means a full collection scan — add an index. The ratio of docsExamined to nReturned is the key efficiency metric: if you examine 10000 docs to return 10, your index is poor. allPlansExecution shows why the planner chose its plan. Index your query's equality, sort, and range fields (ESR rule).
# three verbosity modes
db.users.find({age:30}).explain("queryPlanner") # plan only
db.users.find({age:30}).explain("executionStats") # + timing
db.users.find({age:30}).explain("allPlansExecution") # + rejected plans
# key metrics to check:
# winningPlan.stage: COLLSCAN (bad) vs IXSCAN (good)
# totalKeysExamined: index entries scanned
# totalDocsExamined: documents fetched
# nReturned: documents returned
# executionTimeMillis: total time
# works: number of work units
# ideal ratio: docsExamined ≈ nReturned
# if docsExamined >> nReturned -> missing or bad indexWorking Set & Memory
Performance depends on the working set (data accessed frequently) fitting in RAM. If wiredTiger.cache 'pages read into cache' is high, you're hitting disk — add RAM or add indexes to reduce scans. mongotop shows which collections are I/O hot; mongostat shows query/insert/update rates and faults. compact reclaims wasted disk space from deleted/updated documents.
# check the working set size
db.serverStatus().wiredTiger.cache
# look at: "maximum bytes configured", "current bytes"
# "pages requested from disk" vs "pages read into cache"
# check storage stats
db.stats()
db.users.stats()
db.users.totalSize() # data + index size in bytes
# mongotop: time spent reading/writing per collection
mongotop 30 # every 30 seconds
# mongostat: live server stats
mongostat --rowcount 10 # 10 iterations
# compact a collection (reclaims disk, blocks writes pre-4.4)
db.runCommand({ compact: "users" })Connection Pooling
Each MongoDB connection consumes server memory — thousands of connections degrade performance. Use connection pooling (drivers maintain a reusable pool) with maxPoolSize tuned to your load (50-100 is typical). Share a single MongoClient across your app — never create one per request. Set maxIdleTimeMS to recycle stale connections behind load balancers.
# each connection consumes memory (~1MB) on the server
# limit connections in the driver's connection string
mongodb://host:27017/mydb?maxPoolSize=50
# Node.js driver example
const { MongoClient } = require("mongodb")
const client = new MongoClient(uri, {
maxPoolSize: 50, // max connections
minPoolSize: 5, // keep-alive connections
maxIdleTimeMS: 30000, // close idle conns after 30s
serverSelectionTimeoutMS: 5000
})
# view current connections on the server
db.serverStatus().connections
# { current: 120, available: 880, totalCreated: 200 }Bulk Writes & Batch Sizes
Bulk writes group operations into a single round-trip — far faster than individual ops. unordered:true maximizes throughput (continues past errors, parallelizes on sharded clusters). For large imports, batch in groups of 1000-5000 documents to balance memory and round-trips. insertMany with ordered:false is the fastest way to load data.
# ordered bulk write (stops on first error)
db.users.bulkWrite([
{ insertOne: { document: { name: "A" } } },
{ updateOne: { filter: {n:"B"}, update: {$set:{age:1}} } },
{ deleteOne: { filter: { n: "C" } } }
], { ordered: true })
# unordered bulk write (continues on errors, faster)
db.users.bulkWrite(ops, { ordered: false })
# insertMany is a bulk insert — batch large loads
const docs = Array.from({length: 100000}, (_,i) => ({ i }))
const batch = 1000
for (let i = 0; i < docs.length; i += batch) {
db.big.insertMany(docs.slice(i, i + batch), { ordered: false })
}Change Streams
Watching Changes
Change streams let applications react to data changes in real time by tailing the oplog. Each event includes the operation type, the full document (for inserts/updates with fullDocument option), and a resume token. Store the resume token to resume after a restart without missing or duplicating events. Requires a replica set or sharded cluster.
# open a change stream on a collection
const stream = db.users.watch()
# process change events (Node.js driver)
stream.on("change", (event) => {
console.log(event.operationType) // insert|update|delete|replace
console.log(event.fullDocument) // the new document
console.log(event.documentKey) // the _id
console.log(event.updateDescription) // updated fields
})
# resume token enables restart from last position
const token = stream.resumeToken
// store token, then resume later:
const stream2 = db.users.watch([], { resumeAfter: token })
# close the stream
stream.close()Filtering Change Events
Filter change streams with an aggregation pipeline — only matching events reach your app, reducing noise. By default, updates only include the changed fields, not the full document; fullDocument:'updateLookup' fetches the current document (extra query per event). fullDocumentBeforeChange (6.0+) includes the pre-change document — useful for audit logs.
# filter with an aggregation pipeline
const stream = db.users.watch([
{ $match: { operationType: "update" } },
{ $match: { "updateDescription.updatedFields.status": "active" } },
{ $project: { fullDocument: 1, documentKey: 1 } }
])
# watch specific operations
{ $match: { operationType: { $in: ["insert", "update"] } } }
# full document lookup on update (fetches current doc)
db.users.watch([], {
fullDocument: "updateLookup" // include latest doc on updates
})
# fullDocumentBeforeChange (MongoDB 6.0+)
db.users.watch([], {
fullDocumentBeforeChange: "whenAvailable"
})Change Stream on Database/Cluster
Change streams can watch a single collection, an entire database, or the whole cluster. Database and cluster streams include the namespace (ns) in each event so you know where the change occurred. Use startAtOperationTime to replay changes from a specific point in the oplog — useful for backfilling or recovering missed events after a downtime.
# watch all collections in a database
const stream = db.watch()
# watch all databases in a cluster
const stream = client.watch()
# events include the ns (namespace) field
stream.on("change", (e) => {
console.log(e.ns.db + "." + e.ns.coll)
console.log(e.operationType)
})
# useful for audit logging, cache invalidation,
# cross-system sync, and real-time analytics
# start at a specific timestamp
db.users.watch([], {
startAtOperationTime: Timestamp(1700000000, 1)
})Resume Tokens & Reliability
Resume tokens guarantee at-least-once delivery — persist them after processing each event so a restart resumes exactly where you left off. resumeAfter and startAfter are similar; startAfter works even after collection invalidation events (like a drop). Always handle errors and reconnect with the last token. The token is opaque — treat it as a black box.
# each event has a resume token (_id)
stream.on("change", (e) => {
saveToStore(e._id) // persist the token
processEvent(e)
})
# after a restart, resume from the last token
const lastToken = loadFromStore()
const stream = db.users.watch([], {
resumeAfter: lastToken // resume after this event
})
# alternative: start after a cluster time
db.users.watch([], { startAfter: lastToken })
# handle errors and reconnect
stream.on("error", (err) => {
console.error("stream error", err)
// reconnect with the last saved token
})
# the token is opaque — store it, don't parse itChange Stream Use Cases
Change streams power real-time architectures: cache invalidation (delete stale cache entries), live notifications (WebSocket pushes), audit logging (record every change for compliance), and ETL/sync (replicate to Elasticsearch, a data warehouse, or another system). They replace manual polling and triggers with a clean, reactive event-driven model.
# 1. Cache invalidation: clear cache when data changes
stream.on("change", (e) => {
redis.del("user:" + e.documentKey._id)
})
# 2. Real-time notifications: push to WebSocket clients
stream.on("change", (e) => {
io.emit("user-updated", e.fullDocument)
})
# 3. Audit log: record all changes to a separate store
stream.on("change", (e) => {
auditLog.insertOne({
op: e.operationType,
doc: e.fullDocument,
at: new Date()
})
})
# 4. ETL: sync to a search index or warehouse
stream.on("change", async (e) => {
await elasticsearch.index(e.fullDocument)
})Administration & Tools
Server Status & Monitoring
serverStatus() is the primary monitoring source — track connections, opcounters, cache hit ratio, and replication lag. currentOp shows running operations; killOp terminates a long-running query by its opid. Set a monitoring agent (Prometheus, Datadog, Atlas) to collect these metrics continuously. Key alerts: connection count nearing limit, cache hit ratio dropping, replication lag spiking.
# overall server status
db.serverStatus()
# key metrics to monitor:
db.serverStatus().connections # active/idle connections
db.serverStatus().opcounters # queries/inserts/updates/sec
db.serverStatus().wiredTiger.cache # cache hit ratio (aim >95%)
db.serverStatus().metrics.document # docs inserted/returned/deleted
db.serverStatus().repl # replication state
# current operations (find long-running ops)
db.currentOp({ secs_running: { $gte: 5 } })
# kill a long-running operation
db.killOp(opid)
# host info (CPU, OS, memory)
db.hostInfo()Database & Collection Stats
stats() shows storage details: dataSize (logical), storageSize (physical, after WiredTiger compression — often 50-80% smaller), and indexSize. estimatedDocumentCount is instant (from collection metadata) but approximate; countDocuments({}) is accurate but scans. Use estimated counts for dashboards and exact counts for billing. Monitor storage growth trends to plan capacity.
# database-level stats
db.stats()
db.stats(1024*1024) # in MB
# collection-level stats
db.users.stats()
db.users.dataSize() # uncompressed data size
db.users.storageSize() # on-disk size (with compression)
db.users.totalIndexSize() # total index size
db.users.totalSize() # data + indexes
# count documents (accurate, may be slow)
db.users.countDocuments({})
# estimated count (fast, from metadata)
db.users.estimatedDocumentCount()
# list collections with sizes
db.runCommand({ listCollections: 1 })Compact & Repair
WiredTiger reuses freed disk space, so compact is rarely needed — only after deleting a large fraction of a collection. compact blocks operations on the collection (pre-4.4) or runs online (4.4+). repairDatabase is a destructive last resort that rewrites all data — always back up first. For space reclamation on a replica set, resync a secondary (delete its data and let it re-replicate).
# compact: reclaim disk space from deleted/updated docs
# (WiredTiger reuses freed space, so rarely needed)
db.runCommand({ compact: "users" })
# fullDatabase compact (compact every collection)
db.getCollectionNames().forEach(c => {
db.runCommand({ compact: c })
})
# repairDatabase (legacy, rewrites all data — last resort)
# Stops the server! Back up first.
mongod --repair --dbpath /data/db
# check data integrity (validates BSON)
db.users.validate({ full: true })
# reclaim space by re-syncing a secondary instead
# (safer than repair on a running primary)mongotop & mongostat
mongostat gives a real-time overview of server load (inserts, queries, cache usage, connections) — like top/vmstat for MongoDB. mongotop shows which collections consume the most I/O time — invaluable for finding hot collections. Both are bundled with MongoDB. For production monitoring, ship the mongod log and serverStatus metrics to a monitoring system for historical analysis.
# mongostat: live server stats (like vmstat for MongoDB)
mongostat
mongostat --host=host:27017 --rowcount 10
# columns: insert/query/update/delete/getmore/command
# dirty (cache %), used (cache %), conn, time
# mongotop: time spent per collection (read/write)
mongotop
mongotop 30 # every 30 seconds
mongotop --host=host:27017
# columns: ns (namespace), total/read/write time (ms)
# helps identify which collections are I/O hot
# log slow queries in the mongod log
# set in mongod.conf:
# operationProfiling:
# mode: slowOp
# slowOpThresholdMs: 100Configuration & Startup
Production MongoDB runs via a config file (mongod.conf in YAML). Key settings: cacheSizeGB limits WiredTiger's RAM (default uses 50% of RAM — set explicitly on shared servers), bindIp restricts network access, authorization enables auth, and replSetName/clusterRole enable replication/sharding. Always review and version-control your mongod.conf. serverCmdLineOpts shows the active configuration.
# start mongod with a config file
mongod --config /etc/mongod.conf
# typical mongod.conf (YAML)
storage:
dbPath: /var/lib/mongodb
wiredTiger:
engineConfig:
cacheSizeGB: 4 # limit RAM usage
journal:
enabled: true
net:
port: 27017
bindIp: 127.0.0.1
security:
authorization: enabled
replication:
replSetName: rs0
sharding:
clusterRole: shardsvr
# view current configuration
db.serverCmdLineOpts()
db.runCommand({ getCmdLineOpts: 1 })Drivers & Integration
Node.js Driver
The Node.js driver is async/await-based with full ES6+ support. Always use connection pooling (one MongoClient per app, reused). Cursors are async iterables. Use for-await-of to iterate without loading all documents into memory. Close the client on shutdown. The driver supports transactions, change streams, and all aggregation features.
const { MongoClient } = require("mongodb")
const client = new MongoClient(
"mongodb://localhost:27017",
{ maxPoolSize: 50 }
)
async function main() {
await client.connect()
const db = client.db("mydb")
const users = db.collection("users")
// insert
await users.insertOne({ name: "Alice", age: 30 })
// find with a cursor
const cursor = users.find({ age: { $gte: 18 } })
for await (const doc of cursor) {
console.log(doc.name)
}
// update
await users.updateOne(
{ name: "Alice" },
{ $inc: { age: 1 } }
)
}
main().finally(() => client.close())Python Driver (PyMongo)
PyMongo is the official Python driver. It's synchronous; for async use Motor (asyncio) or Beanie (ODM). PyMongo dictionaries are plain Python dicts — no special objects needed. Aggregation pipelines are lists of dicts. The context-manager transaction pattern ensures automatic commit/abort. Always use connection pooling (the default MongoClient pools connections).
from pymongo import MongoClient
client = MongoClient("mongodb://localhost:27017")
db = client["mydb"]
users = db["users"]
# insert
users.insert_one({"name": "Alice", "age": 30})
# find
for doc in users.find({"age": {"$gte": 18}}).sort("age", -1):
print(doc["name"])
# update
users.update_one(
{"name": "Alice"},
{"$inc": {"age": 1}}
)
# aggregation
pipeline = [
{"$group": {"_id": "$city", "count": {"$sum": 1}}},
{"$sort": {"count": -1}}
]
for doc in users.aggregate(pipeline):
print(doc)
# use a context manager for transactions
with client.start_session() as session:
with session.start_transaction():
users.update_one({"n":"A"}, {"$inc":{"bal":-100}}, session=session)
users.update_one({"n":"B"}, {"$inc":{"bal":100}}, session=session)Indexes from Drivers
Create indexes from your application's setup/migration scripts, not on every request. createIndexes (plural) creates multiple indexes in one call. In production, index builds on large collections can be slow — MongoDB 4.2+ builds indexes non-blocking by default. Version your index creation scripts alongside schema migrations.
// Node.js: create indexes
await db.collection("users").createIndex({ email: 1 }, { unique: true })
await db.collection("users").createIndex({ name: "text" })
// Python
users.create_index([("email", 1)], unique=True)
users.create_index([("name", "text")])
// list indexes
const indexes = await db.collection("users").indexes()
// drop an index
await db.collection("users").dropIndex("email_1")
// create indexes in bulk
await db.collection("orders").createIndexes([
{ key: { customerId: 1 } },
{ key: { status: 1, date: -1 }, name: "status_date" }
])Connection String Options
The connection string carries all client config as query parameters. mongodb+srv uses DNS SRV records to discover hosts — Atlas uses this format; it simplifies connection strings and auto-updates when nodes change. retryWrites=true (default in modern drivers) auto-retries idempotent writes after network blips. Set reasonable timeouts to avoid hung connections.
# standard connection string
mongodb://user:pass@host1:27017,host2:27017/mydb?replicaSet=rs0
# common options (as query params):
# replicaSet=rs0 - replica set name
# readPreference=secondary - route reads to secondaries
# readConcernLevel=majority - read committed data
# w=majority - write concern
# journal=true - wait for journal flush
# maxPoolSize=50 - max connections
# connectTimeoutMS=5000 - connection timeout
# socketTimeoutMS=30000 - socket timeout
# authSource=admin - db for credentials
# tls=true - enable TLS
# retryWrites=true - retry on network errors
# SRV connection string (Atlas, uses DNS)
mongodb+srv://user:[email protected]/mydbODMs & Higher-Level Libraries
ODMs (Object Document Mappers) add schemas, validation, and a higher-level API. Mongoose (Node.js) is the most popular — it enforces structure that MongoDB lacks natively. Beanie is the leading async Python ODM. ODMs are great for rapid development and team consistency, but they add overhead and can hide MongoDB's power. For performance-critical apps, use the raw driver.
// Mongoose (Node.js ODM with schemas)
const mongoose = require("mongoose")
const userSchema = new mongoose.Schema({
name: { type: String, required: true },
age: { type: Number, min: 0 }
})
const User = mongoose.model("User", userSchema)
await User.create({ name: "Alice", age: 30 })
await User.find({ age: { $gte: 18 } })
# Beanie (Python async ODM)
from beanie import Document
class User(Document):
name: str
age: int
await User(name="Alice", age=30).insert()
await User.find(User.age >= 18).to_list()
# Mongose vs raw driver:
# ODMs add schemas, validation, and middleware
# but add overhead and a learning curve
# use the raw driver for max performance/controlGridFS & Large Files
GridFS Basics
GridFS stores files larger than the 16MB BSON document limit by splitting them into chunks (default 255KB) stored in fs.chunks, with metadata in fs.files. It's effectively a filesystem on top of MongoDB. Use it for files that exceed 16MB but still benefit from MongoDB's replication and sharding. For truly large media, consider S3 or a CDN instead.
# GridFS stores files larger than 16MB (the BSON doc limit)
# by splitting them into chunks
# store a file via mongofiles (CLI)
mongofiles --db=mydb put large_video.mp4
mongofiles --db=mydb list
mongofiles --db=myfs get large_video.mp4
# two collections are used:
# fs.files - file metadata (one doc per file)
# fs.chunks - file data in 255KB chunks
# query file metadata
db.fs.files.find().pretty()
db.fs.files.findOne({ filename: "large_video.mp4" })
# query chunks for a file
db.fs.chunks.find({ files_id: fileId }).sort({ n: 1 })GridFS from Drivers
Drivers expose GridFS as a streaming API (GridFSBucket) — ideal for large files as they never load entirely into memory. Upload/download are streams you pipe to/from files or HTTP responses. metadata on uploads is queryable like any document field. GridFS chunks are automatically reassembled on download. The bucket name ('fs' by default) is configurable.
// Node.js: upload and download with streams
const { MongoClient, GridFSBucket } = require("mongodb")
const fs = require("fs")
const bucket = new GridFSBucket(db)
// upload a file
fs.createReadStream("video.mp4")
.pipe(bucket.openUploadStream("video.mp4", {
metadata: { uploadedBy: "alice" }
}))
// download a file
bucket.openDownloadStreamByName("video.mp4")
.pipe(fs.createWriteStream("downloaded.mp4"))
// delete a file
await bucket.delete(fileId)
// find files
const files = await bucket.find({}).toArray()Time Series Collections
Time series collections (5.0+) are optimized for sequential, time-ordered data (IoT sensors, metrics, logs). They store data in columnar-like format under the hood, achieving 30-50% compression and faster range queries than regular collections. granularity should match your data interval. TTL (expireAfterSeconds) auto-purges old data. Use $dateTrunc for time-bucket aggregation.
# create a time series collection (MongoDB 5.0+)
db.createCollection("temperatures", {
timeseries: {
timeField: "timestamp",
metaField: "sensorId",
granularity: "seconds" // seconds|minutes|hours
},
expireAfterSeconds: 2592000 // auto-delete after 30 days
})
# insert measurements
db.temperatures.insertMany([
{ timestamp: ISODate(), sensorId: "s1", value: 22.5 },
{ timestamp: ISODate(), sensorId: "s1", value: 22.7 }
])
# query by time range
db.temperatures.find({
sensorId: "s1",
timestamp: { $gte: ISODate("2025-01-01") }
})
# aggregate averages per hour
db.temperatures.aggregate([
{ $group: {
_id: { $dateTrunc: { date: "$timestamp", unit: "hour" } },
avg: { $avg: "$value" }
}}
])Capped Collections
Capped collections maintain insertion order and overwrite the oldest documents when full — perfect for logs, recent-activity feeds, and ring buffers. They support high-throughput inserts and tailable cursors (like tail -f). Limitations: you can't delete individual documents or grow documents beyond their original size. Change streams are the modern alternative for real-time tailing.
# capped collections have a fixed size and overwrite old docs
db.createCollection("logs", {
capped: true,
size: 5242880, # 5MB max
max: 10000 # optional: max document count
})
# inserts only (no arbitrary deletes/updates)
db.logs.insertOne({ msg: "started", t: new Date() })
# high-throughput tail with a tailable cursor
const cursor = db.logs.find().addOption(2) // tailable
while (true) {
if (cursor.hasNext()) printjson(cursor.next())
}
# convert a regular collection to capped
db.runCommand({
convertToCapped: "events",
size: 1000000
})BSON Size & Document Limits
The 16MB BSON document limit is a hard cap — use Object.bsonsize() to check document sizes. Large arrays (thousands of elements) bloat documents and slow updates (MongoDB rewrites the whole document on update). For unbounded data (reviews, comments, log entries), use a separate collection with a reference. For binary blobs over 16MB, use GridFS or external storage with a URL reference.
# maximum BSON document size: 16MB
# check a document's size (in bytes)
Object.bsonsize(db.users.findOne())
# large arrays/fields should be referenced, not embedded
# a product with 10000 reviews -> use a separate collection
# check if a document is too large
const doc = { /* ... huge doc ... */ }
const size = Object.bsonsize(doc)
if (size > 16 * 1024 * 1024) {
throw new Error("Document exceeds 16MB BSON limit")
}
# GridFS bypasses the limit by chunking
# alternatively, store large blobs externally (S3)
# and keep only a URL reference in the documentSQL Comparison & Migration
SQL to MongoDB Mapping
This mapping is the fastest way for SQL developers to learn MongoDB. The key difference: MongoDB has no fixed schema (documents in a collection can differ), and 'joins' are replaced by either embedding related data or using $lookup (slower than SQL joins — design your schema to minimize them). SQL GROUP BY maps to the $group aggregation stage.
# SQL -> MongoDB terminology
# database -> database
# table -> collection
# row -> document
# column -> field
# index -> index
# join -> $lookup (or embed)
# primary key -> _id (ObjectId by default)
# foreign key -> reference (ObjectId field)
# group by -> $group
# select -> find / $project
# where -> $match
# SQL query -> MongoDB query
# SELECT * FROM users WHERE age >= 18
db.users.find({ age: { $gte: 18 } })
# SELECT name, age FROM users
db.users.find({}, { name: 1, age: 1, _id: 0 })
# SELECT * FROM users ORDER BY age DESC LIMIT 10
db.users.find().sort({ age: -1 }).limit(10)
# SELECT city, COUNT(*) FROM users GROUP BY city
db.users.aggregate([
{ $group: { _id: "$city", count: { $sum: 1 } } }
])Migrating from SQL
Migration is not a 1:1 table-to-collection mapping — the schema should be redesigned for MongoDB's document model. Embed small, exclusive, frequently-accessed-together data (a user's profile + addresses). Keep references for large, shared, or unbounded data (orders for a customer). MongoDB's Relational Migrator tool automates much of this analysis. Plan the schema, don't just port it.
# step 1: export SQL data to JSON/CSV
# (mysqldump, pg_dump, or a script)
# step 2: transform relational data to documents
# SQL:
# users(id, name, dept_id)
# departments(id, name)
# MongoDB (embed small, related data):
db.users.insertOne({
_id: 1,
name: "Alice",
department: { id: 10, name: "Engineering" } # embedded
})
# step 3: for large related data, keep references
db.orders.insertOne({
_id: 101,
customerId: 1, # reference, not embedded
items: [...] # embed order items (bounded)
})
# tools: MongoDB Relational Migrator (GUI), mongoimport
# https://www.mongodb.com/.../relational-migratorTransactions vs SQL ACID
MongoDB provides ACID: single-document operations are always atomic (no transaction needed), and multi-document transactions (4.0+) provide full ACID across collections. The key difference: SQL defaults to ACID on every statement, while MongoDB encourages schema design that keeps related data in one document (atomic by default) and uses transactions only when truly needed. Prefer embedding to avoid transaction overhead.
# SQL: ACID by default on every statement
BEGIN TRANSACTION;
UPDATE accounts SET balance = balance - 100 WHERE name = 'Alice';
UPDATE accounts SET balance = balance + 100 WHERE name = 'Bob';
COMMIT;
# MongoDB: single-document ops are atomic (ACID)
db.accounts.updateOne({name:"Alice"}, {$inc:{balance:-100}})
# (one document = one atomic operation)
# MongoDB: multi-document ACID (transactions, replica set)
const s = db.getMongo().startSession()
s.startTransaction()
db.accounts.updateOne({name:"Alice"},{$inc:{balance:-100}},{session:s})
db.accounts.updateOne({name:"Bob"},{$inc:{balance:100}},{session:s})
s.commitTransaction()
# prefer embedding to avoid transactions entirely
# (embedded data updates atomically in one document)Normalization vs Denormalization
SQL normalizes to avoid duplication; MongoDB denormalizes for read speed. The trade-off: denormalized data is faster to read but harder to update (multiple documents). Denormalize data that changes rarely (country names, product catalogs) and reference data that changes often (prices, inventory). The right balance depends on your read/write ratio — monitor and adjust.
# SQL: normalize to avoid duplication (3NF)
# users(id, name, city_id)
# cities(id, name, country_id)
# countries(id, name)
# -> 4 joins to get a user's country
# MongoDB: denormalize for read performance
db.users.insertOne({
name: "Alice",
city: "NYC",
country: "USA" # denormalized — duplicate but fast reads
})
# trade-off: updates must touch multiple documents
# when a city's name changes:
db.users.updateMany(
{ "city": "New York" },
{ $set: { "city": "NYC" } }
)
# rule: duplicate data that changes rarely,
# reference data that changes oftenWhen to Use MongoDB vs SQL
There's no universal 'best' database. MongoDB excels at flexible schemas, hierarchical data, and horizontal scaling. SQL excels at relational integrity, complex joins, and heavy transactional workloads. Many modern apps are polyglot — MongoDB for catalogs/content/user-generated data, PostgreSQL for financial transactions. Match the database to the workload, not the hype.
# CHOOSE MongoDB when:
# - schema is flexible or evolving rapidly
# - data is naturally hierarchical (embed)
# - you need horizontal scaling (sharding)
# - reads >> writes and you can denormalize
# - document-oriented domain (CMS, product catalogs, IoT)
# CHOOSE SQL (PostgreSQL/MySQL) when:
# - schema is fixed and relational (banking, ERP)
# - you need complex multi-table joins frequently
# - transactions span many tables constantly
# - strong consistency is non-negotiable
# - your team knows SQL well
# HYBRID: use both — MongoDB for product catalog/comments,
# SQL for orders/payments. Many large apps are polyglot.
# AVOID MongoDB for:
# - heavy multi-entity transactional workloads
# - apps requiring complex cross-table joins as core featureAdvanced Aggregation
$merge & $out (Materialized Views)
$out replaces a target collection with the pipeline output (destructive). $merge is more flexible — it can insert, replace, or combine with existing documents, enabling incremental materialized views. $merge is preferred for scheduled refreshes: only new/changed data is updated. Materialized views pre-compute expensive aggregations for fast dashboard queries.
# $out: write aggregation results to a NEW collection
db.orders.aggregate([
{ $group: { _id: "$customerId", total: { $sum: "$amount" } } },
{ $out: "customer_totals" }
])
# $merge: merge results into an EXISTING collection (flexible)
db.orders.aggregate([
{ $group: { _id: "$customerId", total: { $sum: "$amount" } } },
{ $merge: {
into: "customer_totals",
on: "_id",
whenMatched: "replace", // replace|keepExisting|merge|fail
whenNotMatched: "insert" // insert|discard|fail
}}
])
# build a materialized view refreshed on a schedule
# (cron job or change-stream triggered)
db.daily_stats.aggregate([
{ $match: { date: today } },
{ $group: { _id: "$category", count: { $sum: 1 } } },
{ $merge: { into: "daily_stats", whenMatched: "replace" } }
])Window Functions ($setWindowFields)
$setWindowFields (5.0+) brings SQL window functions to MongoDB: running totals, rankings, moving averages, and lag/lead. partitionBy is PARTITION BY, sortBy is ORDER BY, and window defines the frame. Range windows use values (e.g., last 7 days) while documents windows use row counts. Essential for time-series analytics and reporting.
# $setWindowFields: window functions (MongoDB 5.0+)
# like SQL OVER() — compute values across a window of docs
db.sales.aggregate([{
$setWindowFields: {
partitionBy: "$region", # like PARTITION BY
sortBy: { date: 1 }, # like ORDER BY
output: {
runningTotal: { $sum: "$amount", window: { documents: ["unbounded", "current"] } },
rank: { $rank: {} },
denseRank: { $denseRank: {} },
movingAvg: { $avg: "$amount", window: { range: [-7, "current"], unit: "day" } },
lagValue: { $shift: { by: -1, output: "$amount" } }
}
}
}])
# window: documents (count-based) or range (value-based)
# ["unbounded", "current"] = from start to current row
# [-3, "current"] = last 3 docs including current$map, $filter & $reduce
$map, $filter, and $reduce are functional array operators for in-document transformations. $map transforms each element, $filter keeps matching elements, and $reduce aggregates an array into one value. The 'as' variable (default $$this) names the current element. These avoid $unwind for many array operations — faster and cleaner.
# $map: transform each array element
{ $project: {
squares: { $map: {
input: "$nums",
as: "n",
in: { $multiply: ["$$n", "$$n"] }
}}
}}
# $filter: keep elements matching a condition
{ $project: {
adults: { $filter: {
input: "$users",
as: "u",
cond: { $gte: ["$$u.age", 18] }
}}
}}
# $reduce: fold an array into a single value
{ $project: {
total: { $reduce: {
input: "$items",
initialValue: 0,
in: { $add: ["$$value", "$$this.price"] }
}}
}}
# variables: $$value (accumulator), $$this (current element), $$n (named)$facet & Multi-Pipeline
$facet runs multiple sub-pipelines on the same input in a single query — perfect for dashboards needing several views of the data (breakdowns, stats, top N). Each sub-pipeline is independent. The result is one document with a field per sub-pipeline. This replaces multiple round-trips and ensures a consistent snapshot.
# run several aggregations on the same data in one pass
db.products.aggregate([{
$facet: {
"byCategory": [
{ $group: { _id: "$category", count: { $sum: 1 } } },
{ $sort: { count: -1 } }
],
"priceStats": [
{ $group: { _id: null, avg: { $avg: "$price" }, max: { $max: "$price" } } }
],
"topRated": [
{ $sort: { rating: -1 } },
{ $limit: 5 },
{ $project: { name: 1, rating: 1, _id: 0 } }
],
"totalCount": [{ $count: "n" }]
}
}])
# result: a single doc with all four arrays/objects$bucket & $bucketAuto
$bucket categorizes documents into explicit value ranges (like SQL CASE WHEN) — great for price tiers, age groups, and histograms. $bucketAuto lets MongoDB compute boundaries to produce a target number of buckets (good for data exploration when you don't know the distribution). Boundaries are left-inclusive, right-exclusive.
# $bucket: group documents into explicit ranges
db.products.aggregate([{
$bucket: {
groupBy: "$price",
boundaries: [0, 50, 100, 500, 1000],
default: "other",
output: {
count: { $sum: 1 },
names: { $push: "$name" },
avgPrice: { $avg: "$price" }
}
}
}])
# buckets: [0,50), [50,100), [100,500), [500,1000), other
# $bucketAuto: let MongoDB choose the boundaries
db.products.aggregate([{
$bucketAuto: {
groupBy: "$price",
buckets: 5, # target 5 buckets
output: { count: { $sum: 1 }, avg: { $avg: "$price" } }
}
}])
# good for histograms and data explorationMonitoring & Troubleshooting
Key Metrics to Monitor
The four golden signals for MongoDB: connections (approaching the limit signals leaks or missing pooling), cache hit ratio (pages read from disk indicates RAM pressure), replication lag (a lagging secondary risks failover data loss), and queue depth (lock contention). Set alerts on all four. A dropping cache hit ratio is the #1 performance warning sign.
# connections (watch for connection leaks)
db.serverStatus().connections
# { current, available, totalCreated }
# operation rates (queries/inserts/updates/deletes per sec)
db.serverStatus().opcounters
# WiredTiger cache (disk reads = cache misses)
db.serverStatus().wiredTiger.cache
# "bytes currently in the cache" vs "maximum bytes configured"
# "pages read into cache" should be low (cache hits)
# replication lag (critical for HA)
rs.printSecondaryReplicationInfo()
# queue depth (operations waiting for locks)
db.serverStatus().globalLock.currentQueue
# open cursors (watch for leaks)
db.serverStatus().metrics.cursorResolving Lock Contention
MongoDB uses document-level locking (WiredTiger), so contention is rare — but unindexed queries, huge sorts, and long transactions still block. currentOp finds the culprit; killOp stops it. The root cause is usually a missing index (run explain). Schema migrations on large collections can lock for a long time — perform them in small batches or during low traffic.
# check current queue (operations waiting)
db.serverStatus().globalLock.currentQueue
# { readers, writers, total }
# find long-running operations
db.currentOp({
secs_running: { $gte: 5 },
op: { $ne: "none" }
})
# kill a long-running op blocking others
db.killOp(opid)
# common causes of lock contention:
# - unindexed queries (COLLSCAN holds locks)
# - large in-memory sorts
# - long transactions
# - schema migrations on large collections
# check for slow ops in the log
db.adminCommand({ getLog: "global" }).log
.filter(l => l.includes(" Slow query"))Memory & Disk Analysis
WiredTiger's cache is the memory frontier — if it's full of dirty pages, writes slow down (flushing to disk). If 'pages read into cache' is high, add RAM or indexes to reduce scans. Drop unused indexes — they consume disk AND memory AND slow writes. Monitor storage growth to predict capacity needs. On sharded clusters, jumbo chunks (too large to migrate) indicate a poor shard key.
# check storage engine memory usage
db.serverStatus().wiredTiger.cache
# "maximum bytes configured" = cache limit (default 50% RAM)
# "tracked dirty bytes" = dirty pages awaiting flush
# target: dirty < 20%, cache usage < 95%
# disk space per collection
db.users.stats().storageSize
db.users.totalIndexSize()
# find the largest collections
db.getCollectionNames().map(c => ({
name: c, size: db.getCollection(c).storageSize()
})).sort((a,b) => b.size - a.size).slice(0, 5)
# check for jumbo chunks (sharded clusters)
sh.status(true)
# review index sizes (drop unused indexes)
db.users.getIndexes().map(i => ({
name: i.name, size: db.users.stats().indexSizes[i.name]
}))Log Analysis
The mongod log records slow operations, elections, errors, and more. Set slowOpThresholdMs to capture queries above your SLO. getLog retrieves the in-memory ring buffer; the on-disk log persists across restarts. logRotate archives the current log (use with logrotate for daily rotation). Increase component verbosity temporarily for deep debugging (e.g., replication: 2 for election issues), then reset.
# view recent log entries
db.adminCommand({ getLog: "global" })
# log slow queries (set threshold)
db.adminCommand({
setParameter: 1,
logComponentVerbosity: { query: { verbosity: 1 } }
})
# mongod.conf: log slow operations
# operationProfiling:
# mode: slowOp
# slowOpThresholdMs: 100
# slowOpSampleRate: 1.0
# grep the mongod log for slow queries
grep "Slow query" /var/log/mongodb/mongod.log
grep "COMMAND" /var/log/mongodb/mongod.log | tail -50
# rotate logs
db.adminCommand({ logRotate: 1 })
# set log verbosity for a component
db.adminCommand({
setParameter: 1,
logComponentVerbosity: { replication: 2 }
})Common Errors & Solutions
These are the most common MongoDB errors. Duplicate key errors come from _id or unique index violations — handle them in app code. Aggregation memory limits are fixed with allowDiskUse (slower but works). WriteConflict in transactions is normal under contention — retry. Cursor timeouts mean your batch processing is too slow — use smaller batches or noCursorTimeout (but always close cursors).
# ERROR: "E11000 duplicate key"
# cause: _id or unique index conflict
# fix: check the duplicate value, use upsert, or catch the error
# ERROR: "QueryExceededMemoryLimitNoDiskUseAllowed"
# cause: aggregation stage exceeded 100MB RAM
# fix: add { allowDiskUse: true } or add $match early
db.big.aggregate([...], { allowDiskUse: true })
# ERROR: "BSONObjTooLarge"
# cause: document > 16MB
# fix: use GridFS or split the document
# ERROR: "WriteConflict" (transaction)
# cause: two transactions modified the same doc
# fix: retry the transaction (TransientTransactionError)
# ERROR: "ReplicaSetNoPrimary"
# cause: no primary (election in progress)
# fix: wait for election; check rs.status()
# ERROR: "cursor not found"
# cause: cursor timed out (10 min default)
# fix: use noCursorTimeout or process faster関連する MongoDB スニペット
Copy-paste ready code for common tasks.
Insert & Find
Insert documents and query a collection.
Query Operators
Use comparison, logical, array, and regex operators.
Aggregation Pipeline
Build multi-stage pipelines for analytics.
Indexes
Create single, compound, and text indexes for performance.
Update & Delete
Modify and remove documents with update operators.
Collections & Schema
Manage collections, validators, and capped collections.
Replica Set
Configure a replica set for high availability.
Sharding
Distribute data across shards by a shard key.
Was this helpful?