MongoDB Aggregation Pipeline

The aggregation pipeline is one of MongoDB's most powerful features. It processes documents through a sequence of stages, where each stage transforms the data. Think of it as an assembly line — each stage does one job and passes the result to the next.

---

Core Aggregation Stages

---

$lookup (Join) Explained

$lookup is MongoDB's version of a SQL JOIN:

{
  $lookup: {
    from: 'users',       // Collection to join (lowercase, plural = collection name)
    localField: 'owner', // Field in current document
    foreignField: '_id', // Field in the joined collection
    as: 'ownerInfo',     // Array field to store results
  }
}
// Result: ownerInfo is an array of matching User documents
// Use $first or $unwind to get a single object from the array

---

Sub-Pipelines: Nested $lookup with pipeline Option

The pipeline option inside $lookup allows you to filter, project, or further transform documents within the join — without fetching all fields into memory first:

{
  $lookup: {
    from: 'users',
    localField: 'owner',
    foreignField: '_id',
    as: 'owner',
    pipeline: [
      // Project only needed fields from the joined User
      { $project: { fullName: 1, username: 1, avatar: 1 } },
    ],
  },
}

This is much more efficient than fetching the full User document and then projecting — the sub-pipeline runs before the documents are joined.

---

Example 1: Get User Channel Profile

User.aggregate([
  { $match: { username: 'johndoe' } },
  {
    $lookup: {
      from: 'subscriptions',
      localField: '_id',
      foreignField: 'channel',
      as: 'subscribers',
    },
  },
  {
    $lookup: {
      from: 'videos',
      localField: '_id',
      foreignField: 'owner',
      as: 'videos',
      pipeline: [
        { $match: { isPublished: true } },
        { $count: 'count' },
      ],
    },
  },
  {
    $addFields: {
      subscribersCount: { $size: '$subscribers' },
      videoCount: { $first: '$videos.count' },
      isSubscribed: { $in: [req.user._id, '$subscribers.subscriber'] },
    },
  },
  {
    $project: {
      username: 1, fullName: 1, avatar: 1, coverImage: 1,
      subscribersCount: 1, videoCount: 1, isSubscribed: 1,
    },
  },
]);

---

Example 2: Get Watch History with Nested Lookup

User.aggregate([
  { $match: { _id: new mongoose.Types.ObjectId(userId) } },
  {
    $lookup: {
      from: 'videos',
      localField: 'watchHistory',
      foreignField: '_id',
      as: 'watchHistory',
      pipeline: [
        { $match: { isPublished: true } },
        {
          $lookup: {
            from: 'users',           // Nested lookup: video owner info
            localField: 'owner',
            foreignField: '_id',
            as: 'owner',
            pipeline: [
              { $project: { fullName: 1, username: 1, avatar: 1 } },
            ],
          },
        },
        { $addFields: { owner: { $first: '$owner' } } },
        { $project: { title: 1, thumbnail: 1, duration: 1, views: 1, owner: 1 } },
      ],
    },
  },
]);

---

Route Organisation by Feature

Organising routes by feature (or resource) is the industry-standard approach:

src/routes/
├── user.routes.js       — /api/v1/users/*
├── video.routes.js      — /api/v1/videos/*
├── subscription.routes.js — /api/v1/subscriptions/*
├── like.routes.js       — /api/v1/likes/*
├── comment.routes.js    — /api/v1/comments/*
├── playlist.routes.js   — /api/v1/playlists/*
└── tweet.routes.js      — /api/v1/tweets/*

Router chaining (multiple HTTP methods on same path):

router.route('/').get(getAllVideos).post(verifyJWT, upload.fields([...]), publishVideo);
router.route('/:videoId').get(getVideoById).patch(verifyJWT, updateVideo).delete(verifyJWT, deleteVideo);
router.route('/toggle/publish/:videoId').patch(verifyJWT, togglePublishStatus);

Mounting all routers in app.js:

app.use('/api/v1/users', userRouter);
app.use('/api/v1/videos', videoRouter);
app.use('/api/v1/subscriptions', subscriptionRouter);
app.use('/api/v1/likes', likeRouter);
app.use('/api/v1/comments', commentRouter);
app.use('/api/v1/playlists', playlistRouter);
app.use('/api/v1/tweets', tweetRouter);

---

Aggregation vs Regular Query