Robo 3T, Dangerous default value for "Create Index in Background"

Indexes in mongo must be created in Background. Not doing it is dangerous enough to bring your entire MongoDB cluster down, and beyond repair.

Problem

Create and index, followed by any modification to the index in a large collection with the Advanced > Create Index in Background option set to false.

The Cluster propagates those actions and becomes unresponsive!

Github Issue

"Create index in background" checked by default · Issue #1656 · Studio3T/robomongo

History

Not once, but twice, I have created and shortly after modified an index in different collections that were both large.

In the first occurrence, the server has become largely unavailable, but after a few minutes, managed to recover.

In the second occurrence, it was terminal. We literally had to create another cluster and restore from backup.

How we knew what was going on

By looking at the MongoDB Atlas Dashboard, the charts in the resources made it quite clear what was going on:

1. The index creation had been already propagated to the entire cluster
2. Soon after all the nodes became 100% consumed by the index creation.

Dharshan explains in his article Foreground index creation:

When you build an index in the foreground it blocks all other operations on the database — on a large collection, this can be several hours. This implies that your database is down for the duration of the index build. Given that this is the default mode of building indexes it is not surprising that a lot of developers shoot themselves in the foot triggering accidental index builds. There is really no good reason to trigger a foreground index build on a production server (unless you know that the collection has a small amount of data).

Available knowledge on the situation

Apart from the fact that Create Indexes is never a trivial job and caution is always advised for that task, "The Perils of Building Indexes on MongoDB" are known.

MongoDB Atlas has also advised (via customer support) that indexes should always be created in "Background", unless we have the explicit need to do otherwise.

Take Away

My take aways for you from this problem are:

1. Think twice before you commit an index operations over large Mongo collections
2. Consider creating indexes in background unless you explicitly need to do otherwise (in Robot 3T, Advanced > "Create index in Background"
3. Time to send Robo 3T a Pull Request changing the default option (already done! here is the link)

Wanna keep in Touch? LinkedIn!

My name is Hudson Mendes (@hudsonmendes), I’m a 38 years old coder, husband, father of 3, ex Startup Founder, ex-Peloton L7 Staff Tech Lead/Manager, nearly BSc in Computer Science by the University of London & Senior AI/ML Engineering Manager.

I’ve been on the Software Engineering road for 22+ years, and occasionally write about topics of interest to other Senior Engineering Managers and Staff Machine Learning Engineering with a bit of focus (but not exclusively) on Natural Language Processing.

Join me there, and I will keep you in the loop with my new learnings in AI/ML/LLMs and beyond!