mongodb (through restheart) insertion is very slow when collection is big, a single command make it 1000 times faster

  • fennng 

One of my mongo db collection has grown to more than 1.5 million docs. And the total size is 677.54 MB.

With mongo db, this is nothing, but the insertion speed is unbelievably slow for that collection. For all other smaller collections, no problem at all. The the problem is obviously related to the size of the collection.

It took about 2 seconds to insert a small doc.

I checked the VM ram and everything else. Used a lot of tools (top, mongostat, mongotop etc.) . I couldn’t find the bottleneck, until I ran the db.currentOP() command in mongo shell.

I noticed that restheart is actually using findAndModify operation rather than insertOne operation behind the scene for http post. And it query for the _etag (_etag is the field added by restheart) for every insertion. Because the collection is big, the query time is long.

Then the solution is obvious, create a index for _etag. So I open mongo shell and execute the following command.

> db.messages.createIndex({"_etag": -1})
        "createdCollectionAutomatically" : false,
        "numIndexesBefore" : 2,
        "numIndexesAfter" : 3,
        "ok" : 1

Now, a insertion only take 2ms, that’s 1000 times faster. A problem which bothered me for a few months solved. I created the collection with mongo express. I am not sure if _etag index will be added automatically if I created collection through restheart api. But anyway, if you are using mongodb with restheart, don’t forget to index all your collection with _etag.