Glimpses of Daniel's world

M202 July 2014 Final weeks

The first few videos of week 7 covered the write concern and some important changes to the basic parts of the system since version 2.4. It is good these subjects are explicitly covered, because MongoDB is going through a lot of functional upgrades. Without the emphasis, you might not realize there is now a more elegant way to do things.

My personal opinion is that write concern should hold no mystery this late in the course. As the other parts, it's information you could find in the release notes.

Snapshot of operations

The next topic of the videos was interesting, but could have had more depth to it. A video was spent on the database command currentOp which will spew data on the operations of the current mongod process. This command generates a lot of data, but the important thing to remember is and most importantly is a snapshot in time. The videos went over some basic scenarios on how to use the database command and the reliability of the information.

This is all nice, until your system has a heavy load and capturing that problem becomes too difficult for a (wo)man and a simple shell. That's when you need tools that are able to capture and process faster then most humans can do.

Mtools

During the course there were some videos on mtools. If you wouldn't know it's related to MongoDB then you stumble onto several other software tools and packages with that name. This gets confusing when you search your favorite package manager, you will probably not find the tool you're looking for.

The mtools mentioned in this course are a collection tools written in Python to simplify several tasks when experimenting with MongoDB or analyzing its output. This software isn't officially supported by MongoDB Inc., but was written by a current employee and found it's way across his peers.

The source and installation can be found on GitHub. You might have some trouble setting up the software correctly, because sometimes library dependencies not enforced to a particular version or need to be forcefully updated. This can be very frustrating and give the idea the tools don't work, where installation of all the right versions of dependencies will make it work as intended.

So what can mtools do? It can set up basic architectures of MongoDB systems. For instance, it's interface to setting up a sharded cluster is quite nice. The course material demonstrated it and I prefer the mtools way to the mongo shell's ShardingTest object. Both mtools and the mongo shell can only start processes on the local system, which is good enough for experiments.

It can also filter, combine and plot log data. There are a lot of tools that can do that, but mtools doesn't need tweaking to understand MongoDB's log format and comes with some default reports that make sense. That being said, there are definitely tools that have a sleeker appearance, create more attractive reports or just integrate better with other tools.

If I would be faced with an immediate problem in a MongoDB cluster, then churning through the logs with mtools would be my preference. MongoDB Management Service be a second choice, because it takes some time for data to show up and it's not as detailed. In time I would be looking for other tools to specifically suit what I need as a tool.

Native authentication

The last videos off the week gave an overview of MongoDB's native authentication and authorization. The evolution and several important changes of authentication and authorization, or auth for short reference, were mentioned. Native auth was never a strong point of the system, but kept evolving through 2.4 and 2.6. There's still some ugly parts to it, but once you understand how it works it's not so bad.

In my opinion any software trying to implement native auth from the ground up is going to fall short the first couple of tries. At the moment, the way native auth works inside MongoDB it is still falling short. It is however possible to create custom ACL to the level of collections.

For me the most important hurdle is understanding how to authenticate in 2.6 on a database level. All credentials are now stored in the admin database, but to create a user who can authenticate against database XYZ you need to create him/her while XYZ is your current database. That user can only authenticate against XYZ, but still have privileges on other databases.

Not specifying the correct database to authenticate against will fail the attempt, which might be confusing at certain times of the day/night. Personally I expect to authenticate against the server/replica set/cluster, not a particular database. The receiving end should abstract authentication.

At the moment I'm less concerned about auth, so I didn't delve deeper than what the videos provided.

Final exam

There were eight final exam assignments, two of them were optional and didn't count for the grade. I failed to give the correct answer on one question, which made me pass the course with just 92% percent instead of 100%. In hindsight I could have seen the problem and answered the question correct, but sometimes you miss the smallest of log messages when the amount of log statements is huge.

I went through the answers of the exam questions so I could to validate my approaches to the assignments and I learned some new things as well.

I realized that I should have studied more on how to avoid downtime during maintenance. In particular on which database commands can be used to force reconfiguration or lock writes without shutting down one or more MongoDB processes.

The approach in the answer usually didn't cause significant downtime, but it was unavoidable to keep all functionality of the cluster or replica set all the time. Production systems often have stricter availability requirements and contain more data then could reasonably be used during the assignments. So aside from the simplest way to complete the assignment there were some footnotes on the difference in approach should it have been a production environment.

During the course there were certain moments where I cursed either myself or the system. Sometimes it's not easy to follow a particular mindset other than your own expectations. It was good to know where to expect the problems and have at least some basic knowledge to solve those problems when they arise. If all else fails, you still need to call in for official support.

At other times I was pleasantly surprised by the progress that has been made. It seems easier to get started with MongoDB, both in programming and administration. There's still that glass ceiling you might break and the shards will be falling down, but it's higher now.