Here's a snip from a comment:
"... over-solving the problem is writing a bunch of custom code (the file readers and writers) when there is already a perfectly good mechanism available for all of the persistence, reliability, repeatability and scalability issues. To me, if my application has already been designed to provide data access to the database, all of my connection, security, data transformation, etc issues have already been solved. Prior to me someone has already taken ODBC/JDBC and worked out the issues within my app. Why would I want to go to a custom code based system where I have to work all of this out?"
In short, since the RDBMS does all this wonderful stuff, why not use it?
Good question. I can tell it's a good question because the question doesn't have a single pat answer. It has two answers.
Choice 1: Go Ahead, Use the RDBMS
Of course, if I need all those wonderful features ("persistence, reliability, repeatability and scalability issues"), then I'll use the RDBMS.
Clearly, the "DB hides a lot of the low level issues from the standard software developer." There are many times when an RDBMS solves our problem.
My point is about applying the gigantic overhead of the RDBMS to a problem for which it is inappropriate.
Choice 2: Don't use the RDBMS
The issue that I want to dwell on is the need for all those wonderful features. I don't think it's appropriate to throw the RDBMS at a problem which requires almost only three of the endless litany of RDBMS features.
Here's the crux of the matter: "The cost of a professional developer throwing a file based mechanism together and working the bugs out of it for a few days may be more that the cost of the DB resources used."
I like to call this the Tomorrow's Dollars Don't Exist™ viewpoint. The big, visible, obvious cost is the up-front cost of a developer. For the next 10 years, however, we have to support, maintain, upgrade, enhance, and pay for this solution.
Three Specific Features
My point centered on three specific features. In a situation where we have more complex requirements, clearly, the RDBMS is likely to be appropriate. But the three specific features aren't costly.
- Reliability. Double writes don't require developer time to debug. They're a first class feature of the OS.
- Scalability. This is -- actually -- not solved by the RDBMS. That was the genesis of my thoughts on this subject. The RDBMS doesn't scale well because it doesn't handle queueing well.
- Recoverability. Again, file system writes and reads don't require developer time to debug. They're the essence of the OS.
What leads to confusion is the All Those Features comparison. We're comparing all of the features of the RDBMS with a problem that requires almost none of the features.
Summary
The convenience factor is aptly summarized as "Why would I want to go to a custom code based system where I have to work all of this out?"
By assuming that Tomorrow's Dollars Don't Exist™, the convenience carries even more weight. Further, if we look at All Those Features™, the RDBMS looks perfect for the problem at hand.
If, however, we look at the few features we need and the long term cost of the features we don't need, the RDBMS is A Bad Thing™ for this class of problems. A RDBMS isn't appropriate for a message queue, and I don't find the convenience argument terribly convincing.