About once a week a question shows up on Stack Overflow that involves loading a database with truly epic volumes of data. For example "billions of rows in a single table for a month".

Billions of rows per month is a minimum insert rate of 385 row per second.

Also …

Great article on "Search" from back in '08 in Forbes. "Why Google Isn't Enough", by Dan Woods. He's talking about "Enterprise Search": why in-house Google-style search is really hard and often unsatisfying.

Here's the cool quote.

enterprise search systems also index and navigate information that may reside in databases, content …

Making the rounds: Droopy: easy file receiving. Apparently, there were some widely-read blog posts about this. Google "Droopy: A Tiny Web Server That Makes Receiving Files a Snap" to see the buzz.

The point here is that 750 lines of Python code can go a long way. It's a complete …

Let's say you've found some new, good way to do business.

JSON, for example. Or Agile Methods in general. Or TDD specifically. Or use of an ORM.

You read up on it. You build a spike solution to show that it's more efficient.

We do a lot of bulk loads. A lot.

So many that we have some standard ETL-like modules for generic "Validate", "Load", "Load_Dimension", "Load_Fact" and those sorts of obvious patterns.

Mostly our business processes amount to a "dimensional conformance and fact load", followed by extracts, followed by a different "dimensional …

Here's the situation.

A vendor sent us three separate files which need to be merged. 70,000+ records each. They're CSV files, so column position doesn't much matter. The column name (in row 1) is what matters.

I looked at three solutions. Two of which are merely OK. The third …

They say "You Can't Manage What You Don't Measure". This isn't quite right, however. You can measure lots of things you can't manage. Rainfall, for example. Software development is like that. You can measure stuff that you can't actually control.

The original Deming quotes are more subtle: there are things …

The SQL database offers a domain-independent NULL value. This is a terrible thing, and should be treated with a depth of respect and fear.

Before using NULL values in a database, read things like "Null Values in Fuzzy Datasets" and "Null Values Revisited in Prospect of Data Integration".

See this …

I can't say enough good things about TDD.

But I'll try.

Due to an epic failure to read the documentation (this, specifically) I couldn't get our RESTful web services to work in Apache.

The entire application system has pretty good test coverage. I use the Python unittest to do integration …

It seems so simple: use the HTTP Digest Authorization with the Quality of Protection set to "auth".

It's an easy algorithm. A nonce that encodes a timestamp can be used to be sure no one is attempting to cache credentials. It's potentially very, very nice.

Except for one thing: Apache …