In Command-Line Applications, we looked at a Python main-import switch which boiled down to this.

for file in args.file:
    with open( file, "r" ) as source:
        process_file( source, args )

The point was that each distinct file on the command-line was processed in a more-or-less uniform way by a single function …

It occurs to me that much of "Big IT" creates a well-oiled organization that makes broken software seem acceptable. The breakage is wrapped in layers of finely-tuned process. Consider a typical Enterprise Application. There's a help desk, ticket tracking, a user support organization that does "ad-hoc" processing, and a development …

I'm old -- I admit it -- and I feel that command-line applications are still very, very important. Linux, for example, is packed full of almost innumerable command-line applications. In some cases, the Linux GUI tools are specifically just wrappers around the underlying command-line applications.

For many types of high-volume data processing …

See this: "Stop hard-coding business rules" in SD Times. Here's what's exasperating: "Memo to developers: Stop hard-coding business rules into applications. Use business rules engines instead." Business Rules Engines? You mean Python? It appears that they don't mean Python. "Developers can use [a BPM suite or rules engine] and be …

Computerworld, September 12, page 10.

MicroburstIT Disasters According to a study of 1,471 big IT projects, 15% turn out to be money pits, with cost overruns averaging 200%.

How is this a politically-charged statement? We hear this kind of thing all the time.

As developers (or project leaders …

"Threads don't promote concurrent I/O."

Kernel threads may. Most of us write user threads. Here's a great summary under Thread (Computer Science).

However, the use of blocking system calls in user threads (as opposed to kernel threads) or fibers can be problematic. If a user thread or a fiber …

Folks like to claim that unit testing absolutely requires each class be tested in isolation using mocks for all dependencies. This is a noble aspiration, but doesn't work out perfectly well in Python.

First, "unit" is intentionally vague. It could be a class, a function, a module or a package …

web2py wins an award -- cool. I really like Django. But...

"Are you implying that a scalable Data Warehouse solution could be implemented using Python and serialised files?"

Not "implying". I'm trying to state it as clearly as I can. A scalable data warehouse solution involves a lot of flat file processing.

ETL, for example, is mostly a flat-file pipeline. It …

We process a fair amount of externally-prepared datasets. 40,000 rows of econometric data that we purchased from a third-party. Mostly, the data is in a usable format: .CSV or .XSLX.

Once in a while, we get CSV with |. A few times, we got fixed-format COBOL-style records.

Recently, we got …