Moved everything from blogger.com to "here" (https://slott56.github.io).

What did I learn?

Check out DZone's Coding zone: https://dzone.com/coding. Hover over the "Coding" drop-down menu.

Notice anything lacking?

I'll give you a hint: Python.

They have "Frameworks", "Java", "Javascript", "Languages" and "Tools".

The "Languages" seems to be general programming, and the posts include Java, JavaScript and numerous other languages.

Let's …

Some background: FAERIE DUST™, Obstinate Idiocy, Obstinate Idiocy, Expanded, and even Permutations, Combinations and Frustrations. I want to set up algorithm design as the diametric opposite of Obstinate Stupidity. To do that, let's look at Obstinate Stupidity.

The theme?

We did something wrong, and we don't want to fix it …

Real Question (somewhat abbreviated): "My algorithm performs badly, do I need asyncio?"

Short answer: No.

Long answer: Sigh. No. Do you need a slap upside the head?

Here's how it plays out:

Q: "We figured that if we 'parallelize' it, then we can apply multiple cores, and it will run …

When I talk about functional programming in Python, folks like to look for place where functional programming isn't appropriate. They latch onto finite-state automata (FSA) because "state" of an automata doesn't seem to fit with stateless objects used in functional programming.

This is a false dichotomy.

It's emphatically false in …

The issue of permutations and combinations is sometimes funny.

Not funny weird. But, funny "haha."

I received an email with 100's of words and 10 attachments. (10. Really.) The subject was how best to enumerate 6! permutations of something or other. With a goal of comparing some optimization algorithm with …

See Obstinate Idiocy for some background.

Here are three warning signs I was able to deduce.

• No Rational Justification
• Ineffective Tool Choice
• Random Whining

To which I can now add two more.

Symptom 4 of Obstinate Idiocy is that all questions are rhetorical and they often come with pre-argued answers …

Here's how to recognize a Faerie Dust request:

1. We have identified a problem. It can be with almost anything: scalability, reliability, auditability, any Quality Measure.
2. We're pursuing a specific technology. Typically, something that has the lowest impact on our architecture.
3. We can't address anything other than this specific technology variation …

This isn't about details of pySpark. This is about the philosophy of testing when working with a large, complex framework, like pySpark, pandas, numpy, or whatever.