There are many good reasons to use shell scripts. Mostly, a script can be useful when it's an alias that launches an application. Beyond that, I have doubts.
Incumbency is a popular argument for bash.
It's not a good argument, however.
Use invoke and you'll be much happier.
This isn't about details of pySpark. This is about the philosophy of testing when working with a large, complex framework, like pySpark, pandas, numpy, or whatever.
Use data subsets.
Write unit tests for the functions that process the data.
Don't test pyspark itself. Test the code you write.
I can't actually share all the code. So this feels incomplete. But I can share what I said about the code. Then you can look at your code and decide if you've got similar problems to fix.
My responses were these. I'll expand on them below.
Mocking is essential to unit testing.
However.
It's also annoyingly difficult to get right.
If we aren't 100% perfectly clear on what we're mocking, we will merely canonize any dumb assumptions into mock objects that don't really work. They work in the sense that they don't crash, but they don't …
more ...See this short discussion on a Stingray Reader feature:
https://sourceforge.net/p/stingrayreader/discussion/COBOL/thread/d2132851/?limit=25#2a3a
This turned into an exercise in pure TDD.
more ...<rant>
I'm not a fan of applying TDD in a strict, death-march fashion.
I see the comments on Stack Overflow that …
Some years ago--never mind how long precisely--having little or no money in my purse... I had a great chance to do some Test-Driven Reverse Engineering on a rather complex C program. I extracted test cases. I worked with the users to gather test cases. And I rewrote their legacy app …
more ...Somewhere, I have a vague recollection of reading advice from someone (Bill Gates?) that it takes three versions to get things right. The context may have been a justification of the wild success of Windows 3.0.
Or, I could be just making it up.
But one thing I have …
more ...The question of testing came up recently. The description of the process sounded like manually "testing" some complex web application. When trying to work out manual "testing", I find it necessary to use scare quotes. I'm not sure there's a place for "manual testing" of any software.
I know that …
more ...Just stumbled across several blog postings on unit test naming.
Essentially the TestCase will name the fixture. That's pretty easy to understand.
The cool part is this: each test method is a two-part clause: condition_"should"_result or "when"_condition_"then"_result.
See https://wiki.openmrs.org/display/docs/Unit …
more ...