I can't emphasize this enough.

Some people, when confronted with a problem, think

“I know, I'll use a spreadsheet.” Now they have two problems.

(This was originally about regular expressions. And AWK. See http://regex.info/blog/2006-09-15/247)

Fiction writer F. L. Stevens got a list of literary agents …

Browsers are required to skip over bad HTML and render something.

Consequently, many web sites have significant HTML errors that don't show up until you try to scrape their content.

Beautiful Soup has a handy hook for doing markup massage prior to parsing. This is a way of fixing site-specific …

Ugh. When you try to parse MS-generated HTML, you find some extension syntax that is completely befuddling.

I've tried a few things in the past, none were particularly good.

In reading a file recently, I found that even Beautiful Soup was unable to prettify or parse it.

The document was …

The problem with Python is the vastness of the Open Source community. You may think you have something cool for HTML parsing , but then someone tells you about Beautiful Soup which already does it.

In my defense, I actually did a version of this HTML parsing back in '02. Indeed …

The mixed content model, defined succinctly in the XML standards, is pleasant enough for human communication, but leaves a lot to be desired. For example, mapping a mixed content model to a relational database is a hard problem.

The problem is made worse when the document is HTML. HTML doesn't …