V2 Pipeline with PySpark

[alvin319]

Summary

• Refactored the existing pipeline to use PySpark
• Swapped to a standard logging system that goes to both stdout and a log file in its respective run directory
• Renamed key column names to have a consistent them, i.e., index -> sequence_id
• Re-did the filters to calculate sequence/token frequencies

Note
To showcase the result, I ran the pipeline with these two filters across all datasets; the result is available on HF.