Skip to content

Latest commit

 

History

History
233 lines (229 loc) · 21.3 KB

HISTORY.md

File metadata and controls

233 lines (229 loc) · 21.3 KB
Raw

PlexCleaner

Utility to optimize media files for Direct Play in Plex, Emby, Jellyfin.

Release History

  • Version 3.4:
    • Updated to .NET 8.0.
    • Updated Debian Docker image to Bookworm.
    • Warn when a newer GitHub Release version is available.
      • Only tests for new release availability if ToolsOptions:AutoUpdate is enabled.
      • Updating the tool itself is still a manual process.
      • Alternatively subscribe to GitHub Release Notifications.
    • Added verify command option to verify media streams in files.
      • Note that only media stream validation is performed, track-, bitrate-, and HDR verification is only performed as part of the process command.
      • The verify command is useful when testing or selecting from multiple available media sources.
  • Version 3.3:
    • Download Windows FfMpeg builds from GyanD FfMpeg GitHub mirror, may help with issue #214.
    • Install Alpine media tools from latest-stable to match the v3.18 base image version, resolves MediaInfo segfault.
    • Add "legacy" osx.13-arm64 build.
    • Make Rider 2023.2.1 happy with current C# linter rules.
  • Version 3.2:
    • Added Ctrl-Q and Ctrl-Z as additional break commands, Ctrl+C may terminate the shell command vs. cleanly exiting the process.
  • Version 3.1:
    • Added --preprocess option to the monitor command, that will pre-process all monitored folders.
  • Version 3.0:
    • Docker builds expanded to include support for linux/amd64, linux/arm64, and linux/arm/v7, on Ubuntu, Debian, Alpine, and Arch.
      • See the Docker README for image and tag usage details.
      • The Ubuntu x64 build now utilizes Rob Savoury's private PPA for up to date FFmpeg and HandBrake builds.
    • Switched from .NET 6 to .NET 7.
      • Utilizing some new capabilities, e.g. GeneratedRegex and LibraryImport.
    • Added additional architectures to the published releases, including win-x64, linux-x64, linux-musl-x64, linux-arm, linux-arm64, and osx-x64.
    • Added support for custom FFmpeg and HandBrake command line arguments.
      • See the Custom FFmpeg and HandBrake CLI Parameters section for usage details.
      • Custom options allows for e.g. AV1 video codec, Intel QuickSync encoding, NVidia NVENC encoding, custom profiles, etc.
      • Removed the ConvertOptions:EnableH265Encoder, ConvertOptions:VideoEncodeQuality and ConvertOptions:AudioEncodeCodec options.
      • Replaced with ConvertOptions:FfMpegOptions and ConvertOptions:HandBrakeOptions options.
      • On v3 schema upgrade old ConvertOptions settings will be upgrade to equivalent settings.
    • Added support for IETF / RFC 5646 / BCP 47 language tag formats.
      • See the Language Matching section usage for details.
      • IETF language tags allows for greater flexibility in Matroska player language matching.
        • E.g. pt-BR for Brazilian Portuguese vs. por for Portuguese.
        • E.g. zh-Hans for simplified Chinese vs. chi for Chinese.
      • Update ProcessOptions:DefaultLanguage and ProcessOptions:KeepLanguages from ISO 639-2B to RFC 5646 format, e.g. eng to en.
        • On v3 schema upgrade old ISO 639-2B 3 letter tags will be replaced with generic RFC 5646 tags.
      • Added ProcessOptions.SetIetfLanguageTags to conditionally remux files using MkvMerge to apply IETF language tags when not set.
        • When enabled all files without IETF tags will be remuxed in order to set IETF language tags, this could be time consuming on large collections of older media that lack the now common IETF tags.
      • FFmpeg and HandBrake removes IETF language tags.
        • Files are remuxed using MkvMerge, and IETF tags are restored using MkvPropEdit, after any FFmpeg or HandBrake operation.
        • If you care and can, please do communicate the need for IETF language support to the FFmpeg and HandBrake development teams.
      • Added warnings and attempt to repair when the Language and LanguageIetf are set and are invalid or do not match.
      • MkvMerge --identify added the --normalize-language-ietf extlang option to report e.g. zh-cmn-Hant vs. cmn-Hant.
        • Existing sidecar metadata can be updated using the updatesidecar command.
    • Added ProcessOptions:KeepOriginalLanguage to keep tracks marked as original language.
    • Added ProcessOptions:RemoveClosedCaptions to conditionally vs. always remove closed captions.
    • Added ProcessOptions:SetTrackFlags to set track flags based on track title keywords, e.g. SDH -> HearingImpaired.
    • Added createschema command to create the settings JSON schema file, no longer need to use Sandbox project to create the schema file.
    • Added warnings when multiple tracks of the same kind have a Default flag set.
    • Added --logwarning commandline option to filter log file output to warnings and errors, console still gets all output.
    • Added updatesidecar commandline option to update sidecar files using current media tool information.
    • Added getversioninfo commandline option to print app, runtime, and media tool versions.
    • Added settings file correctness verification to detect missing but required values.
    • Fixed bitrate calculation packet filter logic to exclude negative timestamps leading to out of bounds exceptions, see FFmpeg avoid_negative_ts.
    • Fixed sidecar media file hash calculation logic to open media file read only and share read, avoiding file access or sharing violations.
    • Updated cover art detection and removal logic to not be dependent on RemoveTags setting.
    • Updated DeleteInvalidFiles logic to delete any file that fails processing, not just files that fail verification.
    • Updated RemoveDuplicateLanguages logic to use MkvMerge IETF language tags.
    • Updated RemoveDuplicateTracks logic to account for Matroska track flags.
    • Refactored JSON schema versioning logic to use record instead of class allowing for derived classes to inherited attributes vs. needing to duplicate all attributes.
    • Refactored track selection logic to simplify containment and use with lambda filters.
    • Refactored verify and repair logic, became too complicated.
    • Removed forced file flush and waiting for IO to flush logic, unnecessarily slows down processing and is ineffective.
    • Removed VerifyOptions:VerifyDuration, VerifyOptions:IdetDuration, VerifyOptions:MinimumDuration, and VerifyOptions:MinimumFileAge configuration options.
    • Removed docker image publishing to GHCR, broken pipe errors too frequently break the build.
    • Changed the process exit code to return 1 vs. -1 in case of error, more conformant with standard exit codes, 0 remains success.
    • Settings JSON schema updated from v2 to v3 to account for new and modified settings.
      • Older settings schemas will automatically be upgraded with compatible settings to v3 on first run.
    • Breaking Change Removed the reprocess commandline option, logic was very complex with limited value, use reverify instead.
    • Breaking Change Refactored commandline arguments to only add relevant options to commands that use them vs. adding global options to all commands.
      • Maintaining commandline backwards compatibility was complicated, and the change is unfortunately a breaking change.
      • The following global options have been removed and added to their respective commands:
        • --settingsfile used by several commands.
        • --parallel used by the process command.
        • --threadcount used by the process command.
      • Move the option from the global options to follow the specific command, e.g.:
        • From: PlexCleaner --settingsfile PlexCleaner.json defaultsettings ...
        • To: PlexCleaner defaultsettings --settingsfile PlexCleaner.json ...
        • From: PlexCleaner --settingsfile PlexCleaner.json --parallel --threadcount 2 process ...
        • To: PlexCleaner process --settingsfile PlexCleaner.json --parallel --threadcount 2 ...
  • Version 2.10:
    • Added the --reverify option, to allow verification and repair of media that previously failed to verify or failed to repair.
      • When enabled the VerifyFailed and RepairFailed states will be removed before processing starts, allowing media to be re-processed.
      • The alternative was to use --reprocess=2, but that would re-process all media, while this option only re-processes media in a failed state.
      • As with the --reprocess option, this option is useful when the tooling changed, and may now be better equipped to verify or repair broken media.
  • Version 2.9:
    • Added remote docker container debug support.
      • develop tagged docker builds use the Debug build target, and will now install the .NET SDK and the VsDbg .NET Debugger.
      • Added a --debug command line option that will wait for a debugger to be attached on launch.
      • Remote debugging in docker over SSH can be done using VSCode or Visual Studio.
    • Updated Dockerfile with latest Linux install steps for MediaInfo and MKVToolNix.
    • Updated System.CommandLine usage to accommodate Beta 4 breaking changes.
  • Version 2.8:
    • Added parallel file processing support:
      • Greatly improves throughput on high core count systems, where a single instance of FFmpeg or HandBrake can't utilize all available processing power.
      • Enable parallel processing by using the --parallel command line option.
      • The default thread count is equal to half the number of system cores.
      • Override the default thread count by using the --threadcount option, e.g. PlexCleaner --parallel --threadcount 2.
      • The executing ThreadId is logged to output, this helps with correlating between sequential and logical operations.
      • Interactive console output from tools are disabled when parallel processing is enabled, this avoids console overwrites.
    • General refactoring, bug fixes, and upstream package updates.
  • Version 2.7:
    • Log names of all processed files that are in VerifyFailed state at the end of the process command.
    • Prevent duplicate entries in ProcessOptions:FileIgnoreList setting when VerifyOptions:RegisterInvalidFiles is set, could happen when using --reprocess 2.
    • Added a JSON schema for the configuration file, useful when authoring in tools that honors schemas.
    • Added a "Sandbox" project to simplify code experimentation, e.g. creating a JSON schema from code.
    • Fixed verify and repair logic when VerifyOptions:AutoRepair is enabled and file is in VerifyFailed state but not RepairFailed, could happen when processing is interrupted.
    • Silenced the noisy tool version mismatch warnings when ProcessOptions:SidecarUpdateOnToolChange is disabled.
    • Replaced FileEx.IsFileReadWriteable() with !FileInfo.IsReadOnly to optimize for speed over accuracy, testing for attributes vs. opening for write access.
    • Pinned docker base image to ubuntu:focal vs. ubuntu:latest until Handbrake PPA ads support for Jammy, tracked as #98.
  • Version 2.6:
    • Fixed SidecarFile.Update() bug that would not update the sidecar when only the State changed, and kept re-verifying the same verified files.
    • Added a --reprocess option to the process command, process --reprocess [0 (default), 1, 2]
      • The --reprocess option can be used to override conditional sidecar state optimizations, e.g. don't verify if already verified.
      • 0: Default behavior, do not do any reprocessing.
      • 1: Re-process low cost operations, e.g. tag detection, closed caption detection, etc.
      • 2: Re-process all operations including expensive operations, e.g. deinterlace detection, bitrate calculation, stream verification, etc.
      • Whenever processing logic is updated or improved (e.g. this release), it is recommended to run with --reprocess 1 at least once.
    • Added workaround for HandBrake that force converts closed captions and subtitle tracks to ASS format.
      • After HandBrake deinterlacing, the original subtitles are added to the output file, bypassing HandBrake subtle logic.
      • Subtitle track formats and attributes are preserved, and closed captions embedded are not converted to subtitle tracks.
      • The HandBrake issue tracked as #95.
    • Added the removal of EIA-608 Closed Captions from video streams.
      • Closed Caption subtitles in video streams are undesired as they cannot be managed, all subtitles should be in discrete tracks.
      • FFprobe fails to set the closed_captions JSON attribute in JSON output mode, but does detect and print Closed Captions in normal output mode.
      • FFprobe issue tracked as #94.
    • Added the ability to bootstrap 7-Zip downloads on Windows, manually downloading 7za.exe is no longer required.
      • Getting started is now easier, just run:
        • PlexCleaner.exe --settingsfile PlexCleaner.json defaultsettings
        • PlexCleaner.exe --settingsfile PlexCleaner.json checkfornewtools
    • The --mediafiles option no longer supports multiple entries per option, use multiple --mediafiles options instead.
      • Deprecation warning initially issued with v2.3.5.
      • Old style: --mediafiles path1 path2
      • New style: --mediafiles path1 --mediafiles path2
    • Improved the metadata, tag, and attachment detection and cleanup logic.
      • FFprobe container and track tags are now evaluated for unwanted metadata.
      • Attachments are now deleted before processing, eliminating problems with cover art being detected as video tracks, or FFMpeg converting covert art into video tracks.
      • Run with process --reprocess 1 at least once to re-evaluate conditions.
    • Removed the upgradesidecar command.
      • Sidecar schemas are automatically upgraded since v2.5.
    • Removed the verify command.
      • Use process --reprocess 2 instead.
    • Removed the getbitrateinfo command.
      • Use process --reprocess 2 instead.
    • Minor code cleanup and improvements.
  • Version 2.5:
    • Changed the config file JSON schema to simplify authoring of multi-value settings, resolves #85
      • Older file schemas will automatically be upgraded without requiring user input.
      • Comma separated lists in string format converted to array of strings.
        • Old: "ReMuxExtensions": ".avi,.m2ts,.ts,.vob,.mp4,.m4v,.asf,.wmv,.dv",
        • New: "ReMuxExtensions": [ ".avi", ".m2ts", ".ts", ".vob", ".mp4", ".m4v", ".asf", ".wmv", ".dv" ]
      • Multiple VideoFormat comma separated lists in strings converted to array of objects.
        • Old:
          • "ReEncodeVideoFormats": "mpeg2video,mpeg4,msmpeg4v3,msmpeg4v2,vc1,h264,wmv3,msrle,rawvideo,indeo5"
          • "ReEncodeVideoCodecs": "*,dx50,div3,mp42,*,*,*,*,*,*"
          • "ReEncodeVideoProfiles": "*,*,*,*,*,Constrained Baseline@30,*,*,*,*"
        • New: "ReEncodeVideo": [ { "Format": "mpeg2video" }, { "Format": "mpeg4", "Codec": "dx50" }, ... ]
    • Replaced GitVersion with Nerdbank.GitVersioning as versioning tool, resolves #16.
      • Main branch will now build using Release configuration, other branches will continue building with Debug configuration.
      • Prerelease builds are now posted to GitHub releases tagged as pre-release, Docker builds continue to be tagged as develop.
    • Docker builds are now also pushed to GitHub Container Registry.
      • Builds will continue to push to Docker Hub while it remains free to use.
    • Added a xUnit unit test project.
      • Currently the only tests are for config and sidecar JSON schema backwards compatibility.
    • Code cleanup and refactoring to make current versions of Visual Studio and Rider happy.
  • Version 2.4.5
    • Update FfMpeg in Linux instructions and in Docker builds to version 5.0.
  • Version 2.4.3
    • Added more robust error and control logic for handling specific AVI files.
      • Detect and ignore cover art and thumbnail video tracks.
      • Perform conditional interlace detection using FfMpeg idet filter.
      • Verify media tool track identification matches.
      • Modify sidecar file hashing to support small files.
    • Use C# 10 file scoped namespaces.
  • Version 2.4.1
    • Added ProcessOptions:RestoreFileTimestamp JSON option to restore the media file modified time to match the original value.
    • Fixed media tool logic to account for WMV files with cover art, and added wmv3 and wmav2 codecs to be converted.
  • Version 2.3.5
    • Deprecation warning for --mediafiles option taking multiple paths, instead use multiple invocations.
      • Old style: --mediafiles path1 path2
      • New style: --mediafiles path1 --mediafiles path2
    • Added removesubtitles command to remove all subtitles, useful when the media contains annoying forced subtitles with ads.
  • Version 2.3.2
    • Warn when the HDR profile is Dolby Vision (profile 5) vs. Dolby Vision / SMPTE ST 2086 (profile 7).
      • Unless using DV capable hardware, profile 5 may play but will result in funky colors on HDR10 hardware.
      • The warning is only logged during the verify step, repair is not possible.
      • To re-verify existing 4K files use the verify command, or reset the state using the createsidecar and process commands.
    • Renamed getsidecar command to getsidecarinfo for consistency with other getxxxinfo commands.
    • Added gettoolinfo command to print media info reported by tools.
    • Refactored duplicate file iteration logic to use lambdas.
  • Version 2.3:
    • Migrated from .NET 5 to .NET 6.
  • Version 2.1:
    • Added backwards compatibility for some older JSON schemas.
    • Added the upgradesidecar command to migrate sidecar files to the current JSON schema version.
    • Sidecar JSON schema changes:
      • Replaced the unreliable file modified timestamp state tracking with a SHA256 hash of parts of the MKV file.
      • Replaced the Verified boolean with State flags to track more granular file state and modification changes.
      • Run the upgradesidecar command to migrate sidecar files to the current schema version.
    • Repairing metadata inconsistencies, e.g. MuxingMode not specified for S_VOBSUB subtitle codecs, by remuxing the MKV file.
    • Added a ToolsOptions:AutoUpdate configuration option to automatically update the tools before each run.
  • Version 2.0:
    • Linux and Docker are now supported platforms.
      • Automatic downloading of tools on Linux is not currently supported, tools need to be manually installed on the system.
      • The Docker build includes all the prerequisite tools, and is easier to use vs. installing all the tools on Linux.
    • Support for H.265 encoding added.
    • All file metadata, titles, tags, and track names are now deleted during media file cleanup.
    • Windows systems will be kept awake during processing.
    • Schema version numbers were added to JSON config files, breaking backwards compatibility.
      • Sidecar JSON will be invalid and recreated, including re-verifying that can be very time consuming.
      • Tools JSON will be invalid and checkfortools should be used to update tools.
    • Tool version numbers are now using the short version number, allowing for Sidecar compatibility between Windows and Linux.
    • Processing of the same media can be mixed between Windows, Linux, and Docker, note that the paths in the FileIgnoreList setting are platform specific.
    • New options were added to the JSON config file.
      • ConvertOptions:EnableH265Encoder: Enable H.265 encoding vs. H.264.
      • ToolsOptions:UseSystem: Use tools from the system path vs. from the Tools folder, this is the default on Linux.
      • VerifyOptions:RegisterInvalidFiles: Add files that fail verify and repair to the ProcessOptions:FileIgnoreList.
      • ProcessOptions:ReEncodeAudioFormats : opus codec added to default list.
    • File logging and console output is now done using structured Serilog logging.
      • Basic console and file logging options are used, configuration from JSON is not currently supported.