Fix multi-byte character handling in read_file

[sidewaysthought]

Background

Related to issue #232 - Multi-byte character handling
User reports UTF files not being read. Some files may be UTF-8 but have a BOM marker which Python can see as invalid, or may not be UTF-8 at all.

Changes

The read_file command has been updated to detect the file encoding, then use the detected encoding to provide more reliable reading of files. The change requires the addition of the chardet dependency. It has been added to requirements.txt

Documentation

Change is minor with what I believe to be obvious intent when viewing the code.

Test Plan

Tested by making agent create a file by output redirect from PowerShell (which includes BOM markers, and previously failed read.) Files now correctly read.

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes

Unfamiliar with generating test cases on a project of this scale. Would rather not horribly break things.

[SpaceTTAngel]

For me it's ok, I've tested the file_operations.py and requirements.txt with several datasets @sidewaysthought

[SpaceTTAngel]

I've tested the code with a several files dataset to stimulate encoding reading, with the following logs 👍

encoded file type: ascii
encoded file content: ciaociao

---

encoded file type: ascii
encoded file content: ciaociao

---

encoded file type: ascii
encoded file content: ciaociao

---

encoded file type: UTF-8-SIG
encoded file content: ciaociao

---

encoded file type: UTF-16
encoded file content: ciaociao

---

encoded file type: UTF-16
encoded file content: ciaociao

---

encoded file type: UTF-16
encoded file content: ciaociao

---

encoded file type: UTF-16
encoded file content: ciaociao

---

encoded file type: UTF-16
encoded file content: ciaociao

---

encoded file type: ascii
encoded file content: premi?re is first
premie?re is slightly different
????????? is Cyrillic
? am Deseret

---

encoded file type: Windows-1254
encoded file content: premiŠre is first
premie?re is slightly different
????????? is Cyrillic
? am Deseret

---

encoded file type: KOI8-R
encoded file content: premi?re is first
premie?re is slightly different
Кириллица is Cyrillic
? am Deseret

---

encoded file type: ISO-8859-1
encoded file content: première is first
premie?re is slightly different
????????? is Cyrillic
? am Deseret

---

encoded file type: Windows-1254

[ashreim-UPL]

I had the same issue so I removed the file and replaced it with txt file. But this is definitely a bug

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		May 1, 2023 4:22pm

[codecov]

Codecov Report

Patch coverage: 81.81% and project coverage change: +0.02 🎉

Comparison is base (94ec4a4) 60.26% compared to head (9df829a) 60.28%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3173      +/-   ##
==========================================
+ Coverage   60.26%   60.28%   +0.02%     
==========================================
  Files          69       69              
  Lines        3143     3150       +7     
  Branches      525      525              
==========================================
+ Hits         1894     1899       +5     
- Misses       1116     1118       +2     
  Partials      133      133              

Impacted Files	Coverage Δ
autogpt/commands/web_selenium.py	86.04% <66.66%> (-1.61%)	⬇️
autogpt/commands/file_operations.py	83.13% <100.00%> (+0.19%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.