Implement HTTP 1.0 & fix TCP_client

[gpotter2]

This PR implements:

• HTTP 1.0 layer. Based on scapy_http but:
  • added sessions support: follow chunks/streams of TCP HTTP (Content_Length and Chunked)
  • added decompression & re-compression support (3 default methods implemented: gzip, lzip, zlib)
  • fixes build: quite a few things were broken
  • add support for all HTTP fields
  • fixed for PEP8
  • improved Python 3 support, general bytes handling (especially with regexes/split)
  • Removed deprecated fields (Let's not import already-deprecated stuff)
  • added tests: test chunk reassembly, content_length threshold, TCPSession, gzip decompression (most common)
• TCPsessions. Used to sort TCP packets via their seq/ack. Used to decode HTTP flows. Could be extended to other TCP protocols
• TCP_client & Automaton: fixes for Automaton: support _ATMT_Supersocket on Windows. Also update it to h'share the StringBuffer of the TCPSession, so that it can also handle TCP fragmentation.
• documentation about HTTP/TCP_client
• add other specific tests (gzip compact utiles...)

Note: because of Python 2-3 compatibility, several utils were needed:

• gzip_decompress/gzip_compress: gzip.decompress/gzip.compress isn't available on Python 2.7.

Important note: sr() is NOT supported YET, because it would require to change sndrcv() to handle Sessions. This is done in #1999. (The 2 PRs must be merged before it may be implemented) (done)

This code was able to decode a JPEG image sent through several HTTP chunks, despite TCP re-transmission. You can check it out by yourself by un-commenting a line in the chunked-image test

Rationale

• Why scapy_http ?

It has been around for quite some time. It allows users/projects that were using it to migrate seamlessly, to use the new features.
I do not want to change the API, and it allowed to start on an existing basis.

• What's next ?

This PR should be able to fully replace scapy_http which development has been paused (stopped?). That would allow to deprecate scapy_http and continue any further development here.
I've asked the maintainers if they want to merge the projects, therefore release a last scapy_http version with a warning deprecating it. I doubt they want to invest more time into maintaining it 😄

See invernizzi/scapy-http#46

[codecov]

Codecov Report

Merging #1925 into master will increase coverage by 0.18%.
The diff coverage is 87.08%.

@@            Coverage Diff            @@
##           master   #1925      +/-   ##
=========================================
+ Coverage   87.12%   87.3%   +0.18%     
=========================================
  Files         198     199       +1     
  Lines       44679   45078     +399     
=========================================
+ Hits        38926   39356     +430     
+ Misses       5753    5722      -31

Impacted Files	Coverage Δ
scapy/contrib/isotp.py	89.44% <ø> (+5.43%)	⬆️
scapy/plist.py	86.22% <ø> (ø)	⬆️
scapy/pipetool.py	89.97% <100%> (+0.06%)	⬆️
scapy/compat.py	100% <100%> (ø)	⬆️
scapy/config.py	84.16% <100%> (+0.03%)	⬆️
scapy/sendrecv.py	84.29% <100%> (+0.02%)	⬆️
scapy/arch/pcapdnet.py	70.4% <100%> (+0.75%)	⬆️
scapy/error.py	100% <100%> (ø)	⬆️
scapy/layers/inet.py	72.84% <100%> (+0.02%)	⬆️
scapy/automaton.py	87.2% <70.17%> (+1.76%)	⬆️
... and 9 more

[gpotter2]

@guedou @p-l- I've finally finished this PR. Feel free to review it 😄

I would have liked to know your thoughts before actually taking any action such as contacting scapy_http maintainer.

On the licensing question: That's GPLv2-GPLv2 so there's no problem.

[gpotter2]

@guedou This is ready for review. p-l- has already reviewed a version (not the latest, but there have only been slight bug fixes since). It appears that tox is failing to install on Travis with 3.4 - unrelated.

[guedou]

Nice PR. See my comments.

[gpotter2]

@guedou Tests now pass. this is mergeable

[guedou]

@gpotter2 do you want me to squash the commits or keep them?

[gpotter2]

Squashing will be better. There has been some stuff that was reverted

[p-l-]

@guedou this is OK for me.

[guedou]

Cool PR. I made some simple comments.

[gpotter2]

I had actually been so focused on HTTP dissection that I overlooked if the build was always correct.
Turns out there were a few issues (corrected in the last commit) with Unknown_Headers. This is now fixed.
As both building and dissection are complete, this is (actually) now ready.

Sorry for the last-minute commits

[gpotter2]

@guedou @p-l- This should be mergeable.
The Scapy-http maintainer said we would work deprecationnout once this has been merged, to make sure it's ready for 2.4.3 :-)

[guedou]

I will try to release v2.4.3rc3 by Wednesday.