-
-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Syntax test of RPZ complaiant zone records(file) #86
Comments
Did you saw 609d506 ? |
Note to self We should just improve the logic behind the previously mentionned commit in order to implement the RPZ specifig check. We could also make |
hrm hrm.... thought i could remember something about that 😭 but as noticed i couldn't find it. I think the wildcard should stay as it is a valid AUTH zone record type, where RPZ differs in some ways, especially when it comes to the
|
This patch fixes #86. Also: * Fix typo. Contributors: * @spirillen
@spirillen Can you test d603982 before I merge it into the |
I can give it a try... But I'm kinda confused why you left out the testing of Give me a bit of time. Test results from the labTested directly in Debian Buster (10) (conda was not used) Results = failure is the only option 😃 Installed by: pip3 install --user git+https://github.com/funilrys/PyFunceble@rpz-syntax-check Test file drill axfr rpz.mypdns.cloud @axfr.mypdns.cloud >> rpz.mypdns.cloud # IPv6 only
drill axfr rpz.mypdns.cloud @axfr.ipv4.mypdns.cloud -p 5306 >> rpz.mypdns.cloud # IPv4 supported on port 5306 Test string: /home/$user/.local/bin/pyfunceble --syntax --rpz -f rpz.mypdns.cloud Test result: Traceback (most recent call last):
File "/home/$user/.local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1226, in tool
raise exception
File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1115, in tool
PyFunceble.CONFIGURATION.rpz = preset.switch("rpz")
File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/config/preset.py", line 165, in switch
raise Exception(to_print % (repr(variable), PyFunceble.LINKS.repo + "/issues."))
Exception: Impossible to switch 'rpz'. Please post an issue to https://github.com/funilrys/PyFunceble/issues. However, even as PyF requested to update the
Test 2 global installationSame test environment, but this time I've install by sudo -H pip3 install git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
pyfunceble --version
pyfunceble 3.3.2.dev (Teal Blauwbok: Centipede) Appended The test runpyfunceble --syntax --rpz -f rpz.mypdns.cloud
A configuration key is missing or a new version is available.
Try to merge upstream configuration file into '/home/$user/.config/PyFunceble/.PyFunceble.yaml'? [y/n] y The following errors occured Your version: 3.3.2.dev (Teal Blauwbok: Centipede)
Upstream version: 3.3.3.dev (Teal Blauwbok: Tick)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1226, in tool
raise exception
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1218, in tool
generate_all_results_only=args.generate_all_files_from_database,
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
file_path, generate_results_only, generate_all_results_only
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
PyFunceble.output.Constructor()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 92, in __init__
self.restore()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 535, in restore
structure = self._get_structure()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
PyFunceble.LINKS.dir_structure.replace("master", "dev")
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 289, in get
result = self.session.get(url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 162, in send
response = super().send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known')) Conda test envComming up, but quessing you have something to start with here... |
Conda test env resultsSame test box as previously, using global conda activate "${pyfunceblePackageName}"
pyfunceble --syntax --rpz -f /tmp/rpz.mypdns.cloud
Your version: 3.3.2.dev (Teal Blauwbok: Centipede)
Upstream version: 3.3.3.dev (Teal Blauwbok: Tick) Subject Status Source
---------------------------------------------------------------------------------------------------- ----------- ----------
3600 INVALID SYNTAX
in INVALID SYNTAX
soa INVALID SYNTAX
auth1.fi.matrix.rocks. VALID SYNTAX
need.to.know.only. INVALID SYNTAX
2020091304 INVALID SYNTAX
60 INVALID SYNTAX
604800 INVALID SYNTAX
345600 INVALID SYNTAX
in INVALID SYNTAX
a INVALID SYNTAX
116.203.32.67 VALID SYNTAX
in INVALID SYNTAX
a INVALID SYNTAX
95.216.209.53 VALID SYNTAX
600 INVALID SYNTAX
in INVALID SYNTAX
ns INVALID SYNTAX
ns0.dns.matrix.rocks. VALID SYNTAX
in INVALID SYNTAX
ns INVALID SYNTAX
ns1.dns.matrix.rocks. VALID SYNTAX
in INVALID SYNTAX
aaaa INVALID SYNTAX
2a01:4f8:1c0c:5f61::53 VALID SYNTAX
in INVALID SYNTAX
aaaa INVALID SYNTAX
2a01:4f9:c010:410e::53 VALID SYNTAX
172800 INVALID SYNTAX
in INVALID SYNTAX
cname INVALID SYNTAX
. INVALID SYNTAX
in INVALID SYNTAX
cname INVALID SYNTAX
in INVALID SYNTAX
cname INVALID SYNTAX
in INVALID SYNTAX
cname INVALID SYNTAX
in INVALID SYNTAX head /tmp/rpz.mypdns.cloud
rpz.mypdns.cloud. 3600 IN SOA auth1.fi.matrix.rocks. need.to.know.only. 2020091304 3600 60 604800 60
rpz.mypdns.cloud. 345600 IN A 116.203.32.67
rpz.mypdns.cloud. 345600 IN A 95.216.209.53
rpz.mypdns.cloud. 600 IN NS ns0.dns.matrix.rocks.
rpz.mypdns.cloud. 600 IN NS ns1.dns.matrix.rocks.
rpz.mypdns.cloud. 345600 IN AAAA 2a01:4f8:1c0c:5f61::53
rpz.mypdns.cloud. 345600 IN AAAA 2a01:4f9:c010:410e::53
ccp.ac.rpz.mypdns.cloud. 172800 IN CNAME .
*.ccp.ac.rpz.mypdns.cloud. 172800 IN CNAME .
directory.ac.rpz.mypdns.cloud. 172800 IN CNAME . What happens here is you are first testing the UPDATE Sorry for the bash version, but would for sure like to see the |
This patch touches and fixes #86. Contributors: * @spirillen
@spirillen it's a separate branch so a lot will go wrong with the configuration! But once in Can you test my latest improvement? |
Same test environment as previously (Debian 10, Buster) First I test by installing the pyfunceble --versionpyfunceble 3.3.3.dev (Teal Blauwbok: Gander) Test file...drill axfr rpz.mypdns.cloud @axfr.mypdns.cloud >> rpz.mypdns.cloud # IPv6 only
drill axfr rpz.mypdns.cloud @axfr.ipv4.mypdns.cloud -p 5306 >> rpz.mypdns.cloud # IPv4 supported on port 5306 Test strings/home/$user/.local/bin/pyfunceble --syntax --rpz -f rpz.mypdns.cloud
pyfunceble --syntax --rpz -f rpz.mypdns.cloud Rpz variable in .PyFunceble_production.yaml
Test 1 --user installationpip3 install --user -U git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
Collecting git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
Cloning https://github.com/funilrys/PyFunceble (to revision rpz-syntax-check) to ./pip-req-build-nws6jm56
Requirement already satisfied, skipping upgrade: PyMySQL in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.9.3)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: alembic in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.6.1)
Requirement already satisfied, skipping upgrade: dnspython in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.16.0)
Requirement already satisfied, skipping upgrade: domain2idna in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.9.1)
Requirement already satisfied, skipping upgrade: inflection in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (4.2.3)
WARNING: python-box 4.2.3 does not provide the extra 'all'
Requirement already satisfied, skipping upgrade: python-dotenv in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.13.0)
Requirement already satisfied, skipping upgrade: requests<3 in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.21.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (40.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: Mako in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: python-dateutil in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.10)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in $HOME/.local/lib/python3.7/site-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->alembic->PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml->python-box[all]->PyFunceble-dev==3.3.3) (0.2.0)
Building wheels for collected packages: PyFunceble-dev
Building wheel for PyFunceble-dev (setup.py) ... done
Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=333752 sha256=f89285b4d0130fed58f753f95b7e25222bae6e4a2880a71150251e7fcf07a1b3
Stored in directory: /tmp/pip-ephem-wheel-cache-shhh3y4j/wheels/3c/53/e0/88775abcdbc2d7b6f8689506120c7072041feedec7ad2c6cc7
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3 Test 1 --user resultsTraceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "$HOME/.local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1226, in tool
raise exception
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1218, in tool
generate_all_results_only=args.generate_all_files_from_database,
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
file_path, generate_results_only, generate_all_results_only
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
PyFunceble.output.Constructor()
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 92, in __init__
self.restore()
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 535, in restore
structure = self._get_structure()
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
PyFunceble.LINKS.dir_structure.replace("master", "dev")
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/lookup/requests.py", line 289, in get
result = self.session.get(url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/lookup/requests.py", line 162, in send
response = super().send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known')) Test 2 global installationInstall stringsudo -H pip3 install -U git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
Collecting git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
Cloning https://github.com/funilrys/PyFunceble (to revision rpz-syntax-check) to ./pip-req-build-dd2ow06g
Requirement already satisfied, skipping upgrade: PyMySQL in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.9.3)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: alembic in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.6.1)
Requirement already satisfied, skipping upgrade: dnspython in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.16.0)
Requirement already satisfied, skipping upgrade: domain2idna in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.9.1)
Requirement already satisfied, skipping upgrade: inflection in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (4.2.3)
WARNING: python-box 4.2.3 does not provide the extra 'all'
Requirement already satisfied, skipping upgrade: python-dotenv in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.13.0)
Requirement already satisfied, skipping upgrade: requests<3 in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.21.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (40.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: python-dateutil in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: Mako in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.10)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->alembic->PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml->python-box[all]->PyFunceble-dev==3.3.3) (0.2.0)
Building wheels for collected packages: PyFunceble-dev
Building wheel for PyFunceble-dev (setup.py) ... done
Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=333752 sha256=92c5496cff0e61975ab6204a9af5c9287653b1574693ed5e2a19045075a1c89c
Stored in directory: /tmp/pip-ephem-wheel-cache-i5gqayrd/wheels/3c/53/e0/88775abcdbc2d7b6f8689506120c7072041feedec7ad2c6cc7
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3
pyfunceble --version
pyfunceble 3.3.3.dev (Teal Blauwbok: Gander) ResultTraceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1226, in tool
raise exception
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1218, in tool
generate_all_results_only=args.generate_all_files_from_database,
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
file_path, generate_results_only, generate_all_results_only
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
PyFunceble.output.Constructor()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 92, in __init__
self.restore()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 535, in restore
structure = self._get_structure()
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
PyFunceble.LINKS.dir_structure.replace("master", "dev")
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 289, in get
result = self.session.get(url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 162, in send
response = super().send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known')) Conda TestConda setup#!/usr/bin/env bash
condaInstallDir="${HOME}/miniconda"
pyfunceblePackageName="pyfunceble-rpz"
source "${condaInstallDir}/etc/profile.d/conda.sh"
hash conda
conda update -q conda
conda activate "${pyfunceblePackageName}"
pip install --upgrade pip -q
pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@rpz-syntax-check'
python --version # Python 3.8.5
pyfunceble --version # pyfunceble 3.3.3.dev (Teal Blauwbok: Gander)
pip list
export PYFUNCEBLE_AUTO_CONFIGURATION=yes
pyfunceble --syntax --rpz -f /tmp/rpz.mypdns.cloud ResultsSubject Status Source
---------------------------------------------------- ----------- ----------
rpz.mypdns.cloud. VALID SYNTAX
ccp.ac.rpz.mypdns.cloud. VALID SYNTAX
*.ccp.ac.rpz.mypdns.cloud. VALID SYNTAX
directory.ac.rpz.mypdns.cloud. VALID SYNTAX
fix.ac.rpz.mypdns.cloud. VALID SYNTAX
gay.fix.ac.rpz.mypdns.cloud. VALID SYNTAX
forum.ac.rpz.mypdns.cloud. VALID SYNTAX
scarlatinastudyo.forum.ac.rpz.mypdns.cloud. VALID SYNTAX
www.scarlatinastudyo.forum.ac.rpz.mypdns.cloud. VALID SYNTAX
kimono.ac.rpz.mypdns.cloud. VALID SYNTAX
www.kimono.ac.rpz.mypdns.cloud. VALID SYNTAX
link.ac.rpz.mypdns.cloud. VALID SYNTAX
*.link.ac.rpz.mypdns.cloud. VALID SYNTAX
www.link.ac.rpz.mypdns.cloud. VALID SYNTAX
primewire.ac.rpz.mypdns.cloud. VALID SYNTAX
www.primewire.ac.rpz.mypdns.cloud. VALID SYNTAX
putlocker.ac.rpz.mypdns.cloud. VALID SYNTAX
www.putlocker.ac.rpz.mypdns.cloud. VALID SYNTAX
sms.ac.rpz.mypdns.cloud. VALID SYNTAX
reg.sms.ac.rpz.mypdns.cloud. VALID SYNTAX
theacademy.ac.rpz.mypdns.cloud. VALID SYNTAX
www.theacademy.ac.rpz.mypdns.cloud. VALID SYNTAX
tube.ac.rpz.mypdns.cloud. VALID SYNTAX
*.tube.ac.rpz.mypdns.cloud. VALID SYNTAX
twit.ac.rpz.mypdns.cloud. VALID SYNTAX
upload.ac.rpz.mypdns.cloud. VALID SYNTAX
*.upload.ac.rpz.mypdns.cloud. VALID SYNTAX
apple-ios.academy.rpz.mypdns.cloud. VALID SYNTAX
www.apple-ios.academy.rpz.mypdns.cloud. VALID SYNTAX
...
^C
Thanks for using PyFunceble!
Share your experience on Twitter with #PyFunceble!
Have a feedback, an issue or an improvement idea?
Let us know on GitHub! Conda record test (Not syntax)I'm doing this test as it appears (tl;dr;) that you have implanted it in ee5e89e Your implementation of this is a good choice and you are actually sparing me a supplement to the feature request to Several input sources at once Test stringpyfunceble --rpz -f /tmp/adult.mypdns.cloud # As rpz.mypdns.cloud is to big for a full test CommentFrom the on-screen layout, it can appear that you are not stripping the zone from the test records... and this can lead to a misconception of what is test and how. These records are valid, however they was marked 'INVALID`, and found a few more alike. annuairesexeporno.com.adult.mypdns.cloud. ACTIVE XXX
*.annuairesexeporno.com.adult.mypdns.cloud. INVALID XXX
antiquetwat.com.adult.mypdns.cloud. ACTIVE XXX
*.antiquetwat.com.adult.mypdns.cloud. INVALID XXX Test resultThis is a complement to the above comment Status Percentage Numbers
----------- ------------ ------------
ACTIVE 97% 2377
INACTIVE 1% 35
INVALID 1% 29 tree --prune -f output/
output
├── output/continue.json
├── output/hosts
│ ├── output/hosts/ACTIVE
│ │ └── output/hosts/ACTIVE/hosts
│ ├── output/hosts/INACTIVE
│ │ └── output/hosts/INACTIVE/hosts
│ └── output/hosts/INVALID
│ └── output/hosts/INVALID/hosts
├── output/logs
│ └── output/logs/percentage
│ └── output/logs/percentage/percentage.txt
└── output/splited
├── output/splited/ACTIVE
├── output/splited/INACTIVE
└── output/splited/INVALID head output/hosts/INVALID/hosts
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:31:43.402102
0.0.0.0 *.onlyrealzoo.club.adult.mypdns.cloud.
0.0.0.0 *.igayporn.tv.sds.rncdn7.co.adult.mypdns.cloud.
0.0.0.0 *.amateursfuckporn.com.adult.mypdns.cloud. head output/hosts/ACTIVE/hosts
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:29:34.868796
0.0.0.0 adult.mypdns.cloud.
0.0.0.0 *.adult.adult.mypdns.cloud.
0.0.0.0 yespornplease.adult.adult.mypdns.cloud.
0.0.0.0 *.00webcams.com.adult.mypdns.cloud.
0.0.0.0 *.0xxx.ws.adult.mypdns.cloud.
0.0.0.0 *.1cam.webcam.adult.mypdns.cloud.
0.0.0.0 *.1watchmygf.com.adult.mypdns.cloud. head output/hosts/INACTIVE/hosts
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:31:29.186047
0.0.0.0 greentoons.club.adult.mypdns.cloud.
0.0.0.0 adultadworld.com.adult.mypdns.cloud.
0.0.0.0 amateurxfilms.com.adult.mypdns.cloud.
0.0.0.0 badgirlsblog.com.adult.mypdns.cloud.
0.0.0.0 beautifulhairypussy.com.adult.mypdns.cloud.
0.0.0.0 bestjavporn.com.adult.mypdns.cloud.
0.0.0.0 bimbolive.com.adult.mypdns.cloud. Test with
|
Note to self: |
This patch touches #86. Contributors: * @spirillen
This patch touches #86. Indeed, before this patch, we were converting (when needed) wildcard and RPZ only inside the `PyFunceble.Check` class. From now, the `PyFunceble.Check` class will not do any conversion. All the conversion (wildcard, rpz) are now made on a deeper level. This change let us have access to the conversion (wildcard, rpz) with any kind of test (availability, syntax, reputation, etc.). Contributors: * @spirillen
Test string: pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 --mining -m -ex -a -nf --complements --share-logs Error outputTraceback (most recent call last): Test string: pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex -a -nf --complements --share-logs Error outputTraceback (most recent call last): Test string: pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex -a Unexspected prints of dots... Test string: pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex Error outputSubject Status HTTP Code Traceback (most recent call last): |
@spirillen 5628bf3 fixes the I'm now looking at the issue with the multiprocessing. |
This patch touches #86. Contributors: * @spirillen
This issue is ready for dev deployment. It's waiting for your next report @spirillen. |
well that one would be rather short.... as you can see on keybase the pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade pyfunceble-dev OR pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@dev' OR pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@rpz-syntax-check' Test string: (With the pyfunceble -db --database-type mariadb --syntax -m -a -f /tmp/adult.mypdns.cloud Same error:Subject Status Expiration Date Source Analyze Date
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- --------------------
Traceback (most recent call last):
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
result.read()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
first_packet = self.connection._read_packet()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
packet.raise_for_error()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1209, in tool
raise exception
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1194, in tool
Dispatcher(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 131, in __init__
self.dispatch_file_test(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 174, in dispatch_file_test
PyFunceble.core.Multiprocess(file_path, "domain").run_test()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 566, in run_test
self.__run_multiprocess_test(shadow_file, manager, tracker=tracker)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 422, in __run_multiprocess_test
self.__start_process(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 303, in __start_process
if subject in self.autocontinue:
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/engine/auto_continue.py", line 152, in __contains__
db_session.query(Status)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3463, in one
ret = self.one_or_none()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3432, in one_or_none
ret = list(self)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3508, in __iter__
return self._execute_and_instances(context)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
result.read()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
first_packet = self.connection._read_packet()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
packet.raise_for_error()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
[SQL: SELECT pyfunceble_status.id AS pyfunceble_status_id, pyfunceble_status.created AS pyfunceble_status_created, pyfunceble_status.modified AS pyfunceble_status_modified, pyfunceble_status.file_id AS pyfunceble_status_file_id, pyfunceble_status.tested AS pyfunceble_status_tested, pyfunceble_status._status AS pyfunceble_status__status, pyfunceble_status.status AS pyfunceble_status_status, pyfunceble_status._status_source AS pyfunceble_status__status_source, pyfunceble_status.status_source AS pyfunceble_status_status_source, pyfunceble_status.domain_syntax_validation AS pyfunceble_status_domain_syntax_validation, pyfunceble_status.expiration_date AS pyfunceble_status_expiration_date, pyfunceble_status.http_status_code AS pyfunceble_status_http_status_code, pyfunceble_status.ipv4_range_syntax_validation AS pyfunceble_status_ipv4_range_syntax_validation, pyfunceble_status.ipv4_syntax_validation AS pyfunceble_status_ipv4_syntax_validation, pyfunceble_status.ipv6_range_syntax_validation AS pyfunceble_status_ipv6_range_syntax_validation, pyfunceble_status.ipv6_syntax_validation AS pyfunceble_status_ipv6_syntax_validation, pyfunceble_status.subdomain_syntax_validation AS pyfunceble_status_subdomain_syntax_validation, pyfunceble_status.url_syntax_validation AS pyfunceble_status_url_syntax_validation, pyfunceble_status.is_complement AS pyfunceble_status_is_complement, pyfunceble_status.test_completed AS pyfunceble_status_test_completed, pyfunceble_status.tested_at AS pyfunceble_status_tested_at
FROM pyfunceble_status INNER JOIN pyfunceble_file ON pyfunceble_file.id = pyfunceble_status.file_id
WHERE pyfunceble_status.tested = %(tested_1)s AND pyfunceble_file.path = %(path_1)s]
[parameters: {'tested_1': '3600', 'path_1': '/tmp/adult.mypdns.cloud'}]
(Background on this error at: http://sqlalche.me/e/13/f405) And here is something a bit more weird... Clean DB, no tables.. pip uninstall -yq pyfunceble pyfunceble-dev
WARNING: Skipping pyfunceble as it is not installed.
:/tmp$ pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@dev' ErrorCollecting git+https://github.com/funilrys/PyFunceble@dev
Cloning https://github.com/funilrys/PyFunceble (to revision dev) to ./pip-req-build-jbh7bd53
Requirement already satisfied, skipping upgrade: alembic in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (3.1)
Requirement already satisfied, skipping upgrade: dnspython in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (2.0.0)
Requirement already satisfied, skipping upgrade: domain2idna in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: inflection in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: PyMySQL in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.10.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (5.1.1)
Requirement already satisfied, skipping upgrade: python-dotenv in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.14.0)
Requirement already satisfied, skipping upgrade: PyYAML in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: requests<3 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (2.24.0)
Requirement already satisfied, skipping upgrade: setuptools in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: sqlalchemy in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: Mako in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-dateutil in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cryptography->PyFunceble-dev==3.3.3) (1.15.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cryptography->PyFunceble-dev==3.3.3) (1.14.2)
Requirement already satisfied, skipping upgrade: toml; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.12)
Requirement already satisfied, skipping upgrade: msgpack; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (1.0.0)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (2020.6.20)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (1.25.10)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (2.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (3.0.4)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: pycparser in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cffi!=1.11.3,>=1.8->cryptography->PyFunceble-dev==3.3.3) (2.20)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from ruamel.yaml; extra == "all"->python-box[all]->PyFunceble-dev==3.3.3) (0.2.2)
Building wheels for collected packages: PyFunceble-dev
Building wheel for PyFunceble-dev (setup.py) ... done
Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=320945 sha256=a6e60709481d21cd77e91482300d11f62cd5404bb1752ba97a4f017ff218cba9
Stored in directory: /tmp/pip-ephem-wheel-cache-9rw_8j2s/wheels/49/58/3a/77d097367acd934fcfa7ff9e67a79eed3dd9610929f3b5a4fe
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3
pyfunceble -db --database-type mariadb --syntax -m -a -f /tmp/adult.mypdns.cloud
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Subject Status Expiration Date Source Analyze Date
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- --------------------
Traceback (most recent call last):
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
result.read()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
first_packet = self.connection._read_packet()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
packet.raise_for_error()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1209, in tool
raise exception
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1194, in tool
Dispatcher(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 131, in __init__
self.dispatch_file_test(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 174, in dispatch_file_test
PyFunceble.core.Multiprocess(file_path, "domain").run_test()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 566, in run_test
self.__run_multiprocess_test(shadow_file, manager, tracker=tracker)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 422, in __run_multiprocess_test
self.__start_process(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 303, in __start_process
if subject in self.autocontinue:
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/engine/auto_continue.py", line 152, in __contains__
db_session.query(Status)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3463, in one
ret = self.one_or_none()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3432, in one_or_none
ret = list(self)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3508, in __iter__
return self._execute_and_instances(context)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
result.read()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
first_packet = self.connection._read_packet()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
packet.raise_for_error()
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
err.raise_mysql_exception(self._data)
File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
[SQL: SELECT pyfunceble_status.id AS pyfunceble_status_id, pyfunceble_status.created AS pyfunceble_status_created, pyfunceble_status.modified AS pyfunceble_status_modified, pyfunceble_status.file_id AS pyfunceble_status_file_id, pyfunceble_status.tested AS pyfunceble_status_tested, pyfunceble_status._status AS pyfunceble_status__status, pyfunceble_status.status AS pyfunceble_status_status, pyfunceble_status._status_source AS pyfunceble_status__status_source, pyfunceble_status.status_source AS pyfunceble_status_status_source, pyfunceble_status.domain_syntax_validation AS pyfunceble_status_domain_syntax_validation, pyfunceble_status.expiration_date AS pyfunceble_status_expiration_date, pyfunceble_status.http_status_code AS pyfunceble_status_http_status_code, pyfunceble_status.ipv4_range_syntax_validation AS pyfunceble_status_ipv4_range_syntax_validation, pyfunceble_status.ipv4_syntax_validation AS pyfunceble_status_ipv4_syntax_validation, pyfunceble_status.ipv6_range_syntax_validation AS pyfunceble_status_ipv6_range_syntax_validation, pyfunceble_status.ipv6_syntax_validation AS pyfunceble_status_ipv6_syntax_validation, pyfunceble_status.subdomain_syntax_validation AS pyfunceble_status_subdomain_syntax_validation, pyfunceble_status.url_syntax_validation AS pyfunceble_status_url_syntax_validation, pyfunceble_status.is_complement AS pyfunceble_status_is_complement, pyfunceble_status.test_completed AS pyfunceble_status_test_completed, pyfunceble_status.tested_at AS pyfunceble_status_tested_at
FROM pyfunceble_status INNER JOIN pyfunceble_file ON pyfunceble_file.id = pyfunceble_status.file_id
WHERE pyfunceble_status.tested = %(tested_1)s AND pyfunceble_file.path = %(path_1)s]
[parameters: {'tested_1': '3600', 'path_1': '/tmp/adult.mypdns.cloud'}]
(Background on this error at: http://sqlalche.me/e/13/f405) DB visual layout after this as a site note I can confirm this is happening, when I try to load the sql dump, do to the keyrefs vs the read-in order directly with With:
I don't know if it me, who is tired here, but it looks like at step back from previously... |
@spirillen, Can you test this with 4.0.0 ? |
Closing. Feel free to test and reopen if not fixed or if necessary. |
Could you make a temporary "Re-test" project and throw these into... I can't keep up. |
@spirillen Okay. Done. @\me and I'll add them in there. |
/REOPEN idna_subject LIKE "%mypdns.cloud%" tested with the smaller adult.mypdns.cloud Result,
The zone should be subtracted from the test, only validating the actually record. In this case
Still have to test with |
Results of Test file
Versus the My POV, this test test failed. |
@spirillen, I understand your POV about I'm ready to implement the testing without the |
Appendix to previous comment as I have expanded the number of test records
bash /var/storage/repos/github/PyFunceble-Templates/pyfunceble-miniconda/pyfunceble4-arm7.sh /home/pi/Downloads/adult.list
I'm your hungry cheese missing a piece...
Are you ready to go hunting the red pills
so we can hunt down evil ghosts
Are you ready to start?
loading kacman...
Enter output directory for test results: /tmp/pyfunceble4/2343
What version would you like to use?
1. master
2. dev
3. 4.alpha
Which version of PyFunceble would you like to use?: pyfunceble4: 3
Already on '4.0.0-dev'
Your branch is up to date with 'origin/4.0.0-dev'.
Enter any custom test string: --syntax --rpz --no-files
Would you like to use your default pyfunceble enviroment?: [Y/n] Y
pyfunceble 4.0.0a21.dev (Blue Duckling)
Fatal Error: Unexpected '/' in '127.0.0.53/32' |
Let's not disagree about that, but it would be needed if we should be able to load a zone directly from a DNS server at some point. I'm open to make a discussion about this 😃 see DISC: #149
This will actually also make it possible to test whether an RPZ is set up correct 😋 delete the SOA and hammer your DNS, now everything should be in compliance with your RPZ zone 👍 |
This patch fixes #86. Indeed, before this patch, the SOA was not split of a given RPZ policy. This patch fixes that by collecting the SOAs and then splitting them while decoding the RPZ policies. Contributors: * @spirillen
Unfortunately I'm probably not able to test this until morning, the URL test is still running |
TEST OK |
Is your feature request related to a problem? Please describe.
As I just do a
--syntax
test on my rpz.mypdns.org zone file I can see a problem....The challenge is that a RPZ record not have the same rules for how to be syn-taxed as RFC:1034, it have it's own style which looks almost identically to a standard zone, but yet not.
As the test is going on, i see more and more valid Response Policy Zone records as INVALID
These records are valid (beside the two with
rpz-passthru.
) they should have been listed in thepointer
column.Describe the solution you'd like
I request you to add a new switch
--rpz
which is compliant with the draft-vixie-dns-rpz-04Describe alternatives you've considered
None, currently
Additional context
I have made a Easier to read and understand of how RPZ records should be formed here https://www.mypdns.org/w/rpz/#the-quot-passthru-quot-action
With the
--rpz
switch it should also be possible to validate an entire RPZ zone file (you will love this on your own DNS setups 😃 ) to quickly find any typos, or other malformed records that might occur from a program.--rpz
could/should support the following sources--rpz -f
= Local file--rpz --something (hostname|IPv(4|6))(:port)
= take the zone from the remote DNS server.The current test is done with pyfunceble 3.0.8.dev (Teal Blauwbok)
The text was updated successfully, but these errors were encountered: