-
-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updatating the DNS examples to non tracking #122
Changes from 1 commit
67c329e
525dfee
1aa1b75
7aef5c7
8a953f2
3c6f5d2
f1338d9
7c16e21
0c75af5
4d37e11
e237226
3bd95cc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are still room for improvement as I currently don't know the full intention behind all the tables 😒 @funilrys Please enlighten me, so we can get the docs up-to-date
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,6 +6,7 @@ Why do we use "databases"? | |
|
||
We use databases to store data while we run the tests. When globally talking | ||
about databases, we are indirectly talking about the following subsystems. | ||
(JSON) | ||
|
||
* Autocontinue | ||
* InactiveDB | ||
|
@@ -26,41 +27,45 @@ Warnings around Database (self) management | |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
.. warning:: | ||
If you plan to delete everything and still manage to use PyFunceble in the | ||
future, please use the :code:`--clean-all` argument. | ||
If you plan to delete everything and still manage to use PyFunceble | ||
in the future, please use the :code:`--clean-all` argument. | ||
|
||
Indeed, it will delete everything which is related to what we generated, | ||
except things like the whois database file/table which saves (almost) | ||
static data which can be reused in the future. | ||
|
||
Deleting, for example, the whois database file/table will just make | ||
your test run for a much longer time if you retest subject that used to be | ||
indexed into the whois database file/table. | ||
your test run for a much longer time if you retest subject that used | ||
to be indexed into the whois database file/table. | ||
|
||
Databases types | ||
^^^^^^^^^^^^^^^ | ||
|
||
Since PyFunceble :code:`2.0.0` (equivalent of :code:`>=1.18.0.dev`), | ||
we offer multiple database types which are (as per configuration) :code:`json` | ||
(default), :code:`mariadb` and :code:`mysql`. | ||
we offer multiple database types which are (as per configuration) | ||
:code:`json` (default), :code:`mariadb` and :code:`mysql`. | ||
|
||
We however only offers our support time for opensource software, hence | ||
Oracles MySql are not supported. if it works, great. However they should | ||
be pretty good covered through SQLAlchemy <https://docs.sqlalchemy.org/> | ||
|
||
Why different database types? | ||
""""""""""""""""""""""""""""" | ||
|
||
With the introduction of the multiprocessing logic, it became natural to | ||
introduce other database format as it's a nightmare to update a JSON formatted | ||
file. | ||
introduce other database format as it's a nightmare to update a JSON | ||
formatted file. | ||
|
||
In order to write or use a JSON formatted database, we have to load it and | ||
overwrite it completely. | ||
It's great while working with a single CPU/process but as soon as we get out of | ||
that scope it become unmanageable. | ||
It's great while working with a single CPU/process but as soon as we get | ||
out of that scope it become unmanageable. | ||
|
||
How to use the :code:`mysql` or :code:`mariadb` format? | ||
How to use the :code:`mariadb` or :code:`mysql` format? | ||
""""""""""""""""""""""""""""""""""""""""""""""""""""""" | ||
|
||
1. Create a new user, password and database (optional) for PyFunceble to work | ||
with. | ||
1. Create a new user, password and database (optional) for PyFunceble to | ||
work with. | ||
|
||
2. Create a :code:`.pyfunceble-env` file at the root of your configuration | ||
directory. | ||
|
@@ -77,21 +82,21 @@ How to use the :code:`mysql` or :code:`mariadb` format? | |
PYFUNCEBLE_DB_USERNAME=pyfunceble | ||
|
||
.. note:: | ||
Since version :code:`2.4.3.dev` it is possible to use the UNIX socket | ||
for the :code:`PYFUNCEBLE_DB_HOST` environment variable. | ||
Since version :code:`2.4.3.dev` it is possible to use the UNIX | ||
socket for the :code:`PYFUNCEBLE_DB_HOST` environment variable. | ||
|
||
The typical location for :code:`mysqld.sock` is | ||
:code:`/var/run/mysqld/mysqld.sock`. | ||
|
||
This have been done to make | ||
|
||
1. It easier to use the :code:`socket` in conjunction with a supported CI | ||
environment/platform. | ||
1. It easier to use the :code:`socket` in conjunction with a | ||
supported CI environment/platform. | ||
|
||
2. Leaving more space on the IP-stack on local DB installations. | ||
|
||
3. The :code:`UNIX:SOCKET` is usually faster than the IP connection on | ||
local runs. | ||
3. The :code:`UNIX:SOCKET` is usually faster than the IP | ||
connection on local runs. | ||
|
||
:: | ||
|
||
|
@@ -102,10 +107,71 @@ How to use the :code:`mysql` or :code:`mariadb` format? | |
PYFUNCEBLE_DB_PORT=3306 | ||
PYFUNCEBLE_DB_USERNAME=pyfunceble | ||
|
||
4. Switch the :code:`db_type` index of your configuration file to :code:`mysql` | ||
or :code:`mariadb`. | ||
4. Switch the :code:`db_type` index of your configuration file to | ||
:code:`mariadb` or :code:`mysql`. | ||
5. Play with PyFunceble! | ||
|
||
.. note:: | ||
If the environment variables are not found, you will be asked to prompt the | ||
information. | ||
If the environment variables are not found, you will be asked to | ||
prompt the information. | ||
|
||
SQL Layout: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As of a possible merge of https://github.com/funilrys/PyFunceble/tree/issue-118 in the coming days, All INT (except for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll take note of that. Why I've left it with a "open" wildcard What about my question marks in the Another thing I just noticed, it the lines L73-L108 even relevant anymore after changing to sqlachemy? |
||
^^^^^^^^^^^ | ||
|
||
The layout and data within the Sql database and how they are used should | ||
currently be following this patterns. | ||
|
||
alembic_version | ||
- version_num The Current version of Pyfunceble in number | ||
|
||
pyfunceble_file | ||
- :code:`id` Primary key, auto_increment | ||
- :code:`created` creation date of the record | ||
- :code:`modified` Data the record was last tested, altered | ||
- :code:`path` source of the file tested. URI or File_path | ||
- :code:`test_completed` (bool) this data is used for picking up a | ||
interrupted (broken) test or in CI for auto-continue :code:`-c` | ||
|
||
pyfunceble_mined | ||
- :code:`id` Primary key, auto_increment | ||
- :code:`created` creation date of the record | ||
- :code:`modified` Data the record was last tested, altered | ||
- :code:`subject_id` key_ref to :code:`pyfunceble_status.id` | ||
- :code:`file_id` key_ref to :code:`pyfunceble_file.id` | ||
- :code:`mined` the full fqdns results of a :code:`--mining` response | ||
|
||
pyfunceble_status | ||
- :code:`id` Primary key, auto_increment | ||
- :code:`created` creation date of the record | ||
- :code:`modified` Data the record was last tested, altered | ||
- :code:`file_id` (one to many relation) to :code:`pyfunceble_file.id` | ||
This is used to extract where a record comes from. | ||
- :code:`tested` IS the actual record tested in full (domain/URI) | ||
- :code:`_status` ACTIVE/INACTIVE status from the PyFunceble test (Twice??) | ||
- :code:`status` ACTIVE/INACTIVE status from the PyFunceble test (Twice??) | ||
- :code:`_status_source` The technique to determine the status WHOIS/DNSLOOKUP (Twice??) | ||
- :code:`status_source` The technique to determine the status WHOIS/DNSLOOKUP (Twice??) | ||
- :code:`domain_syntax_validation` (*INT???) Would expect a (bool(true,false)). Here I'm in doubt: Does this mean there was performed a :code:`--syntax` test OR if it (0= failed, 1= past) syntax test? | ||
- :code:`expiration_date` domain expiration date from a successful WHOIS response (shouldn't it be served true the whois table???) | ||
- :code:`http_status_code` the HTTP code from a lookup, example: 200 = | ||
succes, 404 file not found (suggested to be moved to new table see <https://www.mypdns.org/T1250#19039> for reusable data) | ||
- :code:`ipv4_range_syntax_validation` (*INT???) Would expect a (bool(true,false)) | ||
- :code:`ipv4_syntax_validation` (*INT???) Would expect a (bool(true,false)) | ||
- :code:`ipv6_range_syntax_validation` (*INT???) Would expect a (bool(true,false)) | ||
- :code:`ipv6_syntax_validation` (*INT???) Would expect a (bool(true,false)) | ||
- :code:`subdomain_syntax_validation` ?? but from current data set I would again expect a (bool) and not (*INT) as it is 0 OR 1 | ||
- :code:`url_syntax_validation` (*INT???) Would expect a (bool(true,false)). Here I'm in doubt: Does this mean there was performed a :code:`--syntax` test OR if it (0= failed, 1= past) syntax test? | ||
- :code:`is_complement` is this record from a :code:`--complement` test. (*INT???) Would expect a (bool(true,false)) | ||
- :code:`test_completed` (*INT???) Would expect a (bool(true,false)) Have we done testing this record since last commit for test. | ||
- :code:`tested_at` The date for last succeeded tested. | ||
|
||
pyfunceble_whois_record | ||
- :code:`id` Primary key, auto_increment | ||
- :code:`created` creation date of the record | ||
- :code:`modified` Data the record was last tested, altered | ||
- :code:`subject` the domain for which this record is stored | ||
- :code:`expiration_date` The domain expiration data according to the WHOIS | ||
- :code:`epoch` The domain expiration data according to the WHOIS. **just in EPOC format** | ||
- :code:`state` the domain state based on `expiration_date` and/or `epoc` future/past | ||
- :code:`record` [NULL]?? would expect a key_ref to :code:`pyfunceble_status.id` and as replacement for :code:`pyfunceble_whois_record.subject` | ||
- :code:`server` the whois server holding the WHOIS data (Should be altered to separate table/(DB) for reusable data and gaining from db.cache and minimize I/O & DB size) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where does this come from?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Line break. (The 80 chars for consoles support)
I added it as this part is about JSON and does have nothing to do with the SQL layout