Skip to content

Commit

Permalink
V0.6.0 Pre
Browse files Browse the repository at this point in the history
  • Loading branch information
naibo committed Dec 4, 2023
1 parent 82dac5a commit cffaeeb
Show file tree
Hide file tree
Showing 25 changed files with 369 additions and 229 deletions.
2 changes: 1 addition & 1 deletion .temp_to_pub/EasySpider_Linux_x64/软件使用说明.txt
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ B站最新版特性视频已上传,新视频非常有用,推荐大家观看
注意,v0.3.1版本任务task文件夹内`.json`文件和之前所有版本均不兼容,请重新设计v0.3.1版本任务。

## 更新说明
1. 高级操作
1. 自定义操作
- 可以在任务流程中**执行自定义脚本**,包括在浏览器中**执行Javascript指令**以及**操作系统级别的脚本调用**并可**得到命令返回值并记录**,大大扩展了可操作空间。

![image](https://github.com/NaiboWang/EasySpider/assets/30287768/06e63a06-328d-4339-b40b-2d57c94cee66)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ B站最新版特性视频已上传,新视频非常有用,推荐大家观看
注意,v0.3.1版本任务task文件夹内`.json`文件和之前所有版本均不兼容,请重新设计v0.3.1版本任务。

## 更新说明
1. 高级操作
1. 自定义操作
- 可以在任务流程中**执行自定义脚本**,包括在浏览器中**执行Javascript指令**以及**操作系统级别的脚本调用**并可**得到命令返回值并记录**,大大扩展了可操作空间。

![image](https://github.com/NaiboWang/EasySpider/assets/30287768/06e63a06-328d-4339-b40b-2d57c94cee66)
Expand Down
2 changes: 1 addition & 1 deletion .temp_to_pub/EasySpider_windows_x32/软件使用说明.txt
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ B站最新版特性视频已上传,新视频非常有用,推荐大家观看
注意,v0.3.1版本任务task文件夹内`.json`文件和之前所有版本均不兼容,请重新设计v0.3.1版本任务。

## 更新说明
1. 高级操作
1. 自定义操作
- 可以在任务流程中**执行自定义脚本**,包括在浏览器中**执行Javascript指令**以及**操作系统级别的脚本调用**并可**得到命令返回值并记录**,大大扩展了可操作空间。

![image](https://github.com/NaiboWang/EasySpider/assets/30287768/06e63a06-328d-4339-b40b-2d57c94cee66)
Expand Down
2 changes: 1 addition & 1 deletion .temp_to_pub/EasySpider_windows_x64/软件使用说明.txt
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ B站最新版特性视频已上传,新视频非常有用,推荐大家观看
注意,v0.3.1版本任务task文件夹内`.json`文件和之前所有版本均不兼容,请重新设计v0.3.1版本任务。

## 更新说明
1. 高级操作
1. 自定义操作
- 可以在任务流程中**执行自定义脚本**,包括在浏览器中**执行Javascript指令**以及**操作系统级别的脚本调用**并可**得到命令返回值并记录**,大大扩展了可操作空间。

![image](https://github.com/NaiboWang/EasySpider/assets/30287768/06e63a06-328d-4339-b40b-2d57c94cee66)
Expand Down
Binary file modified ElectronJS/EasySpider_en.crx
Binary file not shown.
Binary file modified ElectronJS/EasySpider_zh.crx
Binary file not shown.
6 changes: 3 additions & 3 deletions ElectronJS/src/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ <h5 style="margin-top: 20px">选择语言/Select Language</h5>
<h4 style="margin-top: 20px">Copyright and Disclaimer</h4>
<p>Please carefully read the following instructions regarding the use of the software and commercial payments. If you agree, please accept the agreement.</p>
<textarea class="form-control" style="margin:0 auto;width:90%; color:black; height: 450px; min-height: 200px; background: white" readonly>
This software is intended for educational and communication purposes only. It is strictly prohibited to use the software for any illegal activities or operations, such as crawling government/military websites that are not allowed to be crawled. The user bears all consequences resulting from the use of this software and the author shall not be held responsible or liable in any way. Furthermore, the software is protected by patent rights. If you intend to use it for commercial purposes or profit-making activities, such as using the software for client orders, selling the collected data, please contact Hangzhou Tianqin Intellectual Property Agency Co., Ltd. (http:https://www.tqip.com/) for patent authorization and payment operations: https://www.patentguru.com/cn/search?q=一种自定义提取流程的服务封装系统
This software is intended for educational and communication purposes only. It is strictly prohibited to use the software for any illegal activities or operations, such as crawling government/military websites that are not allowed to be crawled. The user bears all consequences resulting from the use of this software and the author shall not be held responsible or liable in any way. Furthermore, the software is protected by patent rights. If you intend to use it for commercial purposes or profit-making activities, such as using the software for client orders, selling the collected data, please contact author: naibowang@foxmail.com for patent authorization and payment operations: https://www.patentguru.com/cn/search?q=一种自定义提取流程的服务封装系统
For individual users, EasySpider is a completely free and ad-free open-source software. The development and maintenance of the software rely solely on the author's voluntary efforts. Therefore, you can choose to support the author, allowing them to have more enthusiasm and energy to maintain this software. Alternatively, if you have profited from using this software, you are welcome to support the author through the following methods:

1. PayPal account: naibowang, or scan the QR code provided in the software package.
Expand Down Expand Up @@ -142,9 +142,9 @@ <h4 style="margin-top: 20px">Specify user data folder</h4>
<div v-else-if="lang=='zh'">
<div v-if="step == -1">
<h4 style="margin-top: 20px">版权声明和注意事项</h4>
<p>请仔细阅读下方有关软件使用和商用付费的说明,同意请接受协议</p>
<p>请接受下方使用协议以使用软件,不同意请退出</p>
<textarea class="form-control" style="margin:0 auto;width:90%; color:black; height: 480px; min-height: 200px; background: white" readonly>
本软件仅供学习交流使用,严禁使用软件进行任何违法违规的操作,如爬取不允许爬取的政府/军事机关网站等。使用本软件所造成的一切后果由使用者自负,与作者本人无关,作者不会承担任何责任。同时,软件受到专利权保护,如要用于商业用途,如使用软件进行盈利接单,出售采集到的数据等,请联系杭州天勤知识产权代理有限公司(http:https://www.tqip.com/)进行专利授权等付费操作:https://www.patentguru.com/cn/search?q=一种自定义提取流程的服务封装系统
本软件仅供学习交流使用,严禁使用软件进行任何违法违规的操作,如爬取不允许爬取的政府/军事机关网站等。使用本软件所造成的一切后果由使用者自负,与作者本人无关,作者不会承担任何责任。同时,软件受到专利权保护,如要用于商业用途,如使用软件进行盈利接单,用于公司业务,或出售采集到的数据等,请邮件联系作者:[email protected]进行专利授权等付费操作:https://www.patentguru.com/cn/search?q=一种自定义提取流程的服务封装系统

对于个人使用者来说,易采集EasySpider是一款完全免费无广告的开源软件,软件开发和维护全靠作者用爱发电,因此您可以选择支持作者让作者有更多的热情和精力维护此软件,或者您使用了此软件进行了盈利,欢迎您通过下面的方式支持作者:

Expand Down
33 changes: 25 additions & 8 deletions ElectronJS/src/taskGrid/FlowChart.html
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,12 @@ <h4 class="modal-title">Equivalent XPath</h4>
<input onkeydown="inputDelete(event)" class="form-control" v-model.number="nowNode['parameters']['scrollCount']" type="number" required></input>
<label>Wait time after scrolling (in seconds):</label>
<input onkeydown="inputDelete(event)" class="form-control" v-model.number="nowNode['parameters']['scrollWaitTime']" type="number" required></input>
<label>Way to handle pop-up windows after clicking:</label>
<p><select v-model='nowNode["parameters"]["alertHandleType"]' class="form-control">
<option :value = 0>No pop-up window</option>
<option :value = 1>Accept pop-up window</option>
<option :value = 2>Reject pop-up window (only for Confirm pop-up window)</option>
</select></p>
<p style="margin-top: 10px">
<a class="btn btn-primary" data-toggle="collapse" href="#collapseExample" role="button" aria-expanded="false" aria-controls="collapseExample">
Click here to expand/collapse advanced operations
Expand Down Expand Up @@ -324,7 +330,7 @@ <h4 class="modal-title">Equivalent XPath</h4>
<div class="elements" v-if="nodeType==4">
<p><input onkeydown="inputDelete(event)" type="checkbox" v-model='nowNode["parameters"]["iframe"]'></input>Element is inside iframe</p>
<div v-if="nowNode['isInLoop']">

<!-- 如果在循环内且循环内是固定文本才显示此行元素 -->
<p><input onkeydown="inputDelete(event)" type="checkbox" v-model='useLoop'></input>Use text from the loop (If unchecked, the text entered each time will be the text from the "Input Value" text box below. If checked, it will use the text set within the loop.)</p>
<p v-if="useLoop">
<label>Index value (0 represents using the entire current loop text. If greater than 0, it represents the text index value separated by "~" within the current loop. For example, if the current loop text value is A~B, index value 1 represents inputting A, 2 represents inputting B, and 0 represents inputting A~B)</label>
Expand Down Expand Up @@ -372,11 +378,14 @@ <h4 class="modal-title">Equivalent XPath</h4>
<option value = 5>Run Python code on current environment (the "exec" operation)</option>
<option value = 6>Get value of a Python expression (the "eval" operation)</option>
<option value = 7>Pause program execution (such as when the captcha box appears)</option>
<option value = 8>Refresh page</option>
</select>
<div v-if='nowNode["parameters"]["codeMode"] == 7'>
<label>This operation can pause program execution, such as when a captcha box appears, and it will not continue until you manually press and hold the p key.</label>
<label>This operation can pause program execution, such as when a captcha box appears, and it will not continue until you manually press and hold the pause/continue shortcut key (default: key p).</label>
</div>
<div v-if='nowNode["parameters"]["codeMode"] == 8'>
<label>This operation can refresh the current page.</label>
</div>

<div v-if='nowNode["parameters"]["codeMode"] < 3 || nowNode["parameters"]["codeMode"] >= 5 && nowNode["parameters"]["codeMode"] <=6'>
<label>Code (Use Field["FieldName"] to input the lastest value of a field): </label>
<textarea onkeydown="inputDelete(event)" class="form-control" rows="2" v-model='nowNode["parameters"]["code"]' placeholder="Please input a JavaScript command or a system command. For example, document.body.innerText = '1' is an example of a JavaScript command, and python D:/test.py is an example of a system command. If you choose to execute a JavaScript script for the current iteration, you can represent the element of the current iteration using arguments[0]. For instance, arguments[0].style.color = 'blue' sets the color of the element in the current iteration to blue."></textarea>
Expand Down Expand Up @@ -404,6 +413,7 @@ <h4 class="modal-title">Equivalent XPath</h4>
2. Return the value of a custom global variable: `self.myVar`
3. Return the result of a conditional statement: `self.myVar == 1`
4. Determining whether the value extracted from a certain field is equal to the value of a certain variable: self.outputParameters["field name"] == self.myVar

Please note that this feature does not support assigning values to variables. In other words, you cannot write something like `self.myVar = 1`. If you want to perform assignment operations, please select the previous option, "Run Python code on current environment (the "exec" operation)"</pre>
<p style="margin-top: 15px">Whether to record the output/return value of the execution as a field: </p>
<p><select v-model='nowNode["parameters"]["recordASField"]' class="form-control">
Expand Down Expand Up @@ -660,18 +670,23 @@ <h4 class="modal-title" id="myModalLabel">Save Task (Can press Ctrl + S to open
</select>
<label>Export File Name/Database Table Name (Can use ../ to represent relative path to change the file save location,the keyword "current_time" will be replaced with the timestamp when the task is executed):</label>
<input onkeydown="inputDelete(event)" value="current_time" id="saveName" class="form-control"></input>
<label>Is it an extreme anti-scraping website like Cloudflare (<a href="https://www.bilibili.com/video/BV1Ph4y1E7R9/" target="_blank">Watch Tutorial</a>)?</label>
<select id="cloudflare" name="cloudflare" class="form-control">
<option value=0>No</option>
<option value=1>Yes (Only support on Windows x64 platform)</option>
</select>
<!-- <label>Is it an extreme anti-scraping website like Cloudflare (<a href="https://www.bilibili.com/video/BV1Ph4y1E7R9/" target="_blank">Watch Tutorial</a>)?</label>-->
<!-- <select id="cloudflare" name="cloudflare" class="form-control">-->
<!-- <option value=0>No</option>-->
<!-- <option value=1>Yes (Only support on Windows x64 platform)</option>-->
<!-- </select>-->
<label>To modify the input parameters of each operation during execution, read the following Excel (.xlsx) file. Please click the "Read Input Parameters from Excel File" button when calling the task to view the file format:</label>
<input onkeydown="inputDelete(event)" id="inputExcel" name="inputExcel" class="form-control" placeholder="If left empty, input parameters will not be read from Excel. The file path is relative to the EasySpider folder, e.g., inputs/task1.xlsx"></input>
<label>Browser Emulation Type:</label>
<select id="environment" name="environment" class="form-control">
<option value=0>Desktop</option>
<option value=1>Mobile (Not supported under Cloudflare mode)</option>
</select>
<label>Whether to maximize the browser window:</label>
<select id="maximizeWindow" name="maximizeWindow" class="form-control">
<option value = 0>No</option>
<option value = 1>Yes</option>
</select>
<label>Save Data Every N Rows (Specify N below, the larger the value, the faster the scraping speed, but there is a risk of data loss if unexpectedly exited):</label>
<input onkeydown="inputDelete(event)" type="number" value="10" id="saveThreshold" name="saveThreshold" class="form-control"></input>
<label>Do you want to resume execution from the last saved position when unexpectedly exiting and restarting the task (The record interval of the number of collected items is the value set above)?</label>
Expand All @@ -686,6 +701,8 @@ <h4 class="modal-title" id="myModalLabel">Save Task (Can press Ctrl + S to open
<option value = 1>Yes</option>
<option value = 0>No</option>
</select>
<label>Pause/Continue Task Shortcut Key:</label>
<input onkeydown="inputDelete(event)" type="text" value="p" id="pauseKey" class="form-control"></input>

</div>
<div class="modal-footer">
Expand Down
Loading

0 comments on commit cffaeeb

Please sign in to comment.