WO2018110735A1 - Procédé de collecte de données intégrée auprès d'une pluralité d'api ouvertes - Google Patents

Procédé de collecte de données intégrée auprès d'une pluralité d'api ouvertes Download PDF

Info

Publication number
WO2018110735A1
WO2018110735A1 PCT/KR2016/014685 KR2016014685W WO2018110735A1 WO 2018110735 A1 WO2018110735 A1 WO 2018110735A1 KR 2016014685 W KR2016014685 W KR 2016014685W WO 2018110735 A1 WO2018110735 A1 WO 2018110735A1
Authority
WO
WIPO (PCT)
Prior art keywords
open api
request
call
open
data
Prior art date
Application number
PCT/KR2016/014685
Other languages
English (en)
Korean (ko)
Inventor
원종규
권준성
정민규
고광민
Original Assignee
주식회사 와이즈넛
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 와이즈넛 filed Critical 주식회사 와이즈넛
Publication of WO2018110735A1 publication Critical patent/WO2018110735A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present invention relates to an integrated data collection method from a plurality of open API, and more particularly, to a data collection method that can collectively collect data from various open API servers through one integrated open API collector.
  • an open API Open Application Programming Interface
  • an open API refers to a connection protocol that allows other systems to call and utilize a specific function when developing an application program.
  • the open API is to expose the API to the outside world so that an unspecified number of users can easily develop and use the application program easily.
  • program developers can easily develop applications with just a few open APIs without knowing the operating system's detailed features.
  • An object of the present invention is to provide an integrated data collection method from a plurality of open API that can collectively collect and manage a large amount of data from various open API servers using one general purpose open API collector. .
  • Integrated data collection method from a plurality of open API for achieving the above object, to collect the data through the N open API server connected to each of the N different external data server integrated
  • a method comprising: an integrated open API collector, for each of the open API servers, classifying a single call method or a multiple call method according to a calling method for collecting target data; Defining a standardized request variable by standardizing a request variable of a corresponding open API according to the calling method; Generating different request URLs according to the calling method using the standardized request variable; Transmitting the request URL to a corresponding open API server; And parsing and loading a response result from the open API server.
  • the single call method may call the open API once to collect the target data, and the multi-call method may call the plurality of open APIs to collect the target data.
  • the single call request URL may include a base URL for each open API, user authentication information, date information, and a call variable set for the target data.
  • the multi-call request URL may include a basic URL, user authentication information, and paging information for each open API.
  • the method may further include verifying a response result collected from the open API server.
  • the method may further include managing a schedule for transmitting the request URL.
  • the integrated data collection method from the plurality of open APIs it is not necessary to implement a separate data collector for each open API service, and it is possible to integrate and collect data through one integrated open API. have.
  • all open APIs can be integratedly managed by dividing them into a single call method and a multiple call method according to the calling method of each open API service.
  • request variables are different for each Open API service, and common and necessary items are normalized to define standardized request variables (that is, request variables for single call or multiple call request variables). By creating a URL, you can collect data for all open API services.
  • FIG 1 shows an overall system for the practice of the present invention.
  • FIG. 2 illustrates a configuration of an integrated open API collector according to an embodiment of the present invention.
  • FIG. 3 is a flowchart illustrating a method of collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.
  • the component when one component 'transmits' data to another component, the component may directly transmit the data to the other component, or via at least one other component. This means that the data can be transmitted to the other component.
  • one component 'directly transmits' data to another component it means that the data is transmitted from the component to the other component without passing through another component. The same can be applied to the case of 'receiving' data.
  • An application programming interface is a language or message format used when an application communicates with a system program, such as an operating system or database management system.
  • An API is implemented by calling a function that provides a link to a specific subroutine for execution within a program.
  • Open API is an open API that allows Internet users to develop their own applications and services, instead of being provided with web search results and user interfaces (UIs).
  • FIG. 1 shows an overall system for the practice of the present invention.
  • the entire system connects the integrated open API collector 100, the plurality of data servers 10, the open API server 20 connected to each server, and each open API server 20 and the integrated open API collector 100.
  • Network 40 The entire system connects the integrated open API collector 100, the plurality of data servers 10, the open API server 20 connected to each server, and each open API server 20 and the integrated open API collector 100.
  • the data server 10 is a server operated by a content provider such as a public institution, a company, a portal site, and stores content in a database.
  • a content provider such as a public institution, a company, a portal site, and stores content in a database.
  • Each data server 10 has an open API server 20 individually according to the data attributes of the stored content.
  • the open API server 20 When the open API server 20 receives an open API service request from the integrated open API collector 100, the open API server 20 transmits the open API service request to the data server 10 to process the request.
  • the open API server 20 transmits the processed response result to the integrated open API collector 100.
  • the response result is provided in a standardized web data format such as XML (eXtensible Markup Language) or JSON (JavaScript Object Notation).
  • the protocol for transmitting the response result uses a protocol of a web standard, for example, HTTP (hypertext transfer protocol) 1.0 or higher.
  • the open API server 20 may include a database storing registered API service information, a database storing API service statistical information, a database storing user authentication information, and the like.
  • the open API server 20 may use various standards, and one or more of REST (Representational State Transfer), SOAP (Simple object access, Protocol), RSS / ATOM, and Resource Description Framework (RDF) may be used. have.
  • REST Real State Transfer
  • the integrated open API collector 100 includes a collector manager 110, a database 130, and a collection robot 120.
  • the collecting robot 120 requests an open API service from the open API server 20 by receiving a command of the collector manager 110 and collects the result data.
  • the collector manager 110 includes a schedule manager 112, a collection robot manager 114, a data manager 116, and a request setup unit 118.
  • the schedule manager 112 manages a schedule of a crawling operation in which the collecting robot 120 requests an open API service from the open API server 20 and receives the result data. For example, in the case of the data server 10 where new data is updated every day, such as a news website, the schedule manager 112 may set a schedule for the collection robot 120 to request an open API service every day or every fixed period.
  • the collecting robot manager 114 controls all operations of the collecting robot 120 based on the setting contents of the request setting unit 118.
  • the data manager 116 parses and processes the data collected by the collection robot 120 from the open API server 20 and loads the data into the database 130.
  • the request setting unit 118 is configured to open API services based on request parameters and API calling methods required by each open API server 20 before data collection in order to efficiently integrate and manage various open API services. Normalize and save the setting information.
  • setting information includes basic information, standardized request variables (single call request variables, multiple call request variables), request confirmation information, and result parsing information.
  • the request setting unit 118 sets basic information about API rules of each open API service as shown in Table 1 below.
  • Basic Information Explanation API ID Enter an ID value representing the API API name Enter a name for the API collection settings Provider Enter the provider that provides the API Explanation Describe the API to collect SPEC URL Enter a URL that describes the API provider.
  • the manner in which the integrated open API collector 100 calls an open API service to collect target data depends on the characteristics of each open API server 20. For example, you can collect the target data you want with a single call, which is called a single call. Multiple pages must be called before the target data is collected.
  • the request setting unit 118 sets standardized request variables by standardizing request variables required by each open API service. Standardized request variables can be divided into single call request variables and multiple call request variables.
  • Table 2 below shows an example of a single call request variable.
  • Request variable for single call Explanation Call type Divided into single call and multiple call Encoding Set encoding of response data
  • Primary API URL Enter the base URL that comes first when constructing URL API key parameters Enter the parameter name to call the API key API key value Enter a parameter value to call the API key Date Information Register date information as used or not used. In case of use, start date and end date can be set.
  • 'base API URL' represents the base URL for each Open API service
  • 'API key parameters' and 'API key values' represent the user authentication information
  • 'call variable set' represents target data from the off API service. Represents a set of call variables that specify.
  • Table 3 below shows exemplary request variables for multiple calls.
  • Request variable for multiple calls Explanation Call type Divided into single call and multiple call (FETCH) Encoding Set encoding of response data
  • Primary API URL Enter the base URL that comes first when constructing URL API key parameters Enter the parameter name to call the API key API key value Enter the parameter value to call the API key. Multiple keys can be registered by separating them with commas Timeout (ms) Set response wait time when calling URL Total DATA TAG Enter the tag value that contains the value to get the total number of data that can be answered in the API call.
  • Number of Calls per API Key Enter the maximum number of calls to the authorized API key to call the API.
  • Paging parameter name Enter paging parameter name when configuring URL
  • Paging Start Value Enter the paging value that starts when calling the API Paging Increase Width Input the number of data per page when moving page API request URL Check the completed request URL
  • the request setting unit 118 may set request confirmation information for verifying whether the response is successful when calling the API.
  • Request confirmation information Explanation Enable request verification Set to "on” to see if the response was successful
  • Request Confirmation TAG Enter tag name to check when response is successful
  • Request Verification VALUE Enter the value of the tag to check when the response is successful
  • the request setting unit 118 may set the result parsing information as a storage rule for the target data received from the open API server 20, that is, a response element.
  • ROW TAG Enter a repeated tag name to distinguish each data in the data when calling the API Duplicate Check TAG When checking for duplicates when saving data, enter the ID value-If not, do not check duplicates.
  • ELEMENT Enter tag name of desired data-Enter tag value in child tag of repeated row tag DATA TYPE Select the data type of the data-VARCHAR, TEXT, CHAR, DATE, NUMBER, DOUBLE SIZE In case of VARCHAR and CHAR, the size of the relevant data type can be input.
  • META Enter a description of the data
  • FIG. 3 is a flowchart illustrating a method of collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.
  • the integrated open API collector 100 performs the following process to collectively collect and manage data from N open API servers 20 respectively connected to different N external data servers 10.
  • the integrated open API collector 100 classifies each open API server 20 according to a calling method for collecting target data (S10). Specifically, when the target data can be collected in one call, the call is classified into a single call method, and when a plurality of calls are required, the call is classified into a multiple call method.
  • S10 target data
  • the integrated open API collector 100 defines a standardized request variable according to the calling method of the open API service (S20). Specifically, a single call request variable is defined for a single call open API server, and a multiple call request variable is defined for an open API service of a multi call type.
  • the integrated open API collector 100 generates a request URL according to the calling method (S30).
  • the single call request URL is configured to include a base URL, user authentication information, date information, and a call variable set for each open API.
  • the combined expression of the single call request URL may be as shown in Equation 1 below.
  • Equation 2 The combination expression of the request URL for multiple calls may be as shown in Equation 2 below.
  • the integrated open API collector 100 individually transmits each request URL to the open API server 20 corresponding thereto (S40).
  • the schedule manager 112 may manage a schedule for transmitting the request URL, and the collection robot 120 transmits the request URL to the open API server 20 according to a preset schedule.
  • the open API server 20 receiving the request URL transmits the response result of processing the request to the integrated open API collector 100.
  • the integrated open API collector 100 verifies the success of the response result using the request confirmation information (S50).
  • the integrated open API collector 100 parses and processes the response result and loads it in the database (S60).
  • the integrated open API collector and the data collection method using the same may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium.
  • the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination.
  • the program instructions recorded on the media may be those specially designed and constructed for the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts.
  • Examples of computer readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, Hardware devices specifically configured to store and execute program instructions, such as flash memory, may be included. Examples of program instructions include machine code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter. Hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

L'invention concerne un procédé de collecte de données intégrée auprès d'une pluralité d'API ouvertes, le procédé permettant la collecte et la gestion intégrées d'un grand ensemble de données provenant de divers serveurs d'API ouvertes en utilisant un collecteur d'API ouvertes universel unique. Le procédé de collecte de données est un procédé dans lequel une collecte de données intégrée est effectuée au moyen d'un nombre N de serveurs d'API ouvertes connectés à un nombre N de serveurs de données externes respectifs, et comprend les étapes suivantes, exécutées par un collecteur d'API ouvertes intégré : classification de chaque serveur d'API ouvertes comme ayant un procédé d'appel unique ou un procédé d'appel multiple conformément à un procédé d'appel pour collecter des données cibles ; normalisation des variables de demande d'une API ouverte conformément au procédé d'appel correspondant, et définition les variables de demande normalisées ; génération d'URL de demande mutuellement différentes conformément au procédé d'appel au moyen des variables de demande normalisées ; transmission des URL de demande aux serveurs d'API ouvertes correspondants ; puis analyse et chargement des réponses résultantes depuis les serveurs d'API ouvertes.
PCT/KR2016/014685 2016-12-12 2016-12-15 Procédé de collecte de données intégrée auprès d'une pluralité d'api ouvertes WO2018110735A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020160168270A KR101950074B1 (ko) 2016-12-12 2016-12-12 복수의 오픈 api로부터 통합적인 데이터 수집방법
KR10-2016-0168270 2016-12-12

Publications (1)

Publication Number Publication Date
WO2018110735A1 true WO2018110735A1 (fr) 2018-06-21

Family

ID=62558656

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/014685 WO2018110735A1 (fr) 2016-12-12 2016-12-15 Procédé de collecte de données intégrée auprès d'une pluralité d'api ouvertes

Country Status (2)

Country Link
KR (1) KR101950074B1 (fr)
WO (1) WO2018110735A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240273193A1 (en) * 2023-02-15 2024-08-15 Oracle International Corporation Advanced persistent threat detection

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102145448B1 (ko) * 2018-11-15 2020-08-18 (주)에스이랩 기상 기후 자료의 유통 서비스를 제공하는 서비스 제공 장치 및 유통 서비스 제공 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100062785A (ko) * 2008-12-01 2010-06-10 한국전자통신연구원 오픈 ａｐｉ 기반 웹포털 서비스 제공 방법 및 시스템
KR20110070724A (ko) * 2009-12-18 2011-06-24 한국전자통신연구원 오픈 ａｐｉ 통합 검색 및 매쉬업 블록 기반코드 생성 장치 및 그 방법
KR20120136666A (ko) * 2011-06-09 2012-12-20 주식회사 피씨엔 오픈 ａｐｉ 기반 콘텐츠 서비스 인터페이스 제공 시스템 및 방법
US20140297809A1 (en) * 2013-03-27 2014-10-02 Samsung Electronics Co., Ltd. Apparatus and method for processing multiple open apis
KR101600616B1 (ko) * 2015-05-28 2016-03-07 동서울대학교 산학협력단 이종콘텐츠 분석 서비스 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100062785A (ko) * 2008-12-01 2010-06-10 한국전자통신연구원 오픈 ａｐｉ 기반 웹포털 서비스 제공 방법 및 시스템
KR20110070724A (ko) * 2009-12-18 2011-06-24 한국전자통신연구원 오픈 ａｐｉ 통합 검색 및 매쉬업 블록 기반코드 생성 장치 및 그 방법
KR20120136666A (ko) * 2011-06-09 2012-12-20 주식회사 피씨엔 오픈 ａｐｉ 기반 콘텐츠 서비스 인터페이스 제공 시스템 및 방법
US20140297809A1 (en) * 2013-03-27 2014-10-02 Samsung Electronics Co., Ltd. Apparatus and method for processing multiple open apis
KR101600616B1 (ko) * 2015-05-28 2016-03-07 동서울대학교 산학협력단 이종콘텐츠 분석 서비스 방법

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240273193A1 (en) * 2023-02-15 2024-08-15 Oracle International Corporation Advanced persistent threat detection

Also Published As

Publication number Publication date
KR101950074B1 (ko) 2019-02-19
KR20180067018A (ko) 2018-06-20

Similar Documents

Publication Publication Date Title
US8396939B2 (en) Content distribution management device, terminal, program, and content distribution system
KR101622815B1 (ko) 소프트웨어 애플리케이션의 최종사용자에 대한 지원제공 방법
US7334039B1 (en) Techniques for generating rules for a dynamic rule-based system that responds to requests for a resource on a network
JP4865581B2 (ja) コンテンツ流通管理装置及びコンテンツ流通システム
US20130297973A1 (en) Automated Conformance and Interoperability Test Lab
CN102694857A (zh) 用于在数据网络上赠送的方法和设备
US7243138B1 (en) Techniques for dynamic rule-based response to a request for a resource on a network
CN102971707A (zh) 为软件包安装配置计算机系统
CN102043665A (zh) 知晓系统全景的应用间通信基础架构
CN100531195C (zh) 为在不同协议下可用的应用提供服务的系统
JP2023531196A (ja) ネットワーク管理システム、方法、装置及び電子機器
CN105893055A (zh) 流程引擎平台化触发方法
CN110932918A (zh) 日志数据采集方法、装置及存储介质
CN107894945A (zh) 埋点添加方法、移动终端及计算机可读存储介质
WO2018110735A1 (fr) Procédé de collecte de données intégrée auprès d'une pluralité d'api ouvertes
CN116414370A (zh) 基于低代码的平台构建方法、装置、介质及电子设备
KR101695636B1 (ko) 복합 상황 정보 이벤트 처리 방법 및 그 장치
CN105955790A (zh) 数据处理方法及装置
CN111045928A (zh) 一种接口数据测试方法、装置、终端及存储介质
CN107643858A (zh) 一种信息处理方法、终端和服务器
US20070233812A1 (en) Common communication framework for network objects
US8200713B2 (en) Database exploration for building wireless component applications
CN110837612B (zh) 统一资源标识符uri数据的获取方法和装置、存储介质
KR20190080418A (ko) 사물인터넷 서비스 제공 장치 및 사물인터넷 서비스 제공 방법
JP2002222236A (ja) 製品情報提供装置、製品情報の提供方法、そのためのプログラムおよび記録媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16923662

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16923662

Country of ref document: EP

Kind code of ref document: A1