KR101950074B1 - Method for comprehensively collecting data from different Open APIs - Google Patents

Method for comprehensively collecting data from different Open APIs Download PDF

Info

Publication number
KR101950074B1
KR101950074B1 KR1020160168270A KR20160168270A KR101950074B1 KR 101950074 B1 KR101950074 B1 KR 101950074B1 KR 1020160168270 A KR1020160168270 A KR 1020160168270A KR 20160168270 A KR20160168270 A KR 20160168270A KR 101950074 B1 KR101950074 B1 KR 101950074B1
Authority
KR
South Korea
Prior art keywords
open api
request
data
url
api
Prior art date
Application number
KR1020160168270A
Other languages
Korean (ko)
Other versions
KR20180067018A (en
Inventor
원종규
권준성
정민규
고광민
Original Assignee
주식회사 와이즈넛
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 와이즈넛 filed Critical 주식회사 와이즈넛
Priority to KR1020160168270A priority Critical patent/KR101950074B1/en
Priority to PCT/KR2016/014685 priority patent/WO2018110735A1/en
Publication of KR20180067018A publication Critical patent/KR20180067018A/en
Application granted granted Critical
Publication of KR101950074B1 publication Critical patent/KR101950074B1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

An integrated data collection method is provided from a plurality of open APIs capable of collecting and managing a large amount of data collectively from various open API servers using one universal open API collector. This data collection method is a method for collectively collecting data through N open API servers connected to different N external data servers, wherein an integrated open API collector collects target data for each open API server Classifying into a single paging method or a multi-paging method according to a paging method; Defining a standardized request variable by standardizing a request variable of the corresponding open API according to a calling method; Generating different request URLs according to the calling method using standardized request variables; Transmitting a request URL to the corresponding open API server; And parsing and loading the response result from the open API server.

Description

[0002] Methods for collecting data from multiple open APIs [

The present invention relates to an integrated data collection method from a plurality of open APIs, and more particularly, to a data collection method capable of integrally collecting data from various open API servers through one integrated open API collector.

Open API (Open Application Programming Interface) generally refers to a connection protocol that allows other systems to invoke a specific function when developing an application program. That is, the open API is to open the API to the outside so that an unspecified number of users can easily develop and utilize the application program. Using a set of protocols and tools that make it easy to create arbitrary applications, a program developer can develop an application easily with just a few open APIs, even if they do not know the detailed functions of the operating system.

Recently, many public and private companies have provided information to general consumers through open APIs. There are many individuals and companies that provide services such as mobile apps or web services using these open API services.

It is not difficult to construct an additional service system using one open API service. However, it is technically very difficult to provide a service by collectively collecting a large amount of data from a plurality of open API services. This is because the request variables and API calls required by each open API service are different. In this case, conventionally, it is necessary to implement a separate data collector for each open API service. However, since there are too many management items in terms of managing various services, it is difficult to manage them collectively.

An object of the present invention is to provide an integrated data collection method from a plurality of open APIs capable of collecting and managing a large amount of data collectively from various open API servers using one universal open API collector .

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other matters not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method for collecting data from a plurality of open APIs, the method comprising collectively collecting data through N open API servers connected to different N external data servers The method as claimed in claim 1, wherein the integrated open API collector comprises: classifying each open API server into a single call method or a multiple call method according to a call method for collecting target data; Defining standardized request variables by standardizing request variables of the corresponding open API according to the calling method; Generating different request URLs according to the calling method using the standardized request variables; Transmitting the request URL to the corresponding open API server; And parsing and loading the response result from the open API server.

The single call method collects the target data by calling the open API once, and the multiple call method may collect the target data by calling a plurality of open APIs.

The single call request URL may include a base URL for each open API, user authentication information, date information, and a set of call variables for the target data.

The multi-call request URL may include a base URL for each open API, user authentication information, and paging information.

And verifying the response result collected from the open API server after transmitting the request URL.

And managing a schedule for transmitting the request URL.

The details of other embodiments are included in the detailed description and drawings.

As described above, according to the integrated data collection method from a plurality of open APIs according to the present invention, it is possible to collect and manage data integrally through one integrated open API without having to implement a separate data collector for each open API service have. Also, according to the calling method of each open API service, it is possible to manage all the open APIs integrally by classifying them into a single calling method and a multi-calling method. Furthermore, the request variables required for each open API service are different from each other. The standardized request variable (that is, the request parameter for a single call or the request variable for multiple invocation) is defined by normalizing the common and essential items among them, By generating URLs, data can be collectively collected for all open API services.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows the overall system for the practice of the invention.
FIG. 2 illustrates a configuration of an integrated open API collector according to an embodiment of the present invention.
3 is a flowchart illustrating a method for collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Also, in this specification, when any one component 'transmits' data to another component, the component may transmit the data directly to the other component, or may transmit the at least one other component To transmit the data to the other component. On the other hand, when one element 'directly transmits' data to another element, it means that the data is transmitted to the other element without passing through another element in the element. The same can be applied to the case of 'receiving' data.

An API (Application Programming Interface) is a language or message format used when an application communicates with a system program such as an operating system or database management system. An API is implemented by calling a function that provides a connection to a specific subroutine for execution within a program. Open API (Open API) is an API that allows Internet users to directly develop application programs and services without having to be provided with web search results and user interface (UI).

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 shows the overall system for the practice of the invention. The overall system comprises an integrated open API collector 100, a plurality of data servers 10, an open API server 20 connected to each server, and an open API server 20, Network < / RTI >

The data server 10 is a server operated by a content provider such as a public institution, a corporation, or a portal site, and stores contents in a database. Each data server 10 has an open API server 20 according to the data attribute of the stored content.

When the open API server 20 receives an open API service request from the integrated open API collector 100, the open API server 20 transmits the open API service request to the data server 10 to process the request. The open API server 20 transmits the processed response result to the integrated open API collector 100. The response result is provided in a standardized web data format such as XML (Extensible Markup Language) or JSON (JavaScript Object Notation). The protocol for transmitting the response result also uses a protocol of the Web standard, for example, HTTP (hypertext transfer protocol) 1.0 or higher. The open API server 20 may include a database for storing registered API service information, a database for storing API service statistical information, a database for storing user authentication information, and the like. In addition, the open API server 20 can use one or more of REST (Representational State Transfer), SOAP (Simple Object Access Protocol), RSS / ATOM, and RDF (Resource Description Framework) have.

FIG. 2 illustrates a configuration of an integrated open API collector according to an embodiment of the present invention. The integrated open API collector 100 includes a collector manager 110, a database 130, and a collection robot 120.

The collecting robot 120 receives the command of the collector manager 110 and requests the open API server 20 for an open API service and collects result data for the open API service.

The collector manager 110 includes a schedule management unit 112, a collection robot management unit 114, a data management unit 116, and a request setting unit 118.

The schedule management unit 112 manages the schedule of the crawling operation in which the collection robot 120 requests the open API server 20 for the open API service and receives the result data. For example, in the case of the data server 10 in which new data is updated every day, such as a news website, the schedule management unit 112 may schedule the collection robot 120 to request an open API service every day or every predetermined period.

The collection robot management unit 114 controls all the operations of the collection robot 120 based on the setting contents of the request setting unit 118. [

The data management unit 116 parses and processes the data collected by the collection robot 120 from the open API server 20 and loads the data into the database 130. [

In order to efficiently integrate and manage various open API services, the request setting unit 118 sets an open API service based on a request parameter and an API call method requested by each open API server 20 before data collection Normalize the setting information and store it. Such configuration information includes basic information, standardized request variables (request variable for single call, request variable for multiple calls), request confirmation information, and result parsing information.

The request setting unit 118 sets basic information about API rules of each open API service as shown in Table 1 below.

Basic Information Explanation API ID Enter the ID value representing the API API name Enter the API collection setup name Provider Enter provider providing API Explanation Describe the API you are collecting SPEC URL Enter a URL that describes the API provider

The manner in which the integrated open API collector 100 calls the open API service to collect target data depends on the characteristics of each open API server 20. For example, you can collect the desired target data in a single call, which is called a single call method. Calling multiple pages before collecting target data is called multi-call method. According to this calling method, the request setting unit 118 sets the standardized request variable by standardizing the request variable requested by each open API service. Standardized request variables can be divided into request variables for single invocation and request variables for multiple invocation.

Table 2 below shows an example of a request variable for a single call.

Request Variable for Single Call Explanation Call type Separated into single call and multiple call (FETCH) Encoding Set encoding of response data Base API URL Enter the base URL that comes first in the URL configuration API key parameters Enter the name of the parameter that calls the API key API key value Enter the parameter value that calls the API key Date information Register date information with or without use. Start date and end date can be set for use - Automatic generation of date For automatic date generation, it is always automatically created with today's date and added to the URL - Start date parameter Enter the name of the start date parameter to be used in the URL. - end date parameter Enter the parameter name of the whole day to be used in the URL. -- start date If the date is not auto-generated, configure the URL by setting the start date - End date If the date is not auto-generated, configure the URL by setting the end date Timeout (ms) Set the response wait time for URL calls Call variable set When configuring the URL, the calling variable can be arbitrarily specified.
You can register multiple lines with 'ENTER' as delimiters, and each row is combined into a URL and called sequentially. TEST When the button is clicked, the request URL is created with the URL component set at the top. API request URL Check the completed request URL

In Table 2, 'basic API URL' represents a basic URL for each open API service, 'API key parameter' and 'API key value' represent user authentication information, and 'call variable set' To the caller.

Table 3 below shows an example of request variables for multiple invocations.

Request variable for multiple calls Explanation Call type Separated into single call and multiple call (FETCH) Encoding Set encoding of response data Base API URL Enter the base URL that comes first in the URL configuration API key parameters Enter the name of the parameter that calls the API key API key value Enter the parameter value that calls the API key. Can register multiple keys separated by commas Timeout (ms) Set the response wait time for URL calls Total number of DATA TAG Enter the tag value that contains the value to get the total number of data that can be received in the API call Number of times that API key can be called Enter the maximum number of API keys authorized to call the API Paging parameter name Enter the paging parameter name when configuring the URL Paging start value Enter the paging value that is started when the API is called Paging Increase Width Enter the number of data to obtain per page when moving the page API request URL Check the completed request URL

In addition, the request setting unit 118 may set the request confirmation information for verifying whether or not the response was successful when calling the API. Table 4 below shows an example of such request confirmation information.

Request resolution information Explanation Enable request verification Set to "On" to see if the response was successful Request verification TAG Enter the tag name that can be verified when the response is successful. Confirm request VALUE Enter the value of the tag that can be verified when the response is successful.

The request setting unit 118 can set the result parsing information as a storage rule for the target data received from the open API server 20, that is, a response element.

About parsing results Explanation Repeated ROW TAG In the API call, enter the name of the repeated tag that can distinguish each data from the data Confirm duplicate TAG Enter the value that will be ID when duplicate check is done
- Do not check duplicates if not input ELEMENT Enter the tag name of the desired data
- Enter the tag value in the child TAG of the repeated ROW TAG DATA TYPE Enter the DATA TYPE of the data
- Select between VARCHAR, TEXT, CHAR, DATE, NUMBER, DOUBLE SIZE If VARCHAR, CHAR, SIZE of corresponding data type can be input META Enter a description of the data

Hereinafter, the data collection method will be described in detail with reference to FIG. 3 is a flowchart illustrating a method for collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.

The integrated open API collector 100 performs the following process to collectively collect and manage data from the N open API servers 20 connected to the different N external data servers 10, respectively.

The integrated open API collector 100 classifies each open API server 20 according to a calling method for collecting target data (S10). More specifically, when the target data can be collected by a single call, it is classified into a single call method, and when a plurality of calls are required, the multi-call method is classified.

The integrated open API collector 100 defines standardized request variables according to the calling method of the open API service (S20). Specifically, a single call request variable is defined for a single call type open API service, and a multiple call request variable is defined for an open API service of a multiple call type.

The integrated open API collector 100 generates a request URL according to the calling method (S30). The request URL for a single call is configured to include a base URL for each open API, user authentication information, date information, and a set of call variables. For example, a combination formula of a single call request URL may be as shown in Equation 1 below.

[Formula 1]

[Request URL for Single Call] = [Base API URL] + [API Key Parameters] + [API Key Value] + [Date Information (Start Date Parameter + Start Date + End Date Parameter + End Date)

The combination expression of the request URL for multiple invocation can be as shown in Equation 2 below.

[Formula 2]

[Request URL for Multiple Calls] = [Base API URL] + [API Key Parameters] + [API Key Value] + [Paging Parameter Name] + [Paging Start Value]

Then, the integrated open API collector 100 individually transmits each request URL to the corresponding open API server 20 (S40). Here, the schedule management unit 112 can manage a schedule for transmitting a request URL, and the collection robot 120 transmits the request URL to the open API server 20 according to a preset schedule.

The open API server 20 that has received the request URL transmits the response result of processing the request to the integrated open API collector 100. The integrated open API collector 100 verifies the success of the response using the request confirmation information (S50).

If the response is successful, the integrated open API collector 100 parses and processes the response result, and loads the result in the database (S60).

For reference, the integrated open API collector according to various preferred embodiments of the present invention and the data collection method using the same may be implemented in a form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, A hard disk drive, a flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, as well as machine accords such as those produced by a compiler. A hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

10: Data server
20: Open API Server
40: Network
100: Integrated Open API Collector
110: Collector manager
112: schedule management unit
114: Collecting robot management section
116:
118: Request setting section
120: Collecting robots
130: Database

Claims (6)

A method for collectively collecting data through N open API servers connected to different N external data servers, the integrated open API collector comprising:
Categorizing each open API server into a single call method or a multiple call method according to a call method for collecting target data;
Defining standardized request variables by standardizing request variables of the corresponding open API according to the calling method;
Generating different request URLs according to the calling method using the standardized request variables;
Transmitting the request URL to the corresponding open API server; And
Parsing and loading the response result from the open API server,
The request URL for the single call method (referred to as a 'single call request URL') includes a base URL for each open API, user authentication information, date information, and a set of call variables for the target data,
The method of collecting data from a plurality of open APIs including a base URL for each open API, user authentication information, and paging information, is a request URL (referred to as 'multiple call request URL') for the multiple call method. delete delete delete The method according to claim 1,
Further comprising verifying the response result collected from the open API server after transmitting the request URL. The method according to claim 1,
Further comprising managing a schedule for transmitting the request URL. ≪ Desc / Clms Page number 21 >
KR1020160168270A 2016-12-12 2016-12-12 Method for comprehensively collecting data from different Open APIs KR101950074B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020160168270A KR101950074B1 (en) 2016-12-12 2016-12-12 Method for comprehensively collecting data from different Open APIs
PCT/KR2016/014685 WO2018110735A1 (en) 2016-12-12 2016-12-15 Method for integrated data collection from plurality of open apis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020160168270A KR101950074B1 (en) 2016-12-12 2016-12-12 Method for comprehensively collecting data from different Open APIs

Publications (2)

Publication Number Publication Date
KR20180067018A KR20180067018A (en) 2018-06-20
KR101950074B1 true KR101950074B1 (en) 2019-02-19

Family

ID=62558656

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020160168270A KR101950074B1 (en) 2016-12-12 2016-12-12 Method for comprehensively collecting data from different Open APIs

Country Status (2)

Country Link
KR (1) KR101950074B1 (en)
WO (1) WO2018110735A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102145448B1 (en) * 2018-11-15 2020-08-18 (주)에스이랩 Service providing apparatus for providing distribution service of weather and climate related information and method therefor
US20240273193A1 (en) * 2023-02-15 2024-08-15 Oracle International Corporation Advanced persistent threat detection

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101107319B1 (en) * 2008-12-01 2012-01-20 한국전자통신연구원 Method and System for Providing Web Portal Service based on Open Application Program Interface
KR101331854B1 (en) * 2009-12-18 2013-11-21 한국전자통신연구원 Apparatus and method for search Open API and generation mashup block skeleton code
KR101296321B1 (en) * 2011-06-09 2013-08-14 주식회사 피씨엔 A contents service interface providing system based on open APIs and the method thereof
KR20140119240A (en) * 2013-03-27 2014-10-10 삼성전자주식회사 Apparatus and method for processing an open api
KR101600616B1 (en) * 2015-05-28 2016-03-07 동서울대학교 산학협력단 Method for analyzing service of heterogeneous contents

Also Published As

Publication number Publication date
WO2018110735A1 (en) 2018-06-21
KR20180067018A (en) 2018-06-20

Similar Documents

Publication Publication Date Title
US8938436B2 (en) System for and method of providing reusable software service information based on natural language queries
US11538046B2 (en) Page data acquisition method, apparatus, server, electronic device and computer readable medium
TW576982B (en) Programmatic management of software resources in a content framework environment
CA2684822C (en) Data transformation based on a technical design document
CN103415841B (en) Method and system mutual between monitor client system place user and back-end system
US9977788B2 (en) Methods and systems for managing files in an on-demand system
CN103645951A (en) Cross-platform mobile data management system and method
CN114981775B (en) Cloud-based API metadata management method and system for integrated API management
US20200401465A1 (en) Apparatuses, systems, and methods for providing healthcare integrations
CN102713791A (en) Collecting community feedback for collaborative document development
CN113811854B (en) Micro-application function suggestions with cross-application activity relevance
US20130198333A1 (en) Method and device for recording and reproducing web operation
JP2007026441A (en) Apparatus and method for reaching agreement on web service policy
US10769166B1 (en) Distributed integrated platforms as a service network
CN105893055A (en) Method for triggering process engine platformization
CN104184826A (en) Multi-data-center storage environment managing method and system
JP2008130033A (en) Service cooperation method
KR101950074B1 (en) Method for comprehensively collecting data from different Open APIs
Kim et al. Comprehensive Knowledge Archive Network harvester improvement for efficient open‐data collection and management
WO2019043462A1 (en) Systems and methods for creating automated interface transmission between heterogeneous systems in an enterprise ecosystem
US20150365460A1 (en) Dynamic workflow-based composite web service system and method
US20130290245A1 (en) Database history management method and system thereof
CN113709187B (en) Multi-server data request method, device, equipment and storage medium
Yahyaoui et al. A synergy between context-aware policies and AOP to achieve highly adaptable Web services
Bahrami et al. WATAPI: composing web API specification from API documentations through an intelligent and interactive annotation tool

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right
GRNT Written decision to grant