WO2018110735A1 - Method for integrated data collection from plurality of open apis - Google Patents

Abstract

Provided is a method for integrated data collection from a plurality of open APIs, the method allowing integrated collection and management of a large set of data from diverse open API servers by utilizing a single general-purpose open API collector. The data collection method is a method in which integrated data collection is by means of N number of open API servers connected to N number of respective external data servers, and comprises the steps, carried out by an integrated open API collector, of: classifying each open API server as having a single call method or a multiple call method in accordance with a call method for collecting target data; standardizing request variables of an open API in accordance with the corresponding call method, and defining the standardized request variables; generating mutually different request URLs in accordance with the call method by means of the standardized request variables; transmitting the request URLs to the corresponding open API servers; and parsing and loading the resultant responses from the open API servers.

Description

Integrated data collection method from multiple open APIs

The present invention relates to an integrated data collection method from a plurality of open API, and more particularly, to a data collection method that can collectively collect data from various open API servers through one integrated open API collector.

In general, an open API (Open Application Programming Interface) refers to a connection protocol that allows other systems to call and utilize a specific function when developing an application program. In other words, the open API is to expose the API to the outside world so that an unspecified number of users can easily develop and use the application program easily. Using a set of protocols and tools that make it easy to create arbitrary applications, program developers can easily develop applications with just a few open APIs without knowing the operating system's detailed features.

Recently, many public organizations and private companies have provided information to general consumers through open APIs. Many individuals and companies are using these open API services to provide services in the form of mobile apps or web services.

It is not difficult to build an additional service system using one open API service. However, it is technically very difficult to provide a service by collectively collecting a large amount of data from a plurality of open API services. This is because request variables and API calling methods required by each open API service are different. In this case, in the related art, a separate data collector has to be implemented for each open API service, but there are so many management items from the standpoint of managing multiple services that it is difficult to manage them collectively.

An object of the present invention is to provide an integrated data collection method from a plurality of open API that can collectively collect and manage a large amount of data from various open API servers using one general purpose open API collector. .

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

Integrated data collection method from a plurality of open API according to an embodiment of the present invention for achieving the above object, to collect the data through the N open API server connected to each of the N different external data server integrated A method, comprising: an integrated open API collector, for each of the open API servers, classifying a single call method or a multiple call method according to a calling method for collecting target data; Defining a standardized request variable by standardizing a request variable of a corresponding open API according to the calling method; Generating different request URLs according to the calling method using the standardized request variable; Transmitting the request URL to a corresponding open API server; And parsing and loading a response result from the open API server.

The single call method may call the open API once to collect the target data, and the multi-call method may call the plurality of open APIs to collect the target data.

The single call request URL may include a base URL for each open API, user authentication information, date information, and a call variable set for the target data.

The multi-call request URL may include a basic URL, user authentication information, and paging information for each open API.

After transmitting the request URL, the method may further include verifying a response result collected from the open API server.

The method may further include managing a schedule for transmitting the request URL.

Details of other embodiments are included in the detailed description and drawings.

As described above, according to the integrated data collection method from the plurality of open APIs according to the present invention, it is not necessary to implement a separate data collector for each open API service, and it is possible to integrate and collect data through one integrated open API. have. In addition, all open APIs can be integratedly managed by dividing them into a single call method and a multiple call method according to the calling method of each open API service. Furthermore, request variables are different for each Open API service, and common and necessary items are normalized to define standardized request variables (that is, request variables for single call or multiple call request variables). By creating a URL, you can collect data for all open API services.

1 shows an overall system for the practice of the present invention.

2 illustrates a configuration of an integrated open API collector according to an embodiment of the present invention.

3 is a flowchart illustrating a method of collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

In addition, in the present specification, when one component 'transmits' data to another component, the component may directly transmit the data to the other component, or via at least one other component. This means that the data can be transmitted to the other component. On the other hand, when one component 'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without passing through another component. The same can be applied to the case of 'receiving' data.

An application programming interface (API) is a language or message format used when an application communicates with a system program, such as an operating system or database management system. An API is implemented by calling a function that provides a link to a specific subroutine for execution within a program. Open API is an open API that allows Internet users to develop their own applications and services, instead of being provided with web search results and user interfaces (UIs).

1 shows an overall system for the practice of the present invention. The entire system connects the integrated open API collector 100, the plurality of data servers 10, the open API server 20 connected to each server, and each open API server 20 and the integrated open API collector 100. Network 40.

The data server 10 is a server operated by a content provider such as a public institution, a company, a portal site, and stores content in a database. Each data server 10 has an open API server 20 individually according to the data attributes of the stored content.

When the open API server 20 receives an open API service request from the integrated open API collector 100, the open API server 20 transmits the open API service request to the data server 10 to process the request. The open API server 20 transmits the processed response result to the integrated open API collector 100. The response result is provided in a standardized web data format such as XML (eXtensible Markup Language) or JSON (JavaScript Object Notation). In addition, the protocol for transmitting the response result uses a protocol of a web standard, for example, HTTP (hypertext transfer protocol) 1.0 or higher. The open API server 20 may include a database storing registered API service information, a database storing API service statistical information, a database storing user authentication information, and the like. In addition, the open API server 20 may use various standards, and one or more of REST (Representational State Transfer), SOAP (Simple object access, Protocol), RSS / ATOM, and Resource Description Framework (RDF) may be used. have.

2 illustrates a configuration of an integrated open API collector according to an embodiment of the present invention. The integrated open API collector 100 includes a collector manager 110, a database 130, and a collection robot 120.

The collecting robot 120 requests an open API service from the open API server 20 by receiving a command of the collector manager 110 and collects the result data.

The collector manager 110 includes a schedule manager 112, a collection robot manager 114, a data manager 116, and a request setup unit 118.

The schedule manager 112 manages a schedule of a crawling operation in which the collecting robot 120 requests an open API service from the open API server 20 and receives the result data. For example, in the case of the data server 10 where new data is updated every day, such as a news website, the schedule manager 112 may set a schedule for the collection robot 120 to request an open API service every day or every fixed period.

The collecting robot manager 114 controls all operations of the collecting robot 120 based on the setting contents of the request setting unit 118.

The data manager 116 parses and processes the data collected by the collection robot 120 from the open API server 20 and loads the data into the database 130.

The request setting unit 118 is configured to open API services based on request parameters and API calling methods required by each open API server 20 before data collection in order to efficiently integrate and manage various open API services. Normalize and save the setting information. Such setting information includes basic information, standardized request variables (single call request variables, multiple call request variables), request confirmation information, and result parsing information.

The request setting unit 118 sets basic information about API rules of each open API service as shown in Table 1 below.

Basic Information	Explanation
API ID	Enter an ID value representing the API
API name	Enter a name for the API collection settings
Provider	Enter the provider that provides the API
Explanation	Describe the API to collect
SPEC URL	Enter a URL that describes the API provider.

The manner in which the integrated open API collector 100 calls an open API service to collect target data depends on the characteristics of each open API server 20. For example, you can collect the target data you want with a single call, which is called a single call. Multiple pages must be called before the target data is collected. According to this calling method, the request setting unit 118 sets standardized request variables by standardizing request variables required by each open API service. Standardized request variables can be divided into single call request variables and multiple call request variables.

Table 2 below shows an example of a single call request variable.

Request variable for single call	Explanation
Call type	Divided into single call and multiple call (FETCH)
Encoding	Set encoding of response data
Primary API URL	Enter the base URL that comes first when constructing URL
API key parameters	Enter the parameter name to call the API key
API key value	Enter a parameter value to call the API key
Date Information	Register date information as used or not used. In case of use, start date and end date can be set.
-Automatic date generation	For auto-generated date, always auto-generated to today's date
-Start date parameter	Enter the start date parameter name to be used for the URL
-End date parameter	Enter all day parameter name to be used in URL
-- start date	If not automatic date generation, set the start date to configure the URL
-End date	If not automatic date generation, set the end date to construct the URL
Timeout (ms)	Set response wait time when calling URL
Call variable set	When configuring URL, call variable can be arbitrarily specified. It can be registered on multiple lines with 'ENTER' as a separator, and each line is combined in URL and called sequentially.
TEST	When the button is clicked, the request URL is created with the URL component set at the top.
API request URL	Check the completed request URL

In Table 2, 'base API URL' represents the base URL for each Open API service, 'API key parameters' and 'API key values' represent the user authentication information, and 'call variable set' represents target data from the off API service. Represents a set of call variables that specify.

Table 3 below shows exemplary request variables for multiple calls.

Request variable for multiple calls	Explanation
Call type	Divided into single call and multiple call (FETCH)
Encoding	Set encoding of response data
Primary API URL	Enter the base URL that comes first when constructing URL
API key parameters	Enter the parameter name to call the API key
API key value	Enter the parameter value to call the API key. Multiple keys can be registered by separating them with commas
Timeout (ms)	Set response wait time when calling URL
Total DATA TAG	Enter the tag value that contains the value to get the total number of data that can be answered in the API call.
Number of Calls per API Key	Enter the maximum number of calls to the authorized API key to call the API.
Paging parameter name	Enter paging parameter name when configuring URL
Paging Start Value	Enter the paging value that starts when calling the API
Paging Increase Width	Input the number of data per page when moving page
API request URL	Check the completed request URL

In addition, the request setting unit 118 may set request confirmation information for verifying whether the response is successful when calling the API. Table 4 below lists these request confirmation information as an example.

Request confirmation information	Explanation
Enable request verification	Set to "on" to see if the response was successful
Request Confirmation TAG	Enter tag name to check when response is successful
Request Verification VALUE	Enter the value of the tag to check when the response is successful

The request setting unit 118 may set the result parsing information as a storage rule for the target data received from the open API server 20, that is, a response element.

Result Parsing Information	Explanation
Repeated ROW TAG	Enter a repeated tag name to distinguish each data in the data when calling the API
Duplicate Check TAG	When checking for duplicates when saving data, enter the ID value-If not, do not check duplicates.
ELEMENT	Enter tag name of desired data-Enter tag value in child tag of repeated row tag
DATA TYPE	Select the data type of the data-VARCHAR, TEXT, CHAR, DATE, NUMBER, DOUBLE
SIZE	In case of VARCHAR and CHAR, the size of the relevant data type can be input.
META	Enter a description of the data

Hereinafter, the data collection method will be described in detail with reference to FIG. 3. 3 is a flowchart illustrating a method of collectively collecting data from a plurality of open APIs according to an embodiment of the present invention.

The integrated open API collector 100 performs the following process to collectively collect and manage data from N open API servers 20 respectively connected to different N external data servers 10.

The integrated open API collector 100 classifies each open API server 20 according to a calling method for collecting target data (S10). Specifically, when the target data can be collected in one call, the call is classified into a single call method, and when a plurality of calls are required, the call is classified into a multiple call method.

The integrated open API collector 100 defines a standardized request variable according to the calling method of the open API service (S20). Specifically, a single call request variable is defined for a single call open API server, and a multiple call request variable is defined for an open API service of a multi call type.

The integrated open API collector 100 generates a request URL according to the calling method (S30). The single call request URL is configured to include a base URL, user authentication information, date information, and a call variable set for each open API. For example, the combined expression of the single call request URL may be as shown in Equation 1 below.

[Equation 1]

[Request URL for Single Call] = [Default API URL] + [API Key Parameter] + [API Key Value] + [Date Information (Start Date Parameters + Start Date + End Date Parameters + End Date)] + [Set of Call Variables]

The combination expression of the request URL for multiple calls may be as shown in Equation 2 below.

[Equation 2]

[Request URL for Multicall] = [Default API URL] + [API Key Parameter] + [API Key Value] + [Paging Parameter Name] + [Paging Start Value]

Subsequently, the integrated open API collector 100 individually transmits each request URL to the open API server 20 corresponding thereto (S40). Here, the schedule manager 112 may manage a schedule for transmitting the request URL, and the collection robot 120 transmits the request URL to the open API server 20 according to a preset schedule.

The open API server 20 receiving the request URL transmits the response result of processing the request to the integrated open API collector 100. The integrated open API collector 100 verifies the success of the response result using the request confirmation information (S50).

If the response is successful, the integrated open API collector 100 parses and processes the response result and loads it in the database (S60).

For reference, the integrated open API collector and the data collection method using the same according to various exemplary embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, Hardware devices specifically configured to store and execute program instructions, such as flash memory, may be included. Examples of program instructions include machine code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter. Hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (6)

1. The integrated open API collector is a method of collectively collecting data through N open API servers connected to N different external data servers.

   Classifying each open API server into a single call method or a multiple call method according to a calling method for collecting target data;

   Defining a standardized request variable by standardizing a request variable of a corresponding open API according to the calling method;

   Generating different request URLs according to the calling method using the standardized request variable;

   Transmitting the request URL to a corresponding open API server; And

   Integrated data collection method from a plurality of open API including parsing and loading the response result from the open API server.
2. The method of claim 1,

   The single call method is to call the open API once to collect the target data,

   The multi-call method collects the target data by calling a plurality of open APIs, the integrated data collection method from a plurality of open APIs.
3. The method of claim 1,

   The single call request URL, the integrated data collection method from a plurality of open API including a base URL for each open API, user authentication information, date information and a set of call variables for the target data.
4. The method of claim 1,

   The multi-call request URL, the integrated data collection method from a plurality of open API including a basic URL, user authentication information, and paging information for each open API.
5. The method of claim 1,

   After transmitting the request URL, verifying a response result collected from the open API server.
6. The method of claim 1,

   Integrated data collection method from a plurality of open API further comprising the step of managing a schedule for transmitting the request URL.

Images