CN112506486A - Search system establishing method and device, electronic equipment and readable storage medium - Google Patents
Search system establishing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112506486A CN112506486A CN202011483122.2A CN202011483122A CN112506486A CN 112506486 A CN112506486 A CN 112506486A CN 202011483122 A CN202011483122 A CN 202011483122A CN 112506486 A CN112506486 A CN 112506486A
- Authority
- CN
- China
- Prior art keywords
- component
- building
- index
- search system
- search
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012217 deletion Methods 0.000 claims abstract description 41
- 230000037430 deletion Effects 0.000 claims abstract description 41
- 238000013507 mapping Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 10
- 230000004044 response Effects 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 13
- 230000009471 action Effects 0.000 claims description 4
- 230000000593 degrading effect Effects 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000011161 development Methods 0.000 abstract description 13
- 230000018109 developmental process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000000712 assembly Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/24—Object-oriented
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/235—Update request formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/2445—Data retrieval commands; View definitions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/31—Programming languages or programming paradigms
- G06F8/315—Object-oriented languages
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to data processing, and discloses a search system establishing method, which comprises the following steps: receiving an initial table building statement fed back by a first client user based on a table building statement template; analyzing the initial table building sentence to obtain a table name of a table to be built and a field type of a table head field, and converting the initial table building sentence into a target table building sentence based on a mapping relation between the field type of a table building sentence template and a standard field type of a search engine; establishing an index base based on the target table building statement; respectively obtaining program templates corresponding to the index building component, the search component and the deletion component, generating the index building component, the search component and the deletion component based on the target table building statement and the program templates, and importing the source data into an index base based on the index building component to obtain a target search system. The invention also provides a search system establishing device, electronic equipment and a readable storage medium. The invention reduces the development difficulty of the search system and improves the development efficiency.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and an apparatus for establishing a search system, an electronic device, and a readable storage medium.
Background
A complete search system comprises a data source, an index service and a search service, wherein different business data search systems correspond to different data sources, different index processes, different search logics and log monitoring services. When a search system of business data needs to be newly developed, a whole set of codes needs to be developed, so that the development period is long, the development difficulty is high, and the development efficiency is low. Therefore, a method for establishing a search system is needed to reduce the difficulty of developing the search system and improve the development efficiency.
Disclosure of Invention
In view of the above, it is necessary to provide a search system establishing method, aiming at reducing the difficulty of search system development and improving the development efficiency.
The search system establishment method provided by the invention comprises the following steps:
responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client, and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template;
analyzing the initial table building sentence to obtain a table name of a table to be built and a field type of a table head field, and converting the initial table building sentence into a target table building sentence based on a mapping relation between the field type of a table building sentence template and a standard field type of a search engine;
establishing a search engine server cluster requesting to distribute idleness for the search system and establishing an index database based on the target table building statement and the table name;
respectively obtaining program templates corresponding to the index building component, the search component and the deletion component, generating the index building component, the search component and the deletion component based on the target table building statement and the program templates, and importing source data corresponding to the search system building request into the index database based on the index building component to obtain a target search system.
Optionally, the generating an index building component, a searching component and a deleting component based on the target table building statement and the program template includes:
analyzing the target table building sentence to obtain the name and the format of the table header field of the table to be built, and generating an entity class based on the name and the format of the table header field;
generating an index building component based on the entity class and a program template corresponding to the index building component;
extracting a field name list with the attribute of a searchable field from the target table building statement, and generating a search component based on the field name list, the entity class and a program template corresponding to the search component;
and generating a deleting component based on the entity class and the program template corresponding to the deleting component.
Optionally, the method further includes:
when an index library capacity expansion request sent by a user based on a second client is received, displaying a capacity expansion configuration interface to a display interface of the second client, and receiving a target index library and a field to be fragmented input by the user based on the capacity expansion configuration interface;
and inputting the key value of the field to be fragmented into a preset calculation formula for calculation, and distributing the data in the target index database to different nodes of the search engine server cluster based on the calculated numerical value.
Optionally, the preset calculation formula is:
K％N＝＝0(1)
K％N＝＝1 (2)
…….
K％N＝＝(N-1) (N)
k is the key value of the field to be fragmented, and N is the total number of nodes after capacity expansion;
data satisfying equation (1) will be stored to the first node, data satisfying equation (2) will be stored to the second node, … …, and data satisfying equation (N) will be stored to the nth node.
Optionally, the method further includes:
and acquiring the resource utilization rate of a server to which the search system belongs in real time or at regular time, and executing current limiting and degrading processing on the search system when the resource utilization rate exceeds a first threshold value.
Optionally, the method further includes:
and when a data writing request sent by a user based on a third client is received, judging whether the data volume of the data to be written is larger than a second threshold, and if so, executing asynchronous writing operation on the data to be written.
Optionally, the program template corresponding to the index building component includes an API interface name written into the index library, a source data name, field version information, and action occurrence time information;
the program template corresponding to the search component comprises a sorting parameter, a paging parameter, a search field and a return field.
In order to solve the above problem, the present invention further provides a search system establishment apparatus, including:
the response module is used for responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template;
the analysis module is used for analyzing the initial table building sentence to obtain the table name of the table to be built and the field type of the table head field, and converting the initial table building sentence into a target table building sentence based on the mapping relation between the field type of the table building sentence template and the standard field type of the search engine;
the distribution module is used for establishing a search engine server cluster requesting to distribute idle for the search system based on the target table building statement and the table name and establishing an index library;
and the generating module is used for respectively acquiring program templates corresponding to the index building component, the searching component and the deleting component, generating the index building component, the searching component and the deleting component based on the target table building statement and the program templates, and importing the source data corresponding to the searching system building request into the index library based on the index building component to obtain the target searching system.
In order to solve the above problem, the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a search system setup program executable by the at least one processor, the search system setup program being executed by the at least one processor to enable the at least one processor to perform the search system setup method described above.
In order to solve the above problems, the present invention also provides a computer-readable storage medium having stored thereon a search system setup program executable by one or more processors to implement the above search system setup method.
Compared with the prior art, the method comprises the steps of firstly sending a table building statement template to a first client, receiving an initial table building statement fed back by a first client user based on the table building statement template, and converting the initial table building statement into a target table building statement based on the mapping relation between the field type of the table building statement template and the standard field type of a search engine; then, distributing idle search engine server clusters, and establishing an index base based on the target table building sentences, the invention provides a table building sentence template, so that a user does not need to additionally learn an index base establishing method, and the establishment of the index base can be realized only by filling corresponding parameters into the table building sentence template, thereby greatly improving the establishment efficiency of the index base; and finally, generating an index building component, a search component and a deletion component based on the target table building statement and program templates corresponding to the index building component, the search component and the deletion component, and importing the source data into the index base based on the index building component to obtain the target search system. Therefore, the invention reduces the development difficulty of the search system and improves the development efficiency.
Drawings
Fig. 1 is a schematic flowchart of a search system establishing method according to an embodiment of the present invention;
fig. 2 is a schematic block diagram of a search system setup apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device implementing a search system establishment method according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
The invention provides a search system establishing method. Fig. 1 is a schematic flow chart of a search system establishing method according to an embodiment of the present invention. The method may be performed by an electronic device, which may be implemented by software and/or hardware.
In this embodiment, the method for establishing the search system includes:
s1, responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client, and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template.
Currently, an index database (similar to a data table) is usually built by using ES (elastic search) library building statements, however, the ES library building statements are complex and the number of statements is huge. The embodiment provides a table building statement template, the syntax of the table building statement template is MySQL syntax, and MySQL syntax is more common syntax, so that developers do not need to learn an ES database building method, and MySQL syntax can realize a table building function in a simpler language (by verification, for a certain table building requirement, MySQL table building statements only have ten rows, while ES table building statements require seventy multiple rows).
S2, analyzing the initial table building sentence to obtain the table name of the table to be built and the field type of the table head field, and converting the initial table building sentence into the target table building sentence based on the mapping relation between the field type of the table building sentence template and the standard field type of the search engine.
And analyzing the initial table building sentence to obtain a table name of a table to be built, a field name of a table header field, a field type and a field format, wherein the contents are filled into a table building sentence template by a first client user, the field type comprises a character type, a numerical value type and the like, and the field format is a format of a value corresponding to a field. Meanwhile, the initial table building statement also comprises whether each field needs to be segmented, stored and searchable fields.
The standard field type of the search engine is the field type of the ES table building statement, the generated target table building statement is the ES table building statement, and the initial table building statement of the MySQL grammar can be converted into the target table building statement of the ES format through the mapping relation between the field type of the table building statement template and the standard field type of the search engine.
S3, based on the target table building statement and the table name, building a search engine server cluster which is idle and is distributed for the search system building request, and building an index database.
The generated index library is similar to a data table, and the field name is the field name of the header part of the data table, namely the generated index library is a blank data table only containing the header.
In this embodiment, an idle search engine server cluster node may be randomly allocated to the search system establishment request to establish an index library, or the index library may be established on a cluster node designated by a user.
S4, program templates corresponding to the index building component, the search component and the deletion component are respectively obtained, the index building component, the search component and the deletion component are generated based on the target table building statement and the program templates, and source data corresponding to the search system building request are led into the index base based on the index building component to obtain the target search system.
The generating an indexing component, a searching component and a deleting component based on the target table building statement and the program template comprises:
a11, analyzing the target table building sentence to obtain the table head field name and the format of the table to be built, and generating an entity class based on the table head field name and the format;
the format of the header field is the specific configuration of the field, for example, a date type field, and has a format attribute indicating in what format the value of the field is input and output.
The entity class defines a method for acquiring the value of each field in the header of the index database, and the value corresponding to each field in the index database can be captured from the source data through the entity class. The entity class (model class) is generated by mapping the index base in the ES to a POJO class in java (which can be directly generated according to the target table building statement).
A12, generating an index building component based on the entity class and the program template corresponding to the index building component;
the indexing component comprises an indexing API component and an indexing response component, the indexing API component is used for importing data and storing the data into an index database, and the indexing response component is used for outputting indexing response result information (response success or failure, response time consumption and response failure reason).
The program template corresponding to the index building component defines an API interface name (automatically generated by the system according to the name and the operation type (increasing, deleting, modifying and checking)) written into the index library, a source data name, field version information (one is added to the field version after each update, so that the problem of inconsistent data update results caused by concurrent access is solved), and timestamp action occurrence time information.
A13, extracting a field name list with the attribute of a searchable field from the target table building statement, and generating a search component based on the field name list, the entity class and a program template corresponding to the search component;
for example, the target table building statement has an index attribute, which indicates whether the field needs to be indexed (whether the field is searchable), and when the index attribute is true, the field indicates searchable; when the index attribute is false, it indicates that it is not searchable.
The search component includes a search API component for performing search operations and a search response component for storing retrieved data and returning error codes and error information if an execution error occurs.
The program template corresponding to the search component defines a sorting parameter, a paging parameter, a search field and a return field.
And A14, generating a deletion component based on the program template corresponding to the entity class and the deletion component.
The deletion component comprises a deletion API component and a deletion response component, the deletion API component is used for executing deletion operation, and the deletion response component is used for recording deletion results (deletion success or failure, if deletion fails, an error code and error report information are returned).
The index building assembly, the searching assembly and the deleting assembly are functional assemblies of the searching system, source data can be imported into a built index base according to the index building assembly, the building of the searching system is completed at this moment, a subsequent user can update data in the index base through the index building assembly, search data corresponding to a retrieval field in the index base through the searching assembly, and delete data to be deleted through the deleting assembly.
In this embodiment, the method further includes:
b11, when an index library capacity expansion request sent by a user based on a second client is received, displaying a capacity expansion configuration interface to a display interface of the second client, and receiving a target index library and a field to be fragmented input by the user based on the capacity expansion configuration interface;
b12, inputting the key value of the field to be fragmented into a preset calculation formula for calculation, and distributing the data in the target index database to different nodes of the search engine server cluster based on the calculated numerical value.
In this embodiment, the preset calculation formula is as follows:
K％N＝＝0(1)
K％N＝＝1 (2)
…….
K％N＝＝(N-1) (N)
k is the key value of the field to be fragmented, and N is the total number of nodes after capacity expansion;
data satisfying equation (1) may be stored to a first node, data satisfying equation (2) may be stored to a second node, … …, and data satisfying equation (N) may be stored to an nth node.
For example, if the "hospital" index repository was originally stored in the node a of the ES cluster, and the "hospital" index repository needs to be expanded to the node B (i.e., stored in A, B two nodes), N is 2, data corresponding to K divided by 2 with no remainder may be stored in the node a, and data corresponding to K divided by 2 with remainder 1 may be stored in the node B.
The new data of the index database are directly stored in A, B nodes according to the rule, and the original old data are re-split according to the rule and are stored in two nodes respectively.
In another embodiment of the present invention, the method further comprises:
and acquiring the resource utilization rate of a server to which the search system belongs in real time or at regular time, and executing current limiting and degrading processing on the search system when the resource utilization rate exceeds a first threshold value.
The current limit may be to limit a maximum concurrent access amount; the downgrading can be to downgrade complex operations to normal operations (e.g., display inventory on the goods page in real time, and update the display inventory on a regular basis when the server running load is too high).
In this embodiment, the method further includes:
and when a data writing request sent by a user based on a third client is received, judging whether the data volume of the data to be written is larger than a second threshold, and if so, executing asynchronous writing operation on the data to be written.
The embodiment shows that the search system establishment method provided by the invention comprises the steps of firstly, sending a table building statement template to a first client, receiving an initial table building statement fed back by a first client user based on the table building statement template, and converting the initial table building statement into a target table building statement based on the mapping relation between the field type of the table building statement template and the standard field type of a search engine; then, distributing idle search engine server clusters, and establishing an index base based on the target table building sentences, the invention provides a table building sentence template, so that a user does not need to additionally learn an index base establishing method, and the establishment of the index base can be realized only by filling corresponding parameters into the table building sentence template, thereby greatly improving the establishment efficiency of the index base; and finally, generating an index building component, a search component and a deletion component based on the target table building statement and program templates corresponding to the index building component, the search component and the deletion component, and importing the source data into the index base based on the index building component to obtain the target search system. Therefore, the invention reduces the development difficulty of the search system and improves the development efficiency.
Fig. 2 is a schematic block diagram of a search system setup apparatus according to an embodiment of the present invention.
The search system establishment apparatus 100 of the present invention may be installed in an electronic device. According to the implemented functions, the search system establishing apparatus 100 may include a response module 110, a parsing module 120, an allocation module 130, and a generation module 140. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the response module 110 is configured to respond to a search system establishment request sent by a user based on a first client, send a table establishment statement template to the first client, and receive an initial table establishment statement fed back by the first client based on the table establishment statement template.
Currently, an index database (similar to a data table) is usually built by using ES (elastic search) library building statements, however, the ES library building statements are complex and the number of statements is huge. The embodiment provides a table building statement template, the syntax of the table building statement template is MySQL syntax, and MySQL syntax is more common syntax, so that developers do not need to learn an ES database building method, and MySQL syntax can realize a table building function in a simpler language (by verification, for a certain table building requirement, MySQL table building statements only have ten rows, while ES table building statements require seventy multiple rows).
The parsing module 120 is configured to parse the initial table building statement to obtain a table name of the table to be built and a field type of the table header field, and convert the initial table building statement into a target table building statement based on a mapping relationship between the field type of the table building statement template and a standard field type of a search engine.
And analyzing the initial table building sentence to obtain a table name of a table to be built, a field name of a table header field, a field type and a field format, wherein the contents are filled into a table building sentence template by a first client user, the field type comprises a character type, a numerical value type and the like, and the field format is a format of a value corresponding to a field. Meanwhile, the initial table building statement also comprises whether each field needs to be segmented, stored and searchable fields.
The standard field type of the search engine is the field type of the ES table building statement, the generated target table building statement is the ES table building statement, and the initial table building statement of the MySQL grammar can be converted into the target table building statement of the ES format through the mapping relation between the field type of the table building statement template and the standard field type of the search engine.
An allocating module 130, configured to allocate an idle search engine server cluster for the search system establishment request and establish an index library based on the target table building statement and the table name.
The generated index library is similar to a data table, and the field name is the field name of the header part of the data table, namely the generated index library is a blank data table only containing the header.
In this embodiment, an idle search engine server cluster node may be randomly allocated to the search system establishment request to establish an index library, or the index library may be established on a cluster node designated by a user.
The generating module 140 is configured to obtain program templates corresponding to the index building component, the search component, and the deletion component, respectively, generate the index building component, the search component, and the deletion component based on the target table building statement and the program template, and import source data corresponding to the search system building request into the index library based on the index building component to obtain the target search system.
The generating an indexing component, a searching component and a deleting component based on the target table building statement and the program template comprises:
a21, analyzing the target table building sentence to obtain the table head field name and the format of the table to be built, and generating an entity class based on the table head field name and the format;
the format of the header field is the specific configuration of the field, for example, a date type field, and has a format attribute indicating in what format the value of the field is input and output.
The entity class defines a method for acquiring the value of each field in the header of the index database, and the value corresponding to each field in the index database can be captured from the source data through the entity class. The entity class (model class) is generated by mapping the index base in the ES to a POJO class in java (which can be directly generated according to the target table building statement).
A22, generating an index building component based on the entity class and the program template corresponding to the index building component;
the indexing component comprises an indexing API component and an indexing response component, the indexing API component is used for importing data and storing the data into an index database, and the indexing response component is used for outputting indexing response result information (response success or failure, response time consumption and response failure reason).
The program template corresponding to the index building component defines an API interface name (automatically generated by the system according to the name and the operation type (increasing, deleting, modifying and checking)) written into the index library, a source data name, field version information (one is added to the field version after each update, so that the problem of inconsistent data update results caused by concurrent access is solved), and timestamp action occurrence time information.
A23, extracting a field name list with the attribute of a searchable field from the target table building statement, and generating a search component based on the field name list, the entity class and a program template corresponding to the search component;
for example, the target table building statement has an index attribute, which indicates whether the field needs to be indexed (whether the field is searchable), and when the index attribute is true, the field indicates searchable; when the index attribute is false, it indicates that it is not searchable.
The search component includes a search API component for performing search operations and a search response component for storing retrieved data and returning error codes and error information if an execution error occurs.
The program template corresponding to the search component defines a sorting parameter, a paging parameter, a search field and a return field.
And A24, generating a deletion component based on the program template corresponding to the entity class and the deletion component.
The deletion component comprises a deletion API component and a deletion response component, the deletion API component is used for executing deletion operation, and the deletion response component is used for recording deletion results (deletion success or failure, if deletion fails, an error code and error report information are returned).
The index building assembly, the searching assembly and the deleting assembly are functional assemblies of the searching system, source data can be imported into a built index base according to the index building assembly, the building of the searching system is completed at this moment, a subsequent user can update data in the index base through the index building assembly, search data corresponding to a retrieval field in the index base through the searching assembly, and delete data to be deleted through the deleting assembly.
In this embodiment, the generating module 140 is further configured to:
b21, when an index library capacity expansion request sent by a user based on a second client is received, displaying a capacity expansion configuration interface to a display interface of the second client, and receiving a target index library and a field to be fragmented input by the user based on the capacity expansion configuration interface;
b22, inputting the key value of the field to be fragmented into a preset calculation formula for calculation, and distributing the data in the target index database to different nodes of the search engine server cluster based on the calculated numerical value.
In this embodiment, the preset calculation formula is as follows:
K％N＝＝0(1)
K％N＝＝1 (2)
…….
K％N＝＝(N-1) (N)
k is the key value of the field to be fragmented, and N is the total number of nodes after capacity expansion;
data satisfying equation (1) may be stored to a first node, data satisfying equation (2) may be stored to a second node, … …, and data satisfying equation (N) may be stored to an nth node.
For example, if the "hospital" index repository was originally stored in the node a of the ES cluster, and the "hospital" index repository needs to be expanded to the node B (i.e., stored in A, B two nodes), N is 2, data corresponding to K divided by 2 with no remainder may be stored in the node a, and data corresponding to K divided by 2 with remainder 1 may be stored in the node B.
The new data of the index database are directly stored in A, B nodes according to the rule, and the original old data are re-split according to the rule and are stored in two nodes respectively.
In another embodiment of the present invention, the generating module 140 is further configured to:
and acquiring the resource utilization rate of a server to which the search system belongs in real time or at regular time, and executing current limiting and degrading processing on the search system when the resource utilization rate exceeds a first threshold value.
The current limit may be to limit a maximum concurrent access amount; the downgrading can be to downgrade complex operations to normal operations (e.g., display inventory on the goods page in real time, and update the display inventory on a regular basis when the server running load is too high).
In this embodiment, the generating module 140 is further configured to:
and when a data writing request sent by a user based on a third client is received, judging whether the data volume of the data to be written is larger than a second threshold, and if so, executing asynchronous writing operation on the data to be written.
As can be seen from the foregoing embodiments, the search system building apparatus 100 according to the present invention first sends a table building statement template to a first client, receives an initial table building statement fed back by a first client user based on the table building statement template, and converts the initial table building statement into a target table building statement based on a mapping relationship between a field type of the table building statement template and a standard field type of a search engine; then, distributing idle search engine server clusters, and establishing an index base based on the target table building sentences, the invention provides a table building sentence template, so that a user does not need to additionally learn an index base establishing method, and the establishment of the index base can be realized only by filling corresponding parameters into the table building sentence template, thereby greatly improving the establishment efficiency of the index base; and finally, generating an index building component, a search component and a deletion component based on the target table building statement and program templates corresponding to the index building component, the search component and the deletion component, and importing the source data into the index base based on the index building component to obtain the target search system. Therefore, the invention reduces the development difficulty of the search system and improves the development efficiency.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a search system establishment method according to an embodiment of the present invention.
The electronic device 1 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a command set or stored in advance. The electronic device 1 may be a computer, or may be a single network server, a server group composed of a plurality of network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing, where cloud computing is one of distributed computing and is a super virtual computer composed of a group of loosely coupled computers.
In the present embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which are communicatively connected to each other through a system bus, wherein the memory 11 stores a search system establishment program 10, and the search system establishment program 10 is executable by the processor 12. Fig. 3 only shows the electronic device 1 with the components 11-13 and the search system setup program 10, and it will be understood by those skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or may combine certain components, or a different arrangement of components.
The storage 11 includes a memory and at least one type of readable storage medium. The memory provides cache for the operation of the electronic equipment 1; the readable storage medium may be a non-volatile storage medium such as flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk provided on the electronic device 1, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. In this embodiment, the readable storage medium of the memory 11 is generally used for storing an operating system and various application software installed in the electronic device 1, for example, codes of the search system establishment program 10 in an embodiment of the present invention. Further, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 12 is generally configured to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is configured to run the program code stored in the memory 11 or process data, for example, run the search system establishment program 10.
The network interface 13 may comprise a wireless network interface or a wired network interface, and the network interface 13 is used for establishing a communication connection between the electronic device 1 and a client (not shown).
Optionally, the electronic device 1 may further include a user interface, the user interface may include a Display (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface may further include a standard wired interface and a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The search system setup program 10 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 12, enable:
responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client, and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template;
analyzing the initial table building sentence to obtain a table name of a table to be built and a field type of a table head field, and converting the initial table building sentence into a target table building sentence based on a mapping relation between the field type of a table building sentence template and a standard field type of a search engine;
establishing a search engine server cluster requesting to distribute idleness for the search system and establishing an index database based on the target table building statement and the table name;
respectively obtaining program templates corresponding to the index building component, the search component and the deletion component, generating the index building component, the search component and the deletion component based on the target table building statement and the program templates, and importing source data corresponding to the search system building request into the index database based on the index building component to obtain a target search system.
Specifically, the processor 12 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the search system establishment program 10, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer readable medium may be non-volatile or non-volatile. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The computer-readable storage medium stores a search system establishment program 10, where the search system establishment program 10 may be executed by one or more processors, and a specific implementation manner of the computer-readable storage medium of the present invention is substantially the same as that of each embodiment of the search system establishment method, and is not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A search system establishment method, the method comprising:
responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client, and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template;
analyzing the initial table building sentence to obtain a table name of a table to be built and a field type of a table head field, and converting the initial table building sentence into a target table building sentence based on a mapping relation between the field type of a table building sentence template and a standard field type of a search engine;
establishing a search engine server cluster requesting to distribute idleness for the search system and establishing an index database based on the target table building statement and the table name;
respectively obtaining program templates corresponding to the index building component, the search component and the deletion component, generating the index building component, the search component and the deletion component based on the target table building statement and the program templates, and importing source data corresponding to the search system building request into the index database based on the index building component to obtain a target search system.
2. The search system building method of claim 1, said generating an indexing component, a search component, and a deletion component based on said target table building statement and said program template, comprising:
analyzing the target table building sentence to obtain the name and the format of the table header field of the table to be built, and generating an entity class based on the name and the format of the table header field;
generating an index building component based on the entity class and a program template corresponding to the index building component;
extracting a field name list with the attribute of a searchable field from the target table building statement, and generating a search component based on the field name list, the entity class and a program template corresponding to the search component;
and generating a deleting component based on the entity class and the program template corresponding to the deleting component.
3. The search system establishment method of claim 1, wherein the method further comprises:
when an index library capacity expansion request sent by a user based on a second client is received, displaying a capacity expansion configuration interface to a display interface of the second client, and receiving a target index library and a field to be fragmented input by the user based on the capacity expansion configuration interface;
and inputting the key value of the field to be fragmented into a preset calculation formula for calculation, and distributing the data in the target index database to different nodes of the search engine server cluster based on the calculated numerical value.
4. The search system establishment method according to claim 3, wherein the predetermined calculation formula is:
K％N＝＝0(1)
K％N＝＝1 (2)
…….
K％N＝＝(N-1) (N)
k is the key value of the field to be fragmented, and N is the total number of nodes after capacity expansion;
data satisfying equation (1) will be stored to the first node, data satisfying equation (2) will be stored to the second node, … …, and data satisfying equation (N) will be stored to the nth node.
5. The search system establishment method of claim 1, wherein the method further comprises:
and acquiring the resource utilization rate of a server to which the search system belongs in real time or at regular time, and executing current limiting and degrading processing on the search system when the resource utilization rate exceeds a first threshold value.
6. The search system establishment method of claim 1, wherein the method further comprises:
and when a data writing request sent by a user based on a third client is received, judging whether the data volume of the data to be written is larger than a second threshold, and if so, executing asynchronous writing operation on the data to be written.
7. The method for establishing a search system according to claim 1, wherein the program template corresponding to the indexing component includes an API interface name written in an index library, a source data name, field version information, and action occurrence time information;
the program template corresponding to the search component comprises a sorting parameter, a paging parameter, a search field and a return field.
8. A search system establishment apparatus, the apparatus comprising:
the response module is used for responding to a search system establishment request sent by a user based on a first client, sending a table establishment statement template to the first client and receiving an initial table establishment statement fed back by the first client based on the table establishment statement template;
the analysis module is used for analyzing the initial table building sentence to obtain the table name of the table to be built and the field type of the table head field, and converting the initial table building sentence into a target table building sentence based on the mapping relation between the field type of the table building sentence template and the standard field type of the search engine;
the distribution module is used for establishing a search engine server cluster requesting to distribute idle for the search system based on the target table building statement and the table name and establishing an index library;
and the generating module is used for respectively acquiring program templates corresponding to the index building component, the searching component and the deleting component, generating the index building component, the searching component and the deleting component based on the target table building statement and the program templates, and importing the source data corresponding to the searching system building request into the index library based on the index building component to obtain the target searching system.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a search system setup program executable by the at least one processor to enable the at least one processor to perform the search system setup method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a search system setup program executable by one or more processors to implement the search system setup method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483122.2A CN112506486A (en) | 2020-12-15 | 2020-12-15 | Search system establishing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483122.2A CN112506486A (en) | 2020-12-15 | 2020-12-15 | Search system establishing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112506486A true CN112506486A (en) | 2021-03-16 |
Family
ID=74972482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011483122.2A Pending CN112506486A (en) | 2020-12-15 | 2020-12-15 | Search system establishing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506486A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111641A (en) * | 2021-04-20 | 2021-07-13 | 上海渠杰信息科技有限公司 | Data operation method and equipment based on full-text search engine |
| CN113722319A (en) * | 2021-08-05 | 2021-11-30 | 平凯星辰(北京)科技有限公司 | Data storage method based on learning index |
| CN114116664A (en) * | 2021-11-19 | 2022-03-01 | 招联消费金融有限公司 | Database table building statement processing method and device, computer equipment and storage medium |
| CN116028598A (en) * | 2023-03-30 | 2023-04-28 | 紫金诚征信有限公司 | File generation method and device based on ES and computer readable medium |
-
2020
- 2020-12-15 CN CN202011483122.2A patent/CN112506486A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111641A (en) * | 2021-04-20 | 2021-07-13 | 上海渠杰信息科技有限公司 | Data operation method and equipment based on full-text search engine |
| CN113722319A (en) * | 2021-08-05 | 2021-11-30 | 平凯星辰(北京)科技有限公司 | Data storage method based on learning index |
| CN114116664A (en) * | 2021-11-19 | 2022-03-01 | 招联消费金融有限公司 | Database table building statement processing method and device, computer equipment and storage medium |
| CN114116664B (en) * | 2021-11-19 | 2024-04-02 | 招联消费金融股份有限公司 | Database table-building sentence processing method, device, computer equipment and storage medium |
| CN116028598A (en) * | 2023-03-30 | 2023-04-28 | 紫金诚征信有限公司 | File generation method and device based on ES and computer readable medium |
| CN116028598B (en) * | 2023-03-30 | 2023-05-30 | 紫金诚征信有限公司 | File generation method and device based on ES and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112506486A (en) | Search system establishing method and device, electronic equipment and readable storage medium | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN112001179A (en) | Named entity recognition method and device, electronic equipment and readable storage medium | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN114185895A (en) | Data import and export method and device, electronic equipment and storage medium | |
| CN112464619B (en) | Big data processing method, device and equipment and computer readable storage medium | |
| CN115543198A (en) | Method and device for lake entering of unstructured data, electronic equipment and storage medium | |
| CN115203435A (en) | Entity relation generation method and data query method based on knowledge graph | |
| CN116089535A (en) | Data synchronization method, device, equipment and storage medium | |
| CN114398346A (en) | Data migration method, device, equipment and storage medium | |
| CN113434542B (en) | Data relationship identification method and device, electronic equipment and storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN113722533A (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113590703A (en) | ES data importing method and device, electronic equipment and readable storage medium | |
| CN113254455A (en) | Dynamic configuration method and device of database, computer equipment and storage medium | |
| CN112256472A (en) | Distributed data calling method and device, electronic equipment and storage medium | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| CN114756564B (en) | Data processing method, device, equipment and medium for stream computing | |
| CN115114297A (en) | Data lightweight storage and search method and device, electronic equipment and storage medium | |
| CN115033605A (en) | Data query method and device, electronic equipment and storage medium | |
| CN115687384A (en) | UUID (user identifier) identification generation method, device, equipment and storage medium | |
| CN115186188A (en) | Product recommendation method, device and equipment based on behavior analysis and storage medium | |
| CN114911479A (en) | Interface generation method, device, equipment and storage medium based on configuration | |
| CN112667570A (en) | File access method, device, equipment and readable storage medium | |
| CN113868138A (en) | Method, system, equipment and storage medium for acquiring test data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |