KR20240106049A - Vehicle remote control method and control server performing the same - Google Patents

Abstract

A method of controlling a vehicle by a control server includes: acquiring an issue message of a vehicle controlled by the control server; In response to the issue message, activating objects associated with a teleoperation mode for remotely controlling the vehicle; And the teleoperation mode, which will include an operation of remotely controlling the vehicle in response to an input to the teleoperation mode by an administrator using the control server, remotely controls the vehicle based on predetermined commands. first mode; And it may include a second mode for remotely controlling the vehicle based on real-time input from the manager.

Description

Vehicle remote control method and control server performing the same {VEHICLE REMOTE CONTROL METHOD AND CONTROL SERVER PERFORMING THE SAME}

The disclosure below relates to a vehicle remote control method and a control server that performs the same.

An autonomous vehicle is a car that can drive on its own without driver intervention. Self-driving vehicles use various sensors to recognize the surrounding environment and drive autonomously when a destination is designated.

As autonomous vehicles develop, control systems that remotely control autonomous vehicles are also developing to ensure safe driving of autonomous vehicles. The control system monitors autonomous vehicles and can remotely control self-driving vehicles that have problems.

A method of controlling a vehicle by a control server according to an embodiment includes: acquiring an issue message of a vehicle controlled by the control server; In response to the issue message, activating objects associated with a teleoperation mode for remotely controlling the vehicle; And the teleoperation mode, which will include an operation of remotely controlling the vehicle in response to an input to the teleoperation mode by an administrator using the control server, remotely controls the vehicle based on predetermined commands. first mode; And it may include a second mode for remotely controlling the vehicle based on real-time input from the manager.

The first mode may be a mode in which one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again is received from the manager.

The second mode may be a mode that receives input for at least one of the manager's steering wheel, pedals, or gear.

The issue message is generated based on information detected by sensors mounted on the vehicle; Alternatively, it may be generated in response to issue reporting by a safety driver who assists the autonomous driving of the vehicle.

The activating operation may include providing an alarm for the vehicle within a real-time monitoring interface provided by the control server. The real-time monitoring interface includes information on a plurality of vehicles controlled by the control server; And a plurality of real-time image information captured by cameras mounted on the plurality of vehicles can be provided.

The activating operation may further include providing a vehicle centralized monitoring interface based on the manager's input in response to the alarm.

The vehicle intensive monitoring interface may provide status information of the vehicle and issue information of the vehicle.

The vehicle intensive monitoring interface may further provide a remote control object for displaying the objects. The activating operation may further include activating the objects associated with the teleoperation mode within the vehicle centralized monitoring interface in response to the administrator input for the remote control object.

The activating operation may include activating only some of the objects based on surrounding information of the vehicle or information detected by sensors mounted on the vehicle.

The objects may include an object associated with the first mode and an object associated with the second mode.

The controlling operation may include transmitting one of the predetermined commands to the vehicle in response to the manager's input for an object associated with the first mode.

The controlling operation may include providing an issue-generating vehicle control interface in response to the manager's input for the object associated with the second mode.

The issue vehicle control interface includes real-time image information captured by a camera mounted on the vehicle; And status information of the vehicle may be provided.

A control server according to an embodiment includes: a memory including instructions; and a processor for executing the instructions. When the instructions are executed by the processor, the processor may perform a plurality of operations. The plurality of operations may include obtaining an issue message of a vehicle controlled by the control server; In response to the issue message, activating objects associated with a teleoperation mode for remotely controlling the vehicle; and an operation of remotely controlling the vehicle in response to an input to the teleoperation mode by an administrator using the control server. The teleoperation mode includes: a first mode for remotely controlling the vehicle based on predetermined commands; And it may include a second mode for remotely controlling the vehicle based on real-time input from the manager.

The first mode may be a mode in which one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again is received from the manager.

The second mode may be a mode that receives input for at least one of the manager's steering wheel, pedals, or gear.

The issue message is generated based on information detected by sensors mounted on the vehicle; Alternatively, it may be generated in response to issue reporting by a safety driver who assists the autonomous driving of the vehicle.

The activating operation may include providing an alarm for the vehicle within a real-time monitoring interface provided by the control server. The real-time monitoring interface includes information on a plurality of vehicles controlled by the control server; And a plurality of real-time image information captured by cameras mounted on the plurality of vehicles can be provided.

The activating operation may further include providing a vehicle centralized monitoring interface based on an input of the existing manager in response to the alarm. The vehicle intensive monitoring interface may provide status information of the vehicle and issue information of the vehicle.

The vehicle intensive monitoring interface may further provide a remote control object for displaying the objects.

The activating operation may further include activating the objects associated with the teleoperation mode within the vehicle centralized monitoring interface in response to the administrator input for the remote control object.

The objects may include an object associated with the first mode and an object associated with the second mode.

1 is a diagram for explaining an autonomous driving method according to an embodiment.
Figure 2 is a block diagram showing hardware included in an autonomous driving device according to an embodiment.
Figure 3 is a diagram for explaining a control system for controlling an autonomous vehicle according to an embodiment.
Figure 4 is a block diagram showing hardware included in a control server according to an embodiment.
Figure 5 is a diagram for explaining a real-time monitoring interface according to an embodiment.
Figure 6 is a diagram for explaining a vehicle centralized monitoring interface according to an embodiment.
FIG. 7 is a diagram illustrating a control interface for a vehicle in which an issue occurs according to an embodiment.
Figure 8 shows a flowchart of a vehicle control method according to one embodiment.
FIG. 9 is a block diagram illustrating hardware included in an electronic device according to an embodiment.
Figure 10 is a diagram for explaining the main interface according to one embodiment.
11 and 12 are diagrams for explaining a reporting operation according to an embodiment.
Figure 13 shows a flowchart of a method for reporting driving issues of a vehicle according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. In this specification, terms such as "comprise" or "have" are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but are not intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

The term '~unit' used in this document refers to software or hardware components such as FPGA or ASIC, and '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. For example, '~part' refers to software components, object-oriented software components, components such as class components and task components, processes, functions, properties, procedures, May include subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card. Additionally, '~ part' may include one or more processors.

Hereinafter, 'vehicle' may refer to any type of transportation used to move people or objects with engine, such as a car, bus, motorcycle, kickboard, or truck.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

FIG. 1 is a diagram for explaining an autonomous driving method according to an embodiment, and FIG. 2 is a block diagram showing hardware included in an autonomous driving device according to an embodiment.

Referring to FIG. 1 , according to one embodiment, an autonomous driving device (eg, self-driving device 40 of FIG. 2 ) may be mounted on a vehicle to implement an autonomous vehicle 10 . The autonomous vehicle 10 may be a car that can drive on its own without driver intervention.

The autonomous driving device 40 mounted on the autonomous vehicle 10 may include various sensors (eg, the sensor unit 41 in FIG. 2) to collect surrounding situation information.

The autonomous driving device 40 may detect the movement of the preceding vehicle 20 running in front through an image sensor and/or an event sensor mounted on the front of the autonomous vehicle 10. The autonomous driving device 40 may further include sensors for detecting not only the front of the autonomous vehicle 10, but also other driving vehicles 30 running in the lane next to the autonomous vehicle 10 and pedestrians around the autonomous vehicle 10. You can.

At least one of the sensors for collecting situational information around the autonomous vehicle 10 may have a predetermined field of view (FoV), as shown in FIG. 1 . When the sensor mounted on the front of the autonomous vehicle 10 has a field of view (FoV) as shown in FIG. 1, information detected at the center of the sensor may have relatively high importance. This may be because the information detected at the center of the sensor includes most of the information corresponding to the movement of the preceding vehicle 20.

The autonomous driving device 40 controls the movement of the autonomous vehicle 10 by processing the information collected by the sensors of the autonomous vehicle 10 in real time, while at least some of the information collected by the sensors is stored in a memory device (e.g. : Can be stored in the memory system 47 of FIG. 2).

Referring to FIG. 2, according to one embodiment, the autonomous driving device 40 may include a sensor unit 41, a processor 46, a memory system 47, and a vehicle body control module 48.

The sensor unit 41 may include a plurality of sensors 42-45. The plurality of sensors 42-45 may include an image sensor, an event sensor, an illumination sensor, a GPS device, an acceleration sensor, etc. Data collected by sensors 42-45 may be transmitted to processor 46.

The processor 46 may store data collected by the sensors 42-45 in the memory system 47. The processor 46 may determine the movement of the vehicle by controlling the body control module 48 based on data collected by the sensors 42-45.

The memory system 47 may include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices may be provided as one semiconductor chip. In addition to the system controller of the memory system 47, each of the memory devices included in the memory system 47 may include a memory controller. The memory controller may include artificial intelligence (AI) computing circuitry, such as a neural network. The memory controller may generate calculation data by assigning a predetermined weight to data received from the sensors 42 - 45 or the processor 46 and store the calculation data in a memory chip.

The vehicle body control module 48 may receive commands from the processor 46 and control the movement of the vehicle.

Figure 3 is a diagram for explaining a control system for controlling an autonomous vehicle according to an embodiment.

Referring to FIG. 3, according to one embodiment, the control system 300 may include an autonomous vehicle 310 (eg, the autonomous vehicle 10 of FIG. 1) and a control server 320.

The autonomous vehicle 310 and the control server 320 may communicate using a network. For example, networks include Local Area Network (LAN), Wide Area Network (WAN), Value Added Network (VAN), mobile radio communication network, satellite communication network, and others. It may include a mutual combination of. The network is a comprehensive data communication network that allows each network constituent shown in FIG. 3 (e.g., autonomous vehicle 310 and control server 320) to communicate smoothly with each other, and includes wired Internet and wireless Internet. and mobile wireless communication networks. In addition, wireless communications include, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, ZigBee, WFD (Wi-Fi Direct), UWB (ultra-wideband), and infrared communication (IrDA). , infrared Data Association), NFC (Near Field Communication), etc., but are not limited to these.

The autonomous vehicle 300 refers to a car that can drive on its own without driver intervention. The self-driving vehicle 300 can recognize the surrounding environment using various sensors and drive autonomously when a destination is designated.

The control server 320 may be a server that monitors the autonomous vehicle 300 in real time and remotely controls the autonomous vehicle 300. Hereinafter, the operation of the control server 320 along with the hardware of the control server 320 will be described in detail.

Figure 4 is a block diagram showing hardware included in a control server according to an embodiment.

Referring to FIG. 4, according to one embodiment, the control server 320 may perform remote control of a vehicle (e.g., the autonomous vehicle 10 of FIG. 1 and the autonomous vehicle 300 of FIG. 3). . The control server 320 can perform remote control of the vehicle through teleoperation mode. The teleoperation mode may include a plurality of modes for adaptively performing remote control corresponding to the vehicle situation. The control server can adaptively perform remote control of the vehicle based on a teleoperation mode composed of a plurality of modes.

The control server 320 may support real-time monitoring of vehicles (e.g., autonomous vehicle 10 in FIG. 1 and autonomous vehicle 300 in FIG. 3). The control server 320 can support efficient monitoring of vehicles by providing an interface (e.g., user interface) for interaction with an administrator (e.g., a manager using the control server). The control server 320 may display an alarm for a vehicle in which an issue has occurred within the real-time monitoring interface. The control server 320 can support the manager's judgment of the situation and decision on response measures by clearly providing various information about the vehicle in which an issue occurs to the manager. The control server 320 can increase the manager's control accuracy and convenience by providing the manager with various information about the vehicle causing the issue through the form of a user interface.

The control server 320 may include a processor 330 and memory 340.

The processor 330 may process data stored in the memory 340. The processor 330 may execute computer-readable code (eg, software) stored in the memory 340 and instructions triggered by the processor 330.

The processor 330 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The memory 340 may be implemented as a volatile memory device or a non-volatile memory device.

Volatile memory devices may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Non-volatile memory devices include Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque (STT)-MRAM (MRAM), and Conductive Bridging RAM (CBRAM). , FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM (Nanotube RRAM), Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, molecular electronic memory device, or insulation resistance change memory.

The control server 320 provides administrators (e.g., managers using the control server 320) with interfaces (e.g., real-time monitoring interface, vehicle A centralized monitoring interface and an issue-occurring vehicle control interface) can be provided. The control server 320 can support remote control and monitoring of vehicles by providing interactive interfaces to managers. Descriptions related to the interface will be explained in detail through FIGS. 5 to 7.

The control server 320 may receive an issue message from a vehicle controlled by the control server (eg, the control server 320 in FIG. 3). The issue message is generated in response to issue reporting by a safety driver who assists the autonomous driving of the vehicle, or is information detected by sensors mounted on the vehicle (e.g., sensor unit 41 in FIG. 2). It may have been created based on .

The control server 320 may activate objects related to a teleoperation mode for remotely controlling a vehicle in response to an issue message.

The teleoperation mode may include a first mode of remotely controlling the vehicle based on predetermined commands. The first mode may be a mode in which one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again is received from the manager. The teleoperation mode may include a second mode in which the vehicle is remotely controlled based on the manager's real-time input. The second mode may be a mode that receives input for at least one of the manager's steering wheel, pedals, or gear. The second mode may be a mode for performing remote control of the vehicle even in situations where the first mode is not supported.

The control server 320 can remotely control the vehicle in response to the manager's input into the teleoperation mode. The control server 320 may transmit any one of predetermined commands to the vehicle in response to the manager's input related to the first mode. The predetermined commands may be commands that control the operation (e.g., driving) of the vehicle. The control server 320 may control the vehicle based on the manager's input associated with the second mode (eg, the manager's input to at least one of the steering wheel, pedal, or gear). Below, the interfaces provided by the control server 320 and the manager's interaction will be described in detail.

Figure 5 is a diagram for explaining a real-time monitoring interface according to an embodiment.

Referring to FIG. 5, according to one embodiment, a control server (e.g., control server 320 of FIG. 4) provides a real-time monitoring interface 500 that interacts with an administrator (e.g., an administrator using the control server). You can.

The manager interacts with an object included in the first area 510 of the real-time monitoring interface 500 to control a vehicle (e.g., the control server 320 of FIG. 3). Search conditions for the autonomous vehicle 310 can be set. By interacting with objects included in the first area 510, the manager can set conditions for operation area, operating company, operation route, vehicle type product, control status, driver name, vehicle information, and operation information.

The control server 320 may display information on a plurality of vehicles (eg, vehicles operating in Sejong) that meet the search conditions set by the administrator in the second area 520 of the real-time monitoring interface 500. The second area 520 may display the driving status, driver name, operating company, vehicle number, and code (eg, code associated with vehicle identification) of the vehicle that meets the search conditions. For example, referring to area 521, the vehicle's operating status (e.g., issue occurred), driver name (e.g., Director Kim), operating company (e.g., 42dot), vehicle number (e.g., Lim 0000), and code (eg CG21) may be shown.

The control server 320 sends a plurality of real-time image information captured by cameras mounted on a plurality of vehicles (e.g., vehicles that meet the search conditions set by the administrator) to the third area 530 of the real-time monitoring interface 500. ) can be displayed. The third area 530 may include an object 533 that can select a display method of vehicle-related information. When the manager interacts with the object 533 and selects a map view, the control server 320 may display real-time map information representing a plurality of vehicles in the third area 530.

When the control server 320 receives an issue message from one of a plurality of vehicles (e.g., a vehicle that meets the search conditions set by the administrator), the control server 320 generates an issue within the real-time monitoring interface 500. Alarms for vehicles can be provided. For example, the control server 320 may change the color of the area 521 containing information about the vehicle causing the issue within the second area 520 (eg, change it to red). Additionally, the control server 320 may change the color of the area 531 containing the image information of the vehicle causing the issue (eg, change it to red) within the third area 530.

In response to the administrator's input in response to the alarm (e.g., clicking on the area 521 or area 531), the control server 320 may activate the object 532. Object 532 may be an object for converting to an interface that can intensively monitor a specific vehicle. In response to the administrator's input (e.g., click) on the activated object 532, the control server 320 may provide an intensive monitoring interface for the selected vehicle (e.g., the vehicle in which an issue occurs).

Figure 6 is a diagram for explaining a vehicle centralized monitoring interface according to an embodiment.

Referring to FIG. 6, according to one embodiment, a control server (e.g., control server 320 of FIG. 4) provides a vehicle centralized monitoring interface 600 that interacts with an administrator (e.g., a manager using the control server). can do.

The vehicle intensive monitoring interface 600 may include information related to the selected vehicle (eg, the vehicle with an issue selected in FIG. 5 ). Like FIG. 5 , the control server 320 may maintain the color of the area 601 containing information about the vehicle causing the issue (e.g., keep it red).

The control server 320 may display status information of a vehicle (eg, a vehicle in which an issue occurs) in the first area 610 of the vehicle intensive monitoring interface 600. The vehicle's status information includes driving issue information (e.g., emergency on the road), vehicle shape, speed limit (e.g., 50 km), vehicle's current speed (e.g., 42 km), and vehicle gear information (e.g., D). can do.

The control server 320 may display a remote control object 611 in the first area 610 of the vehicle intensive monitoring interface 600. The administrator may interact (e.g., click) with the remote control object 611 to activate objects associated with a teleoperation mode (e.g., a teleoperation mode for remotely controlling a vehicle).

The teleoperation mode may include a first mode in which a vehicle (eg, a vehicle in which an issue occurs) is remotely controlled based on predetermined commands. The first mode may be a mode in which one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again is received from the manager. The teleoperation mode may include a second mode in which the vehicle is remotely controlled based on the manager's real-time input. The second mode may be a mode that receives input for at least one of the manager's steering wheel, pedals, or gear. The second mode may be a mode for performing remote control of the vehicle even in situations where the first mode is not supported.

Objects associated with a teleoperation mode may include an object associated with a first mode and an object associated with a second mode. The control server 320 may activate only some of the objects associated with the teleoperation mode based on information about the vehicle's surroundings or information detected by sensors mounted on the vehicle. For example, if there is no shoulder around the vehicle, the control server 320 may not activate the shoulder stopping object. For example, when other vehicles driving quickly near the vehicle are detected, the control server 320 may not activate the waiting object.

The administrator can remotely control the vehicle (e.g., the vehicle in question) by interacting (e.g., clicking) with objects associated with the teleoperation mode. The vehicle intensive monitoring interface 600 may include additional information to support the manager's judgment of the situation and decision of response measures for the vehicle in which an issue occurs. The control server 320 may provide images captured by a camera mounted on a vehicle (eg, a vehicle in which an issue occurs) through the second area 620 of the vehicle intensive monitoring interface 600.

The control server 320 may display real-time map information representing a vehicle (eg, a vehicle causing an issue) through the third area 630 of the vehicle intensive monitoring interface 600. The control server 320 may display an exclamation point object 631 next to the vehicle in which an issue occurs. The manager can quickly check the real-time location of the vehicle in question through the exclamation mark object 631.

The control server 320 may provide vehicle issue information through the area 631. The issue information of the vehicle may include the driver's name, vehicle number, current location of the vehicle, current number of passengers in the vehicle, and time of issue occurrence. The administrator may release the issue by interacting (e.g., clicking) with the issue release object 632 included in the area 631.

FIG. 7 is a diagram illustrating a control interface for a vehicle in which an issue occurs according to an embodiment.

Referring to FIG. 7, according to one embodiment, a control server (e.g., control server 320 of FIG. 4) uses an issue vehicle control interface 700 to interact with an administrator (e.g., a manager using the control server). can be provided.

In response to the manager's interaction (eg, clicking) with an object associated with the second mode, the control server 320 may provide the issue-generating vehicle control interface 700. As described above, the second mode may be a mode in which the vehicle is remotely controlled based on the manager's real-time input (eg, input to at least one of the steering wheel, pedal, or gear). The issue vehicle control interface 700 can maximize and display real-time images captured by a camera attached to the issue vehicle. The issue-generating vehicle control interface 700 may display status information of the vehicle in the area 710. The vehicle's status information may include current speed (e.g., 52 km), speed limit (e.g., 50 km), and gear information (e.g., D).

The trigger for controlling the issue-generating vehicle based on the second mode is not limited to the manager's selection (eg, clicking on an object associated with the second mode). For example, the second mode may be initiated based on the safety driver's selection to assist autonomous driving of the vehicle. Additionally, the second mode may be initiated according to the self-determination algorithm of the control server 320.

Figure 8 shows a flowchart of a vehicle control method according to one embodiment.

Referring to FIG. 8, according to one embodiment, operations 810 to 830 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

In operation 810, the control server (e.g., control server 320 in FIG. 4) controls a vehicle (e.g., autonomous vehicle 310 in FIG. 3) controlled by the control server (e.g., control server 320 in FIG. 3). You can receive issue messages from .

In operation 820, the control server 320 may activate objects associated with a teleoperation mode for remotely controlling a vehicle in response to an issue message.

In operation 830, the control server 320 may remotely control the vehicle in response to an input to the teleoperation mode from an administrator using the control server.

Operations 810 to 830 may be substantially the same as the operations of the control server 320 described with reference to FIGS. 1 to 7 . Accordingly, detailed description will be omitted.

FIG. 9 is a block diagram illustrating hardware included in an electronic device according to an embodiment.

Referring to FIG. 9, according to one embodiment, the electronic device 900 may be the same as the autonomous driving device (e.g., the autonomous driving device 40 of 2) and may be implemented as part of the autonomous driving device 40. It may be possible. The electronic device 900 may be mounted on an autonomous vehicle (eg, the autonomous vehicle 310 of FIG. 3).

The electronic device 900 may support immediate reporting of driving issues occurring in a vehicle (eg, autonomous vehicle). The electronic device 900 can contribute to responding to driving issues within the golden time by supporting immediate driving issue reporting. The electronic device 900 can improve the reliability of driving issue information by supporting immediate driving issue reporting.

The electronic device 900 can contribute to systematic management of driving issues. Driving issue information generated by the electronic device 900 may be automatically preprocessed within the control system (e.g., the control system 300 of FIG. 3). Driving issue information generated by the electronic device 900 can be automatically classified and quantified. The electronic device 900 can contribute to systematic management of driving issues by generating driving issue information in a unified format.

Driving issue information generated by the electronic device 900 may be used by a control server (e.g., control server 320 in FIG. 3) to set a teleoperation mode for remotely controlling the vehicle. The electronic device 900 may contribute to adaptive remote control corresponding to the situation of the vehicle (eg, autonomous vehicle).

The electronic device 900 may include a processor 910 and a memory 920.

The processor 910 may process data stored in the memory 920. The processor 910 may execute computer-readable code (eg, software) stored in the memory 920 and instructions triggered by the processor 910 .

The processor 910 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The memory 920 may be implemented as a volatile memory device or a non-volatile memory device.

Volatile memory devices may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Non-volatile memory devices include Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque (STT)-MRAM (MRAM), and Conductive Bridging RAM (CBRAM). , FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM (Nanotube RRAM), Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, molecular electronic memory device, or insulation resistance change memory.

The electronic device 900 provides a main interface (e.g., FIG. 10 main interfaces (1000) can be provided. The main interface will be described in detail with reference to FIGS. 10 to 12.

The electronic device 900 may generate driving issue information of the vehicle based on the interaction between the safety driver and the main interface. Driving issue information may include the type of issue, the severity of the issue, and/or the time of occurrence of the issue.

Issue types may include vehicle (e.g., autonomous vehicle) perception issues, steering issues, acceleration/deceleration issues, system issues, and other issues. Issue types may include vehicle (e.g., autonomous vehicle) recognition issues, stopping issues, acceleration issues, deceleration issues, steering issues, traffic light issues, lane change issues, V2X issues, GPS issues, and system issues.

The electronic device 900 may suggest a type of issue to a safe driver based on information detected by sensors mounted on the vehicle or information surrounding the vehicle. For example, when a problem occurs with a GPS sensor, the electronic device 900 may suggest that the type of issue occurring in the vehicle is a GPS issue. For another example, when visibility is difficult due to fog around the vehicle, the processor 910 may suggest that the type of issue occurring in the vehicle is a cognitive issue.

The severity of the issue may be determined by the safety driver himself. Additionally, processor 910 may suggest a severity range for the issue. The severity range of the issue may be generated based on information detected by sensors mounted on the vehicle (e.g., sensor unit 41 in FIG. 2).

The time of occurrence of an issue may be recorded in the form of a timestamp. The time of occurrence of an issue may correspond to the time of issue reporting by a safe driver. also. The time of occurrence of an issue may correspond to the time of occurrence of an issue recognized by the vehicle.

Figure 10 is a diagram for explaining the main interface according to one embodiment.

Referring to FIG. 10 , according to one embodiment, an electronic device (eg, the electronic device 900 of FIG. 9 ) may provide a main interface 1000 with which a safety driver can interact.

The electronic device 900 may display information about the driving route in the first area 1010 of the main interface 1000. The first area 1010 may display the surrounding environment of the vehicle 1011 (eg, a vehicle to which the electronic device 900 is attached).

The first area 1010 may include an icon corresponding to the vehicle 1011. The icon corresponding to the vehicle 1011 may be based on vehicle information. For example, when the vehicle 1011 is a passenger car, the icon corresponding to the vehicle 1011 may be an icon in the shape of a passenger car. For example, if the vehicle 1011 is a van, the icon corresponding to the vehicle 1011 may be a van-shaped icon.

The first area 1010 may include a lane. Lanes may be created based on road information (eg, road information received based on location information of the vehicle 1011). Lanes may be created based on information on lanes around the vehicle 1011.

The first area 1010 may include icons corresponding to other vehicles located around the vehicle 1011. The shape of the icon corresponding to another vehicle may be determined based on information about the other vehicle. The location of the icon corresponding to another vehicle may be determined based on the location information of the other vehicle (eg, location information relative to the reference vehicle 1011). Information about other vehicles or location information of other vehicles may be recognized from a sensor attached to the vehicle 1011. The electronic device 900 may place icons corresponding to other vehicles at positions relative to the other vehicles with respect to the vehicle 1011.

The first area 1010 may include icons corresponding to pedestrians located around the vehicle 1011. Pedestrian location information may be recognized from a sensor attached to the vehicle 1011. The electronic device 900 may place an icon corresponding to the pedestrian at the pedestrian's relative position with respect to the vehicle 1011.

The electronic device 900 may generate an icon corresponding to the vehicle 1011, a lane, an icon corresponding to another vehicle, or an icon corresponding to a pedestrian. Based on the received or collected information, the electronic device 900 arranges a lane, an icon corresponding to another vehicle, or an icon corresponding to a pedestrian in the first area 1010, centered on the icon corresponding to the vehicle 1011. can do.

The electronic device 900 may display a guidance route in the second area 1020 of the main interface 1000. The second area 1020 may include information about the destination, a navigation map, and driving method guidance. The guidance route of the second area 1020 may be generated based on information about the destination, location information of the vehicle, road information, information about traffic volume, etc.

The electronic device 900 may display the status of the autonomous driving function in the third area 1030 of the main interface 1000. The state of the autonomous driving function may refer to the operating state of one or more devices, functions, etc. related to autonomous driving that are mounted to implement autonomous driving of the vehicle. The electronic device 900 may display whether one or more devices, functions, etc. related to autonomous driving are operating normally through the third area 1030. This is provided as a simple example, and any suitable way to indicate the operating status of a device, function, etc. may be applied.

The electronic device 900 may display a report object 1040 within the main interface 1000. In response to the safety driver's input (eg, touch input) on the report object 1040, the electronic device 900 may display an issue type object within the main interface 1000. The issue type object will be explained with reference to FIGS. 11 and 12.

11 and 12 are diagrams for explaining a reporting operation according to an embodiment.

Referring to FIG. 11, according to one embodiment, an electronic device (e.g., the electronic device 900 of FIG. 9) includes an issue type object 1041 with which a safety driver can interact (e.g., touch input). ) can be displayed in the main interface 1000. Issue type objects 1041 may each indicate issue types included in driving issue information. Issue type objects 1041 may include objects corresponding to vehicle recognition issues, steering issues, acceleration/deceleration issues, system issues, and other issues.

Referring to FIG. 12 , according to one embodiment, the electronic device 900 may display issue type objects 1042 in the main interface 1000. The number of issue type objects may not be limited to 5 as shown in FIG. 11. Issue type objects 1042 may include objects corresponding to vehicle recognition issues, stopping issues, acceleration issues, deceleration issues, steering issues, traffic light issues, lane change issues, V2X issues, GPS issues, and system issues. .

The electronic device 900 may suggest a type of issue to the safe driver based on information detected by sensors mounted on the vehicle or information around the vehicle. For example, when a problem occurs with a GPS sensor, the electronic device 900 may suggest that the type of issue occurring in the vehicle is a GPS issue. For another example, when visibility is difficult due to fog around the vehicle, the processor 910 may suggest that the type of issue occurring in the vehicle is a cognitive issue.

The severity of the issue may be determined by the safety driver himself. Additionally, processor 910 may suggest a severity range for the issue. The severity range of the issue may be generated based on information detected by sensors mounted on the vehicle (e.g., sensor unit 41 in FIG. 2).

As described above, the electronic device 900 may generate vehicle driving issue information based on the interaction between the safety driver and the main interface. Driving issue information may include the type of issue, the severity of the issue, and/or the time of occurrence of the issue. Among the driving issue information, the time of issue occurrence may be recorded in the form of a timestamp. The time of occurrence of an issue may correspond to the time of issue reporting by a safe driver. also. The time of occurrence of an issue may correspond to the time of occurrence of an issue recognized by the vehicle.

The electronic device 900 may support immediate reporting of driving issues occurring in a vehicle (eg, autonomous vehicle). The electronic device 900 can contribute to responding to driving issues within the golden time by supporting immediate driving issue reporting. The electronic device 900 can improve the reliability of driving issue information by supporting immediate driving issue reporting.

Figure 13 shows a flowchart of a method for reporting driving issues of a vehicle according to an embodiment.

Referring to FIG. 13, according to one embodiment, operations 1310 to 1330 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

In operation 1310, an electronic device (e.g., electronic device 900 of FIG. 9) is controlled by a vehicle (e.g., autonomous vehicle 310 of FIG. 3) controlled by a control server (e.g., control server 320 of FIG. 3). Through the internal console, it is possible to provide a main interface through which the safety driver who assists the vehicle's autonomous driving can interact.

In operation 1320, the electronic device 900 may generate driving issue information of the vehicle based on the interaction between the safety driver and the main interface.

In operation 1330, the electronic device 900 may transmit driving issue information to the control server that controls the vehicle.

Operations 1310 to 1330 may be substantially the same as the operations of the electronic device 900 described with reference to FIGS. 9 to 12 . Accordingly, detailed description will be omitted.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (20)

In the vehicle control method of the control server,
Obtaining an issue message of a vehicle controlled by the control server;
In response to the issue message, activating objects associated with a teleoperation mode for remotely controlling the vehicle; and
An operation of remotely controlling the vehicle in response to an input to the teleoperation mode by an administrator using the control server.
Including,
The teleoperation mode is,
A first mode for remotely controlling the vehicle based on predetermined commands; and
A second mode for remotely controlling the vehicle based on real-time input from the manager
Including a vehicle control method.
According to paragraph 1,
The first mode is,
A mode for receiving, from the manager, any one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again,
How to control the vehicle.
According to paragraph 1,
The second mode is,
A mode for receiving input for at least one of the manager's steering wheel, pedals, or gear,
How to control the vehicle.
According to paragraph 1,
The issue message is,
It is generated based on information detected by sensors mounted on the vehicle; or
Generated in response to issue reporting by a safety driver who assists the autonomous driving of the vehicle,
How to control the vehicle.
According to paragraph 1,
The activating operation is,
An operation of providing an alarm for the vehicle within a real-time monitoring interface provided by the control server.
Including,
The real-time monitoring interface is,
Information on a plurality of vehicles controlled by the control server; and
A plurality of real-time image information captured by cameras mounted on the plurality of vehicles, respectively
providing a vehicle control method.
According to clause 5,
The activating operation is,
An operation of providing a vehicle centralized monitoring interface based on the manager's input in response to the alarm.
It further includes,
The vehicle intensive monitoring interface is,
A vehicle control method that provides status information of the vehicle and issue information of the vehicle.
According to clause 6,
The vehicle intensive monitoring interface is,
further providing a remote control object for displaying the objects,
The activating operation is,
In response to the administrator input for the remote control object, activating the objects associated with the teleoperation mode within the vehicle centralized monitoring interface.
A vehicle control method further comprising:
According to paragraph 1,
The activating operation is,
An operation of activating only some of the objects based on surrounding information of the vehicle or information detected by sensors mounted on the vehicle
Including a vehicle control method.
According to paragraph 1,
The objects are,
A vehicle control method including an object associated with the first mode and an object associated with the second mode.

According to clause 9,
The controlling operation is,
An operation of transmitting any one of the predetermined commands to the vehicle in response to the manager's input for an object associated with the first mode.
Including a vehicle control method.
According to clause 9,
The controlling operation is,
An operation of providing a vehicle control interface in which an issue occurs in response to the manager's input for an object associated with the second mode.
Including a vehicle control method.
According to clause 11,
The vehicle control interface where the issue occurred is,
Real-time video information captured by a camera mounted on the vehicle; and
Status information of the vehicle
providing a vehicle control method.
In the control server,
memory containing instructions; and
Processor for executing the instructions
Including,
When the instructions are executed by the processor, the processor performs a plurality of operations,
The plurality of operations are:
Obtaining an issue message of a vehicle controlled by the control server;
In response to the issue message, activating objects associated with a teleoperation mode for remotely controlling the vehicle; and
An operation of remotely controlling the vehicle in response to an input to the teleoperation mode by an administrator using the control server.
Including,
The teleoperation mode is,
A first mode for remotely controlling the vehicle based on predetermined commands; and
A second mode for remotely controlling the vehicle based on real-time input from the manager
Including a control server.
According to clause 13,
The first mode is,
A mode for receiving, from the manager, any one of inputs corresponding to stopping the vehicle on the shoulder, waiting, or driving again,
Control server.
According to clause 13,
The second mode is,
A mode for receiving input for at least one of the manager's steering wheel, pedals, or gear,
Control server.
According to clause 13,
The issue message is,
It is generated based on information detected by sensors mounted on the vehicle; or

Generated in response to issue reporting by a safety driver who assists the autonomous driving of the vehicle,
Control server.
According to clause 13,
The activating operation is,
An operation of providing an alarm for the vehicle within a real-time monitoring interface provided by the control server.
Including,
The real-time monitoring interface is,
Information on a plurality of vehicles controlled by the control server; and
A plurality of real-time image information captured by cameras mounted on the plurality of vehicles, respectively
A control server that provides.
According to clause 17,
The activating operation is,
An operation of providing a vehicle centralized monitoring interface based on the manager's input in response to the alarm.
It further includes,
The vehicle intensive monitoring interface is,
A control server that provides status information of the vehicle and issue information of the vehicle.
According to clause 18,
The vehicle intensive monitoring interface is,
further providing a remote control object for displaying the object,
The activating operation is,
In response to the administrator input for the remote control object, activating the objects associated with the teleoperation mode within the vehicle centralized monitoring interface.
A control server further comprising:
According to clause 13,
The objects are,
A control server including an object associated with the first mode and an object associated with the second mode.

Images