GB2579539A

GB2579539A - Vehicle controller

Info

Publication number: GB2579539A
Application number: GB1816174.5A
Authority: GB
Inventors: Hasedzic Elvir; Munoz Mauricio; Valentin Gheorghe Ionut
Original assignee: Jaguar Land Rover Ltd
Current assignee: Jaguar Land Rover Ltd
Priority date: 2018-10-04
Filing date: 2018-10-04
Publication date: 2020-07-01
Anticipated expiration: 2038-10-04
Also published as: GB2579539B

Abstract

The vehicle controller 100 comprises: an input 6, 7 which receives data 61, 71 concerning an area around a vehicle; a processing module 14 which analyzes the data to detect the presence of a user and determine one or more body landmarks of the user within the data and subsequently determine the user's intent in accordance with the position of the or each body landmark; and a control module 15 which issues a control signal to initiate an vehicular action dependent on the user's intent. The body landmarks are preferably body parts. The user’s intent is inferred from user’s movement, such as the user approaching the vehicle or making a gesture. Preferably, the initiated action is to open a door of the vehicle. Preferably, the body landmark positions are tracked relative to the vehicle over a time period, where a user trajectory can be determined, and the tracked movement of the body landmarks can be compared with a gait pattern. The vehicle controller preferably receives a signal from a keyfob, the action only initiated if the signal from the keyfob matches a stored keyfob identifier. A method determining a user’s intent to access a vehicle is also disclosed.

Description

Vehicle Controller

TECHNICAL FIELD

The present disclosure relates to a controller and control system for initiating actions on a vehicle. Aspects of the invention relate to a vehicle controller, a vehicle control system, to a vehicle and to a method.

BACKGROUND

It is an ongoing challenge within the automotive industry to improve vehicle functionality and design, and in turn enhance the perceived quality and 'feel' of vehicles, without adding significant additional cost.

One area that has received interest in this respect is vehicle access systems. There is a growing desire to provide a seamless entry process that creates an effect of 'welcoming' a user to a vehicle, while maintaining the robustness of vehicle access systems against misuse, to preserve vehicle security.

More widely, vehicle personalisation, namely the alignment of vehicle functions and features with specific user requirements, is an increasingly common aim.

In this context, keyless vehicle door access systems are well-known, in which the doors of a vehicle are automatically unlocked upon detection of a key fob carried by an approaching user in the vicinity of the vehicle. For example, the position of the key fob may be monitored using a GPS tracking signal. Such door access systems create a seamless entry experience, insofar as the user need not take any action to unlock the vehicle other than to carry the key fob.

Some more advanced door access systems take this principle a step further by dispensing with the need to carry a key fob. For example, a facial recognition algorithm may be used to compare an image of an approaching user captured by a dedicated on-board image capture device with a user identifier held in memory, the user identifier corresponding to an authenticated user registered with the system. If the approaching user is determined to be authenticated, the vehicle doors are unlocked.

In each of the above examples, although door unlocking may be automated the user must still open a selected vehicle door manually. To address this, it is known for door access systems to enable a user to open a vehicle door remotely by pressing a button on a key fob, although this reintroduces a requirement for the user to take positive action and operate the key fob to achieve entry.

In other known systems, users may open a vehicle door by performing particular gestures' (e.g. hand waving, etc.). On detecting such a gesture, the door access system opens the appropriate door for the user. Known systems of this type rely on the user interfering with infrared beam light gates created locally around the vehicle, the system detecting a gesture when the beams are broken in a particular sequence corresponding to a predetermined gesture. In an alternative approach, other systems apply facial recognition or related techniques to images captured by a camera or other sensor to detect gestures. In each of these approaches, however, the user must stand very close to the vehicle for a gesture to be detected reliably This restricts the versatility and usefulness of this type of door access system, particularly as the user must be so close to the vehicle for a gesture to be detected that they could simply open the vehicle door manually.

Furthermore, the requirement for a user to stop in front of the vehicle to perform a door-unlocking gesture, or to allow time for facial recognition, prevents a truly seamless entry.

It is against this background that the present invention has been devised.

SUMMARY OF THE INVENTION

Aspects of the present invention provide a vehicle controller, a vehicle control system, a vehicle, a method, a processor, a computer program product and a non-transitory computer-readable medium.

According to an aspect of the present invention there is provided a vehicle controller configured to determine a user's intent to access a vehicle, the controller comprising: an input configured to receive data relating to an area in the vicinity of the vehicle; a processing module configured to analyse the data to detect the presence of a user and determine one or more body landmarks of the user within the data, and determine the user's intent in accordance with the position or spatial arrangement of the or each body landmark; and a control module configured to issue a control signal to initiate an action on the vehicle in dependence on the user's intent.

By determining body landmarks for a user identified in the data, the pose and any movement of the user can be tracked when the user is at a significant distance from the vehicle. This allows the controller to analyse the behaviour of an approaching user to determine their intent over a longer period than in prior art systems that can only monitor users stood directly adjacent to the vehicle. Moreover, by monitoring a user from a greater distance, the controller can initiate an action before the user arrives.

Accordingly, determining the user's intent in accordance with the position of each body landmark may entail analysing a volumetric spatial arrangement of the body landmarks, as extracted from multiple time snap-shots of sensory input for a particular user, so that the intent of vehicle ingress can be inferred from a kinematic model. It follows that the action on the vehicle may be initiated in accordance with the spatial and temporal position of the or each body landmark.

Each body landmark of the user may correspond to a respective predetermined body part, including joints. For example, the body landmarks may be defined as virtual points mapped onto joints belonging to a virtual skeletal approximation of the anthropometric form of the user. The relative distances between these body landmarks may be characterised as being unique to a particular user.

In an embodiment, the data relates to the area adjacent to the vehicle over a time period, in which case the processing module is configured to track the position of the or each body landmark relative to the vehicle over the time period.

The processing module may be configured to determine the user's intent by comparing the tracked movement of each body landmark to a respective predetermined movement.

In some embodiments, the processing module is configured to determine the user's intent by comparing the tracked movement of the or each body landmark with a gait pattern.

The processing module may be configured to analyse the tracked movement of the or each body landmark to determine a trajectory of the user relative to the vehicle, in which case the processing module may be configured to determine the user's intent by determining whether the trajectory of the user indicates that the user is approaching the vehicle. For example, the processing module may be configured to determine the user's intent by determining whether the trajectory of the user indicates that the user is approaching a body aperture closure member of the vehicle, such as a door, boot lid, bonnet or fuel filler cap.

In some embodiments, the processing module is configured to determine an identity of the user based on the tracked movement of the or each body landmark.

Determining the user's intent may comprise analysing the one or more body landmarks to detect a gesture.

The data may comprise image data corresponding with one or more images acquired by one or more image capture devices, in which case the processing module may be configured to analyse at least one of the one or more images to determine an identity of the user. In embodiments in which the identity of the user is determined, the control module is optionally configured to issue the control signal to initiate the action on the vehicle in dependence on the identity of the user.

The data may comprise any one or more of the following: data acquired by a camera; point cloud data; data acquired by a radar system; data acquired by a LIDAR system; data acquired by a Bluetooth system; and data acquired by an ultrasonic system.

The action on the vehicle may comprise operating a body aperture closure member of the vehicle, for example by opening or closing a door, and/or locking or unlocking the door.

In some embodiments, the processing module is configured to analyse the data to detect one or more hazards in the area in the vicinity of and adjacent to the vehicle, and to inhibit unlocking and/or opening of the door if a hazard is detected.

The processing module optionally includes a neural network.

The processing module may be configured to determine a plurality of body landmarks of the user, and to generate a skeletal representation of the user based on the plurality of body landmarks. The skeletal representation may be realised as a kinematic model, for example.

The vehicle controller may comprise a further input configured to receive a notification signal.

The notification signal may comprise a signal from a key fob for the vehicle, in which case the vehicle controller comprises a key fob identification module configured to receive the notification signal, and the control module is configured to issue the control signal to initiate the action on the vehicle only if a signal received by the key fob identification module corresponds to a stored key fob identifier.

The input for receiving data relating to an area in the vicinity of the vehicle may comprise an electronic processor having an electrical input for receiving said data, and an electronic memory device electrically coupled to the electronic processor and having instructions stored therein. The processing module may be configured to access the memory device and execute the instructions stored therein such that it is operable to determine the user's intent.

The invention also extends to a vehicle control system comprising the controller of the above aspect.

The vehicle control system may comprise one or more scanning devices, in which case the control module is configured to issue control signals to operate the or each scanning device to acquire the data.

Another aspect of the invention provides a vehicle comprising the vehicle control system of the above aspect.

According to another aspect of the present invention, there is provided a method of determining a user's intent to access a vehicle, the method comprising acquiring data relating to an area in the vicinity of the vehicle; analysing the data to identify a user and determine one or more body landmarks of the user within the data; determining the user's intent in accordance with the position of the or each body landmark; and initiating an action on the vehicle in dependence on the user's intent.

Other aspects of the invention provide a computer program product executable on a processor so as to implement the method of the above aspect, a non-transitory computer readable medium loaded with such a computer program product, and a processor arranged to implement the method or the computer program product of the above aspects.

Within the scope of this application it is expressly intended that the various aspects, embodiments, examples and alternatives set out in the preceding paragraphs, in the claims and/or in the following description and drawings, and in particular the individual features thereof, may be taken independently or in any combination. That is, all embodiments and/or features of any embodiment can be combined in any way and/or combination, unless such features are incompatible. The applicant reserves the right to change any originally filed claim or file any new claim accordingly, including the right to amend any originally filed claim to depend from and/or incorporate any feature of any other claim although not originally claimed in that manner.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 is a schematic diagram of a vehicle comprising a vehicle control system according to an embodiment of the invention; Figure 2 is a schematic block diagram of a vehicle door access controller of the vehicle of Figure 1; Figure 3 is a simplified representation of an image stored by the controller of Figure 2; and Figure 4 is a chart representing stages in a process for triggering an action on the vehicle of Figure 1.

DETAILED DESCRIPTION

The present invention relates to an approach for initiating an action on a vehicle based on a determination of an approaching user's intentions, as derived from analysis of sensor data obtained from on-board sensors.

For example, the initiated action may include operating a door access control system to unlock and open a particular vehicle door, if it is determined that the user intends to access the vehicle through that door. The vehicle doors may be controlled independently of one another or may be controlled as a group.

For purposes of this disclosure, it is to be understood that the control system described herein may comprise a control unit or computational device having one or more electronic processors. The vehicle and/or a system thereof may comprise a single control module or electronic controller or alternatively different functions of the controller(s) may be embodied in, or hosted in, different control modules or controllers.

As used herein, the terms 'controller' or 'control module' will be understood to include both a single control module or controller and a plurality of control modules or controllers collectively operating to provide the required control functionality. A set of instructions could be provided which, when executed, cause said control modules(s) to implement the control techniques described herein (including the method(s) described below). The set of instructions may be embedded in one or more electronic CPU or processors.

Alternatively, the set of instructions could be provided as software to be executed by one or more electronic processor(s). For example, a first control module may be implemented in software run on one or more electronic processors, and one or more other control modules may also be implemented in software run on one or more electronic processors, optionally the same one or more processors as the first control module. It will be appreciated, however, that other arrangements are also useful, and therefore, the present invention is not intended to be limited to any particular arrangement.

In any event, the set of instructions described above may be embedded in a computer-readable storage medium (e.g. a non-transitory storage medium) that may comprise any mechanism for storing information in a form readable by a machine or electronic processors/computational device, including, without limitation: a magnetic storage medium (e.g., floppy diskette); optical storage medium (e.g., CD-ROM); magneto-optical storage medium; read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or electrical or other types of medium for storing such information/instructions.

Referring now to Figures 1 and 2, a vehicle 10 suitable for use in embodiments of the invention includes a front door la and a rear door 1 b, each disposed on a driver's side of the vehicle 10 so that the doors may be referred to as the driver's door 1 a and the passenger door 1 b.

The driver's door la includes a lower door panel 2a beneath an upper door frame 2b that defines an opening for housing a driver's window 3.

Each door 1 a, 1 b has a door handle 4 located towards the rear of the door panel 2a, the door handle 4 being operable between open and closed positions to allow a user to open and close the door is when the door 1a is unlocked, to access the vehicle 10. Each door 1 a, 1 b further includes an electro-mechanical door opening assembly (not shown) of a kind that will be familiar to the skilled reader that is operable to open its respective door 1 a, 1 b, unlocking the door 1 a, 1 b first if necessary, on receipt of a door unlocking and/or opening control signal.

In one embodiment, the opening assemblies may employ electric actuators or motors to effect door unlocking and/or opening. Alternatively, unlocking and/or opening of the doors 1 may be facilitated by pressure springs or gas spring struts, namely gas-powered rods or bars designed to resist compression that facilitate the spring-like opening of doors, typically used in known vehicle-boot or vehicle-bonnet door unlocking and/or opening systems.

A lock assembly for the door handle (not shown) is electronically controlled and is operable by means of a key fob 9 carried by the user in the conventional manner. For example, the key fob 9 may transmit a wireless signal 91 that is received by an on-board vehicle controller in the form of a door access controller 100 (as described further below), which is configured to determine whether the key fob 9 is an authorised key fob for the vehicle 10 and to control the state of the door locking and door opening assemblies accordingly. The door access controller 100 therefore forms part of a vehicle control system also including the locking and opening assemblies.

The vehicle control system also includes first (front) and second (rear) image capture devices 6, 7, which are located along an upper edge of the lower door panel 2a, immediately below the window 3.

In this embodiment, the image capture devices 6, 7 are stereoscopic cameras, although in other embodiments the image capture devices 6, 7 may include any suitable type of sensor that is configured to generate data that can represent a two-dimensional image of an area adjacent to and in the vicinity of the vehicle 10 towards which the device that produced the data is directed. Such data is hereafter referred to as 'scan data'. Suitable sensors include alternative types of camera, proximity sensors (e.g. ultrasonic proximity detectors), light-detecting sensors such as Light Detection and Ranging (LIDAR) sensors, or Radio Detection and Ranging (RADAR) sensors, for example. It follows that the scan data 61, 71 may comprise any one or more of the following: photographic images; point cloud data; data acquired by a radar system; data acquired by a LIDAR system; data acquired by a Bluetooth system; and data acquired by an ultrasonic system.

The rear image capture device 6 is located immediately behind the front image capture device 7, along a longitudinal axis of the vehicle 10, and is oriented to capture two-dimensional scan data in the form of still or moving images (e.g. as a video feed) relating to a user approaching the vehicle 10 from the rear. The front image capture device 7 is oriented in an opposite manner to capture scan data in the form of still or moving images relating to a user approaching the vehicle 10 from the front.

The scan data 61, 71 detected by the front and rear image capture devices 6, 7 is transmitted to the door access controller 100 either wirelessly or by hardwiring through the vehicle door la. It is particularly convenient to mount the image capture devices 6, 7 in this position as they can be connected to an electrical wiring harness already present within the door 1 a for the electrical windows.

The front and rear image capture devices 6, 7 have a collective panoramic field of view that spans 180 degrees around the vertical. In this way, the front and rear image capture devices 6, 7 are able to produce scan data covering all possible angles of trajectory of approach that a user may take when approaching vehicle doors la and/or 1b. Additional corresponding front and rear image capture devices may be situated on the left-side of the vehicle 10 (not shown in figures) to generate scan data covering all possible angles of trajectory of approach that a user may take when approaching the left-side vehicle doors.

The front image capture device 7 is angled to provide a field of view that captures a user travelling towards the front of the vehicle 10 as they approach a door 1 of the vehicle 10. Also, the image capture device 7 is angled so that an adequate still image of the face of a user standing next to the vehicle 10 may be captured. The rear image capture device 6 is positioned and oriented in a similar fashion.

In other embodiments only one image capture device with a wider field of view may be used. However, the use of dual image capture devices 6, 7 provides the advantage that the distance between the image capture device and the approaching user and the height and physical dimensions of the approaching user, may be determined more accurately than with a mono camera, in turn enhancing the security level of the door access controller 100.

The field of view of the image capture devices 6, 7 may typically cover a scan range of several meters in radius, as measured from the centre of the vehicle 10. For example, the image capture devices 6, 7 ideally have a field of view that is sufficient to detect an oncoming object travelling at high speed towards the vehicle 10, which may be at some distance from the vehicle 10 at the time of detection. The ability to detect such hazards enables enhanced control of initiation of actions on the vehicle 10, as shall

become clear in the description that follows.

The image capture devices 6, 7 are configurable in a 'standby' state in which they are idle, and a 'record' state in which they actively capture scan data. The state of the image capture devices 6, 7 is controlled by the door access controller 100, which is now described in more detail with reference to Figure 3.

The door access controller 100 comprises a key fob identification module 11 configured to receive a wireless signal 91 from the key fob 9 when it enters the vicinity of the vehicle 10. The wireless signal 91 may be an infrared or radio wave signal, for example, and is unique to the particular key fob and is typically registered to the vehicle at the time of purchase; although other systems allow subsequent key fob registration to the vehicle.

The key fob identification module 11 is continually in a 'standby' state, in which it is ready to receive a wireless signal 91 from a key fob 9 in the vicinity of the vehicle 10.

The key fob identification module 11 compares the signals 91 it receives with a key fob identifier (e.g. an identification or serial number) stored in a memory 12 of the door access controller 100. If there is a match, the key fob identification module 11 sends a correspondence signal 51 to a processing module 14. Upon receipt of the correspondence signal 51, the processing module 14 is configured to send a first authorisation signal 33 to a control module 15. In turn, the control module 15 forwards the first authorisation signal 33 to a door opening control module 13, which generates an unlock signal 34 to activate unlocking of the locking assembly of the doors 1.

The door access controller 100 is also configured to initialise or 'start-up' the image capture devices 6, 7 from their 'standby' state if the wireless signal 91 received by the key fob identification module 11 corresponds to a stored key fob identifier. Specifically, the control module 15 is configured to control the record status of the image capture devices 6, 7 via control signals 62, 72. On receiving a first authorisation signal 33 from the processing module 14, the control module 15 sends control signals 62, 72 to activate the image capture devices 6, 7 to start recording. In this way, the quantity of scan data that is recorded and processed is minimised, as scan data is only recorded, collated, and processed if there is a positive indication that a user is approaching with the authorised key fob 9.

Once activated, the scan data 61, 71 may be retrieved by the image capture devices 6, 7 over a pre-defined time period or an indefinite time period. The scan data 61, 71 may comprise a single image or a series of images over regular or otherwise defined time intervals. A series of images may be recorded at relatively high frequency, for example 50Hz, so that the series forms a 'moving image'.

In other embodiments, the door access controller 100 activates the image capture devices 6, 7 to be in a continuous 'record' state, where they actively capture a continuous stream of scan data 61, 71. In turn, the processing module 14 continuously analyses this scan data 61, 71.

The door access controller 100 communicates with the image capture devices 6, 7 that capture scan data 61, 71. The captured scan data 61, 71 (e.g. camera images in this embodiment, or RADAR or LIDAR signals, etc. in other embodiments) may be communicated directly to the processing module 14 for processing and analysis, or may be collated by a scan data collator 16 of the door access controller 100 before forwarding to the processing module 14.

The processing and analysis performed by the processing module 14 includes filtering out any spurious signals and background objects in the captured scene that do not relate to features of the user that are required for authorisation or intent detection purposes. In addition, if the image capture devices 6, 7 include a wide angle lens, this may give rise to image distortion, and so the processing module 14 may also be configured to correct for such distortion effects before the scan data is used in further steps of the process.

Once any unwanted elements of the collated scan data 61, 71 have been removed or corrected, the processing module 14 further processes the collated scan data 61, 71 in order to identify and determine the presence of a user within the or each image contained in the data and to perform body landmark recognition to identify body (including facial) landmarks on the images of the user.

Each body landmark is a virtual point that represents a specific, predefined body part such as the major joints belonging to virtual skeletal approximations of the anthropometric form of a potential user, and so by identifying a set of body landmarks the size, position and movement of the user can be determined. The multiple virtual points may define a three-dimensional point cloud in a coordinate system (e.g. Cartesian coordinate system) intended to represent the external surfaces of a user and/or object.

The door access controller 100 may alternatively or additionally use a 'flat world' system in its model approximation of the scanned area, typically only focussing on three-dimensional objects in the field of view of the image capture devices 6, 7 that may be mapped by one or more three-dimensional point clouds with respect to a 'flat' reference plane, typically the foreground within the field of view. This advantageously reduces the processing burden on the processing module 14 by filtering out differences in possible terrain heights in the scanned area, for example.

Relative distances between each body landmark of a given user are characterised as unique to that user, such that the set of body landmarks assigned to a user may be used to identify the user subsequently.

Once body landmarks have been identified within the image or series of images forming the scan data 61, 71, the processing module 14 communicates with the memory 12 to identify an authenticated user by comparing the body landmarks defined in the scan data with stored user identifiers that represent body landmarks of a preregistered user. In an example embodiment, the user identifiers are stored on the memory 12 in the form of still and/or moving images of body landmarks associated with a respective user.

In this respect, Figure 3 provides a simplified illustration of an image 41 of a user in which user identifiers 42 representing body landmarks are defined over an area of the image.

Returning to Figure 2, in the event that there is correspondence between one or more of the assigned body landmarks in the captured still or moving images with the user identifiers of the stored still or moving images of the authorised user, a second authorisation signal 33 is output from the processing module 14 to the control module 15, which is forwarded by the control module 15 to the door opening control module 13 (solid arrow 33 in Figure 3). Upon receipt of the second authorisation signal 33, the door opening control module 13 transmits an unlocking and/or opening signal 35 (solid arrow 35 in Figure 3) to activate the unlocking and/or opening of one or more of the vehicle doors 1 to permit user access to the vehicle 10.

The door access controller 100 has two modes of operation: a 'registration mode', in which a user may program into the door access controller 100 user identifiers relating to authorised users; and an 'authorisation mode', in which user identification based on the user identifiers stored during operation in the registration mode is used to control the automatic unlocking and/or opening of one or more of the vehicle doors 1. Upon first use of the vehicle 10 by an authorised user (for example, at the time of purchase of the vehicle), the user of the vehicle 10 must carry out a registration process.

Typically, operation of the image capture devices 6, 7, for example to switch between the registration mode and the authorisation mode, is controlled through a human machine interface (HMI) (not shown) located in a cabin of the vehicle 10.

In the process described previously, the processing module 14 may be configured to determine an identity of the user based on the tracked movement of each body landmark. During the registration process in 'registration' mode, the image capture devices 6, 7 capture scan data relating to a scan area adjacent to the vehicle 10. During this registration process, the authorised user must at least be in the field of view of the image capture devices 6, 7.

The processing module 14 analyses the captured scan data 61, 71 to identify any user(s) in the field of view of image capture devices 6, 7, using an algorithm that is able to recognise an anthropometric form and extrapolate it from the captured scan data 61, 71. This allows the processing module to identify a potential human user from the captured scan data 61, 71, and distinguish that potential human user from any other non-human or inanimate objects (e.g. lampposts, bollards, other animals, etc.) captured in the scan data 61, 71 by the image capture devices 6, 7.

Once a user has been identified within the scan data 61, 71, the processing module 14 determines one or more body landmarks of that user.

The processing module 14 may fit a virtual skeletal approximation to the potential user's anthropometric anatomy. The processing module 14 assigns authorised user identifiers to the user corresponding to the body landmarks defined for that user. This may be sufficient in itself to enable subsequent identification of the user.

However, in an enhancement, during the registration process the door access controller 100 may determine the user's default gait pattern or walking style (e.g. bipedal gait cycle) as being the gait pattern of that user when they are walking towards the vehicle 10 in their typical manner. The determined gait pattern may therefore be used subsequently to identify the user.

One way to achieve this is for the user to record an authorised moving image of their typical gait pattern or walking style using the appropriate controls on image capture devices 6, 7. The scan data 61, 71 is processed by the processing module 14 to remove background noise or image clutter and is transmitted to the memory 12 where it is stored as an authorised user identifier for that particular user. Alternatively, the door access controller 100 may initially identify a user using another technique, such as facial recognition, and then associate a gait pattern with that user, thereby enhancing the data stored for that user and in turn improving subsequent identification.

Aside from identifying an authorised user from the one or more facial or body landmarks belonging to the authorised user, in embodiments of the invention a vehicle control system is configured to determine the intent of a user based on body landmarking analysis of the scan data obtained from the image capture devices 6, 7, and to trigger a vehicle action based on that determined intent.

In the present example where the vehicle control system is embodied as a door access controller 100, the system operates to determine whether the user intends to access the vehicle 10, for example by tracking the trajectory of approach of the user through monitoring the body landmarks identified for that user.

Alternatively, or in parallel, the door access controller 100 may use the identified body landmarks to detect that the user has made a pre-determined gesture indicating their intention to access the vehicle 10 through a particular door 1, and determine the user's intent on that basis.

In either case, if the analysis reveals that the user is approaching or otherwise intends to enter the vehicle 10 through a particular door 1, the door access controller 100 opens the door 1 prior to the user's arrival -provided that other conditions are also satisfied -thereby creating a seamless entry process.

In this embodiment, the vehicle action, namely unlocking and/or opening of one or more doors 1, is only triggered in the event that the user is found to be an authorised user. However, in other embodiments the vehicle action may trigger for any user approaching the vehicle 10 with the key fob 9.

In order for the door access controller 100 to pre-empt an authorised user's intent, the processing module 14 includes a 'neural network' that identifies and analyses the body landmarks relating to the user. As the skilled reader will appreciate, artificial neural network architecture refers to a computer processing mainframe and system that is designed to progressively self-improve its own performance over time, without requiring task-specific programming.

The door access controller 100 implements neural networks to enable the processing module 14 to 'learn' the typical intentions of an authorised user over successive analyses, for example by identifying repeated behaviour trends of a particular authorised user, thereby improving the performance of the system 100 in determining intent.

For example, the processing module 14 may learn over time that a particular authorised user often prefers to ingress the vehicle 10 via a particular vehicle door. In this example, the processing module 14 would identify that particular user from one or more body landmarks and pre-empt the 'intent' of that authorised user by automatically unlocking and opening that user's preferred door. To achieve this, the control module 15 outputs a third authorisation signal 33 to the door opening control module 13. On receipt of the third authorisation signal 33, the door opening control module 13 is configured to send an unlocking and/or opening signal 35 to activate the unlocking and/or opening of the required door 1.

In a modification to the process described previously, the control module 15 is configured to initiate an action on the vehicle 10 in dependence on the identity of the user.

In one example, the door access controller 100 determines whether the authorised user is carrying objects (e.g. luggage, shopping bags, etc.) by comparing the tracked movement of each of the user's body landmarks with a stored gait pattern associated with that user. By analysing the scan data 61, 71, the processing module 14 may be able to calculate any deviations in the user's movement relative to their usual default gait pattern. The neural network 'learning' of the controller 100 may enable it to determine that the user intends to access the vehicle's boot to deposit the object(s) that they are carrying. In this case, the door access controller 100 may automatically open the vehicle's boot as that user approaches the vehicle 10, effectively pre-empting that user's intent.

As noted above, the processing module 14 may track the movement of each body landmark to determine the trajectory of a user relative to the vehicle 10. In this respect, the processing module 14 actively tracks one or more of the assigned body landmarks detected in the scan data 61, 71. The processing module 14 is able to track the movement of each body landmark in real-time or near-real time, in order to determine the trajectory of the user relative to the vehicle 10.

In one example, by tracking the movement of each body landmark of an approaching user, the processing module 14 may determine, from the travelling trajectory of the user relative to the vehicle 10, that the user is walking towards the driver door 1a. In this example, the neural network 'learning' of the controller 100 enables it to determine that the user intends to enter the vehicle 10 via the driver door 1 a, most probably to drive the vehicle 10. Subsequently, the door access controller 100 may automatically open the driver door la as the user approaches the vehicle 10, effectively pre-empting the user's intent.

For the complementary approach, in which the door access controller 100 detects from the scan data 61, 71 that a user has made a pre-determined gesture to signal their intent, it is noted that such gestures are identified through cross-referencing with a set of stored gestures, each having a respective associated vehicle action. Typical examples include hand waving or other hand signals. For example, raising one finger towards the vehicle 10 may correspond to unlocking and/or opening the driver's door 1a, whereas raising two fingers may trigger unlocking and/or opening of both the driver's door la and the passenger door 1 b.

The door access controller 100 compares the position of the user's body parts, as indicated by the body landmarks defined in processed scan data 61, 71, with the stored gestures to detect a match. On detecting a particular gesture, the door access controller 100 takes appropriate action, for example to open the driver's door if that is what is required by the detected gesture.

It is noted that a gesture may be detected in a single image within the scan data 61, 71, or in a series of images.

Aside from pre-determined gestures stored in the memory 12, a user can define their own gestures during the registration process. Specifically, through the HMI the user can control the image capture devices 6, 7 to capture images of the user while they make a particular gesture, and then that gesture can be recorded and associated with a specific vehicle action. In this way, the door access controller 100 is customisable.

It is not necessary to require a simultaneous authorisation of every image contained within the scan data 61, 71; confirmation of an authorised user in any one of a series of images, in combination with a positive identification of the correct key fob 9, may be adequate to trigger an action on the vehicle 10.

The door access controller 100 can also control the closing and/or locking of the vehicle doors 1, either at the time of entry to the vehicle 10, to close a door behind a user once they are safely within the confines of the vehicle 10, or when the vehicle 10 is vacated and a user leaves the vehicle 10, to close and lock the vehicle doors 1 behind them.

Accordingly, in a complementary process to that described above, the image capture devices 6, 7 begin recording scan data 61, 71 after one or more users vacate the vehicle 10. In this respect, the door access controller 100 may initially determine whether a user is vacating the vehicle 10 by detecting that a door 1 has been opened. Subsequently, analysis of scan data 61, 71 captured by the image capture devices 6, 7 may identify the user and their intent to leave the vehicle 10, again using the body landmarking based techniques already described.

If analysis of the scan data 61, 71 reveals that the user is moving away from the vehicle 10, the door access controller 100 closes the door previously opened by the user after a predetermined time period has elapsed. This creates a seamless exit process to complement the entry process.

In this embodiment, to avoid opening a vehicle door 1 onto another object, the processing module 14 is configured to analyse the scan data 61, 71 to detect one or more hazards in the area adjacent to the vehicle 10, and to inhibit opening of the door 1 if a hazard is detected. The door access controller 100 may use other on-board sensors in addition to the image capture devices to provide indications of possible hazards, such as ultrasonic proximity detectors associated with a park-assist system.

As noted previously, the processing module 14 analyses the captured scan data 61, 71 to determine all the objects (both human and non-human) within the field of view of the image capture devices 6, 7. Using the previously mentioned facial recognition and body landmark identification, the processing module 14 determines whether or not each detected object is a human. The processing module 14 is also able to determine the size, proximity and trajectory of any detected object in the vicinity of the vehicle 10.

Based on this and the known area required for each door to open into, the processing module 14 is able to determine whether there are obstructions to any one of the vehicle's doors 1. If so, opening of the obstructed door is inhibited even if the system 100 finds that the user intends to open that door.

For example, the processing module 14 may be programmed with an acceptable threshold distance from the vehicle 10, and if any object is detected at a distance from the vehicle 10 that is below the threshold, opening of the doors 1 is inhibited. Alternatively, threshold distances may be applied to each door individually.

If a door is found to be obstructed, the processing module 14 communicates a hazard signal 43 to the control module 15. The hazard signal is in turn forwarded by the control module 15 to the door opening control module 13 (dashed arrows 43 in Figure 3). Upon receipt of the hazard signal 43, the door opening control module 13 generates a lock signal 44 (dashed arrow 44 in Figure 3) to activate the locking of the corresponding vehicle door 1. Typically, the control module 15 acts only to lock the specific vehicle door 1 that is obstructed, thereby ensuring the safety of a user attempting to access or vacate the vehicle 10 through that particular door, whilst safeguarding against opening the vehicle door 1 onto a passing person or animal, for example, or colliding with another object that may damage the door 1. Alternatively, the control module 15 might lock all of the doors 1.

Once a detected obstruction is found to have moved away from the vicinity of the vehicle through continued analysis of the scan data 61, 71, the door access controller acts to unlock the or each door 1 that had previously been locked due to the obstruction.

When a door 1 is locked for this purpose, the control module 15 transmits a signal to the HMI that triggers an alarm or other indication to occupants of the vehicle 10 that one or more of the doors 1 are obstructed and therefore temporarily locked. This indication may be an audible indication (e.g. buzzer, voice indication), a visible indication (e.g. a flashing light), or displayed on the HMI itself.

The image capture devices 6, 7 may only be required to capture scan data 61, 71 relating to the vehicle's surroundings in certain circumstances, and correspondingly may be disabled in some circumstances. For example, automated operation of the doors 1 may not be desirable if the ignition is started and the engine is running. Accordingly, the control module 15 is configured to receive indication of vehicle operating parameters such as the engine state to enable it to determine whether to operate the image capture devices 6, 7.

Referring finally to Figure 4, the above described process by which the door access controller 100 triggers unlocking and/or opening of a door 1 is represented in Gantt chart form, which illustrates how each of the various sub-processes involved in reaching a decision to open a door 1 fit together in practice. Specifically, the Gantt chart of Figure 4 illustrates the parallel analyses involved in determining a user's intent to access the vehicle 10, determining whether that user is authorised and determining whether it is safe to open a door 1 of the vehicle 10, before triggering the unlocking and/or opening of one or more of the vehicle's doors 1.

In the Figure 4 chart, the steps execute sequentially from bottom to top, with steps forming a vertical stack defining a path in which lower steps represent precursors for higher steps. Meanwhile, horizontally neighbouring steps positioned at the same vertical point with respect to the Y-axis of the Gantt chart may be performed concurrently.

The first step in the process shown in Figure 4 is a 'wake up strategy', namely activation or initiation of the door access controller 100 and the image capture devices 6, 7, if necessary.

Next, two parallel groups of steps initiate and execute concurrently: a first group 50 associated with determining a user's identity and intent, to ascertain whether door opening might be desired by an authenticated user; and a second group 52 concerned with assessing the environment around the vehicle to detect static and dynamic hazards and inhibit door opening if required.

In the first group of steps 50, the door access controller 100 initially performs three parallel complementary processes to determine whether an authorised user intends to access the vehicle 10, namely: (i) monitoring, by the key fob identification module 11, for a wireless signal 91 received from a key fob 9; (ii) detection of an authorised user by analysing scan data 61, 71 captured by the image capture devices 6, 7 using the body landmarking process described above; and (iii) monitoring and tracking of other pedestrians around the vehicle 10.

Once a user has been identified within the scan data using body landmarking analysis, two further concurrent processes a triggered to perform facial recognition and gait recognition, and thereby determine the identity of the user.

In parallel, the body landmarking analysis also feeds into an intent detection module, which determines the intent of a user identified within the scan data 61, 71 through continued monitoring of that user's movements, including gesture detection. The intent detection module also monitors other people identified within the scan data 61, 71, namely pedestrians around the vehicle, to predict their future movements to ascertain whether they might pose a hazard with respect to opening a vehicle door 1.

Once the identity of the user has been confirmed, this information is then passed together with an authentication indication provided by the key fob identification module 11 to an authentication algorithm that confirms whether the identified user is an authenticated user. This confirmation, in combination with the determined intent of the user as derived by the intent detection module, forms an output from the first group of steps 50 that will trigger automatic opening of a vehicle door 1 provided that such opening is not inhibited by the output of the second group of steps 52 that identifies potential hazards.

In this respect, the second group of steps 52 includes four parallel processes that each checks for a respective condition that would preclude door opening. Three of these steps check for dynamic hazards, specifically oncoming vehicles, oncoming cyclists, and generic moving objects respectively. The fourth step checks for static obstacles around the vehicle within range of the vehicle door 1.

The three algorithms checking for dynamic hazards act as precursors for a dynamic obstacle detection and/or hazard perception module, which assesses whether any object detected by the three precursor algorithms represents a hazard to door opening, for example if an oncoming vehicle could collide with a door 1, if opened.

Meanwhile, the check for static obstacles continues in parallel so that the second group of steps ultimately outputs separate respective indications of dynamic and static hazards.

Towards the top of the chart, a decision making function of the door access controller triggers an action based on the output of each of the two groups of steps 50, 52 beneath it. In this embodiment, that action entails automatic unlocking and opening of one or more of the vehicle's doors 1 in order to facilitate vehicle ingress, or inhibiting the unlocking and/or opening of the vehicle's doors 1, for example as part of a hazard perception contingency.

Completion of the process is indicated as contributing towards a 'utopia concept', namely the provision of a vehicle 10 that actively 'welcomes' an approaching user by opening a door 1 in readiness for their arrival.

The above described process is a robust embodiment that includes multiple checks to authorise an approaching user as well as a wide range of hazard perception techniques. The skilled reader will appreciate that many variations on this process are possible. For example, a user carrying a key fob 9 may be authenticated without the need for facial and/or gait recognition. Also, the ability of the vehicle 10 to perform hazard perception will depend on the nature of available scanning devices, and so in some implementations only static or slow-moving objects may be considered.

Many modifications may be made to the above examples without departing from the scope of the present invention as defined in the accompanying claims.

For example, the processing module may be configured to compare features and characteristics of scan data other than those mentioned, including any biometric characteristic of a user. In each case, users have the opportunity to store, at the registration phase, a user identifier that corresponds to the gesture or physical characteristic that they wish to use as an authorising characteristic each time they wish to trigger a vehicle action.

Although the above description assumes that the vehicle control system includes all functionality required to initiate a vehicle action based on a detected user intent, in other embodiments some functions may be provided externally to the vehicle, for example on a remote server or on a portable device. By way of example, the resource intensive task of processing scan data may be outsourced to a remote server, which may then return a simple indication of user intent to the vehicle control system.

Claims

CLAIMS1. A vehicle controller configured to determine a user's intent to access a vehicle, the controller comprising: an input configured to receive data relating to an area in the vicinity of the vehicle; a processing module configured to: analyse the data to detect the presence of a user and determine one or more body landmarks of the user within the data; and determine the user's intent in accordance with the position of the or each body landmark; and a control module configured to issue a control signal to initiate an action on the vehicle in dependence on the user's intent.
2. The vehicle controller as claimed in claim 1, wherein the data relates to the area in the vicinity of the vehicle over a time period, and wherein the processing module is configured to track the position of the or each body landmark relative to the vehicle over the time period.
3. The vehicle controller as claimed in claim 2, wherein the processing module is configured to determine the user's intent by one or more of: comparing the tracked movement of each body landmark to a respective predetermined movement, comparing the tracked movement of the or each body landmark with a gait pattern and analysing the one or more body landmarks to detect a gesture.
4. The vehicle controller as claimed in claim 2 or claim 3, wherein the processing module is configured to analyse the tracked movement of the or each body landmark to determine a trajectory of the user relative to the vehicle.
5. The vehicle controller as claimed in claim 4, wherein the processing module is configured to determine the user's intent by determining whether the trajectory of the user indicates that the user is approaching the vehicle. 6. 7. 8. 9. 10. 11. 12. 13.
The vehicle controller as claimed in claim 5, wherein the processing module is configured to determine the user's intent by determining whether the trajectory of the user indicates that the user is approaching a body aperture closure member of the vehicle.
The vehicle controller as claimed in any of claims 2 to 6, wherein the processing module is configured to determine an identity of the user based on the tracked movement of the or each body landmark.
The vehicle controller as claimed in any preceding claim, wherein the data comprises image data corresponding with one or more images acquired by one or more image capture devices.
The vehicle controller as claimed in claim 8, wherein the processing module is configured to analyse at least one of the one or more images to determine an identity of the user.
The vehicle controller of claim 7 or claim 9, wherein the control module is configured to issue the control signal to initiate the action on the vehicle in dependence on the identity of the user.
The vehicle controller as claimed in any preceding claim, wherein the data comprises any one or more of the following: data acquired from a camera; point cloud data; data acquired by a radar system; data acquired by a lidar system; data acquired by a Bluetooth system; and data acquired by an ultrasonic system.
The vehicle controller as claimed in any preceding claim, wherein the action on the vehicle comprises operating a body aperture closure member of the vehicle.
The vehicle controller of claim 12, wherein the processing module is configured to analyse the data to detect one or more hazards in the area in the vicinity of 14. 15. 16. 17. 18. 19. 20.the vehicle, and to inhibit opening of the body aperture closure member if a hazard is detected.
The vehicle controller as claimed in any preceding claim, wherein each body landmark of the user corresponds to a respective predetermined body part.
The vehicle controller as claimed in any preceding claim, wherein the processing module is configured to determine a plurality of body landmarks of the user, and to generate a skeletal representation of the user based on the plurality of body landmarks.
The vehicle controller as claimed in any preceding claim, comprising a further input configured to receive a notification signal, and wherein the notification signal comprises a signal from a key fob for the vehicle, the vehicle control system comprising a key fob identification module configured to receive the notification signal, wherein the control module is configured to issue the control signal to initiate the action on the vehicle only if a signal received by the key fob identification module corresponds to a stored key fob identifier.
A vehicle control system comprising the controller of any preceding claim.
The vehicle control system as claimed in claim 17, comprising one or more scanning devices, wherein the control module is configured to issue control signals to operate the or each scanning device to acquire the data.
A vehicle comprising the vehicle controller of any of claims 1 to 16, or the vehicle control system of claim 17 or claim 18.
A method of determining a user's intent to access a vehicle, the method comprising: acquiring data relating to an area in the vicinity of the vehicle; analysing the data to identify a user and determine one or more body landmarks of the user within the data; determining the user's intent in accordance with the position of the or each body landmark; and initiating an action on the vehicle in dependence on the user's intent.
21. A computer program product executable on a processor so as to implement the method of claim 20.
22. A non-transitory computer readable medium loaded with the computer program product of claim 21.
23. A processor arranged to implement the method of claim 20, or the computer program product of claim 21.