Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No Mater what version I use, Constant shut down and restart cycle. #89

Open
3 tasks
Strangedave opened this issue Feb 18, 2024 · 6 comments
Open
3 tasks
Assignees
Labels
question Further information is requested

Comments

@Strangedave
Copy link

Strangedave commented Feb 18, 2024

Area of Concern

  • [Server version: 2.5.4]
  • [Constant rebooting of server. YOLOv5-6.2 ,YOLOv5 .NET, YOLOv8]
  • [CodeProject.AI-Server-win-x64-2.5.4] Installer
  • Python3.7, .NET
  • Module packages [e.g. PyTorch)
  • Something else

Describe the bug
For the last year or so I had been running YOLOv5-6.2 with out much issue. I did see an occasional stoppage, but usually after many weeks of uptime. After installing Server version 2.5.1 I was experiencing frequent stopages, on some occasions the system would restart, but that was not most of the time. After many hours of trouble shooting and reading of online forums, I decided to just flush the system and start over. After reinstalling Blue Iris 5.8.6.6 and CPAI 2.5.1 I was getting about 1/2 a day of uptime, but didnt really have the ambition to chase the problem any more. Today I received an update notification, and thought thank god, hopefully someone solved the issue. After installing the update to 2.5.4 and Blue Iris 5.8.6.7 I have tried 3 modules YOLOv5-6.2 ,YOLOv5 .NET, YOLOv8. all are restarting frequently. YOLOv5.net has been the most stable unfortunately, not catching most things.

Expected behavior
Just want the system to stay running.

LOG
15:39:26:System: Windows
15:39:26:Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
15:39:26:CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel)
15:39:26: 1 CPU x 10 cores. 20 logical processors (x64)
15:39:26:GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA)
15:39:26: Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN:
15:39:26:System RAM: 48 GiB
15:39:26:Platform: Windows
15:39:26:BuildConfig: Release
15:39:26:Execution Env: Native
15:39:26:Runtime Env: Production
15:39:26:.NET framework: .NET 7.0.15
15:39:26:Default Python:
15:39:26:App DataDir: C:\ProgramData\CodeProject\AI
15:39:26:Video adapter info:
15:39:26: NVIDIA GeForce RTX 3060:
15:39:26: Driver Version 31.0.15.5123
15:39:26: Video Processor NVIDIA GeForce RTX 3060
15:39:26:STARTING CODEPROJECT.AI SERVER
15:39:26:RUNTIMES_PATH = C:\Program Files\CodeProject\AI\runtimes
15:39:26:PREINSTALLED_MODULES_PATH = C:\Program Files\CodeProject\AI\preinstalled-modules
15:39:26:MODULES_PATH = C:\Program Files\CodeProject\AI\modules
15:39:26:PYTHON_PATH = \bin\windows%PYTHON_NAME%\venv\Scripts\python
15:39:26:Data Dir = C:\ProgramData\CodeProject\AI
15:39:26:Server version: 2.5.4
15:39:29:
15:39:29:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2)
15:39:29:Valid: True
15:39:29:Module Path: \modules\ObjectDetectionYOLOv5-6.2
15:39:29:AutoStart: True
15:39:29:Queue: objectdetection_queue
15:39:29:Runtime: python3.7
15:39:29:Runtime Loc: Shared
15:39:29:FilePath: detect_adapter.py
15:39:29:Pre installed: False
15:39:29:Start pause: 1 sec
15:39:29:Parallelism: 0
15:39:29:LogVerbosity:
15:39:29:Platforms: all,!raspberrypi,!jetson
15:39:29:GPU Libraries: installed if available
15:39:29:GPU Enabled: enabled
15:39:29:Accelerator:
15:39:29:Half Precis.: enable
15:39:29:Environment Variables
15:39:29:APPDIR = \modules\ObjectDetectionYOLOv5-6.2
15:39:29:CPAI_MODULE_ENABLE_GPU = True
15:39:29:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models
15:39:29:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets
15:39:29:MODEL_SIZE = Medium
15:39:29:USE_CUDA = True
15:39:29:YOLOv5_AUTOINSTALL = false
15:39:29:YOLOv5_VERBOSE = false
15:39:29:
15:39:29:Started Object Detection (YOLOv5 6.2) module
15:39:32:Server: This is the latest version
15:39:35:Sending shutdown request to python/ObjectDetectionYOLOv5-6.2
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...12765c) ['No objects found'] took 2167ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6bd0e4) ['No objects found'] took 2167ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e408c) ['No objects found'] took 2175ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...179c35) ['No objects found'] took 2162ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e1103) ['No objects found'] took 2196ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a8a28d) ['No objects found'] took 2206ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ba4cfb) ['No objects found'] took 2215ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6d5838) ['No objects found'] took 2231ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...02c90a) ['No objects found'] took 2230ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6f6fe8) ['No objects found'] took 2246ms
15:39:36:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d6e3a6) ['No objects found'] took 140ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...23e424) ['No objects found'] took 527ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...0167ac) ['No objects found'] took 515ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...77ed8a) ['No objects found'] took 534ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...6f0dde) ['No objects found'] took 534ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...cbb408) ['No objects found'] took 525ms
15:39:37:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...5f4e6c) ['No objects found'] took 525ms
15:39:38:detect_adapter.py: Object Detection (YOLOv5 6.2) started.
15:39:38:detect_adapter.py: GPU compute capability is 8.6
15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060'
15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: GPU compute capability is 8.6
15:39:38:detect_adapter.py: Using half-precision for the device 'NVIDIA GeForce RTX 3060'
15:39:38:detect_adapter.py: Inference processing will occur on device 'NVIDIA GeForce RTX 3060'
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: Detecting using ipcam-combined
15:39:38:detect_adapter.py: Retrieved objectdetection_queue command 'custom'
15:39:38:detect_adapter.py: Model Path is C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv5-6.2\custom-models\ipcam-combined.pt
15:39:38:Module ObjectDetectionYOLOv5-6.2 has shutdown
15:39:38:detect_adapter.py: has exited
15:40:08:ObjectDetectionYOLOv5-6.2 went quietly
15:40:08:
15:40:08:Module 'Object Detection (YOLOv5 6.2)' 1.9.1 (ID: ObjectDetectionYOLOv5-6.2)
15:40:08:Valid: True
15:40:08:Module Path: \modules\ObjectDetectionYOLOv5-6.2
15:40:08:AutoStart: True
15:40:08:Queue: objectdetection_queue
15:40:08:Runtime: python3.7
15:40:08:Runtime Loc: Shared
15:40:08:FilePath: detect_adapter.py
15:40:08:Pre installed: False
15:40:08:Start pause: 1 sec
15:40:08:Parallelism: 0
15:40:08:LogVerbosity:
15:40:08:Platforms: all,!raspberrypi,!jetson
15:40:08:GPU Libraries: installed if available
15:40:08:GPU Enabled: enabled
15:40:08:Accelerator:
15:40:08:Half Precis.: enable
15:40:08:Environment Variables
15:40:08:APPDIR = \modules\ObjectDetectionYOLOv5-6.2
15:40:08:CPAI_MODULE_ENABLE_GPU = True
15:40:08:CUSTOM_MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\custom-models
15:40:08:MODELS_DIR = \modules\ObjectDetectionYOLOv5-6.2\assets
15:40:08:MODEL_SIZE = Medium
15:40:08:USE_CUDA = True
15:40:08:YOLOv5_AUTOINSTALL = false
15:40:08:YOLOv5_VERBOSE = false
15:40:08:
15:40:08:Started Object Detection (YOLOv5 6.2) module
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...45f723) ['No objects found'] took 2172ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...129b4c) ['No objects found'] took 2203ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...c16f02) ['No objects found'] took 2194ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...645e6b) ['No objects found'] took 2186ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6e9f5c) ['No objects found'] took 2214ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...360f78) ['No objects found'] took 2246ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e2447e) ['No objects found'] took 2244ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d7ef76) ['No objects found'] took 2283ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...62c31d) ['No objects found'] took 2274ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...062494) ['No objects found'] took 2291ms
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...0ef1b3) ['No objects found'] took 93ms
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...3e9a6d) ['No objects found'] took 110ms
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...3c126f) ['No objects found'] took 325ms
15:40:15:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...ffbe51) ['No objects found'] took 525ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...38d0a9) ['No objects found'] took 508ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...e80d0d) ['No objects found'] took 347ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...669b9a) ['No objects found'] took 494ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...7d4ce5) ['No objects found'] took 478ms
15:40:15:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...07b60f) ['No objects found'] took 473ms
15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...49a13c) ['No objects found'] took 61ms
15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...980b04) ['No objects found'] took 50ms
15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...a7a040) ['No objects found'] took 66ms
15:40:18:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:18:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9debaa) ['No objects found'] took 31ms
15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92b1cf) ['No objects found'] took 36ms
15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...66a9e0) ['No objects found'] took 30ms
15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...7eed63) ['No objects found'] took 43ms
15:40:19:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:19:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...12bb88) ['No objects found'] took 34ms
15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...ab823e) ['No objects found'] took 44ms
15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...1bd57d) ['No objects found'] took 31ms
15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...d1739f) ['No objects found'] took 68ms
15:40:20:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:20:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...20250c) ['No objects found'] took 41ms
15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...92904a) ['No objects found'] took 87ms
15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...9a8375) ['No objects found'] took 41ms
15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...cb51b0) ['No objects found'] took 122ms
15:40:28:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:28:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...74185d) ['No objects found'] took 44ms
15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...e1b343) ['No objects found'] took 75ms
15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8d8369) ['No objects found'] took 37ms
15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...6cf9dc) ['No objects found'] took 73ms
15:40:29:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:29:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...a4f65e) ['No objects found'] took 48ms
15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...8b87bb) ['No objects found'] took 75ms
15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...c83114) ['No objects found'] took 36ms
15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'detect' (...1295e3) ['No objects found'] took 74ms
15:40:30:Object Detection (YOLOv5 6.2): Detecting using ipcam-combined
15:40:30:Response rec'd from Object Detection (YOLOv5 6.2) command 'custom' (...8c62ae) ['No objects found'] took 39ms

YOLOv8 Might be a different issue
18:50:31:Started Object Detection (YOLOv8) module
18:50:31:detect_adapter.py: Traceback (most recent call last):
18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8\detect_adapter.py", line 12, in
18:50:31:detect_adapter.py: from request_data import RequestData
18:50:31:detect_adapter.py: File "C:\Program Files\CodeProject\AI\modules\ObjectDetectionYOLOv8../../SDK/Python\request_data.py", line 8, in
18:50:31:detect_adapter.py: from PIL import Image
18:50:31:detect_adapter.py: ModuleNotFoundError: No module named 'PIL'

Your System (please complete the following information):
Server version: 2.5.4
System: Windows
Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
CPUs: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (Intel)
1 CPU x 10 cores. 20 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 3060 (12 GiB) (NVIDIA)
Driver: 551.23, CUDA: 12.4 (up to: 12.4), Compute: 8.6, cuDNN:
System RAM: 48 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
.NET framework: .NET 7.0.15
Default Python:
Video adapter info:
NVIDIA GeForce RTX 3060:
Driver Version 31.0.15.5123
Video Processor NVIDIA GeForce RTX 3060
System GPU info:
GPU 3D Usage 0%
GPU RAM Usage 1.6 GiB
Global Environment variables:
CPAI_APPROOTPATH =
CPAI_PORT = 32168

Additional context
Add any other context about the problem here.

@onlineapps-cloud
Copy link

i have same issue on windows.

@Strangedave
Copy link
Author

It sems like the most recent update is better. maybe give that a try

@onlineapps-cloud
Copy link

It sems like the most recent update is better. maybe give that a try

no, i tried yesterday with latest installer.

@ChrisMaunder
Copy link
Contributor

Are you using Blue Iris? If so, stop Blue Iris from starting/stopping the CodeProject.AI service

@ChrisMaunder ChrisMaunder self-assigned this Apr 18, 2024
@ChrisMaunder ChrisMaunder added the question Further information is requested label Apr 18, 2024
@onlineapps-cloud
Copy link

here is fixed codeproject ai windows version: https://mega.nz/folder/YzgiEIxC#spjBLYks3nNUrdiQkBquyg
proof:
image
my build will install all neded components, do all exactly as in Readme, and you will obtain working version of codeproject.
Soon i will release fixed linux version of codeproject ai.

@Strangedave
Copy link
Author

Will there be a 2.6.3 version with the patch included? My setup is working most of the time on 2.6.2, and I don’t really want to mess with a working thing. My system has been online for 118,479 objects, which is probably a week or so. Running YOLOv8. Was looking for a last restart time stamp, but I don’t see that anywhere. Might be a useful add on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants