Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't install BPCells on MacOS #24

Closed
Dario-Rocha opened this issue May 4, 2023 · 9 comments
Closed

Can't install BPCells on MacOS #24

Dario-Rocha opened this issue May 4, 2023 · 9 comments

Comments

@Dario-Rocha
Copy link

The Error:

remotes::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/RtmpXZPBBi/remotes4497101206fb/bnprks-BPCells-c647fb1/DESCRIPTION’ ...
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (423ms)
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’
   
* installing *source* package ‘BPCells’ ...
** using staged installation
Testing hdf5 by compiling example program...


Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'


Searching for hdf5 in a conda env...
no conda environment found in '$CONDA_PREFIX'


Unable to locate libhdf5. Please install manually or edit compiler flags.
ERROR: configuration failed for package ‘BPCells’
* removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/BPCells’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T//RtmpXZPBBi/file449714bda8bf/BPCells_0.1.0.tar.gz’ had non-zero exit status

First I installed Macports and HDF5 with
sudo port install hdf5
having the base MacOS environment activated (as default in the terminal)
The error persisted

Then I installed HDF5 using homebrew having the base MacOS environment activated (as default in the terminal) following this post. The error persisted.

Then I installed HDF5 with conda having the base env activated
% conda install -c anaconda hdf5
But the error persisted

I've tried, in R

Sys.setenv(HDF5_DIR = "/opt/homebrew/Cellar/hdf5/1.14.0")
Sys.setenv(HDF5_LIBRARIES = "/opt/homebrew/Cellar/hdf5/1.14.0/lib")
remotes::install_github("bnprks/BPCells")

To no avail

What else could I try?
Thank you in advance for your time
@bnprks
Copy link
Owner

bnprks commented May 4, 2023

  • Your main tool for affecting the compilation setup is by changing the ~/.R/Makevars file -- I have not had success using Sys.setenv to set environment variables for R installations. In particular, you should look at setting the CFLAGS, CXXFLAGS, and maybe LDFLAGS in the Makevars file.
  • If you have installed hdf5 through homebrew or macports, I would probably try installing with your conda environment deactivated since you won't be relying on dependencies installed through conda.

It looks like you have already identified the directories containing the hdf5 headers (hfd5.h is presumably in /opt/homebrew/Cellar/hdf5/1.14.0/include) and dynamic libraries (/opt/homebrew/Cellar/hdf5/1.14.0/lib). Therefore, I'd try putting the following in your ~/.R/Makevars file:

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

Then with those changes and conda deactivated, try installing BPCells.

If that continues to not work, there's a newly added debugging tool to help get more error diagnostics from the installation process. On the terminal, run export BPCELLS_DEBUG_INSTALL=true on the command line before starting R on the command line and running remotes::install_github("bnprks/BPCells"). This will print out the commands the installer runs to check for the hdf5 libraries, as well as any error messages printed out during the test compilations.

@bnprks
Copy link
Owner

bnprks commented May 4, 2023

To help make these problems less common I just tried making a fix in 5d9e78c, which will also try to use h5cc -show and h5cc -showconfig to try to automatically infer the required compiler flags.

If h5cc -show prints something for you in the terminal, you might try just re-installing with no special configuration set to see if this fix works for you.

(p.s. if you still are ending up with issues, could you let me know your MacOS version and the compiler version you're using?)

@Dario-Rocha
Copy link
Author

Thank you for your reply, I spotted some mistakes I made and tried again, though without sucess

When trying to manually modify .R/Makevars with a text editor I realized that the file simply didn't exist. So I:
-Uninstalled hdf5 with homebrew
-Deactivated the base conda environment
-Installed hdf5 with homebrew
-Confirmed hdf5 installation directory

~ % brew info llvm
==> llvm: stable 16.0.3 (bottled), HEAD [keg-only]
Next-gen compiler infrastructure
https://llvm.org/
/opt/homebrew/Cellar/llvm/16.0.3 (6,779 files, 1.5GB)

-Confirmed the location of clang and clang++

 ~ % ls /opt/homebrew/Cellar/llvm/16.0.3/bin/clang
/opt/homebrew/Cellar/llvm/16.0.3/bin/clang
~ % ls /opt/homebrew/Cellar/llvm/16.0.3/bin/clang++
/opt/homebrew/Cellar/llvm/16.0.3/bin/clang++

Created /Users/dariorocha/ .R/Makevars.rtf using a text editor

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib
CC=/opt/homebrew/Cellar/llvm/16.0.3/bin/clang
CXX=/opt/homebrew/Cellar/llvm/16.0.3/bin/clang++

-ran this on the terminal
export BPCELLS_DEBUG_INSTALL=true

-then tried to install BPCells again

remotes::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ───────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/Rtmp6Qj2kr/remotes3f2510dca9b5/bnprks-BPCells-98536c9/DESCRIPTION’ ...
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (461ms)
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’
   
* installing *source* package ‘BPCells’ ...
** using staged installation
Testing hdf5 by compiling example program...


Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'


Searching for config information with h5cc -showconfig and h5cc -show...
Found working hdf5
HDF5_CFLAGS=' -I/Users/dariorocha/miniconda3/include'
HDF5_LIBS=' -Wl,-rpath,/Users/dariorocha/miniconda3/lib -L/Users/dariorocha/miniconda3/lib -lcrypto -lcurl -lpthread -lz -ldl -lm -Wl,-rpath,/Users/dariorocha/miniconda3/lib -lhdf5'

Testing architecture flag support support...'-march=native' succeeded
ARCH_FLAG='-march=native'


Unable to compile program with C++17 std::filesystem.
Please install a newer compiler version and set CC and CXX in ~/.R/Makevars

-moved on to check h5cc

~ % h5cc -show                                     
clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm

clang version is

~ % clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

~ % clang++ --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

System info is:

% sw_vers
ProductName:		macOS
ProductVersion:		13.3.1
ProductVersionExtra:	(a)
BuildVersion:		22E772610a

@bnprks
Copy link
Owner

bnprks commented May 9, 2023

It looks like the debug log didn't quite come through properly. When you run export BPCELLS_DEBUG_INSTALL=true, you then need to run R and remotes::install_github() in the same terminal. It should print out a bunch of lines starting with "+", interspersed with some program output. That said, I have a few ideas from the information you've provided

  1. I see you're on an ARM Mac. In issue cannot install BPcells on ARM mac- cannot find hdf5 #14, it seems that just using the homebrew-provided hdf5 is likely sufficient, and you are unlikely to need the llvm from homebrew as well.
    • I would try removing the CC and CXX entries from your Makevars
  2. Based on the log you provided, it seems that your compilation used a conda hdf5 install, rather than homebrew as you intended. It's not entirely clear to me why this is happening. You might try removing the CC and CXX entries as I mentioned, or simply deactivating conda in your terminal prior to running R and remotes::install_github(). This will prevent the installation from picking up the conda version of hdf5 at least. Otherwise, double-check CFLAGS, CXXFLAGS, and LDFLAGS
  3. This error "Unable to compile program with C++17 std::filesystem." typically only shows up on older Intel Macs, so I'm not quite sure of the cause. I know mixups between ARM and x86 binaries can happen on ARM Macs as some programs still use Rosetta 2

Hopefully some of that is useful.

If not, please retry the export BPCELLS_DEBUG_INSTALL=true as I described, deactivate conda in the same terminal, then run R and provide the (much more verbose) debug log.

@Dario-Rocha
Copy link
Author

Dario-Rocha commented May 10, 2023 •
edited by bnprks
Loading

Alright, another attempt:

-deleted the " .R/" directory and created it again without the initial blank space, then created the Makevars file and edited it, also adding the .txt extension, following this post:
https://stackoverflow.com/questions/5891365/mac-os-x-doesnt-allow-to-name-files-starting-with-a-dot-how-do-i-name-the-hta/51788836#51788836

-My Makevars looks like this now

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

-Opened the terminal, conda deactivate

-Checked pkg config

~ % pkg-config hdf5 --cflags --libs
Package hdf5 was not found in the pkg-config search path.
Perhaps you should add the directory containing `hdf5.pc'
to the PKG_CONFIG_PATH environment variable
No package 'hdf5' found

-Rechecked that hdf5 is installed with homebrew

~ % brew info hdf5
==> hdf5: stable 1.14.0 (bottled)
File format designed to store large amounts of data
https://www.hdfgroup.org/HDF5
Conflicts with:
  hdf5-mpi (because hdf5-mpi is a variant of hdf5, one can only use one or the other)
/opt/homebrew/Cellar/hdf5/1.14.0 (283 files, 20.6MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-05-10 at 11:36:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/hdf5.rb
License: BSD-3-Clause
==> Dependencies
Build: autoconf ✘, automake ✘, libtool ✘
Required: gcc ✔, libaec ✔
==> Analytics
install: 613 (30 days), 12,491 (90 days), 391,867 (365 days)
install-on-request: 157 (30 days), 3,285 (90 days), 93,417 (365 days)
build-error: 0 (30 days)

-Checked that pkg config is installed

~ % brew info pkg-config
==> pkg-config: stable 0.29.2 (bottled)
Manage compile and link flags for libraries
https://freedesktop.org/wiki/Software/pkg-config/
/opt/homebrew/Cellar/pkg-config/0.29.2_3 (11 files, 676.2KB) *
  Poured from bottle using the formulae.brew.sh API on 2023-05-04 at 14:55:43
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pkg-config.rb
License: GPL-2.0-or-later
==> Analytics
install: 4,778 (30 days), 81,050 (90 days), 2,104,367 (365 days)
install-on-request: 756 (30 days), 16,512 (90 days), 242,040 (365 days)
build-error: 17 (30 days)

-There is no pkgconfig directory in /opt/homebrew/Cellar/hdf5/1.14.0/lib

~ % ls /opt/homebrew/Cellar/hdf5/1.14.0/lib
libhdf5.310.dylib		libhdf5_hl.a
libhdf5.a			libhdf5_hl.dylib
libhdf5.dylib			libhdf5_hl_cpp.310.dylib
libhdf5_cpp.310.dylib		libhdf5_hl_cpp.a
libhdf5_cpp.a			libhdf5_hl_cpp.dylib
libhdf5_cpp.dylib		libhdf5_hl_fortran.a
libhdf5_fortran.310.dylib	libhdf5hl_fortran.310.dylib
libhdf5_fortran.a		libhdf5hl_fortran.a
libhdf5_fortran.dylib		libhdf5hl_fortran.dylib
libhdf5_hl.310.dylib

-Following chatbot instructions

sudo mkdir -p /opt/homebrew/lib/pkgconfig
sudo nano /opt/homebrew/lib/pkgconfig/hdf5.pc

-Then copied and pasted this in the file

prefix=/opt/homebrew/Cellar/hdf5/1.14.0
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: hdf5
Description: Hierarchical Data Format 5 (HDF5)
Version: 1.14.0

Libs: -L${libdir} -lhdf5
Cflags: -I${includedir}

-and save the file by pressing Ctrl+X, then Y, and finally Enter.

-now I can successfully run the command pkg-config hdf5 --cflags --libs

~ % pkg-config hdf5 --cflags --libs
-I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5

So, within the terminal and with the base environment deactivated, I run R and try to install BPCells

Install log (EDIT by bnprks to make into collapsible section) 
> remotes::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ─────────────────────────────────────────────────────────────────
   checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/R✔  checking for file ‘/private/var/folders/vs/2n7hpqjd7s70rsg0ys8kzgc40000gn/T/RtmphisDvG/remotes3c9d3c210373/bnprks-BPCells-32fb937/DESCRIPTION’
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (511ms)
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’
   
* installing *source* package ‘BPCells’ ...
** using staged installation
+ ENABLE_INSTALL_COUNTING=yes
+ '[' -n yes ']'
+ curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure
+ true
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CC
+ CC='clang -mmacosx-version-min=10.13'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXX
+ CXX='clang++ -mmacosx-version-min=10.13 -std=gnu++14'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS
+ CFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS
+ CXXFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS
+ LDFLAGS=-L/usr/local/lib
+ echo 'Testing hdf5 by compiling example program...'
Testing hdf5 by compiling example program...
+ HDF5_CFLAGS=
+ HDF5_LIBS=-lhdf5
+ HDF5_OK=
+ clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -lhdf5 -o tools/h5write
tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found
#include "hdf5.h"
         ^~~~~~~~
1 error generated.
+ '[' -z ']'
+ printf '\n\nRetrying with pkg-config flags...\n'


Retrying with pkg-config flags...
++ pkg-config hdf5 --cflags
+ HDF5_CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
++ pkg-config hdf5 --libs
+ HDF5_LIBS='-L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5'
+ clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write
tools/h5write.c:31:12: warning: variable 'status' set but
      not used [-Wunused-but-set-variable]
    herr_t status;
           ^
1 warning generated.
ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "_H5Dclose", referenced from:
      _main in h5write-e7dbda.o
  "_H5Dcreate2", referenced from:
      _main in h5write-e7dbda.o
  "_H5Dwrite", referenced from:
      _main in h5write-e7dbda.o
  "_H5Fclose", referenced from:
      _main in h5write-e7dbda.o
  "_H5Fcreate", referenced from:
      _main in h5write-e7dbda.o
  "_H5Sclose", referenced from:
      _main in h5write-e7dbda.o
  "_H5Screate_simple", referenced from:
      _main in h5write-e7dbda.o
  "_H5T_NATIVE_INT_g", referenced from:
      _main in h5write-e7dbda.o
  "_H5Tclose", referenced from:
      _main in h5write-e7dbda.o
  "_H5Tcopy", referenced from:
      _main in h5write-e7dbda.o
  "_H5Tset_order", referenced from:
      _main in h5write-e7dbda.o
  "_H5check_version", referenced from:
      _main in h5write-e7dbda.o
  "_H5open", referenced from:
      _main in h5write-e7dbda.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
+ '[' -z ']'
+ printf '\n\nSearching for config information with h5cc -showconfig and h5cc -show...\n'


Searching for config information with h5cc -showconfig and h5cc -show...
++ h5cc -showconfig
+ H5CC_CONFIG='	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.14.0
                  Configured on: Mon May  1 09:04:43 UTC 2023
                  Configured by: [email protected]
                    Host system: aarch64-apple-darwin22.4.0
              Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5/1.14.0

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: 
                     H5_LDFLAGS:  -Wl,-commons,use_dylibs
                     AM_LDFLAGS:  -L/opt/homebrew/opt/libaec/lib
                Extra libraries: -lsz -lz -ldl -lm 
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: clang ( Apple clang version 14.0.3 )
                       CPPFLAGS: 
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                        C Flags: 
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0)
                  Fortran Flags: 
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Warray-temporaries -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -O3
               AM Fortran Flags: 
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: yes
                   C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                      C++ Flags: 
                   H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no


Features:
---------
                     Parallel HDF5: no
  Parallel Filtered Dataset Writes: no
                Large Parallel I/O: no
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib),szip(encoder)
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
            Function stack tracing: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no'
++ echo '	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.14.0
                  Configured on: Mon May  1 09:04:43 UTC 2023
                  Configured by: [email protected]
                    Host system: aarch64-apple-darwin22.4.0
              Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5/1.14.0

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: 
                     H5_LDFLAGS:  -Wl,-commons,use_dylibs
                     AM_LDFLAGS:  -L/opt/homebrew/opt/libaec/lib
                Extra libraries: -lsz -lz -ldl -lm 
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: clang ( Apple clang version 14.0.3 )
                       CPPFLAGS: 
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                        C Flags: 
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0)
                  Fortran Flags: 
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Warray-temporaries -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -O3
               AM Fortran Flags: 
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: yes
                   C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                      C++ Flags: 
                   H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no


Features:
---------
                     Parallel HDF5: no
  Parallel Filtered Dataset Writes: no
                Large Parallel I/O: no
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib),szip(encoder)
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
            Function stack tracing: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no'
++ awk -F: '/FLAGS|Extra libraries:/ {printf("%s ", $2)}'
+ H5CC_CONFIG='    -Wl,-commons,use_dylibs   -L/opt/homebrew/opt/libaec/lib  -lsz -lz -ldl -lm   cr      -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling   -I/opt/homebrew/opt/libaec/include '
++ h5cc -show
+ H5CC_CONFIG2='clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
+ HDF5_CFLAGS=
+ HDF5_LIBS=
+ HDF5_RPATH_LAST_ARG=false
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr false = true
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr '' : '.*-L/opt/homebrew/opt/libaec/lib'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib' : '.*-lsz'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz' : '.*-lz'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz' : '.*-ldl'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' : '.*-lm'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr '' : '.*-I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling'
+ HDF5_CFLAGS=' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling' : '.*-I/opt/homebrew/opt/libaec/include'
+ HDF5_CFLAGS=' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' : '.*-I/opt/homebrew/opt/libaec/include'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' : '.*-L/opt/homebrew/Cellar/hdf5/1.14.0/lib'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-L/opt/homebrew/opt/libaec/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lsz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-ldl'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib' : '.*-lm'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5'
+ clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/private/tmp/hdf5-20230501-8407-pu0d18/hdf5-1.14.0/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write
tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found
#include "hdf5.h"
         ^~~~~~~~
1 error generated.
+ '[' -z ']'
+ printf '\n\nSearching for hdf5 in a conda env...\n'


Searching for hdf5 in a conda env...
+ '[' -z '' ']'
+ echo 'no conda environment found in '\''$CONDA_PREFIX'\'''
no conda environment found in '$CONDA_PREFIX'
+ '[' -z ']'
+ printf '\n\nUnable to locate libhdf5. Please install manually or edit compiler flags.\n'


Unable to locate libhdf5. Please install manually or edit compiler flags.
+ exit 1
ERROR: configuration failed for package ‘BPCells’
* removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/BPCells’

@bnprks
Copy link
Owner

bnprks commented May 10, 2023

Thanks for the debug log, that gave a lot of useful information.

It looks like your main problem is having a mixup between software for the x86_64 architecture, and software for the arm_64 architecture. x86_64 is emulated through Rosetta2 on ARM macs, while arm_64 is what the computer runs natively. Depending on whether your version of R is running as x86_64 or arm_64, you will need to match your compilers and libraries to that architecture. As I don't own one of these Macs personally, I don't have many useful tips on how to accomplish that, but it is definitely the source of the error in the log you just sent.

This issue has a bit of useful discussion about having mismatched x86_64/arm_64 when installing BPCells: #6. In that case, switching to running R natively in the arm_64 architecture helped solve the problem. If using x86_64 is important to you for some reason, then it appears my advice to use homebrew hdf5 over conda hdf5 is probably bad, though I've also noticed mac users having trouble with trying to install with hdf5 through conda

Some of my observations:

  1. In your error message, I notice this part in particular:

    + clang -mmacosx-version-min=10.13 tools/h5write.c -Wall -g -O2 -L/usr/local/lib -I/opt/homebrew/Cellar/hdf5/1.14.0/include -L/opt/homebrew/Cellar/hdf5/1.14.0/lib -lhdf5 -o tools/h5write
tools/h5write.c:31:12: warning: variable 'status' set but
      not used [-Wunused-but-set-variable]
    herr_t status;
           ^
1 warning generated.
ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.0/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

    This bit "building for macOS-x86_64 but attempting to link with file built for macOS-arm64" is why you have the linker errors just below, and why I've diagnosed your main problem this way.

  2. Your makevars file must have exactly the name ~/.R/Makevars, and it cannot have a txt extension. These lines of the log indicate that R is not reading the CFLAGS, CXXFLAGS, and LDFLAGS you have set, likely due to the file having the wrong name:

    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS
+ CFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS
+ CXXFLAGS='-Wall -g -O2'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS
+ LDFLAGS=-L/usr/local/lib

  3. Don't worry about making pkg-config work -- that's mainly for use on linux. BPCells has several options for how to try to find hdf5, so it's okay if pkgconfig doesn't work. I would recommend you don't bother making a hdf5.pc file manually. If you set up ~/.R/Makevars correctly, you should succeed after the line Testing hdf5 by compiling example program..., and you shouldn't ever get to the line Retrying with pkg-config flags....

  4. Similarly, it looks like the h5cc -show option doesn't yield useful compiler flags on your system, so don't worry too much about it and just focus on getting the method with ~/.R/Makevars to work.

  5. For future error logs, you can use a collapsible section as described here to take up a bit less scroll space

@Dario-Rocha
Copy link
Author

Sorry for the long logs, I was unaware of the collapsible section

Thank you for pointing out it was about the architecture. I decided to go from a clean slate, uninstalled R and Rstudio, then from the terminal deactivated the 'base' conda environment, installed R with homebrew, installed Rstudio with homebrew, created the .R directory mkdir -p ~/.R created the Makevars file touch ~/.R/Makevars edited the Makevars file from Rstudio file.edit("~/.Makevars")

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

installed BPCells from Rstudio remotes::install_github("bnprks/BPCells")
Installation was successful, haven't yet tested if everything works alright.

Thank you so much for your time

@bnprks
Copy link
Owner

bnprks commented May 12, 2023

Great, I'm glad you've gotten it installed finally! I'd suggest walking through this tutorial if you want to check everything is working.

@bnprks bnprks closed this as completed May 12, 2023
@bnprks bnprks mentioned this issue Jun 8, 2023
Closed
@rhodesch
Copy link

rhodesch commented Feb 26, 2024 •
edited
Loading

Sorry for the long logs, I was unaware of the collapsible section

Thank you for pointing out it was about the architecture. I decided to go from a clean slate, uninstalled R and Rstudio, then from the terminal deactivated the 'base' conda environment, installed R with homebrew, installed Rstudio with homebrew, created the .R directory mkdir -p ~/.R created the Makevars file touch ~/.R/Makevars edited the Makevars file from Rstudio file.edit("~/.Makevars")

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.0/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.0/lib

installed BPCells from Rstudio remotes::install_github("bnprks/BPCells") Installation was successful, haven't yet tested if everything works alright.

Thank you so much for your time

Just confirming comment above works for me too (as would @bnprks comment on May 4, 2023) with:
MacOS 14.3.1, M1
R version 4.3.2
hdf5 1.14.3 (via homebrew)
no conda active during installation steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bnprks @rhodesch @Dario-Rocha