Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build from source if the ta-lib is not installed #135

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

mckelvin
Copy link
Contributor

@mckelvin mckelvin commented Feb 5, 2017

This PR is a break-out part of #111 (comment) after PR #132 , which will include the TA-Lib C code at compile time if the library is not installed. It will make it easier to install the python ta-lib module by just run pip install TA-Lib without worrying about how to setup the TA-Lib dependency.

The first 2 commits are generated by git subtree add --prefix vendor/ta-lib https://github.com/TA-Lib/ta-lib.git v0.4.0 --squash (https://github.com/TA-Lib/ta-lib/releases/tag/v0.4.0).

This PR has been verified (by running make test) on macOS(EI Capitan) and Linux(Ubuntu 14.04). I'll appreciate if anyone could help to test on Windows.

setup.py Outdated
from distutils.dist import Distribution

display_option_names = Distribution.display_option_names + ['help', 'help-commands']
query_only = any('--' + opt in sys.argv for opt in display_option_names) or len(sys.argv) < 2 or sys.argv[1] == 'egg_info'

# Use setuptools for querying the package, normal builds use distutils
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It breaks on python setup.py sdist if using distutils here in my environment:

$ python setup.py sdist
setup.py:81: UserWarning: Cannot find ta-lib library, will try to build from source.
  'Cannot find ta-lib library, will try to build from source.'
/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'setup_requires'
  warnings.warn(msg)
/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
running sdist
running check
error: [Errno 2] No such file or directory: 'venv/lib/python2.7/lib-dynload/lib-dynload'

@mrjbq7 Would you please detail why the following code block for setup is essential here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed it to work for some commands only available in setuptools and not distutils, for example python setup.py egg_info. But more importantly, I wanted to be able to run "query only" commands like python setup.py --help without having import errors if numpy and cython weren't available.

I'd love to stop using setuptools and just use distutils if we can refactor the setup.py to still allow --help with no numpy or cython.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrjbq7 The requirement is quite strict. I've left it unchanged. The problem I met should only be happened in my environment and there might be some problem with my virtualenv.

@mrjbq7
Copy link
Collaborator

mrjbq7 commented Feb 5, 2017

Shouldn't we use a git submodule or git subtree?

@mckelvin
Copy link
Contributor Author

mckelvin commented Feb 5, 2017

@mrjbq7 No, we'd better not. Otherwise the vendor code in vendor/ta-lib will not be included in .tar.gz, .zip files listed in https://github.com/mrjbq7/ta-lib/releases . Since there's even no .git/ directory in the compressed files, users can't even run git submodule ... after they download the code.

@mckelvin
Copy link
Contributor Author

mckelvin commented Feb 5, 2017

Oh I missed the git subtree, actually we're using git subtree here.

The first 2 commits are generated by git subtree add --prefix vendor/ta-lib https://github.com/TA-Lib/ta-lib.git v0.4.0 --squash (https://github.com/TA-Lib/ta-lib/releases/tag/v0.4.0).

@mrjbq7
Copy link
Collaborator

mrjbq7 commented Feb 5, 2017

Right, how do you imagine we keep up to date with the underlying C library changes? I'm not trying to fork it or have a stale version included here. Also, does this approach work on Windows?

@mckelvin
Copy link
Contributor Author

mckelvin commented Feb 6, 2017

@mrjbq7 The last commit of TA-Lib in svn is 2013-04-03 . I tried to ask the owner to migrate from SourceForge to GitHub but there's still no reply. I think the library is out of maintain right now. So I suppose the TA-Lib code will not be changed any more.

I've migrated the code from svn to github https://github.com/TA-Lib/ta-lib and tagged v0.4.0 according to https://svn.code.sf.net/p/ta-lib/code/tags/ , v0.5.0, v0.6.0-rc1 according to the commit message.

The latest version released in https://ta-lib.org/hdr_dw.html is v0.4.0. The latest version in source code is 0.6.0, and there is also v0.5.0. I don't known why they're not tagged yet. I think it's OK to use v0.4.0 as the latest version for this python ta-lib .

I also compare the ta-lib-0.4.0-src.tar.gz from the official site and the tagged version in svn trunk. They're not fully identical but identical in source code files:

$ diff -r ta-lib/ dist/ta-lib/
Only in ta-lib/: .git
Only in dist/ta-lib/: CHANGELOG.TXT
Only in dist/ta-lib/: HISTORY.TXT
Only in dist/ta-lib/: Makefile.in
Only in dist/ta-lib/: aclocal.m4
Only in dist/ta-lib/: autom4te.cache
Only in ta-lib/: bin
Only in dist/ta-lib/: config.guess
Only in dist/ta-lib/: config.sub
Only in dist/ta-lib/: configure
diff -r ta-lib/configure.in dist/ta-lib/configure.in
6c6
< AC_INIT([ta-lib], [SVN], [https://sourceforge.net/tracker/?group_id=8903&atid=108903])
---
> AC_INIT([ta-lib], [0.4.0], [https://sourceforge.net/tracker/?group_id=8903&atid=108903])
9c9
< AM_INIT_AUTOMAKE([ta-lib], [SVN])
---
> AM_INIT_AUTOMAKE([ta-lib], [0.4.0])
Only in dist/ta-lib/: depcomp
Only in ta-lib/: ide
Only in dist/ta-lib/include: ta_config.h.in
Only in dist/ta-lib/: install-sh
Only in ta-lib/: lib
Only in dist/ta-lib/: ltmain.sh
Only in ta-lib/: make
Only in dist/ta-lib/: missing
Only in dist/ta-lib/src: Makefile.in
Only in dist/ta-lib/src/ta_abstract: Makefile.in
Only in dist/ta-lib/src/ta_common: Makefile.in
Only in dist/ta-lib/src/ta_func: Makefile.in
Only in dist/ta-lib/src/tools: Makefile.in
Only in dist/ta-lib/src/tools/gen_code: Makefile.in
Only in dist/ta-lib/src/tools/ta_regtest: Makefile.in
Only in ta-lib/: temp

But what if the TA-Lib project is back to life in the future? If it's still maintained in svn, we'll mirror the code in svn trunk to https://github.com/TA-Lib/ta-lib and update the vendor/ta-lib using git subtree. Otherwise, we just need to update vendor/ta-lib using git subtree.


Also, does this approach work on Windows?

It's not tested yet since I don't have a Windows machine. I'll try to setup a Windows virtual machine first and test it later.

mckelvin referenced this pull request Feb 6, 2017
I wish this could be preserved automatically but it seems to need manually
re-adding everytime we regenerate the C file.
@mckelvin
Copy link
Contributor Author

mckelvin commented Feb 7, 2017

Also, does this approach work on Windows?

@mrjbq7 Yes. it's working now.


Unfortunately, I was not able to run python setup.py build_ext --inplace in my environment directly. Only pip wheel can handle the VC compiler variables related issue (vcvarsall.bat). So I have to use pip wheel . to build the package:

C:\Users\kelvin\Dev\python-ta-lib>pip wheel .
Processing c:\users\kelvin\dev\python-ta-lib
Building wheels for collected packages: TA-Lib
  Running setup.py bdist_wheel for TA-Lib ... done
  Stored in directory: c:\users\kelvin\dev\python-ta-lib
Successfully built TA-Lib

and install the package:

C:\Users\kelvin\Dev\python-ta-lib>pip install TA_Lib-0.4.10-cp27-cp27m-win_amd64
.whl
Requirement already satisfied: TA-Lib==0.4.10 from file:https:///C:/Users/kelvin/Dev/p
ython-ta-lib/TA_Lib-0.4.10-cp27-cp27m-win_amd64.whl in c:\users\kelvin\anaconda2
\lib\site-packages

Then, I created a directory tests and copied all the test files there to avoid using the talib directory as talib module since there's no _ta_lib.so file. And run nosetests .:

C:\Users\kelvin\Dev\python-ta-lib\tests>dir
 驱动器 C 中的卷没有标签。
 卷的序列号是 A474-8C50

 C:\Users\kelvin\Dev\python-ta-lib\tests 的目录

02/07/17  12:22    <DIR>          .
02/07/17  12:22    <DIR>          ..
02/07/17  11:58             9,066 test_abstract.py
02/07/17  11:58            16,539 test_data.py
02/07/17  11:58             5,033 test_func.py
02/07/17  11:58             1,332 test_stream.py
               4 个文件         31,970 字节
               2 个目录 102,207,041,536 可用字节

C:\Users\kelvin\Dev\python-ta-lib\tests>nosetests .
........................
----------------------------------------------------------------------
Ran 24 tests in 0.905s

OK

TA_Lib-0.4.10-cp27-cp27m-win_amd64.whl.zip

@mckelvin
Copy link
Contributor Author

ping @mrjbq7

@mrjbq7
Copy link
Collaborator

mrjbq7 commented Feb 14, 2017

Hi @mckelvin, thanks for the ping. I meant to look at this in the last few days but didn't get to it. Will try again soon. The idea is very interesting, the implications I wanted to understand (e.g., pip wheel).

@mckelvin
Copy link
Contributor Author

@mrjbq7 Here is the reason why I have to use pip wheel:

I'm using the default compiler (Microsoft Visual C++ Compiler for Python 2.7) for Python in Windows. When building the python package, it need to find vcvarsall.bat first to activate environments for compiler. I tried a few methods but failed to find a way to find vcvarsall.bat for python setup.py build_ext --inplace. I didn't drill down and turned to pip wheel which can find the vcvarsall.bat.

If you don't like VC++, you may also try MinGW, then setup the distutils.cfg and add g++ to the PATH. In this way, you might be able to run python setup.py build_ext --inplace as normal but I haven't tested yet.

@mckelvin
Copy link
Contributor Author

Hi, @mrjbq7 what's the status of the PR right now? Do you have any advice?

@mrjbq7
Copy link
Collaborator

mrjbq7 commented Apr 7, 2017

The status is I haven't had time to get my windows builder to test it locally and I have been traveling (and for next 10 days). I wish I had a co-maintainer so I wasn't single point of failure here! I want to get this one done for sure but needed time.

@mrjbq7
Copy link
Collaborator

mrjbq7 commented May 4, 2017

I've had a few people give me guidance that forking and maintaining the ta-lib C sources is probably more trouble than it's worth... since most non-Windows package managers have easier ta-lib installation...?

@mckelvin
Copy link
Contributor Author

mckelvin commented May 4, 2017

since most non-Windows package managers have easier ta-lib installation...?

@mrjbq7 Not really. For example, in Ubuntu the ta-lib C library is not included in apt. I have to build it from source and installed without package manager. This makes it not easy to check if the software is installed, to distribute the built library, or to uninstall. In this point of view, the problem is mainly inside the ta-lib C library, however it's out of maintain right now. It would be great if the python ta-lib has the fallback ability to build from source.

$ apt search talib
Sorting... Done
Full Text Search... Done

$ apt search ta-lib
Sorting... Done
Full Text Search... Done

@a904guy
Copy link

a904guy commented May 4, 2017

@mckelvin TA-Lib absolutely is in apt.

DEBIAN-ISH APT TA-LIB REPO

sudo add-apt-repository ppa:mario-mariomedina/talib
sudo apt update
sudo apt install ta-lib

~@

@a904guy
Copy link

a904guy commented May 4, 2017

Look over #142

Windows install is absolutely easy via the installer.
Mac/OSX has homebrew https://brewformulas.org/TaLib
Linux has sudo add-apt-repository ppa:mario-mariomedina/talib

There is only a few ta-libs sources in git, the svn-updater, and a semi-maintained branch trying to reimplement the APIs, and to convert it from stateless. So perhaps there may be better TA-Lib in github soon, until then I would keep the source and wrapper separated.

@a904guy
Copy link

a904guy commented May 4, 2017

If we were to implement a install method, it would be better to write a method that handled it from third party sources depending on their OS/ENV than trying to maintain a copy of the source inside the wrapper's repository.

@mckelvin
Copy link
Contributor Author

mckelvin commented May 4, 2017

@a904guy PPA for ta-lib is a good idea for Ubuntu users, but this PPA is not actively maintained, The latest upload date is 2012-02-29. There's even no corresponding package for Ubuntu Trusty (14.04). And what about CentOS/yum ?

Another case is: When a user doesn't have root privilege and he just want to use the python ta-lib inside his own local virtualenv. This PR will make it easier to do so.

The idea of including the third party source comes from https://github.com/redis/hiredis-py/tree/master/vendor , where hiredis is included as a git submodule, and in this PR, sources of the C ta-lib is included as a git subtree. This's not a elegant idea but it just works.

I think it's not proper to use word maintain since we don't need to and should not do any modification with vendor code.

... depending on their OS/ENV ...

The code has been tested on Windows, macOS and Linux (Ubuntu Trusty), and it's compatible with all these 3 OSs.

@a904guy
Copy link

a904guy commented May 4, 2017

You're obviously not familiar with the ta-lib. It's EOL, dead. They never made 0.5 or 0.6 from the roadmap. The only progress that has been made since 2013 is the ta-lib-rt. Which isn't even the TA-Lib Team anymore.

The TA-Lib hasn't been modified for a stable release since 2012.

Even their own SVN trunk is only at 2013.

[Problem]: There isn't a {version} of Ubuntu specific ppa.

[Answer]: It doesn't matter as long as you have libc6-dev & libc-dev. Any of the deb files from that ppa still work.

Those deb files will still work on a trusty environment, and can be installed without superuser permission.
Same with the source code (--prefix)

A user without superuser privileges can still build the source into their own ~/.local/bin or ~/bin which is part of their $PATH. dpkg-deb -x {ta-lib.deb from ppa}, and move a few files around

Flavors: CentOS/yum/RHEL/Fedora/ect

And your code doesn't work on the 3 OSs, it failed on Windows 10 Pro. Doesn't have dependency checking prior. As it failed to install because of numpy missing. Then when I installed numpy. It tried to build TA-Lib and failed again.

On the other hand, I just ran the windows installer from Ta-Lib and it worked flawlessly...

I haven't tested other OSs yet as I'm doing windows testing on a project at the moment.

Which brings me to my point, it increases the development scope of the project from just the python module, to maintaining a build system as well. Nobody has time for that, especially the windows version...

I'm failing to see the benefit of maintaining a system to build the source inside the wrapper. The wrapper is just that, a wrapper/interface to the library. It isn't the library.

If you really want to make a python installer of a relic c project, why don't you make a new repository project, that is called Ta-Lib-Py-Installer, and subtree this repository, and maintain a build system of your own design?

@a904guy
Copy link

a904guy commented May 4, 2017

Sorry man, as someone who has been close to this project since it was just a thought in @mrjbq7 mind.

I don't agree with adding the source and compilation to this repository. It might of worked well for Redis, but that is an active development project. TA-LIB is not.

So I would vote 👎 on #135
Regarding #142 I would close it out as it can be solved by the 14.0 build tools as I provided the download link for.

@herrold
Copy link

herrold commented May 4, 2017

As to an RPM based installation, I had to research a bit, but it appears I packaged ta-lib into a Source RPM, which may be trivially rebuild into binaries, back in 2008.

File and notes are at:

ftp:https://ftp.owlriver.com/pub/mirror/ORC/ta-lib/

Also, Red Hat for their Open Shift Enterprise later released SRPMs which are freely available at:

https://ftp.redhat.com/redhat/linux/enterprise/6Server/en/RHOSE/SRPMS/ta-lib-0.4.0-1.el6op.src.rpm

for their initial 'Open Shift Enterprise', which goes EOL later this year

Red Hat OpenShift Enterprise
https://rhn.redhat.com/errata/rhel6-rhose-errata.html

I just rebuilt my packaging on CentOS 7 without issue

@mrjbq7
Copy link
Collaborator

mrjbq7 commented May 4, 2017

I started a project to re-implement all the ta-lib indicators in pure python and numpy and pandas. That may be a better long term approach than propping up the unmaintained C library.

I had many of them done, but not the candlesticks. So there is still a feature gap. Not yet open-sourced.

@mckelvin
Copy link
Contributor Author

mckelvin commented May 5, 2017

Those deb files will still work on a trusty environment.

I tried that and it works well. I think it's better than this PR for Debian/Ubuntu users. Thanks for the idea that I didn't try before.

Build in docker image python:2.7.12

wget https://ppa.launchpad.net/mario-mariomedina/talib/ubuntu/pool/main/t/ta-lib/libta-lib0_0.4.0-3_amd64.deb
wget https://ppa.launchpad.net/mario-mariomedina/talib/ubuntu/pool/main/t/ta-lib/ta-lib0-dev_0.4.0-3_amd64.deb
echo "1ef8872a2c12fa56346c877e4fbffba892e88367f89db63c57585a2a388790f5 libta-lib0_0.4.0-3_amd64.deb" |  sha256sum -c -
echo "41424a778c9a85cac654341002794bb672fb04087a6f8fbcd0da71c932c2b1ee ta-lib0-dev_0.4.0-3_amd64.deb" | sha256sum -c -

dpkg -i libta-lib0_0.4.0-3_amd64.deb  # XXX: Have to install this dep first manually
dpkg -i ta-lib0-dev_0.4.0-3_amd64.deb

pip install numpy
pip wheel --wheel-dir=wheelhouse .

Runtime in python:2.7.12-slim (without compile tools)

wget https://ppa.launchpad.net/mario-mariomedina/talib/ubuntu/pool/main/t/ta-lib/libta-lib0_0.4.0-3_amd64.deb
echo "1ef8872a2c12fa56346c877e4fbffba892e88367f89db63c57585a2a388790f5 libta-lib0_0.4.0-3_amd64.deb" |  sha256sum -c -

dpkg -i libta-lib0_0.4.0-3_amd64.deb

pip install numpy
pip install wheelhouse/TA_Lib-0.4.10-cp27-cp27mu-linux_x86_64.whl

Doesn't have dependency checking prior. As it failed to install because of numpy missing.

It's irrelevant with the PR. You can also reproduce with the code in upstream that you have to install numpy manually.

It tried to build TA-Lib and failed again.

Have you ever searched vs14 link ran out of heap space and drill down ? What your MEM size?

Which brings me to my point, it increases the development scope of the project from just the python module, to maintaining a build system as well.

I have to admit that the development scope increased by adding a new build system for the underling C library. Have you ever read 84fe612 ? It doesn't nothing but include C/C++ source code while compiling the Cython extension. Is this scope hard for you?

Nobody has time for that, especially the windows version...

I'm not a Windows user. So I'm nobody?

I don't agree with adding the source and compilation to this repository.

I don't agree neither, but I have to if it comes to the initial question: How can I install the python talib module anywhere without pain?

I known that the TA-Lib C library is out of maintain, and that's the reason why I mirror the source to GitHub to make it easier to add it as a git subtree instead of coping code of the C library. Again it's a fallback feature.

It might of worked well for Redis, but that is an active development project. TA-LIB is not.

I don't understand the relationship between a) the underlying library is active development project or not, b) it worked well to combine underlying c code with wrapper in the wrapper repo.

Both are dirty but works.

If you really want to make a python installer of a relic c project, why don't you make a new repository project, that is called Ta-Lib-Py-Installer, and subtree this repository, and maintain a build system of your own design?

Please read 84fe612 first and let me known how can I do it without changing the code in python ta-lib.


My personal reason why sending this PR is to solve the question:

How can I install the python talib module on Ubuntu/Debian without pain?

And this PR is trying to resolve a more common question (though it's not perfect yet):

How can I install the python talib module anywhere without pain?

For me, it's also acceptable to use deb files provided in mario-mariomedina's PPA. You can close this PR if you don't like this feature.

@mckelvin
Copy link
Contributor Author

mckelvin commented May 5, 2017

@mrjbq7 Awesome! Why not open source it as a WIP project now? I know that @femtotrader also developed a similar project https://github.com/femtotrader/pandas_talib . So what's the difference between the two?

@mrjbq7
Copy link
Collaborator

mrjbq7 commented May 5, 2017

I don't have a strong test suite but maybe if I can automate random inputs against ta-lib that might work. Needs some time and a good name.

@mrjbq7
Copy link
Collaborator

mrjbq7 commented May 5, 2017

Also, my version is more numpy-centric. Seems like @femtotrader has more stuff for pandas which is neat too.

@a904guy
Copy link

a904guy commented May 5, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants