Using nfldb / nflgame with Google AppEngine (et al)

[data-scoop]

After playing around with nfldb and nflgame in the Command Window, I tried to import nfldb into a HelloWorld.py project created in Google AppEngine for my local machine and was unable to use it because I couldn't get the app to find the third party psycopg2 module needed for Postgresql.

"No module named psycopg2._psycopg"

As I mentioned above, both modules work fine when I run in IDLE or Windows Command.

With AppEngine, I used the following successfully with nflgame, but it fails with nfldb due to the psycopg2 dependency:

import sys
sys.path.insert(0, 'C:\Python27\Lib\site-packages')
import nflgame

I'm a total rookie with nfldb and nflgame - Postgres, psycopg2, Python and AppEngine for that matter, but because the psycopg2 directory is a sibling of both nfldb and nflgame I can't think of a reason why it isn't found the way nflgame, and evidently nfldb are found.

Is there a different/better way to approach and accomplish this?

Thank you.

[BurntSushi]

In general, nfldb is higher level and much faster than nflgame. If you're building a web site, then nflgame is probably inadequate. (Unless you generate static content with it.)

I tried to import nfldb into a HelloWorld.py project created in Google AppEngine for my local machine and was unable to use it because I couldn't get the app to find the third party psycopg2 module needed for Postgresql.

I don't really know anything about Google AppEngine. Could you please tell me how you got nfldb on to it? A quick Google search suggests that you should be able to do pip install nfldb. This will install dependencies, including psycopg2.

Note that nfldb also requires a database. Did you read the installation instructions on the wiki?

but because the psycopg2 directory is a sibling of both nfldb and nflgame I can't think of a reason why it isn't found the way nflgame, and evidently nfldb are found.

Ah, OK. In the future, when describing a problem, please include the exact steps (i.e., the commands you ran) to get into your current state. There are so many ways to install Python modules that it is absolutely impossible for me to know what your problem is from a generic ImportError. (pip? easy_install? virtualenv? copied files manually?)

Does this help? http:https://stackoverflow.com/questions/23717834/importerror-no-module-named-psycopg2-after-install

[data-scoop]

I went through the installation process again without incident, following the instructions found at: https://github.com/BurntSushi/nfldb/wiki/Windows-Install

The nflgame and nfldb modules both work as advertised, from the command line. And again, importing nflgame works within AppEngine too when I use the import sys code snippet in my first post.

It's only when I try to import nfldb into an AppEngine file that the dependency psycopg2 is not found.

I guess my question straddles the gamuts of Python and pyscopg2, and it's not actually an nfldb-specific issue. I'm ignorant as a programmer on the various ways one imports modules and their dependencies. I thought you might see something obvious that I didn't do or try.

I realize Python, Psycopg2 and AppEngine are outside your scene, and I'm thankful for your attention thus far :) No worries, I am definitely satisfied and excited with the nflgame, and I can still use nfldb locally without AppEngine.

Thanks again for your time!

For what it's worth:
I've included the following roadmap of my installation steps as a re-fresher.

Windows 7 64bit

1. Install Python27 ( python-2.7.8.amd64.msi )
2. Install pip ( get-pip.py )
3. Install psycopg2 ( psycopg2-2.5.3.win-amd64-py2.7.exe )
4. pip install nfldb ( nfldb nflgame enum34 pytz httplib2 beautifulsoup4 )
5. Install PostgreSQL ( postgresql-9.3.5-1-windows-x64.exe )
   * Finish the wizard without launching Stack Builder
6. Create user nfldb
7. Create new database
8. Enable fuzzystrmatch
9. Download and unzip the nfldb.sql database
10. Import the nfldb.sql database
11. Copy config.ini.sample > config.ini
12. Edit config.ini to reflect Timezone and Password
13. Run nfldb-update

All successful!

[BurntSushi]

I realize Python, Psycopg2 and AppEngine are outside your scene, and I'm thankful for your attention thus far :)

No worries. I'm happy to help at any part of the install process.

Here's the thing. You gave me a pretty good list of what you did to get it working on Windows.

But your problem is with getting it working on Google AppEngine. As far as I can see, you've provided no details on what you've tried other than "import nflgame works and import nfldb does not with a psycopg2 import error." How did you get into that state?

I also linked to a SO post that might be of help. But without knowing more about what you've done on Google AppEngine, it's hard to give more help...

I thought you might see something obvious that I didn't do or try.

Yup. But first, I need to know what you did! :-)

[data-scoop]

I think the problem is that Google AppEngine (GAE) does not support the Psycopg2 module.

To be clear, I'm using AppEngine on my local machine. It provides a server and Python27 support. I have not tried to Deploy to Google's cloud (yet). >>> Sorry for any confusion. <<< Although I will try to deploy nflgame later.

You've asked me what steps got me to my current state, but I haven't done anything more than use the GAE GUI to create a new project on my local machine, and then edit the simple "main.py" file to include the nfl... modules. Main.py imports webapp2 (framework included with GAE) and then creates a Main class to handle HTTP GET requests to the "" url (route).

GAE projects also use an "app.yaml" file where the webapp2 module is ALSO specified, in addition to "main.py"

I didn't have to edit this file to get nflgame to work. However, I tried copying the psycopg2 from \site-packages to the Google AppEngine \lib folder (so as to be a sibling of webapp2), and then edited the app.yaml (a config file) file, and finally got a new error message:

EventError: the library "psycopg2" is not supported

I think that solves that, but I have no idea why nflgame can be imported, but not Pyscopg2. It may be that modules without dependencies cannot be imported, but I dunno.

I think it's wise to stop trying :)

[BurntSushi]

but I haven't done anything more than use the GAE GUI to create a new project on my local machine, and then edit the simple "main.py" file to include the nfl... modules.

OK. That's really hard for me to debug because I don't know what the Google AppEngine GUI is or what it's doing. It's also weird to me that you're running AppEngine stuff locally. I thought GAE was a cloud computing platform.

If you can find a way to use pip with GAE, then everything might work OK.

Copying installed Python modules from one machine to another is _never_ the right thing to do. In the case of psycopg2, it's almost guaranteed to break because it includes a C extension. You might get lucky with a module or two because it's just Python (or Python byte code, which is portable across all supported platforms), but if a module has a C extension, then copying would only work if the source and destination systems are similar (OS and architecture).