-
Notifications
You must be signed in to change notification settings - Fork 263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Any active users here? #302
Comments
I forked nfldb with the idea that I might a) get it off of deprecated python, and then b) play with it eventually. But I would not describe myself as terribly active with it. I'm not really using nflgame repo at all, yet. It has the same HUGE problem of being only deprecated Python, also |
Actively trying to use it. Just found your nflgame fork and now I need to figure out where the most active fork of nfldb is. I'm in the process of trying to figure out how to port this all over to Ruby instead. |
I still use nfldb. S:\>python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import nfldb
>>> db = nfldb.connect()
>>> q = nfldb.Query(db)
>>> q.game(season_year=2018, week=2, season_type='Preseason')
<nfldb.query.Query object at 0x027E63F0>
>>> for game in q.as_games():
... print game
...
Preseason 2018 week 2 on 08/20 at 07:00PM, BAL (17) at IND (10)
Preseason 2018 week 2 on 08/18 at 07:00PM, TB (30) at TEN (14)
Preseason 2018 week 2 on 08/18 at 12:00PM, JAC (14) at MIN (10)
Preseason 2018 week 2 on 08/18 at 03:00PM, OAK (15) at LA (19)
Preseason 2018 week 2 on 08/18 at 09:00PM, SEA (14) at UNK (24)
Preseason 2018 week 2 on 08/18 at 07:00PM, SF (13) at HOU (16)
Preseason 2018 week 2 on 08/18 at 08:05PM, CHI (24) at DEN (23)
Preseason 2018 week 2 on 08/18 at 06:00PM, CIN (21) at DAL (13)
Preseason 2018 week 2 on 08/17 at 07:00PM, ARI (20) at NO (15)
Preseason 2018 week 2 on 08/17 at 06:00PM, NYG (30) at DET (17)
Preseason 2018 week 2 on 08/17 at 06:30PM, BUF (19) at CLE (17)
Preseason 2018 week 2 on 08/17 at 06:30PM, MIA (20) at CAR (27)
Preseason 2018 week 2 on 08/17 at 06:00PM, KC (28) at ATL (14)
Preseason 2018 week 2 on 08/16 at 06:30PM, PHI (20) at NE (37)
Preseason 2018 week 2 on 08/16 at 07:00PM, PIT (34) at GB (51)
Preseason 2018 week 2 on 08/16 at 07:00PM, NYJ (13) at WAS (15)
>>> |
Oy vey... Good luck! I wouldn't recommend that as @ochawkeye as demonstrated, this project is in working order, albiet a bit quirky. I'm in the middle of prepping for this season so i can't really take on any maintenance questions or the like. I'm ASSUMING as a new user you are unable to access new seasons data? |
I'm attempting to use this, but cannot pull updates. See my issue. UPDATE: I resolved this issue by simply downloading the source code and extracting the scripts folder. Additionally, I had to do all the other stuff folks have been discussing, including adding JAX and LAC to the teams (#289 , this solution) , and installing nflgame directly from the git repo (#296 , this solution). All seems in order now. |
@derek-adair, I've been fiddling with nfldb off and on for the last year or so. I've got the new season's data working fine(in python). I haven't gotten to the point of updating data in Rails. I'm still trying to port the postgres db into Rails which is a PITA in itself. |
@derek-adair edit your link to remove '-redux' from the end. The package is redux but it's still a fork of nflgame. |
Ya i mean i'm unsure of the subtleties of naming a project "redux" it's just something i saw some contributors do. It's a temporary repository so the name doesn't really matter. |
I think he means that the link you pasted to your fork in #302 (comment) |
hah. Fixed. |
damn i keep hitting the close button grr. |
Yes, I have returned to update my PostgreSQL nfldb and use nflgame. @ochawkeye and others have provided excellent direction and your (@derek-adair) fork and issue tracking is also helpful. I find @BurntSushi nfldb implementation in PostgreSQL is awesome. His use of custom data types and domains, triggers and procedures is well beyond my abilities; but an absolute joy to use and try to reverse engineer. His ER diagrams are very useful and can assist in building your own db. I think the only stumbling block to using nfldb as is (from the 2016 code) is to account for the JAC-JAX issue, the LA Rams and the LAC Chargers. That is straight forward INSERT SQL. As well, as the various threads indicate it is necessary to alter nflgame_init_.py and nfldb\team.py (both team1 and team2) code. Also, as @ochawkeye points out, make sure you have a clean install with the required packages. |
Is anyone working on a fork of nfldb that is using nflgame-redux from @derek-adair ? |
'nfldb' works with @derek-adair's 'nflgame-redux'. |
Oh yeah I meant basically someone releasing a |
I plan on doing this when i get some extra time Hopefully soon |
@derek-adair , @ochawkeye @jaredkwright , I think @BurntSushi nfldb (0.2.17) is a "hit the ground running" package. Only two lines of code need to be added to the team.py file and you are good to go. All the challenges of getting a fully functional PostgreSQL nfldb database installed and working with historical integrity stem from the confusion generated for the use of multiple forks of the nflgame library and the confusing/contradictory issues arising from different users using slightly different configurations of nflgame. In particular, incomplete schedule.json files, outdated/incomplete json.gz files in the gamecenter-json directory, and a code deficiency in update_players.py have caused multiple, intersecting problems. The direct result is that the issues documentation has become more of a hindrance than a help in getting a proper install of the database. I work on Windows using PyCharm and DataGrip (or DBVisualizer). I use PyCharm to create a Python 2.7 conda env and install the BurntSushi nfldb (0.2.17) and @derek-adair nflgame-redux (1.2.23) libraries into that project. I use "nickfrerichs/fflproject#29" suggestion and update_players.py file by adding the following two statements after line175:
There are other proposed solutions (BurntSushi/nflgame#316). I verify that the schedule.json file is complete. (I have seem versions missing one game, (i.e., 2017111911, the postponed TB@MIA game. In which case you can manually insert
Or, I have seen schedule.json files that have 2018 entries but are missing over 100 games from 2017. (In this case, the nflgame update-schedule script never correctly updates the schedule.json file.) The remedy is to look around at the various forks of nflgame until you find a complete file. The crucial link here is that nfldb uses the nflgame schedule.json file to load/update its tables. So if the schedule.json file is incomplete, the database will be incomplete. Then, if you want a historically complete (from 2009) nfldb you MUST avoid two standard practices:
So, finally, assuming you use @derek-adair nflgame-redux (1.2.23) and you have corrected the update_players and schedule.json files, then you are ready to install nfldb (0.2.17).
You may get additional errors of the type below. However, again they do not affect the integrity of nfldb or nflgame and they will be resolved as the NFL updates their data in gamecenter and as you run future updates.
Now that nflgame is working with no errors and full integrity of the schedule.json file, you are one step away from installing nfldb. You MUST first ensure that the files in the gamecenter-json directory have a one-to-one correspondence to the files listed in schedule.json (with the exception of 2017083158 (never played) and 2017091006 (posponed and played as 2017111911) and the yet-to-be played games following the current date. As of 2018-12-04, there are 3309 entries in a current schedule.json file. There are 3245 games that the schedule considers "played". There are 2343 files that are in the gamecenter-json directory). Either you need a source for a correct list of these 2343 files, or you need to regenerate them as you run nfldb-update.)
The only two lines you are changing are the JAX and LAC lines in teams2. That is all. That is the only change to @BurntSushi nfldb (0.2.17). That is why this is a 'hit the road running' package. When you run nfldb-update script for the first time on an empty PostgreSQL nfldb, it creates a new (version 8) schema (refer to the meta table after your install) and all the tables, indexex, triggers, domains, the agg_play materialized view, etc. The update takes a while. If you had to empty your gamecenter-json directory to ensure integrity, then the necessary _gtd.json files will, in accordance with schedule.json file contents, be downloaded from "http:https://www.nfl.com/liveupdate/game-center" and then batch loaded into the database. If your .gz files are okay (or you do not need a complete nfldb database), then the upgrade will load whatever is in your gamecenter-json directory. If your schedule.json file is missing a file, but it exists in your gamecenter directory; it will NOT be loaded. The schedule.json file drives the process, but the absence of files in the gamecenter directory can affect the outcome. A completer process results in output (truncated in the middle section) as follows:
You now have a clean, hit the road running nfldb. |
@MargareeMan Thank you for this great write-up! |
@MargareeMan - let me just say.... wow. one of the most helpful comments i've read in a very long time. Thank you. It belongs in a wiki somewhere and illustrates perfectly how one may leverage these projects. Do you have any suggestions on how these problems can be fixed? It seems like fork and;
Also, Please note that I intend on removing the dependency on http:https://www.nfl.com/liveupdate/game-center - There is a much more complete stream of data, that seems to be free (just not advertised). This may or may not impact nfldb and this very issue. |
@derek-adair Thank you for your very generous compliment. But, if I have added to this project, it is only because of the efforts of more competent contributors such as yourself and @ochawkeye (and, of course, the solid foundation that inherited from @BurntSushi ). I cannot advise you on how to proceed with updating the current marriage between nfldb-nflgame. And, a marriage it is. The vows were last renewed on September 8th, 2016. Since then, there have been some small ups and down; but they are still together. It is, frankly, a good marriage based on a strong and sophisticated foundation. Nonetheless, it is fragile. And, although the architecture invites some flexibility, the faint of heart need not apply. I think subsequent development direction really depends on the audience, the functionally desired, and the "legal" stance the developer takes toward the source(s) of data consumed. I am not a Python programmer (in fact I am not a programmer at all). And, I have no interest in Fantasy football. I am a simple hobbyist that is thankful that the NFL passively "permits" hobbyists to acquire data through their public-facing endpoints. Personally, I consider Andrew Gallant's nfldb/nflgame impressive (functioning) implementation as a good model architecture for linking a SQL database to data feeds. It remains tied to NFL proprietary data that we, as hobbyists, are "permitted" to use, individually, at the pleasure of the NFL. Therefore, any alteration you want to make to the fork, I think, needs to carefully consider what -- if any -- data you provide along with the application code. |
wow...so you are just really good at installing software??? |
@MargareeMan Thanks for that magnificent writeup, alongside @derek-adair and @ochawkeye for their contributions. Hoping to help once I get more familiar with the codebase and python--I've got a project I want to finish by labor day and hoping to continue on it during the season using nflgame/db. |
hello all, https://github.com/ml-systems/nfldb let us know what you think |
@Dubl - how disappointing to not open source your work. |
They did. It's MIT licensed. |
@derek-adair Yes it’s open source. All the code is there, it just utilizes Azure resources (storage, db, etc) which costs a nominal amount to run (after the free trial) If you wanted to, you could just run the python script on your own machine, to pull from the api and convert to csv, and then load that to your local db. We just wrapped it up so a newcomer doesn’t need to learn and install python and postgres. Just run a few lines of code, download a free copy of ssms, and start querying your db. We’ll do a more thorough writeup on how it works. |
Whoops I misread the project's intent. It looked like a pre-baked image that you were charging for. Very cool. |
I need glasses D: lol |
“Is the az group create line in DeployNFLDB.ps1 creating a new Resource group? Surely not.” And yes ‘dbadmin’ is a hardcoded db user name. So... if you are tired of it, or want to start over, just delete the resource group from the portal, and you’re free to start over or to be done. The ‘functionapp’ argument is just a name to give the function app. (The bolt-in python). Thanks so much for the questions. Happy to explain/write up more |
Folks, this is the nfldb issue tracker. I don't have a problem with dropping a link to a project, but please have a bit of courtesy and take discussions of other projects to their respective issue trackers. |
Sorry @BurntSushi . My last post was coincidental with your admonishment. I also apologize for mixing the Azure discussion into this thread. My bad. I have deleted my comments. |
No. I'm not a lawyer. So you should go talk to one. This topic has been discussed many times. Please consider searching the nflgame and nfldb issue trackers. Failing all of that, use common sense. |
I'm pretty green when it comes to complex installations, but I followed @MargareeMan's instructions on Mac OS 10.14.6 Mojave and I was able to get everything working in a couple hours. It's so cool! I did have to overcome a few issues installing psycopg2 to a conda environment in python 2.7, but I was able to get it up and running with a few stack overflow hints. https://stackoverflow.com/questions/16407995/psycopg2-image-not-found Once I was able to run the update script, everything worked perfect. It took about 2 hours to fully update. Thanks so much for all the help in this thread! I'm off to start exploring stats. |
Congrats, @msmorey that's better than I did on the Mac. I do not know your competence with SQL; but I struggle with JOINS. So I hope I do not embarrass myself (or you) by passing along a query I did last week while watching the BAL-NYJ SELECT g.week AS "wk", |
I've forked nfldb and begun a python3 conversion. Currently digesting the entire project, but running python 2to3 and swapping in nflgame-redux>2.0.2 has it at least running. Any help would be greatly appreciated. |
In light of our great leader @BurntSushi leaving the project, I've forked the nflgame repo.
Is anyone using this project actively?
The text was updated successfully, but these errors were encountered: