Skip to content

Commit

Permalink
initial commit of code with parquet data
Browse files Browse the repository at this point in the history
  • Loading branch information
jmlondon committed Jun 7, 2024
1 parent 838ad91 commit 9485c7b
Show file tree
Hide file tree
Showing 183 changed files with 61 additions and 0 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.Rproj.user
.Rhistory
.RData
.Ruserdata
42 changes: 42 additions & 0 deletions R/get_locs_obs.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
library(DBI)
library(dbplyr)
library(dplyr)
library(RPostgres)
library(sf)
library(arrow)
library(geoarrow)

tryCatch({
con <- dbConnect(
RPostgres::Postgres(),
dbname = 'pep',
host = Sys.getenv('PEP_PG_IP'),
user = keyringr::get_kc_account("pgpep_londonj"),
password = keyringr::decrypt_kc_pw("pgpep_londonj")
)
}, error = function(cond) {
print("Unable to connect to Database.")
})

tbl_deploy <- dplyr::tbl(con,in_schema("telem","tbl_tag_deployments")) |>
dplyr::collect()

locs_obs <- sf::st_read(con,Id("telem","geo_wc_locs_qa")) |>
dplyr::left_join(tbl_deploy) |>
dplyr::filter(meta_project %in% c('Ice Seals'),
species %in% c('Hf','Pl')) |>
dplyr::select(speno,deployid,ptt,instr,tag_family,type,quality,locs_dt,latitude,longitude,
error_radius, error_semi_major_axis,error_semi_minor_axis,
error_ellipse_orientation, project, species, qa_status,deploy_dt, end_dt,
deploy_lat,deploy_long,capture_lat,capture_long, geom)

unlink(here::here('locs_obs'),
recursive = TRUE,
force = TRUE)

write_dataset(locs_obs,
path='locs_obs',
format="parquet",
partitioning = 'speno')

dbDisconnect(con, disconnect = TRUE)
15 changes: 15 additions & 0 deletions cefi-ice-seal-movement.Rproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Version: 1.0

RestoreWorkspace: Default
SaveWorkspace: Default
AlwaysSaveHistory: Default

EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8

RnwWeave: Sweave
LaTeX: pdfLaTeX

AutoAppendNewline: Yes
Binary file added locs_obs/speno=HF2005_5880/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5882/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5884/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5886/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5888/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5890/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5892/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5894/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5896/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2005_5898/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5975/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5978/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5980/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5981/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5982/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5988/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5989/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2006_5991/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6039/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6044/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6047/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6048/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6049/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6050/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6051/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6056/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6065/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6066/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6067/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6068/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6069/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6070/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6071/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6073/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6075/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6082/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6085/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6089/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6090/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6099/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6100/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6101/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6102/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6103/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6109/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2007_6110/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2008_1001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1015/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1016/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1017/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1018/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1019/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1021/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1023/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1024/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_1026/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2012/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2017/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2018/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2021/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2022/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2023/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2024/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2025/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2026/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2027/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2028/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2029/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2030/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2031/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2035/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2037/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2040/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2009_2041/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1007/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1010/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1012/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_1017/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2000/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2006/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2007/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2010/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2015/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2022/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2010_2023/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1003/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1005/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1007/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1008/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1013/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1015/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1018/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1022/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1023/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1025/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2014_1026/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1013/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1018/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1019/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2016_1023/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=HF2022_1005/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5974/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5976/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5977/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5984/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5986/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2006_5990/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6037/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6038/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6040/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6041/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6042/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6055/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6061/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6064/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6078/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6081/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6091/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2007_6097/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1003/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1020/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_1025/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2003/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2004/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2005/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2006/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2010/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2011/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2014/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2016/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2019/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2020/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2032/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2033/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2034/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2038/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2009_2039/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_1002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_1003/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_1018/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_2002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_2004/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_2005/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_2020/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2010_2021/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2014_1017/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2014_1019/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2014_1020/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2014_1027/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2014_1028/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1001/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1003/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1004/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1007/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1016/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2016_1021/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2018_1002/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2018_1009/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2018_1016/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2018_1022/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2018_1027/part-0.parquet
Binary file not shown.
Binary file added locs_obs/speno=PL2022_1004/part-0.parquet
Binary file not shown.

0 comments on commit 9485c7b

Please sign in to comment.