From 4d2d3c9061d2e42894bbbf4912b4fe1e103e72a3 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 10:21:01 -0500 Subject: [PATCH 01/21] Add files via upload --- rearc_logo_rgb.png | Bin 0 -> 10914 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 rearc_logo_rgb.png diff --git a/rearc_logo_rgb.png b/rearc_logo_rgb.png new file mode 100644 index 0000000000000000000000000000000000000000..787b2fc22449941e845caff5a5e23f975eebc241 GIT binary patch literal 10914 zcmb7qbzIZm_dhXUfYcB~#^~;nj?o<=(lrod!01N0l}1uZX{kx4l!DX<=`I~ehk)b{ z-=FvQ_x;sBzx}a$cX#eN=XuU+dz^djGgenyg@}Nf00RSqNKI8y4+8^p?|$wB#Jzv# zm7#LDA22=iROB(LQ8e54^C7stk*ATCri6_Pf*1C}#oCtF7vXv@g@GaED{((X*m}a~ zeGyL19umINjQ>bT+|U0m^D)x@BjV{O&1j^hORwPKZcG1!SD2TdQHFq?o?goRg`I?+ zqVm7n?>|X1+IxDsO7QXd`1tVp2=ThOzvL4T7Z>N_7vvKZhU7YFv>V;XmczH@QGX6F6KVSd)iE#a&hMYbAMRkvl&ll#(C&0_k z_y5}D3Ag)i_x`tCf4}+9wtrCl$C%W8Xe1QeZDF1+?)ok+PBLgoe(y+*j~In@~DdS+hSmV z0!gtVFc^}Au)Z?xIdrsk)-J3rO7;P2u)T?@F7U@eIZZTuye-^rK`%qw!9DVRal;5wGtX%Sx%EvV&<%)TJuBUs=PHEpHM-2*U@&CpxlNfN! zv=^58duvB-Q~EW~ocu3XC~!-e{?SCbohvJ<>)LfqdHN|4hTX|jZb!=#cat~ruk-#D zV^->7hil*ojpEc!x9fAktAup;B*MhGn!dq!Jc`OO{sC8lk7-jR?Q^j9o%D2bQaF?l zX3^-q)Mj8mDQ#Y*-N5oUgY;YgK{iC35i5<`XAIi4$1u&p{kRa8PyaZh3Bk%r@y$Q_a0x+FXg3w#xH$=Q6&y^6`1Vic|gNX#VoHWphF$ z{)}EvW$bz2U9l{v+UCH-mcD>Esyxk>&0F88r~ev6x|FWPobZ{3VvX0^X8n!7I^HmA zc;+8@v5XQ12}^`uoBx9Co8j$fZrsIs2oFjLe8%M)h55;BA=AYn)d@P*-QSqOMy5_f zQJpL4FQF|VOO#^#^*80(QsBcgU;B%5n>&IyZVujNikCwr3Eyvq7KnCzf!&F69fy#L zzUuF;J(RbBf}<^)21Y_d62I6GMuSgG{k$O5kEk;fFaYy;&%~X{%K=;8JTK@mLV0)F zJ&PYR2OnC@F!rXeTG$+ZUiSlA>`y^_oVgOU_0FE&e4%vm_kf}M(ukCDDhHNX99$lk zOdOhbk(9Oc>gnNQt*+YWSZXTeVE0t#OM?FF9r)yk0oo@PXnCl{ayxD}ulqvO`?QnG zA`jD{c$;PQ%#dO}Cv_IyU2#;DGghKgGy6zX9urKmriaj0O8Fv*`phyW0X@t;tGURk z+NLsHTF~Jz*O0a4iW-$iuX%dgE+%vDyb%6qSrdd~VaKlG)+32SH~vFX^`|c@evZpe zCC;Z$MCInsWrP0IThR z$m4!vN0(urK034qo5n7J_Twm+FfjFF?0!&Bd)_q&Vt zgnG#R+WWn5sW!GO*s^Ca#V;YYGQVVXy?QVkTl z*-=*%u7UaW+__#UozvJ(m~7F)eYEYS=!yc5SM&E z@dhBl6_M?d1S^Ptmn6)Wd+~2p@ z(KFqZ0gSsX(&tAV?oEiyu;ib%7nxWhu`;@6R&z5&uUS!onL}3*RWm~EB>LIpF*-@} zDWT7J#k0=8>p%b5jf%_(3MEFEF_f#BqKLBdGvZ8Zuu8(k)#7OIm<^#n6rML$+292M zz-@f8os=IBi}3D~<9hMo31zriE3s6#M9qe0e-}?-Y07G|r?a4mb4jFJ;Qm$DnRSWn z3-(3j9^9d+UvKjBSJa>Fg==KM29;3er2~0BdDO$({rkWYM#P*4-xpP;gGN;&nM{P+ zvM7w#F;$`F(ar=?SuS(vv)9I;x=CQ`5gzwPov_yhsoC|U%-;%`%9$H$$-GIZVvL#+ zSc4&AS?QRClAdC9Dn7g-DY&oXhLIJ4GO%%Qcw*K3EFne}c7n%@vlHn!y8 zDU2b7Vy8&$?9stfgQ}TFkjL@7EV|}Bb@QKCYVAr^cpm=(#qj8TT4ycm0hX8ju8T%K zcHG<0UMC&>+)#jVGIO+YNcAN*vmW9@Vxb4_u-(`IiT-d5e*H-$h{4$*e!E%BCKWrm zFZ&1cNANH!;)QTMH^m4uErD>mDeKSXqi})<+ywu~{9fY7+Vg_yfO0+kP?msYH%O3F zeA5NO@Ux|em&~cK7Ql^-5h2cJH?^fkqFkt#RC#xJ6K8%)V}q6vxttvKYVvqdQdyf3 zCg(K4Th3Kzi_c@LED>oYJap>}y8$Q?KQs1`B%+F(9$zDbr;mA|k;Y8=&f|2N^!u|# zJhB!nD9CWx*QpyU(4Img-NXxVUqgW zrn&}0ph-vXpHsuJxZx_UR9bQ#-u1%eRq|Hb2{|C-bJ>Mcto}M-f&OA8>=Iw63T=bS z!w{4e_XO|nen*VG&WoO|uU#Ao1uI7y*rLkXZl1SmD_hH7JgUB!Cm&U+s22^X0{DMr z1uBlP99m`?_#gBVkdoWEFS5h(dr3l%c{(VPCg^J)T0J@?5iR2P z@oWM;F|=^g2y09+$b9&uta;+7CY#ZbtgfD4VzF!;x#IJMLDpiH2~CFo=I%n{ZX$Px zSkQxsL-u#>kpynC@NzHq54ae$6{|}b0DHV398QK{C%yZl(kmN#PMGjAEsNsJy@H(Y z#+NfD(;KT@ukJ#u9tQ6>*!(&E-)-nRS<9agvMedBQs^iC zC<|3TC%2f{pSMO)ibGdzV1{O)!93_bX1%X7_K`NmeqfLOfQPyZV{dZ2Cag?H@}?p* z*n0iti59LnuYG%@1pQn(f|;}|BBf?IoQQrHF8Bnf>TltgFuMA4tj4^s&M{OHLO_@G zmgS#`pi^v_> z8KD)m$@S@1W!^0NfN8=LR$7m)4IJo0_yquEb|0L2=@uO3|9G8VM73_Lz<9hmbQ znm2E@SL{5YZyo@}WGSvgW=9+uG_{)v4kr})38&L7BZa5bwjbvImWV>J7{IF7rGF&Y z8+!uSNJXYeJ~NRgILTO$6Aa&xN#wtu5xY&GRh?B`^i?D~Xdn)Nu&1=f3X7jEM(x=Xo+4C{wwCCG;DsST8^zVZ{BN%z*)Z3wVSW^ zOU?C|Pe#O4&tNam7>?x}F+kY5vLH%5U^;{e6H6?O|3zVl`B&$zZ~)XN_JM=~=BMH3 z$nhm@1tSIX?n`%r;-#V|{U_R8A()c94Ge$qdDq)d4wndJ1uvlykL(9h%f>}e#yZ(e z(^|9AMsW#^LM{PBintQ%20kwfqWJ>Ou{2XT$wrw`CXyeQey_nZ-bLPsuSyB`eVl>g zf3qoV*^hZNJ&*fpm7!-!om!=sf0%VCL$MI9<(HGV&uqRfHI!K|Np)u$*@YyQ)|^?Z zkDB3ZejzsK+*<&{3FOsPF;}SxKLUKyu%n3igNl~?(A}>A48_q|I?5u9V>rTvyhA!m z;`Jy$cTk#T69?9<1*pDA^P3vyA5?sHxj)1=9G>Mtr0H8_x}S4={GnwK^?_B;DV%AM zVJRa1w{ljV5H^#|c()@2@7!!7c*os%HdTeP;Cq9Jesm^X&3h?=N5_RkxvDs1H zlGB~+%W8xpGan=3r}o9^P#XsCE_A+HyMs#wQ$SV^&#c0t%H#Z&B6xO!_P5|bDj^bd z75kIRQS$ba_e^`${THXr)92kgxti37#FF1NLe#LU^`n{0$3lh4-eBOBFQuXgFlF%1 zeM#he#dLn^Ui%o9_^Z(83lYr1rTwsVTu{^8CeA(X2NB)H&xVo0eyub~BjP^f^lH)c z=b-&>@8w!rM}?^2O(9<>Yx5Fy!oPO+=YJ;n;{ZR#{TVuo2|xQp{FcH^C~CtNPx-1p z)swbIW&V|f%3Kim)e-pCxZM&vBL21T>o#Z81Y{MfQ*2d&*IbYBlW~D=K>*ki1UiU^ zXnn2lGjZSCB35Oxj~yy2F*YaMuXIQ<|4!dq4-CL>1Dh$*mEhuo!2*n3NuI+R^0Vwh zt^*l4To0)OM2u33@g#C!ua4T*&f_~)R0{HGHe)Caa6Cm6@EZ72{f#~gpMQtkNv0lA34ZZ|{u0Lc*IX#+>;Eo5(VWkfKu#H3D$OyX% z4VsENQXpSZO*OpF$yPU}!eG|2*G?0pIA(OKoE7Uy_X4>$#K`}~t^Htl)>7z*?Gu-? zUPk+~rDCu!@N|1)X^%>4o+$v3a;R==)Rt!7lMY{xsRK1$q5}+BECfV0-oyzu1j1HHwP*Yrfro4S?Q zRsA!zP0>>B%pfyJpRTK2$FnN;Y$xP`i3V}+?_)%n3)6ZcieVLCy{y**U`&+gF>9?Fsq5%>A>RnESv7P#+jiR7 zW9xX2#gCbZ$s(t-go;4;+E8E4UTV{Ryn?bL{^l4}$IG1tUcrIZAX=M#{h?Vmv}c_6 zv-0tHj8a>CBc?R0x8=S-&q7j@il*ux9#qVI^XetKPI2{b6oDMS7DhoY>vO&HsBJQA zzSKV$XtT2}jFvGXV7u=>l*oJA7`KqVX42%>@tNqCn0flt)${4TFw`kZbZ0E`g;63@ z(xtysH6ko*U(t?QW2_eTR=OmuC!Rq?SPhV8`Yg#Xvr-f_Lr|Nb247C@xcI z;i>Fp+rHGKIM4aEvh=Ob2 z8M7YoLZPVwSXB7iH~om-ITAr0Xof410KWIxPxpg@h`prr2dva^w%&@&a_P~5o50)% znlBH^wESsLj4Pl!KWq$QKQT-gxi~%7*vGGT{&z73Kw4CrJN^| zc&qFcZr+shX|sA-Dz{o%G{{I~v-g=i-x{BpXy}Gk=b-Bj>_2>cyq6#*!Nr1w(sa>r z(r_&8#V1PVwM5qYt2+=SH~J6TvY6}H%3M`j!t#C5e09IiJDHx}s%7=4g3%o+B*Nh& zDWVDMZX3zmBsCLDOPoW3Z(p2PCpF@tD@s7xCPEcDYDO!MCtp9puLEHcLEsBj_;9y_ zR1wrpF-3}@4D;?vjx6_^;GOqZR2Kcx%Qwp#5AL)idWJW`wVew81Rjf!O;4@~--*2r zZgmvICs`9I>^|maPU3Y^uo}IMK|Fard^UaMWTMn3I_=HlWn=p26A zIY_fRHuUzsF%9ihozA};(mBfy{5^!eLf@**vV-#r)uevK)D>~7D^Gd=MX#Xx)c&-D zp(d63Yq1W^TqYMb9y(oQRd&_&yBa-W%N@vqI6wEViY1Zib=;%%K(2@dshKwj)apiw zucv-^CVP6m@tF*KqfBe*L|8~V8h}T3<{|4J+p%-Nhb)SS-wZrSd$*f^hrhdkt&)kp z*lPa`rQ9ezM3g_~tIxstxNa2QnfvTVM4wre!_dO>7N1&dE~bc_%kqu>D-?PG;~h17 z56y?K-1w4kpLr=;h+np+g1Xrl0yQN1{cSJ(*cQbdVsk5$nb#nL9Lf2S>Wcz$pqi`O^+XK7` zJn+j|YX6h^iCYzAzfQYkv}rn;z%p{8%rn|k?j%pogQYqhkmrLHbBRy?%fUqaZOtnn zFk|WkTC3Qhu|~nL%rw8A;>C-+DPfpR^k#ab^?I~rS-KO z)j)~{)&B5AbLCnYoP`3AvFRjlV_BGEJp*$M&o86W=-)-Tc(BAqP(t(3%CnWnf$}qv zDj3qmu?_LUHZu=-QN8EQup%hu3kNdN9(YE>al!~iT__UKhv&sL*vHVgp?Yci2BR+} zYW%kdnRlaD>M8GiF&{@o2A?0APrhGj|5*ua@|U#GQ&$3RhCy9UqTlF>W%YW%F*)ls zfkxsJ1`NAx2SG$v!{}T8VGWuL&q(pyv~sy}-iQz_ASrBNlDbIaF)7ly=mzK;ZKFCN zs&6S2anRp+>#|$7a=Sm)F;2+5%NO*6=IU#7+tZ0_wB^m9-&t}hvJMY&-@UYYfGHlm z*G(f5kBo&QSEnzs%Qf&L|J-gMUL}#A(rR*4saCMFpK3Eje{l_ENakN>&G`$+0B~yfY z%S7*E4|?S{2rKx-MZY?XOpy1^dcHqh!ue(5I#5M;?TC1Te_6zHmEC9Krc-9q)0@0; z@_M5tSZHGVK}YYn22JiLZerqfw~FzY$h&a(;Vah+eyM(@!Er6t4|l*&D?s-B*@hb+ zn_b-|l-HCa2OK0uf8dbvjy24o+{ekCnojxK$rIkvDY@D9M-V<=kb(DBv-3PN^;qAU zX49bKVRl+HIWeBK_Lz8|io9``IZYR(NU0ZpnQ<*KR)d8lUvEX+SWO>Vdq#@UV>*Ew zWu-+WkQ;b(a$-9FB87g3thrHOMe4pm=HQ<$rth3zU#y_5L{uu)f?#qArrQ^5T}W3p zQE__qG3L@0DLDQ<4$7CLu4PPW1KB^^xHy`of1{|$fpsBcaoX*8ZVX8J zsWeLkLyR%+i>}Mc1V2yMw+ww6L`G7d@bIb2f@hT4`^FS)dHuBG3}&J8b|s?c@I8+U z21X|$RZ3%kk@4FpA7gXls7S2KqNS<$m&#avAn&B!$0!(I@EV7Keox=oyx+EZ-GcLJ zCjr)=cIFF{j6+A)+wErl9_Ra}r2c&BF(n-)RZ>|ec7jjkg=*Z4RZM+dHq*#EBwduM zV>lj2MA?|@H1q`NX)o7B{ww$SJc1}&-i8?9e`P|a9RHEKC{O%dIlBsQQj`|Hs zxLj_d7_9;uEPPzw)Z4?2r^*=S;I>N7)+K-gHZy;dO&N zN&UuHZQh1TD!S=q+dYsK_1j2-(>r}(t{_fNJU_kReD@7pX;Pz2=j`wBUwLXoD=}T?xgCW3!R1r?Li%4!y#b`bnw|=i{dH1vPPSze;$7H*j@F!;>&P58) zrTlP1G^ga7ZFRmN-ZjpIdY({dxKJ+36OK@7Qe71Vo{xBR3l`J#{nTVOMI+LmY~Ogx zxH0*u#3Kcme~AX2cQ{RMKR93GZT~)JAO2$U6E%-rcD1?1)1U*`_G~%4#|c@T6VTA@ zD8UAJOLi!%h3W35?rdFHnbe8_`Alb1#53b)x|tRC0uqgMGoAr|?1gD1SIS=P9eiX$ zWC7ku7^KUknm`~Y^SZu>{jRkilj@po%H6p$CQkz`Pz@ahR_}CU11U-tllLi%7f#yA z@!!nk(d4dk<~4tmIHtQl*_EJ5yyJyHWwqO?%AB^`22SHB{W<6Z2C+0*&}x!*-|H4c zre%Hyn!$#S)7Hh3fqJgq;Z=FZQ{5*)#OhfrSX-IE&%}elsbx%ksiWUh%@R(f-JY(0 z{n|?4hL>_yE!xHqX?nYIN+!Y^InsyjIxZ3UKnpM)A2a?jH8+UzV^Yxo{md~&&$9Mq7S$>$1v-o-*y9x8P-tN+O+z_@UpDhTW>0lu4};2`M#hZF zfr>d^s>tSq(61OynWd9$W+y6Jk023<^;<3xEM$fLZaq1O(KwrRza@Y~R8h$(anA@B zW{})m7dKzRWFY42%=a;qh)q~!v+ydT32fEppvYPTP$o-qJa3sDGqQ)oE<`-X8#&UQ z7TJ8ubq~SEB7~j4YgAP zjh9dUfj;)nra5|~G(+=pF7)Rw7f0oxE+r6|<0+e5kDx|WI7y@cr8LMTtIZ+o8nB5E>E4w;*k_iMJ;YJsJG!+h(~sH;9n+z# zH9dVa=sKh9k|{V4;+4*uKROW?OXqlPuK+E#g&^bwYZXEaeMPWU=`}Eay1ybxa{b=! zl8Iim$rUT^soLg-{gRt>W&ni_^s`Aydnmlc zXd!SrGBtEmZ}CgwqiO}MWG{Eh%ro?q zA4v9VDHQ0C47M>yOyaf5Y}VcP%9#^{53Ga&4#FL+B&3rC07UGCn49=HSnrJwzsNQR zLn>VM(CVjzDH$&h{E(k_mKqAs(i`s%CqzIT$;<*EK7e(oxB|w5!y|dYUYjgjI-l-) z)=SfyFbro#EW54eu8k7U9_CV@{Dt|sg{zK!Hu zw}jCWDmJ&2k#EW6PVu_(YJb_hq9C~Kn^#P6fQwpY0zLOke$GmW;KHZDV#NrkJfK?E z&r{3rZGD6iI=P*oLobH_EMc82@-xZfW0Xxwp`jS)39Z}tT6(O;alT=eF^Ago*TzKK zYN&>hKPE(2r02JT&I-{9RPUt{kR*RudIm58aT-B zOodN?dEdo6@ab*eTy&f79;i~lM5Ohd5kR)I&5kq)Wxa~DEvdg&$8-H#@ktd7TZ6&Lb zk8Xd%!W@_D@8o7q9XqJRK(^+j@gFoT-rd?tJ* zSCUv3Ayulthec|6Er#eQfAoxJ!Yb%pK)d3Bv?s9}Brpm*R**#_wQkt3sv{2!#Q@;O zv_TG=jhg6bAR~cQPVkdG?S0wDX8l z`F@=z;q`^n_Esi+bZoM)cRQbVydadwaWslh(f!O}cb6L_NijbiYbSUU|q+dg{g(=reTOT zH#XKm7Y&*cjk2qe3LJr^5QKMiwRs;m4pC-pK)!3P@*G;Z2yRKe{K15|lI%p>LidRY zrGWEm?yatH`9-K#{Z}7elRdQ+UaSkBwdcN&vytlavw^~NXzY-Lz=fsf{lSq57= zi~RPt{_Q#qlUUpA`UBxK1T>BHFLS5>0DPf6A2WW7O~T{;vr4D1g+^Jq!irPcJj^HZ z+@N3BDOOA z!$s@T2DmdJ@bNN-(RSY2A&>URl~`Sv9s46&9A|L`QFg3V|5_XjfqpL9V~ROhLhTDy z76Oj#p4>Gqb#^?dRh;$~C5Nm0C3Hb!Ns;08co=(r*?isK7PO`;quQ1T$wqNWzPUgAc(A$BJM`sf?Z@&6hMt?oX#Z>R0h1LUoCFIKknoryTpY-NMQSSYB)F8bj7lw&G-Xyd3&p%9$U7?t=+ps0uISgkJIEhBOA*%&(-P z>8nRc61(YJUF@{V)Aqa!ReD+JylV~+lZf} sXPNu|Qpf&b)%}~E|DS;wx_XB>|CviZw8r%8FPl_NNn5d6-YV>W0ElZ*lK=n! literal 0 HcmV?d00001 From bb2c53359a6fda5c00a4b37c089c11da976b3e80 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 10:21:25 -0500 Subject: [PATCH 02/21] Create init.sh --- init.sh | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 init.sh diff --git a/init.sh b/init.sh new file mode 100644 index 0000000..bb5a18f --- /dev/null +++ b/init.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash + +# Exit on error. Append "|| true" if you expect an error. +set -o errexit +# Exit on error inside any functions or subshells. +set -o errtrace +# Do not allow use of undefined vars. Use ${VAR:-} to use an undefined VAR +#set -o nounset +# Catch the error in case mysqldump fails (but gzip succeeds) in `mysqldump |gzip` +set -o pipefail +# Turn on traces, useful while debugging but commented out by default +# set -o xtrace + +# Sets profile variable to an empty value by default, reassigns in while loop below if it was included as a parameter +PROFILE="" + +while [[ $# -gt 0 ]]; do + opt="${1}" + shift; + current_arg="$1" + case ${opt} in + "-s"|"--s3-bucket") export S3_BUCKET="$1"; shift;; + "-d"|"--dataset-name") export DATASET_NAME="$1"; shift;; + "-p"|"--product-name") export PRODUCT_NAME="$1"; shift;; + "-i"|"--product-id") export PRODUCT_ID="$1"; shift;; + "-r"|"--region") export REGION="$1"; shift;; + "-f"|"--profile") PROFILE=" --profile $1"; shift;; + *) echo "ERROR: Invalid option: \""$opt"\"" >&2; exit 1;; + esac +done + +while [[ ${#DATASET_NAME} -gt 53 ]]; do + echo "dataset-name must be under 53 characters in length, enter a shorter name:" + read -p "New dataset-name: " DATASET_NAME + case ${#DATASET_NAME} in + [1-9]|[1-4][0-9]|5[0-3]) break;; + * ) echo "Enter in a shorter dataset-name";; + esac +done + +while [[ ${#PRODUCT_NAME} -gt 72 ]]; do + echo "product-name must be under 72 characters in length, enter a shorter name:" + read -p "New product-name: " PRODUCT_NAME + case ${#PRODUCT_NAME} in + [1-9]|[1-6][0-9]|7[0-2]) break;; + * ) echo "Enter in a shorter product-name";; + esac +done + +#creating a pre-processing zip package, these commands may need to be adjusted depending on folder structure and dependencies +(cd pre-processing/pre-processing-code && zip -r pre-processing-code.zip . -x "*.dist-info/*" -x "bin/*" -x "**/__pycache__/*") + +#upload pre-preprocessing.zip to s3 +echo "uploading pre-preprocessing.zip to s3" +aws s3 cp pre-processing/pre-processing-code/pre-processing-code.zip s3://$S3_BUCKET/$DATASET_NAME/automation/pre-processing-code.zip --region $REGION$PROFILE + +#creating dataset on ADX +echo "creating dataset on ADX" +DATASET_COMMAND="aws dataexchange create-data-set --asset-type "S3_SNAPSHOT" --description file://dataset-description.md --name \"${PRODUCT_NAME}\" --region $REGION --output json$PROFILE" +DATASET_OUTPUT=$(eval $DATASET_COMMAND) +DATASET_ARN=$(echo $DATASET_OUTPUT | tr '\r\n' ' ' | jq -r '.Arn') +DATASET_ID=$(echo $DATASET_OUTPUT | tr '\r\n' ' ' | jq -r '.Id') + +#creating pre-processing cloudformation stack +echo "creating pre-processing cloudformation stack" +CFN_STACK_NAME="producer-${DATASET_NAME}-preprocessing" +aws cloudformation create-stack --stack-name $CFN_STACK_NAME --template-body file://pre-processing/pre-processing-cfn.yaml --parameters ParameterKey=S3Bucket,ParameterValue=$S3_BUCKET ParameterKey=DataSetName,ParameterValue=$DATASET_NAME ParameterKey=DataSetArn,ParameterValue=$DATASET_ARN ParameterKey=ProductId,ParameterValue=$PRODUCT_ID ParameterKey=Region,ParameterValue=$REGION --region $REGION --capabilities "CAPABILITY_AUTO_EXPAND" "CAPABILITY_NAMED_IAM" "CAPABILITY_IAM"$PROFILE + +echo "waiting for cloudformation stack to complete" +aws cloudformation wait stack-create-complete --stack-name $CFN_STACK_NAME --region $REGION$PROFILE + +if [[ $? -ne 0 ]] +then + # Cloudformation stack created + echo "Cloudformation stack creation failed" + exit 1 +fi + +#invoking the pre-processing lambda function to create first dataset revision +echo "invoking the pre-processing lambda function to create first dataset revision" +LAMBDA_FUNCTION_NAME="source-for-${DATASET_NAME}" +# AWS CLI version 2 changes require explicitly declairing `--cli-binary-format raw-in-base64-out` for the format of the `--payload` +LAMBDA_FUNCTION_STATUS_CODE=$(aws lambda invoke --function-name $LAMBDA_FUNCTION_NAME --invocation-type "RequestResponse" --payload '{ "test": "event" }' response.json --cli-binary-format raw-in-base64-out --region $REGION --query 'StatusCode' --output text$PROFILE) + +#grabbing dataset revision status +echo "grabbing dataset revision status" +DATASET_REVISION_STATUS=$(aws dataexchange list-data-set-revisions --data-set-id $DATASET_ID --region $REGION --query "sort_by(Revisions, &CreatedAt)[-1].Finalized"$PROFILE) + +update () { + echo "" + echo "Manually create the ADX product and enter in the Product ID below:" + read -p "Product ID: " NEW_PRODUCT_ID + + # Cloudformation stack update + echo "updating pre-processing cloudformation stack" + aws cloudformation update-stack --stack-name $CFN_STACK_NAME --use-previous-template --parameters ParameterKey=S3Bucket,ParameterValue=$S3_BUCKET ParameterKey=DataSetName,ParameterValue=$DATASET_NAME ParameterKey=DataSetArn,ParameterValue=$DATASET_ARN ParameterKey=ProductId,ParameterValue=$NEW_PRODUCT_ID ParameterKey=Region,ParameterValue=$REGION --region $REGION --capabilities "CAPABILITY_AUTO_EXPAND" "CAPABILITY_NAMED_IAM" "CAPABILITY_IAM"$PROFILE + + echo "waiting for cloudformation stack update to complete" + aws cloudformation wait stack-update-complete --stack-name $CFN_STACK_NAME --region $REGION$PROFILE + + if [[ $? -ne 0 ]] + then + echo "Cloudformation stack update failed" + break + fi + echo "cloudformation stack update completed" +} + +delete () { + echo "Destroying the CloudFormation stack" + aws cloudformation delete-stack --stack-name $CFN_STACK_NAME --region $REGION$PROFILE + + #check status of cloudformation stack delete action + aws cloudformation wait stack-delete-complete --stack-name $CFN_STACK_NAME --region $REGION$PROFILE + if [[ $? -eq 0 ]] + then + # Cloudformation stack deleted + echo "CloudFormation stack successfully deleted" + break + else + # Cloudformation stack deletion failed + echo "Cloudformation stack deletion failed" + exit 1 + fi +} + +if [[ $DATASET_REVISION_STATUS == "true" ]] +then + echo "Dataset revision completed successfully" + echo "" + + while true; do + echo "Do you want use this script to update the CloudFormation stack? If you enter 'n' your CloudFormation stack will be destroyed:" + read -p "('y' to update / 'n' to destroy): " Y_N + case $Y_N in + [Yy]* ) update; exit;; + [Nn]* ) delete; break;; + * ) echo "Enter 'y' or 'n'.";; + esac + done + + echo "Manually create the ADX product and manually re-run the pre-processing CloudFormation template using the following params:" + echo "" + echo "S3Bucket: $S3_BUCKET" + echo "DataSetName: $DATASET_NAME" + echo "DataSetArn: $DATASET_ARN" + echo "Region: $REGION" + echo "S3Bucket: $S3_BUCKET" + echo "" + echo "For the ProductId param use the Product ID of the ADX product" + +else + echo "Dataset revision failed" + cat response.json +fi From 3dbf589e3eed77e12a649303a15e56061d669c1b Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 10:22:54 -0500 Subject: [PATCH 03/21] Create .gitignore --- .gitignore | 183 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..776fa6d --- /dev/null +++ b/.gitignore @@ -0,0 +1,183 @@ +# Created by https://www.gitignore.io/api/linux,macos,python,windows +# Edit at https://www.gitignore.io/?templates=linux,macos,python,windows + +### Linux ### +*~ + +# temporary files which can be created if a process still has a handle open of a deleted file +.fuse_hidden* + +# KDE directory preferences +.directory + +# Linux trash folder which might appear on any partition or disk +.Trash-* + +# .nfs files are created when an open file is removed but is still being accessed +.nfs* + +### macOS ### +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +### Python ### +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +pip-wheel-metadata/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +.hypothesis/ +.pytest_cache/ + +# Translations +*.mo +*.pot + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +target/ + +# pyenv +.python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# celery beat schedule file +celerybeat-schedule + +# SageMath parsed files +*.sage.py + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# Mr Developer +.mr.developer.cfg +.project +.pydevproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +### Windows ### +# Windows thumbnail cache files +Thumbs.db +Thumbs.db:encryptable +ehthumbs.db +ehthumbs_vista.db + +# Dump file +*.stackdump + +# Folder config file +[Dd]esktop.ini + +# Recycle Bin used on file shares +$RECYCLE.BIN/ + +# Windows Installer files +*.cab +*.msi +*.msix +*.msm +*.msp + +# Windows shortcuts +*.lnk + +# End of https://www.gitignore.io/api/linux,macos,python,windows + +**/pre-processing-code/*/ +**/pre-processing-code.zip + +response.json From 43a5851089022fa606bb6e9aab376e8a7aa36fe9 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 10:38:58 -0500 Subject: [PATCH 04/21] Create README.md --- README.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..e75a6ef --- /dev/null +++ b/README.md @@ -0,0 +1,62 @@ + +Rearc Logo + + +Order and Referring Medicare Data | CMS +========================= +You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting [https://aws.amazon.com/marketplace/pp/prodview-ymiystuotpuwa](https://aws.amazon.com/marketplace/pp/prodview-ymiystuotpuwa). + +Main Overview +------------- + +This release provides information of medical specialists, physicians and other healthcare professionals who can be consulted by those in the Medicare program released by +the Centers for Medicare & Medicaid Services (CMS). + +Centers for Medicare & Medicaid provide substantial amounts of data to +the public about their programs, various health care topics, and care +settings. If you are interested in learning more about datasets released +by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). + +#### Data Source + +This resource is presented in both the JSON and CSV formats. The +included dataset contains the following columns: + +`"npi", "last_name", "first_name", "partb", "dme", "hha", "pmd"` + +More Information +---------------- + +- Source - [Centers for Medicare & Medicaid Services + (CMS)](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination) +- [CMS Data Homepage](https://data.cms.gov/) +- [Terms of Use](https://www.usa.gov/government-works) +- Frequency: Weekly +- Formats: CSV, JSON + +Contact Details +--------------- + +- If you find any issues or have enhancements with this product, open + up a GitHub + [issue](https://github.com/rearc-data/Data-Dissmenination-CMS/issues) + and we will gladly take a look at it. Better yet, submit a pull + request. Any contributions you make are greatly appreciated :heart:. +- If you are interested in any other open datasets, please create a + request on our project board + [here](https://github.com/rearc-data/covid-datasets-aws-data-exchange/projects/1). +- If you have questions about this source data, please send the + Center for Medicare & Medicaid Services an email at customerservice@npienumerator.com. +- If you have any other questions or feedback, send us an email at + data@rearc.io. + +About Rearc +----------- + +Rearc is a cloud, software and services company. We believe that +empowering engineers drives innovation. Cloud-native architectures, +modern software and data practices, and the ability to safely experiment +can enable engineers to realize their full potential. We have partnered +with several enterprises and startups to help them achieve agility. Our +approach is simple — empower engineers with the best tools possible to +make an impact within their industry. From 8c5d32d673bd335d3681794198de5e47c54f521c Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 10:40:02 -0500 Subject: [PATCH 05/21] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e75a6ef..90b6607 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Order and Referring Medicare Data | CMS ========================= -You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting [https://aws.amazon.com/marketplace/pp/prodview-ymiystuotpuwa](https://aws.amazon.com/marketplace/pp/prodview-ymiystuotpuwa). +You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting Main Overview ------------- From 1ef1bd34e201986cdd25045a2df216b938b7b4c4 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:04:49 -0500 Subject: [PATCH 06/21] Update README.md --- README.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 90b6607..c3caf2b 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,16 @@ Rearc Logo -Order and Referring Medicare Data | CMS + + +Data Dissmenination | CMS ========================= You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting Main Overview ------------- -This release provides information of medical specialists, physicians and other healthcare professionals who can be consulted by those in the Medicare program released by -the Centers for Medicare & Medicaid Services (CMS). +This release provides information of NPPES health care providers. Centers for Medicare & Medicaid provide substantial amounts of data to the public about their programs, various health care topics, and care @@ -20,9 +21,16 @@ by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). #### Data Source This resource is presented in both the JSON and CSV formats. The -included dataset contains the following columns: +included zip download contains the following files: + +- an endpoint file for Type 1 and 2 NPIs +- NPI data file +- other names file for NPI type 2s which have other names associated with them +- code values +- a read me +- a practice location(pl) file which contains all the non-primary pl for for type 1 and 2 NPIs + -`"npi", "last_name", "first_name", "partb", "dme", "hha", "pmd"` More Information ---------------- From e4974eff94b545b86293f4b6cff54dadc9719e55 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:04:51 -0500 Subject: [PATCH 07/21] Create product-description.md --- product-description.md | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 product-description.md diff --git a/product-description.md b/product-description.md new file mode 100644 index 0000000..1dd3b6b --- /dev/null +++ b/product-description.md @@ -0,0 +1,66 @@ + + +Data Dissmenination | CMS +========================= +You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting + +Main Overview +------------- + +This release provides information of NPPES health care providers. + +Centers for Medicare & Medicaid provide substantial amounts of data to +the public about their programs, various health care topics, and care +settings. If you are interested in learning more about datasets released +by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). + +#### Data Source + +This resource is presented in both the JSON and CSV formats. The +included zip download contains the following files: + +- an endpoint file for Type 1 and 2 NPIs +- NPI data file +- other names file for NPI type 2s which have other names associated with them +- code values +- a read me +- a practice location(pl) file which contains all the non-primary pl for for type 1 and 2 NPIs + + + +More Information +---------------- + +- Source - [Centers for Medicare & Medicaid Services + (CMS)](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination) +- [CMS Data Homepage](https://data.cms.gov/) +- [Terms of Use](https://www.usa.gov/government-works) +- Frequency: Weekly +- Formats: CSV, JSON + +Contact Details +--------------- + +- If you find any issues or have enhancements with this product, open + up a GitHub + [issue](https://github.com/rearc-data/Data-Dissmenination-CMS/issues) + and we will gladly take a look at it. Better yet, submit a pull + request. Any contributions you make are greatly appreciated :heart:. +- If you are interested in any other open datasets, please create a + request on our project board + [here](https://github.com/rearc-data/covid-datasets-aws-data-exchange/projects/1). +- If you have questions about this source data, please send the + Center for Medicare & Medicaid Services an email at customerservice@npienumerator.com. +- If you have any other questions or feedback, send us an email at + data@rearc.io. + +About Rearc +----------- + +Rearc is a cloud, software and services company. We believe that +empowering engineers drives innovation. Cloud-native architectures, +modern software and data practices, and the ability to safely experiment +can enable engineers to realize their full potential. We have partnered +with several enterprises and startups to help them achieve agility. Our +approach is simple — empower engineers with the best tools possible to +make an impact within their industry. From d0452149a820e8721d6233b99986752d342e869b Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:07:00 -0500 Subject: [PATCH 08/21] Create dataset-description.md --- dataset-description.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 dataset-description.md diff --git a/dataset-description.md b/dataset-description.md new file mode 100644 index 0000000..b365222 --- /dev/null +++ b/dataset-description.md @@ -0,0 +1 @@ +This release provides information of NPPES health care providers. From fc92634c08b487ef1641fcca57fb8685585074e6 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:33:47 -0500 Subject: [PATCH 09/21] Update product-description.md --- product-description.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/product-description.md b/product-description.md index 1dd3b6b..bd62291 100644 --- a/product-description.md +++ b/product-description.md @@ -1,6 +1,6 @@ -Data Dissmenination | CMS +NPPES NPI Registry Data | CMS ========================= You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting @@ -16,7 +16,7 @@ by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). #### Data Source -This resource is presented in both the JSON and CSV formats. The +This resource is presented in both the PDF and CSV formats. The included zip download contains the following files: - an endpoint file for Type 1 and 2 NPIs From 54eb32ed19d8803ee8d8862432b317879dae0531 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:33:51 -0500 Subject: [PATCH 10/21] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c3caf2b..8b501a6 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ -Data Dissmenination | CMS +NPPES NPI Registry Data | CMS ========================= You can subscribe to the AWS Data Exchange product utilizing the automation featured in this repository by visiting @@ -20,7 +20,7 @@ by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). #### Data Source -This resource is presented in both the JSON and CSV formats. The +This resource is presented in both the PDF and CSV formats. The included zip download contains the following files: - an endpoint file for Type 1 and 2 NPIs From 94e07a9fb20a8bbb8fa3b0bd870b1a38b3b1b20f Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:34:48 -0500 Subject: [PATCH 11/21] Update product-description.md --- product-description.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product-description.md b/product-description.md index bd62291..8b14c41 100644 --- a/product-description.md +++ b/product-description.md @@ -36,7 +36,7 @@ More Information - [CMS Data Homepage](https://data.cms.gov/) - [Terms of Use](https://www.usa.gov/government-works) - Frequency: Weekly -- Formats: CSV, JSON +- Formats: CSV, PDF Contact Details --------------- From 2bfa83e1922e387220d9bc13021f1ea61aea48c0 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 11:34:55 -0500 Subject: [PATCH 12/21] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8b501a6..7e19db4 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ More Information - [CMS Data Homepage](https://data.cms.gov/) - [Terms of Use](https://www.usa.gov/government-works) - Frequency: Weekly -- Formats: CSV, JSON +- Formats: CSV, PDF Contact Details --------------- From b889aaad23e57f38f739accf9c813387cc416105 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:07:25 -0500 Subject: [PATCH 13/21] Update product-description.md --- product-description.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/product-description.md b/product-description.md index 8b14c41..cd0add2 100644 --- a/product-description.md +++ b/product-description.md @@ -7,7 +7,8 @@ You can subscribe to the AWS Data Exchange product utilizing the automation feat Main Overview ------------- -This release provides information of NPPES health care providers. +This release contains an achieved copy of the National Plan and Provider Enumeration System (NPPES)'s National Provider Identifier (NPI) registry data. NPI numbers are utilized throughout various interactions within the United State medical system to identify individual healthcare providers. + Centers for Medicare & Medicaid provide substantial amounts of data to the public about their programs, various health care topics, and care From b8da39d696e8e1f90114a72cfc96036fab9cb246 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:07:28 -0500 Subject: [PATCH 14/21] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7e19db4..b996db8 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,8 @@ You can subscribe to the AWS Data Exchange product utilizing the automation feat Main Overview ------------- -This release provides information of NPPES health care providers. +This release contains an achieved copy of the National Plan and Provider Enumeration System (NPPES)'s National Provider Identifier (NPI) registry data. NPI numbers are utilized throughout various interactions within the United State medical system to identify individual healthcare providers. + Centers for Medicare & Medicaid provide substantial amounts of data to the public about their programs, various health care topics, and care From 7a5172f70ff657be67fb2dec4d5c3131eb44f36e Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:08:14 -0500 Subject: [PATCH 15/21] Update dataset-description.md --- dataset-description.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dataset-description.md b/dataset-description.md index b365222..ad71f6e 100644 --- a/dataset-description.md +++ b/dataset-description.md @@ -1 +1 @@ -This release provides information of NPPES health care providers. +This release contains an achieved copy of the National Plan and Provider Enumeration System (NPPES)'s National Provider Identifier (NPI) registry data. NPI numbers are utilized throughout various interactions within the United State medical system to identify individual healthcare providers. From 5bb169c12e46530e388f9ffd711b734969c1fb1e Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:22:55 -0500 Subject: [PATCH 16/21] Update product-description.md --- product-description.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/product-description.md b/product-description.md index cd0add2..c0d35f6 100644 --- a/product-description.md +++ b/product-description.md @@ -17,15 +17,7 @@ by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). #### Data Source -This resource is presented in both the PDF and CSV formats. The -included zip download contains the following files: - -- an endpoint file for Type 1 and 2 NPIs -- NPI data file -- other names file for NPI type 2s which have other names associated with them -- code values -- a read me -- a practice location(pl) file which contains all the non-primary pl for for type 1 and 2 NPIs +The provided data files are presented in CSV format, while the associated documentation files are presented PDF format. For details regarding the included files visit [CMS's webpage for NPI Data Dissemination](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination). From 6c1af7f4b54ff3098828d8e76cbc25b857c01e74 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:23:20 -0500 Subject: [PATCH 17/21] Update README.md --- README.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/README.md b/README.md index b996db8..86f5110 100644 --- a/README.md +++ b/README.md @@ -21,15 +21,7 @@ by the CMS, please visit the [CMS Data homepage](https://data.cms.gov/). #### Data Source -This resource is presented in both the PDF and CSV formats. The -included zip download contains the following files: - -- an endpoint file for Type 1 and 2 NPIs -- NPI data file -- other names file for NPI type 2s which have other names associated with them -- code values -- a read me -- a practice location(pl) file which contains all the non-primary pl for for type 1 and 2 NPIs +The provided data files are presented in CSV format, while the associated documentation files are presented PDF format. For details regarding the included files visit [CMS's webpage for NPI Data Dissemination](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination). From 39af1038c944b4fabc425502db96354c4c831d04 Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:31:01 -0500 Subject: [PATCH 18/21] Update product-description.md --- product-description.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product-description.md b/product-description.md index c0d35f6..cb1e4bc 100644 --- a/product-description.md +++ b/product-description.md @@ -26,7 +26,7 @@ More Information - Source - [Centers for Medicare & Medicaid Services (CMS)](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination) -- [CMS Data Homepage](https://data.cms.gov/) +- [CMS Data Homepage](https://www.cms.gov/) - [Terms of Use](https://www.usa.gov/government-works) - Frequency: Weekly - Formats: CSV, PDF From 6dfd62cb1e2ccaf149c946964e62dc2ed68f539b Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:31:17 -0500 Subject: [PATCH 19/21] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 86f5110..597cd6f 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ More Information - Source - [Centers for Medicare & Medicaid Services (CMS)](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination) -- [CMS Data Homepage](https://data.cms.gov/) +- [CMS Homepage](https://www.cms.gov/) - [Terms of Use](https://www.usa.gov/government-works) - Frequency: Weekly - Formats: CSV, PDF From 554976fe84f15b0f7e79dbcf78a76653f188c83f Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:31:28 -0500 Subject: [PATCH 20/21] Update product-description.md --- product-description.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product-description.md b/product-description.md index cb1e4bc..66814eb 100644 --- a/product-description.md +++ b/product-description.md @@ -26,7 +26,7 @@ More Information - Source - [Centers for Medicare & Medicaid Services (CMS)](https://www.cms.gov/Regulations-and-Guidance/Administrative-Simplification/NationalProvIdentStand/DataDissemination) -- [CMS Data Homepage](https://www.cms.gov/) +- [CMS Homepage](https://www.cms.gov/) - [Terms of Use](https://www.usa.gov/government-works) - Frequency: Weekly - Formats: CSV, PDF From 05c306f05d8c81b17062d59e5f1f68b5c17142aa Mon Sep 17 00:00:00 2001 From: AyushSVarma <34693841+AyushSVarma@users.noreply.github.com> Date: Wed, 15 Jul 2020 12:31:55 -0500 Subject: [PATCH 21/21] Update CONTRIBUTING.md --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e1761ad..3ef7a9f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -37,7 +37,7 @@ For more details on the technologies used in our ADX products, please visit [Get ## How can I contribute? #### Report an Issue/Bug or Submit an Improvement/Suggestion -If you have feedback specific to the ADX product featured in this repository, the best way to contact us would be through [opening a GitHub issue](https://github.com/rearc-data/ordering-referring-medicare/issues) in this repository. Before opening an issue please review the existing suggestions to see if your idea is already there. If already present, please comment on the existing issue instead of making a new one. +If you have feedback specific to the ADX product featured in this repository, the best way to contact us would be through [opening a GitHub issue](https://github.com/rearc-data/nppes-npi-registry-data-cms/issues) in this repository. Before opening an issue please review the existing suggestions to see if your idea is already there. If already present, please comment on the existing issue instead of making a new one. When opening an issue please **be as descriptive as possible**. If relevant please **provide information regarding your use-case, development configuration and environment**. The more specific you can be the easier it will be for us to identify and address the situation.