-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to find documentation to set User Agent #22
Comments
Within edarWebR you need to set an env variable. Then, i usually call Also, for those of us that are getting errors due to the SEC shutting it down (despite never exceeding the stated limit on their webiste!), I've been substituting read_html with this:
|
Hello, thanks for developing this package. I set an env variable
could you help me how to proceed from here? |
@mirayoon : I have the same issue, but a try-catch block loop until successful like @xbrl-data mentioned will work. |
Has anyone gotten this to work lately? I've been setting the user agent correctly per the SEC website and I'm still not able to return anything other than the messages above. |
Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with. |
I've managed to hack some of the code to work with a with_config wrapper around a GET call. But things only break down again later...On Sep 29, 2021 4:59 PM, xbrl-data ***@***.***> wrote:
Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with.
—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.
|
It seems like some formatting at the SEC site has changed. E.g. the /feed/company-info node seems empty. Cik_ search works though. Does anyone have a workflow they can share for low frequency requests for data access/parsing/report analysis if this is now busted? I was hoping to pair this with finstr-upd/xbrl-upd to show a college class. Mconsidine |
@xbrl-data : can you give me an idea how you have used those as a workaround? A PM would be fine, if you'd prefer. |
@mirayoon I believe this is the proper way to set the environment variable Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail") where yourcompanyname and youremail are specified by the user. You can check it is properly set using the following Sys.getenv("EDGARWEBR_USER_AGENT") |
For me neither |
Hi @balthasars thank you for sharing your solution but, unfortunately, it does not seem to work for me. |
This has worked for me: library(tidyverse) edgar_agent <- Sys.getenv("EDGARWEBR_USER_AGENT", Haven't tried to figure out if there's redundancy/unused vars being set in that ... HTH, |
Hello |
I used the version found here |
Sorry for only getting back to this now, everyone! So it appears that the install of my proposed fork doesn't work when installing it using remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#>
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/Rtmp2UYpBk/remotes894768da6952/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)
bank_terms <- tibble::tribble(
~bank, ~search_terms,
"BCV", "Banque Cantonale Vaudoise",
"Credit Suisse", "Credit Suisse",
"Edmond de Rotschild", c("Edmond de Rothschild"),
"Julius Baer", c("Julius Baer", "Bank Julius Baer"),
"LGT", "LGT",
"Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
"Pictet", c("Pictet", "Bank Pictet"),
"UBS", c("UBS", "UBS AG"),
"Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
"Vontobel", "Vontobel",
"ZKB", "Zurcher Kantonalbank"
) %>%
unnest(search_terms)
bank_search_results <- bank_terms %>%
mutate(
results = purrr::map(
search_terms,
edgarWebR::company_search,
type = "13F-HR"
)
) %>%
unnest(results)
#> No encoding supplied: defaulting to UTF-8.
#> Error in `mutate()`:
#> ! Problem while computing `results = purrr::map(search_terms,
#> edgarWebR::company_search, type = "13F-HR")`.
#> Caused by error in `check_result()`:
#> ! EDGAR request blocked from Undeclared Automated Tool.
#> Please visit https://www.sec.gov/developer for best practices.
#> See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
#> Consider also setting the environment variable 'EDGARWEBR_USER_AGENT
bank_search_results
#> Error in eval(expr, envir, enclos): object 'bank_search_results' not found
Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "" Created on 2022-05-25 by the reprex package (v2.0.1) BUT: when I clone the repo, build the package in RStudio from there and then run my example, ta-dah: library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)
bank_terms <- tibble::tribble(
~bank, ~search_terms,
"BCV", "Banque Cantonale Vaudoise",
"Credit Suisse", "Credit Suisse",
"Edmond de Rotschild", c("Edmond de Rothschild"),
"Julius Baer", c("Julius Baer", "Bank Julius Baer"),
"LGT", "LGT",
"Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
"Pictet", c("Pictet", "Bank Pictet"),
"UBS", c("UBS", "UBS AG"),
"Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
"Vontobel", "Vontobel",
"ZKB", "Zurcher Kantonalbank"
) %>%
unnest(search_terms)
bank_search_results <- bank_terms %>%
mutate(
results = purrr::map(
search_terms,
edgarWebR::company_search,
type = "13F-HR"
)
) %>%
unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#> bank search_terms name cik fiscal_year_end company_href sic
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 BCV Banque Cant… Banq… 0001… 1231 https://www… <NA>
#> 2 Credit Suisse Credit Suis… CRED… 0001… 1231 <NA> <NA>
#> 3 Credit Suisse Credit Suis… CRED… 0000… <NA> <NA> <NA>
#> 4 Credit Suisse Credit Suis… CRED… 0000… 1231 <NA> <NA>
#> 5 Credit Suisse Credit Suis… CRED… 0001… <NA> <NA> <NA>
#> 6 Credit Suisse Credit Suis… CRED… 0000… 1231 <NA> 6211
#> 7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231 <NA> <NA>
#> 8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> 9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> # state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> # mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> # mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> # business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> # business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> # business_street1 <chr>
Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "" Created on 2022-05-25 by the reprex package (v2.0.1) I am puzzled by this behavior and will need to read up some more on environmental variables. UPDATE: remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#>
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/RtmpvaBIOY/remotes937f2cfc9c69/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)
# edgar_agent <- Sys.setenv(EDGARWEBR_USER_AGENT = "edgarWebR [email protected]")
# ua <- httr::user_agent(edgar_agent)
# options(HTTPUserAgent=ua$options$useragent)
bank_terms <- tibble::tribble(
~bank, ~search_terms,
"BCV", "Banque Cantonale Vaudoise",
"Credit Suisse", "Credit Suisse",
"Edmond de Rotschild", c("Edmond de Rothschild"),
"Julius Baer", c("Julius Baer", "Bank Julius Baer"),
"LGT", "LGT",
"Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
"Pictet", c("Pictet", "Bank Pictet"),
"UBS", c("UBS", "UBS AG"),
"Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
"Vontobel", "Vontobel",
"ZKB", "Zurcher Kantonalbank"
) %>%
unnest(search_terms)
bank_search_results <- bank_terms %>%
mutate(
results = purrr::map(
search_terms,
edgarWebR::company_search,
type = "13F-HR"
)
) %>%
unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#> bank search_terms name cik fiscal_year_end company_href sic
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 BCV Banque Cant… Banq… 0001… 1231 https://www… <NA>
#> 2 Credit Suisse Credit Suis… CRED… 0001… 1231 <NA> <NA>
#> 3 Credit Suisse Credit Suis… CRED… 0000… <NA> <NA> <NA>
#> 4 Credit Suisse Credit Suis… CRED… 0000… 1231 <NA> <NA>
#> 5 Credit Suisse Credit Suis… CRED… 0001… <NA> <NA> <NA>
#> 6 Credit Suisse Credit Suis… CRED… 0000… 1231 <NA> 6211
#> 7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231 <NA> <NA>
#> 8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> 9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231 <NA> <NA>
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> # state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> # mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> # mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> # business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> # business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> # business_street1 <chr>
Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "Balthasar [email protected]" Created on 2022-05-25 by the reprex package (v2.0.1) So something must be overriding the correct value for the user agent variable.... |
I had been able to install and use the Balthazar version by specifying the @fix-user-agent branch in the "remotes" request. This was a few months ago and I'm not at my PC right now. I'm not sure what if anything has changed lately, but a get company info request earlier today worked. Mconsidine |
I had luck by changing the edgar_GET function edgar_GET_V2 <- function(href, ua = httr::user_agent(Sys.getenv('EDGARWEBR_USER_AGENT'))){ environment(edgar_GET_V2) <- asNamespace('edgarWebR') assignInNamespace("edgar_GET", edgar_GET_V2, ns = "edgarWebR") |
@twood886 solution worked for me |
As of last week, @twood886 had the only solution that worked for me. |
|
When retrieving filings, I receive an error for being an Undeclared Automated Tool. For example, when using
latest_filings()
I receive error
I found in the README the following information:
However, no details were given below. Could anyone help me on setting the user agent?
The text was updated successfully, but these errors were encountered: