HTTP Error 403: Forbidden

[anatoliakor]

issue came approximately last month, when i tried to run a script, that work appropriate for 1 year at least.
error came:
Traceback (most recent call last):
File "C:_python_projects\my_finance\stock_quotes.py", line 24, in finam_quote
paper = exporter.lookup(code=ticker, market=market)
File "C:_python_python3.9\lib\site-packages\finam\export.py", line 303, in lookup
return self._meta.lookup(*args, **kwargs)
File "C:_python_python3.9\lib\site-packages\finam\export.py", line 220, in lookup
self._load()
File "C:_python_python3.9\lib\site-packages\finam\export.py", line 167, in _load
meta_url = page.find_meta_file()
File "C:_python_python3.9\lib\site-packages\finam\export.py", line 82, in find_meta_file
html = self._fetcher(self.FINAM_ENTRY_URL)
File "C:_python_python3.9\lib\site-packages\finam\export.py", line 55, in fetch_url
raise FinamDownloadError('Unable to load {}: {}'.format(url, e))
finam.exception.FinamDownloadError: Unable to load https://www.finam.ru/profile/moex-akcii/gazprom/export/: HTTP Error 403: Forbidden
It seems like finam restrict automatic loading, because i can do it from same computer, using web interface

[ffeast]

Yes it seems so. Would be happy to see a pull request because I don't have spare time to figure it out

[miko1ann]

I tryed to replace user-agent but error stil repduces. So problen is in securuty policy of finam api.

[anatoliakor]

Yes it seems so. Would be happy to see a pull request because I don't have spare time to figure it out

I tryed to replace user-agent but error stil repduces. So problen is in securuty policy of finam api.

yes, i did the same and got same result... it seems that security protection distinguishes somehow differently (not by user-agent). i stopped attemts to figure it out and starting to find another way to get stocks quotes

[miko1ann]

Workaround with selenium

`
def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 100).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.split('\n')
return res

...

exporter = Exporter(fetcher = fetcher)

`

[ffeast]

Interesting. Pull request?

[miko1ann]

No nesessary, just copy and use. Also this way contains issue with encoding for cyrilyc symbols.

update

def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 100).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.encode('cp1252').decode('cp1251') # <
res = res.split('\n') # <
return res

UPD 28.01.23: Working solution

[Celeevo]

Добрый день,
если у кого-то получилось исправить доступ с помощью описанной выше заплатки, поделитесь пжлст для тех кто в танке - куда копировать новый fetcher и что дополнительно нужно импортировать (driver, WebDriver..., etc)

[miko1ann]

Complete workaround, contains downloading data.

def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 15).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.encode('cp1252').decode('cp1251')
res = res.split('\n')
return res

def fetcher_download(url, readlines = False):
print ('>>>', url)
r = requests.get(url, headers = {'User-Agent':'Mozilla 8'}, stream=True)
return r.content.decode('utf8')
if lines:
return r.content.decode('utf8').split('\n\r')
else :
return r.content.decode('utf8')

exporter = Exporter(fetcher = fetcher)
exporter._fetcher = fetcher_download

I think this solution is temorary, until Finam changes an autorisation of the web downloading page.
Different way to reanimate project is research workarount to extract token and pass it to request (to awoid webdriver...)
So its looks like strange, because they are produces acess to stock data throw web page(free), and connector with .net lib (login:password), but dont provides api...

[miko1ann]

Thanks @ffeast fo the extensible and clear code.

[am1ter]

@Celeevo I used code above and created new version of the lib. You can install it with: pip install git+https://github.com/am1ter/finam-export
I also created pull request, but before you can use it, you have to wait until @ffeast review it and publish.

[Celeevo]

@am1ter Thanks a lot, it works now!

[ffeast]

Hi
@am1ter commented on the PR, please check. The overall takeaway is that now it looks too complex with 2 different fetchers and all classes being aware of that

[am1ter]

Hi @am1ter commented on the PR, please check. The overall takeaway is that now it looks too complex with 2 different fetchers and all classes being aware of that

Hello @ffeast ! Thank you for a quick response. I don't see any comment to the PR at the moment. I am not sure why.

On one hand, I agree with you that it is not the best decision to have two independant fetchers in Exporter class. By the other hand, it gives a user an option to declare using standart urllib fetcher manually in the case if token-protection will be disabled one day.
Also, as I see there is an alternative way to use second fetcher. For example, try to use urllib-based fetcher and in the case of the error use webdriver-based fetcher. However, I see some downsides in this soltions as well.

What do you think? Or maybe you can even suggest more effective solution?

[ffeast]

Hi
Could you please check if any of the comments are seen on this page or here?

[am1ter]

Hi Could you please check if any of the comments are seen on this page or here?

Hello @ffeast! Unfortunately, not. And it is quite strange. Look at the screenshot 1 and screenshot 2.

[am1ter]

@ffeast Hello again! So, since I don't see any comment on the PR page maybe you can post it here?