-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Login now includes a hidden captcha that prevents python-myfitnesspal from logging-in #144
Comments
I'm having the same problem. It does look like MFP changed something about the login process. |
It looks like there's an invisible recaptcha. |
…n credentials directly from local browsers instead of directly logging in to MyfitnessPal. See #144 for more information.
…n credentials directly from local browsers instead of directly logging in to MyfitnessPal. See #144 for more information.
…n credentials directly from local browsers instead of directly logging in to MyfitnessPal. See #144 for more information.
OK... good and bad news! The good news is that I've updated this library such that it's again able to access MyFitnessPal on your behalf. You can find those changes in version 2.0.0 of this library. The bad news is that, as you might guess from the major version bump, the way we access login credentials had to be changed in ways that may limit how useful this library is to you. I'm really sorry about that, but I don't see an alternative at the moment. Now that MyFitnessPal has added a hidden captcha to their log in flow, this library will no longer be able to log in directly in the manner it was historically doing so. Instead, this library now uses the browser_cookie3 library for gathering cookies from your local browser for use when interacting with MyFitnessPal. To be a little more concrete:
If any of you have any clever ideas of other ways of getting around this limitation, I'd love to hear them! Cheers & good luck! |
There are some people that use this library indirectly via a Home Assistant integration. For some context: helto4real/custom_component_myfitnesspal#25 |
You can absolutely instantiate your own cookiejar and hand it directly to Unfortunately, though, you're still going to need to log-in in an actual browser somewhere to find your session token given that the log-in flow has that invisible captcha. There is unfortunately no avenue forward via which we can return to accepting a username/password combination directly as we were before while a captcha is present. |
I did briefly try to de-compile the latest android APK to see what api endpoint mobile clients were using to request auth as there's no captcha there but ran out of time before I could really get my head around all the decompiled code. Probably for the better, it wouldn't be too difficult for MFP to detect/prevent unauthorized use of the mobile login endpoint :/. Thanks for spelling out the cookie jar details... it shouldn't be too difficult to get the HA <-> MFP integration working again even if it means walking the user through browser debug/inspector tooling :/. |
MFP added a silent captcha so the old package version no longer works (coddingtonbear/python-myfitnesspal#144)
@coddingtonbear howdy and thanks for all your work on this library, know I've gotten a lot of use out of it. I was pulling data on a scheduler in a pod in K8S so the browser option wasn't going to work for me, so I spun up something w/ a headless webdriver that works fairly well. Obviously comes w/ all the standard pitfalls of any web automation like this, but it gets past the reCAPTCHA and logs in consistently. Not sure if something like this would really belong in the library considering the dependency on the webdriver, but figured I would share in case it's useful to others. It supports saving/loading credentials so ultimately should only have to actually login once in a while (think 30 days is the auth token expiration or something like that). Here's an example of how I'm using it: from webdriver_auth import SeleniumAuth
import myfitnesspal
auth = SeleniumAuth(
username='username',
password='password',
webdriver_path=r'C:\chromedriver.exe',
creds_filepath='creds.json',
use_stored_credentials=True)
def get_mfp_client():
if auth.login(max_wait_time=30):
return myfitnesspal.Client(
cookiejar=auth.cookiejar)
mfp_client = get_mfp_client()
meals = mfp_client.get_meals() Auth client: import json
import logging
import os
from typing import Tuple
import requests
from requests.cookies import RequestsCookieJar
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from seleniumwire import webdriver
logger = logging.getLogger()
def first(items, func):
for item in items:
if func(item):
return item
return None
def any(items, func):
for item in items:
if func(item):
return True
return False
def foreach(items, func):
for item in items:
func(item)
def empty(items):
if items is not None and len(items) > 0:
return True
return False
wire_logger = logging.getLogger(
'seleniumwire.handler').setLevel(level=logging.WARNING)
class SeleniumAuthConstants:
LOGIN_URL = 'https://www.myfitnesspal.com/account/login'
LOGGER_NAME = 'seleniumauth'
TERMINAL_REQUEST = 'featured_blog_posts'
class Element:
EMAIL_NAME = 'email'
PASSWORD_NAME = 'password'
SUBMIT_XPATH = "//button[@type='submit']"
TOOLBAR_XPATH = "//span[contains(@class,'MuiContainer-maxWidthLg')]"
class SeleniumAuth:
@property
def cookies(
self
) -> list[dict]:
'''
The raw cookie data returned from the
webdriver requests or stored credentials
'''
return self.__cookies
@property
def cookiejar(
self
) -> RequestsCookieJar:
'''
A `RequestsCookieJar` with the cookies captured
from the webddriver or from loaded credentials.
'''
return self.__cookiejar
def __init__(
self,
username: str,
password: str,
webdriver_path: str,
creds_filepath: str = None,
use_stored_credentials: bool = True
):
self.__username = username
self.__password = password
self.__webdriver_path = webdriver_path
self.__creds_filepath = creds_filepath
self.__use_stored_credentials = use_stored_credentials
self.__driver: webdriver.Chrome = None
self.__cookies = list()
self.__cookiejar = None
def __initialize_driver(self) -> webdriver.Chrome:
''' Create and configure the Chrome webdriver '''
options = webdriver.ChromeOptions()
options.headless = True
driver = webdriver.Chrome(
self.__webdriver_path,
options=options)
driver.set_window_size(1920, 1080)
driver.maximize_window()
self.__driver = driver
def __navigate(
self
) -> None:
'''
Navigate to service login page
'''
self.__driver.get(
url=SeleniumAuthConstants.LOGIN_URL)
def __get_login_input_elements(
self
) -> Tuple[WebElement, WebElement]:
''' Get the login username and password elements '''
logger.info(f'Waiting for email element')
email = self.__get_element(
_type='name',
selector=Element.EMAIL_NAME)
logger.info(f'Waiting for password elemenet')
password = self.__get_element(
_type='name',
selector=Element.PASSWORD_NAME)
return email, password
def __create_cookiejar(
self,
cookies=None
) -> Tuple[RequestsCookieJar, dict]:
'''
Create a cookiejar from the captured driver cookies and
return the created cookie jar and the source dictionary
'''
raw_cookies = cookies or self.__driver.get_cookies()
if self.__driver:
self.__driver.quit()
if not empty(raw_cookies or []):
raise Exception('Failed to capture driver cookies')
# Convert the dictionary of cookie values returned from
# the driver or loaded from stored credentials into a
# requests cookiejar (derived from http.CookieJar used
# by MFP client)
logger.info(f'Creating requests session cookiejar')
session = requests.session()
foreach(raw_cookies, lambda cookie: session.cookies.set(
name=cookie.get('name'),
value=cookie.get('value'))
)
self.__cookiejar = session.cookies
self.__cookies = raw_cookies
if (self.__use_stored_credentials
and self.__creds_filepath is not None):
self.__save_credentials()
def __get_element(
self,
_type: str,
selector: str,
max_wait_time=10
) -> WebElement:
'''
Explicitly wait for an element to become available
on the page and return the element
`type`: element type
`value`: element path
`max_wait_time`: maximum time to wait before timeout
'''
locator = (_type, selector)
element = WebDriverWait(
driver=self.__driver,
timeout=max_wait_time).until(
method=EC.presence_of_element_located((
locator)))
return element
def __get_submit(
self
) -> WebElement:
''' Get the login page submit button element '''
logger.info(f'Locating submit element')
submit = self.__get_element(
_type='xpath',
selector=Element.SUBMIT_XPATH)
logger.info(f'Submit element: {submit}')
if isinstance(submit, list):
return submit[0]
return submit
def __get_distinct_request_urls(
self
) -> list[str]:
'''
Get a list of all distinct request URLs from
the webdriver
'''
urls = set([
req.url for req in self.__driver.requests
])
return list(urls)
def __wait_for_login(
self,
max_wait_time: int
) -> None:
'''
Wait for the login to complete indicated by
the toolbar element becoming available on the
page before exiting. Otherwise seleniumwire
may not capture the full network trace for the
login and the required cookies
`max_wait_time` : maximum time to wait on the
toolbar element, this may
'''
# Use the XHR request that fetches the spash page's blog
# post list as an idicator that we've captured all the
# auth-related requests. Otherwise the driver may exit
# before this happens
def terminal_request_receieved(driver: webdriver.Chrome):
distinct_urls = self.__get_distinct_request_urls()
if any(items=distinct_urls,
func=lambda url: SeleniumAuthConstants.TERMINAL_REQUEST in url):
logger.info(f'Terminal request received')
return True
WebDriverWait(
driver=self.__driver,
timeout=max_wait_time).until(
method=terminal_request_receieved)
logger.info(f'Terminal login request receieved')
def __login(
self,
max_wait_time: int
) -> RequestsCookieJar:
'''
Internal login routine, to be wrapped in try/except
to capture and return specific exceptions concerning
timeouts
'''
# Attempt to load stored credentials and use those if
# available
if (self.__use_stored_credentials
and os.path.exists(self.__creds_filepath)):
logger.info('Using stored credentials')
return self.load_credentials()
logger.info('No stored credentials found, using webdriver')
self.__initialize_driver()
self.__navigate()
usr, pwd = self.__get_login_input_elements()
usr.send_keys(self.__username)
pwd.send_keys(self.__password)
submit = self.__get_submit()
submit.click()
self.__wait_for_login(
max_wait_time=max_wait_time)
self.__create_cookiejar()
def __save_credentials(
self
) -> None:
'''
Export and save a JSON file containing the raw
cookie data
'''
with open(self.__creds_filepath, 'w') as file:
file.write(json.dumps(self.__cookies))
def load_credentials(
self
) -> bool:
'''
Load credentials from the provided filepath if
the file exists
'''
if not os.path.exists(self.__creds_filepath):
raise Exception(
f"Could not find saved credentials at path: '{self.__creds_filepath}'")
with open(self.__creds_filepath, 'r') as file:
cookies = json.loads(file.read())
self.__cookiejar = self.__create_cookiejar(
cookies=cookies)
self.__cookies = cookies
return True
def login(
self,
max_wait_time: int,
) -> bool:
'''
If `use_stored_credentials` is enabled:
Attempt to fetch credentials from credential filepath and
if the file does not exist then login using the webdriver
and store the captured cookies
If `use_stored_credentials` is not enabled::
Login using a headless webdriver then capture driver cookies
in a `requests` session cookiejar
`max_wait_time` : max time to wait for the driver to
return a response
'''
logger.info('Starting webdriver login')
try:
self.__login(
max_wait_time=max_wait_time)
return True
except TimeoutException as ex:
self.__driver.save_screenshot('timeout.png')
logger.exception(
msg='Webdriver timeout exception')
raise Exception(
f'Login timed out, consider extending maximum wait times') |
Hi @danleonard-nj, I am trying your solution, but I am having problems with the Cookies message "Your Choices Regarding Cookies on this Site" and I neet to click in the agree button, but I cannot achieve that. How did you pass that popup? It is the one with the class="truste_overlay"
|
@goliath888 I was able to get past that by maximizing the webdriver and setting a larger window size. |
Tried with driver.set_window_size(2560, 1440) and driver.set_window_size(3840, 2160), but the problem remains. I think that the div only goes away If you click on the agree button. |
@goliath888 any luck here? I'm running into the same "is not clickable" exception. @danleonard-nj what was the exact window size you selected? |
I solved this. MFP added a cookie iframe at the bottom. Simply had to modify the driver to switch to it, click it, and proceed with the rest of the code. Adding here if its helpful. `import json import requests logger = logging.getLogger() def first(items, func): def any(items, func): def foreach(items, func): def empty(items): wire_logger = logging.getLogger( class SeleniumAuthConstants: class Element: class SeleniumAuth:
` |
@schambersnh do you have a more updated version of this that works? couldnt get it to work, failing at multiple parts |
Morning,
I wonder if something API-side has changed? Up to last night I was connecting fine, now this:
Any thoughts?
The text was updated successfully, but these errors were encountered: