Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with Dataflow: create() missing 1 required positional argument: 'drivername' #39

Open
manupeco opened this issue Apr 30, 2021 · 0 comments

Comments

@manupeco
Copy link

manupeco commented Apr 30, 2021

versions

beam-nuggets==0.18.0
pg8000==1.16.5
SQLAlchemy==1.4.0

configuration in the code

source_config = relational_db.SourceConfiguration(
        drivername='postgresql+pg8000',
        host=known_args.host,
        port=known_args.port,
        username=known_args.username,
        password=known_args.password,
        database='geospatial',
 )
   file = (p | 'States' >> beam.Create(['Vermont'])
            | 'Links' >> beam.ParDo(Links())
            | 'Download from Github to GCS' >> beam.ParDo(DownloadAsItIs())
            | 'Read the file from GCS' >> beam.io.ReadAllFromText()
            | 'Transform lines' >> beam.Map(lambda s: data_ingestion.parse_line(s))
            | 'Filter lines' >> beam.Filter(is_validGeometry)
            | 'Transform for Postgres' >> beam.Map(lambda s: postgres_ingestion.parse_line(s))
            | 'Writing to DB' >> relational_db.Write(
        source_config=source_config,
        table_config=table_config)
    )

Error

Error message from worker: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 279, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/url.py", line 100, in __new__ return URL.create(*arg, **kw) TypeError: create() missing 1 required positional argument: 'drivername' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 649, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "apache_beam/runners/worker/operations.py", line 695, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 697, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 698, in apache_beam.runners.worker.operations.DoOperation.start File "apache_beam/runners/worker/operations.py", line 308, in apache_beam.runners.worker.operations.Operation.start File "apache_beam/runners/worker/operations.py", line 314, in apache_beam.runners.worker.operations.Operation.start File "apache_beam/runners/worker/operations.py", line 644, in apache_beam.runners.worker.operations.DoOperation.setup File "apache_beam/runners/worker/operations.py", line 645, in apache_beam.runners.worker.operations.DoOperation.setup File "apache_beam/runners/worker/operations.py", line 289, in apache_beam.runners.worker.operations.Operation.setup File "apache_beam/runners/worker/operations.py", line 303, in apache_beam.runners.worker.operations.Operation.setup File "apache_beam/runners/worker/operations.py", line 779, in apache_beam.runners.worker.operations.DoOperation._get_runtime_performance_hints File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 283, in loads return dill.loads(s) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 275, in loads return load(file, ignore, **kwds) File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load return Unpickler(file, ignore=ignore, **kwds).load() File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/url.py", line 100, in __new__ return URL.create(*arg, **kw) TypeError: create() missing 1 required positional argument: 'drivername'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant