Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when running beat sample Laquila #21

Closed
vic-chou opened this issue Dec 18, 2018 · 5 comments
Closed

Error when running beat sample Laquila #21

vic-chou opened this issue Dec 18, 2018 · 5 comments

Comments

@vic-chou
Copy link

vic-chou commented Dec 18, 2018 •
edited
Loading

Hi, when I run the command "beat sample Laquila", I meet a error as follows
beat sample Laquila
config - INFO All parameter-priors ok!
config - INFO All hyper-parameters ok!
config - INFO All hierarchical-parameters ok!
models - INFO ... Initialising Geometry Optimizer ...

models - INFO Analysing problem ...
models - INFO ---------------------

/usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1341: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result.
out_full[ind] += zi
/usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1344: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result.
out = out_full[ind]
/usr/local/python35/lib/python3.5/site-packages/scipy/signal/signaltools.py:1350: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result.
zf = out_full[ind]
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap!
heart - INFO Number of seismic datasets for wavemap: any_P_0: 35
geodetic - INFO Number of geodetic datasets: 2
geodetic - INFO Number of geodetic data points: 419
geodetic - WARNING Covariance estimation not implemented (yet)! Using imported covariances!
geodetic - INFO Fit residual ramp selected!
models - INFO ... Building model ...

models - INFO Optimization for 3 hyperparameters in total!
seismic - INFO Seismic optimization on:
north_shift, nucleation_y, east_shift, width, nucleation_x, strike, time, slip, length, dip, rake, depth
seismic - INFO Retrieving seismic data-covariances with structure "non-toeplitz" for any_P_0 ...
seismic - INFO Preparing data of "any_P_0" for optimization
seismic - INFO Initializing synthesizer for "any_P_0"
geodetic - INFO Geodetic optimization on:
north_shift, east_shift, width, strike, slip, length, dip, rake, depth
geodetic - INFO Estimating ramp for each dataset...
geodetic - INFO Initialized 4 hierarchical parameters (ramps).
models - INFO Model building was successful!

models - INFO ... Initiate Sequential Monte Carlo ...
n_chains=1000, tune_interval=10, n_jobs=4, proposal_distribution: MultivariateCauchy,

models - INFO Compilation time: 1598.274327
models.base - INFO ... Starting SMC ...

backend - INFO Reloading existing results ...
backend - INFO Loading multitrace from /test_beat/Laquila/geometry/stage_0
backend - INFO Checking for corrupted files ...
smc - INFO Sample initial stage: ...
smc - INFO Beta: 0.000000 Stage: 0
backend - INFO Loading multitrace from /test_beat/Laquila/geometry/stage_0
smc - INFO Updating Covariances ...
seismic - INFO Updating data-covariances ...
seismic - INFO Retrieving seismic data-covariances with structure "non-toeplitz" for any_P_0 ...
seismic - INFO Not updating seismic velocity model-covariances because number of model variations is too low! < 5
geodetic - INFO Not updating geodetic velocity model-covariances because number of model variations is too low! < 5
sampler - INFO Updating last samples ...
sampler - INFO in /test_beat/Laquila/geometry/trans_stage_0
sampler - INFO Initialising 1000 chain traces ...
sampler - INFO Serial time per sample: 19.111001
sampler - INFO Data to be memory shared:
sampler - INFO No data to be memshared!
sampler - INFO Sampling ...
parallel - INFO Worker timeout after 90 second(s)
parallel - INFO Overseer timeout after 7500 second(s)
parallel - INFO Chunksize: 250
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last): | 0/1 [00:00<?, ?it/s]
File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 903, in call
self.fn() if output_subset is None else
File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/op.py", line 892, in rval
r = p(n, [x[0] for x in i], o)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/theanof.py", line 400, in perform
arrival_times=arrival_times)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/heart.py", line 2574, in seis_synthetics
targets=targets, nprocs=nprocs)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 3146, in process
nthreads=nthreads):
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2692, in process_dynamic
source, target, components, dsource_cache, nthreads)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2989, in base_seismogram
nthreads=nthreads)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 1907, in seismogram
'c', optimization)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 856, in _sum
int(itmin), int(nsamples)))
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 116, in timeout_handler
raise TimeoutException(traceback.format_stack())
beat.parallel.TimeoutException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 279, in _sample
for strace in sampling:
File "/usr/local/python35/lib/python3.5/site-packages/tqdm-4.28.1-py3.5.egg/tqdm/_tqdm.py", line 979, in iter
for obj in iterable:
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 333, in _iter_sample
point, out_list = step.step(point)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/backend.py", line 93, in step
apoint, alist = self.astep(self.bij.map(point))
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/metropolis.py", line 207, in astep
l_new = self.logp_forw(q0)
File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 917, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/python35/lib/python3.5/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/local/python35/lib/python3.5/site-packages/theano/compile/function_module.py", line 903, in call
self.fn() if output_subset is None else
File "/usr/local/python35/lib/python3.5/site-packages/theano/gof/op.py", line 892, in rval
r = p(n, [x[0] for x in i], o)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/theanof.py", line 400, in perform
arrival_times=arrival_times)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/heart.py", line 2574, in seis_synthetics
targets=targets, nprocs=nprocs)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 3146, in process
nthreads=nthreads):
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2692, in process_dynamic
source, target, components, dsource_cache, nthreads)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/seismosizer.py", line 2989, in base_seismogram
nthreads=nthreads)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 1907, in seismogram
'c', optimization)
File "/usr/local/python35/lib/python3.5/site-packages/pyrocko/gf/store.py", line 856, in _sum
int(itmin), int(nsamples)))
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 116, in timeout_handler
raise TimeoutException(traceback.format_stack())
beat.parallel.TimeoutException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/python35/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/local/beat/python35/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 175, in _pay_worker
return overseer(worker.timeout)(worker.run)()
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 123, in wrapped_f
result = func(*args, **kwargs)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 164, in run
return self.function(*self.work)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 291, in _sample
strace.record_buffer()
UnboundLocalError: local variable 'strace' referenced before assignment
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/python35/bin/beat", line 11, in
load_entry_point('beat==1.0rc1', 'console_scripts', 'beat')()
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/apps/beat.py", line 1588, in main
globals()'command_' + command
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/apps/beat.py", line 755, in command_sample
sample(step, problem)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/models/base.py", line 174, in sample
rm_flag=pa.rm_flag)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/smc.py", line 471, in smc_sample
homepath, step, progressbar, model, n_jobs, rm_flag)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 586, in update_last_samples
mtrace = iter_parallel_chains(**sample_args)
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/sampler/base.py", line 483, in iter_parallel_chains
for res in p:
File "/.local/lib/python3.5/site-packages/beat-1.0rc1-py3.5-linux-x86_64.egg/beat/parallel.py", line 252, in paripool
chunksize=chunksize, callback=callback).get(pool_timeout)
File "/usr/local/python35/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
UnboundLocalError: local variable 'strace' referenced before assignment

Can you help me solve this problem? Thank you!
@hvasbath
Copy link
Owner

Hi again! Thanks for reporting and not giving up hope ;) !
Your processes timed out and got killed by the process manager. Do you follow exactly the Tutorial-or did you change some values? May I ask for a list of your system hardware? The forward calculation is very slow in your case- 19s, it should rather be around 1-2s .
You may also try turning off processbars-please set the processbar flag to false in the SMC config.
Also please try running:
THEANO_FLAGS=optimizer=fast_compile beat sample Laquila
(instead of beat sample Laquila) will update the FAQ with this respect.

@vic-chou
Copy link
Author

vic-chou commented Dec 19, 2018 •
edited
Loading

I list the system hardware as follows:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Stepping: 4
CPU MHz: 2699.939
BogoMIPS: 5399.29
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
I runing the THEANO_FLAGS=optimizer=fast_compile beat sample Laquila, and now it can runing normally, but I don't know whether it will crash halfway. I will report the results when it finish running.

@hvasbath
Copy link
Owner

hvasbath commented Jan 5, 2019

Added to FAQ here: https://hvasbath.github.io/beat/faq.html
please reopen if issue persists

@hvasbath hvasbath closed this as completed Jan 5, 2019
@nanfang0422
Copy link

nanfang0422 commented Jan 30, 2019 •
edited by hvasbath
Loading

Deleted and transferred to new issue.

@hvasbath
Copy link
Owner

This is the new issue:
#32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hvasbath @vic-chou @nanfang0422