Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web workers #1993

Merged
merged 27 commits into from
Apr 1, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6481082
Web workers pass 1
afinch7 Mar 23, 2019
368bbfd
Make linter happy
afinch7 Mar 23, 2019
d28a44d
fmt
afinch7 Mar 23, 2019
84e56d8
Refactored the way worker polling is scheduled and errors are handled.
afinch7 Mar 25, 2019
df12a00
Some cleanup and handle worker close on host end.
afinch7 Mar 26, 2019
99f6db1
Merge branch 'master' into web_workers
afinch7 Mar 26, 2019
a1b048c
Share the worker future as a Shared
afinch7 Mar 26, 2019
4f52d6b
Merge branch 'web_workers' of https://github.com/afinch7/deno into we…
afinch7 Mar 26, 2019
8d30513
Fixed some merge issues and added a worker test
afinch7 Mar 26, 2019
ec2149b
compile_sync returns errors now
afinch7 Mar 27, 2019
2103d32
Refactored compile_sync again
afinch7 Mar 27, 2019
1a6be92
fixed bugs and moved compiler error exit back to compile_sync
afinch7 Mar 27, 2019
4eb80ab
lots of comments for compile_sync
afinch7 Mar 28, 2019
1e2c184
refactored compile_sync again to use a future that is less blocking.
afinch7 Mar 28, 2019
cfd3289
Merge branch 'master' into web_workers
afinch7 Mar 28, 2019
e245db1
enable debug to find this problem
afinch7 Mar 28, 2019
58de310
remove old comments
afinch7 Mar 28, 2019
e6a3bab
maybe same tokio runtime for all compiler tasks?
afinch7 Mar 28, 2019
fa72bc9
remove debug from tests
afinch7 Mar 28, 2019
32d7ad4
worker ts lib
afinch7 Mar 30, 2019
a9c8271
requested change: workers test assert
afinch7 Mar 31, 2019
5b101a8
renamed WebWorkerBehavior to UserWorkerBehaivor and moved to workers.rs
afinch7 Mar 31, 2019
8388fa2
Merge remote-tracking branch 'upstream/master' into web_workers
afinch7 Mar 31, 2019
53fcd75
fixed remaining merge issues
afinch7 Mar 31, 2019
6e173b8
remove thread spawn from worker tests
afinch7 Mar 31, 2019
5815927
removed worker specific snapshot/bundle
afinch7 Apr 1, 2019
b0881d4
forgot to remove the startup data function
afinch7 Apr 1, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed bugs and moved compiler error exit back to compile_sync
  • Loading branch information
afinch7 committed Mar 27, 2019
commit 1a6be9206d8f0ede6b454d8915d2582524f3e907
38 changes: 26 additions & 12 deletions cli/compiler.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Copyright 2018-2019 the Deno authors. All rights reserved. MIT license.
use core::ops::Deref;
use crate::isolate_state::*;
use crate::js_errors::JSErrorColor;
use crate::msg;
use crate::ops;
use crate::resources;
Expand All @@ -22,6 +23,7 @@ use serde_json;
use std::str;
use std::sync::Arc;
use std::sync::Mutex;
use tokio::runtime::Runtime;

pub type CompilerResult = Result<ModuleMetaData, JSError>;
type CompilerInnerResult = Result<ModuleMetaData, Option<JSError>>;
Expand All @@ -35,6 +37,7 @@ struct CompilerShared {

lazy_static! {
static ref C_SHARED: Mutex<Option<CompilerShared>> = Mutex::new(None);
static ref C_RUNTIME: Mutex<Runtime> = Mutex::new(Runtime::new().unwrap());
Copy link
Member

@ry ry Mar 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runtime is different than the tokio runtime? (defined by crate::tokio_util::run)
Ideally we could get all isolates using the same runtime.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that, but this leads to tokio waiting indefinitely for the compiler worker to exit. The compiler worker only has two ways to exit right now: calling workerClose as the worker or throwing a uncaught error. Maybe work on a better way to terminate the compiler worker later?

}

pub struct CompilerBehavior {
Expand Down Expand Up @@ -131,8 +134,11 @@ fn lazy_start(parent_state: Arc<IsolateState>) -> CompilerShared {
// results from worker future
let (err_sender, err_receiver) =
oneshot::channel::<CompilerInnerResult>();
tokio::spawn(lazy(move || {
worker.then(|result| -> Result<(), ()> {
let mut runtime = C_RUNTIME.lock().unwrap();
runtime.spawn(lazy(move || {
let resource = worker.resource.clone();
worker.then(move |result| -> Result<(), ()> {
resource.close();
match result {
Err(err) => err_sender.send(Err(Some(err))).unwrap(),
_ => err_sender.send(Err(None)).unwrap(),
Expand All @@ -153,6 +159,11 @@ fn lazy_start(parent_state: Arc<IsolateState>) -> CompilerShared {
}).clone()
}

fn show_compiler_error(err: JSError) -> ModuleMetaData {
eprintln!("{}", JSErrorColor(&err).to_string());
std::process::exit(1);
}

fn req(specifier: &str, referrer: &str) -> Buf {
json!({
"specifier": specifier,
Expand All @@ -167,7 +178,7 @@ pub fn compile_sync(
specifier: &str,
referrer: &str,
module_meta_data: &ModuleMetaData,
) -> Result<ModuleMetaData, JSError> {
) -> ModuleMetaData {
let req_msg = req(specifier, referrer);

let shared = lazy_start(parent_state);
Expand All @@ -178,19 +189,23 @@ pub fn compile_sync(
let compiler_rid = shared.rid.clone();
let module_meta_data_ = module_meta_data.clone();

tokio::spawn(lazy(move || {
tokio::spawn(lazy(move || -> Result<(), ()> {
debug!("Running rust part of compile_sync");
let send_future = resources::post_message_to_worker(compiler_rid, req_msg);
debug!("Waiting on a message to be sent to compiler worker");
match send_future.wait() {
Ok(_) => {} // Do nothing if result is ok else send back None error
_ => return Ok(local_sender.send(Err(None)).unwrap()),
};
debug!("Message sent to compiler worker");

let recv_future = resources::get_message_from_worker(compiler_rid);
debug!("Waiting on a message from the compiler worker");
let res_msg = match recv_future.wait() {
Ok(Some(v)) => v,
_ => return Ok(local_sender.send(Err(None)).unwrap()),
};
debug!("Received result message from the compiler worker");

let res_json = std::str::from_utf8(&res_msg).unwrap();
Ok(
Expand Down Expand Up @@ -229,13 +244,14 @@ pub fn compile_sync(
// local_receiver finished first
let mut rest_mut = rest;
match ((*result.deref()).clone(), i) {
(Ok(v), 1) => Ok(v),
(Err(Some(err)), _) => Err(err),
(Ok(v), 1) => v,
(Err(Some(err)), _) => show_compiler_error(err),
(Err(None), 1) => {
debug!("Compiler local exited with None error first!");
match (*rest_mut.remove(0).wait().unwrap().deref()).clone() {
Ok(v) => Ok(v),
Err(Some(err)) => Err(err),
Err(None) => panic!("Odd compiler worker result!"),
Ok(v) => v,
Err(Some(err)) => show_compiler_error(err),
Err(None) => panic!("Compiler exit for an unknown reason!"),
}
}
(_, i) => panic!("Odd compiler result for future {}!", i),
Expand Down Expand Up @@ -270,14 +286,12 @@ mod tests {
maybe_source_map: None,
};

let out_result = compile_sync(
out = compile_sync(
Arc::new(IsolateState::mock()),
specifier,
&referrer,
&mut out,
);
assert!(out_result.is_ok());
out = out_result.unwrap();
assert!(
out
.maybe_output_code
Expand Down
9 changes: 1 addition & 8 deletions cli/isolate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ use crate::errors::RustOrJsError;
use crate::isolate_state::IsolateState;
use crate::isolate_state::IsolateStateContainer;
use crate::js_errors;
use crate::js_errors::JSErrorColor;
use crate::msg;
use crate::tokio_util;
use deno_core;
Expand Down Expand Up @@ -190,13 +189,7 @@ fn fetch_module_meta_data_and_maybe_compile_async(
|| state_.flags.recompile
{
debug!(">>>>> compile_sync START");
out = match compile_sync(state_.clone(), &specifier, &referrer, &out) {
Ok(v) => v,
Err(err) => {
eprintln!("{}", JSErrorColor(&err).to_string());
std::process::exit(1);
}
};
out = compile_sync(state_.clone(), &specifier, &referrer, &out);
debug!(">>>>> compile_sync END");
state_.dir.code_cache(&out)?;
}
Expand Down