-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Bidirectional for RNN layer. #687
Comments
I think Bidirectional RNNs can be implemented as: Parallel(
<fuse_op>, # User provided
Recurrence(<rnn cell>; return_sequence=true),
Chain(
ReverseSequence(), # Needs to be implemented
Recurrence(<rnn cell>; return_sequence=true)
)
) It would be nice to have a I will have to check what other frameworks do but I am not sure if the two direction parameters are shared between the layers.
For sure! |
Hi! I would like to ask what role does |
So I think So it should like: x = [1 2; 3 4]
model = ReverseSequence(1)
y,st_new = model(x,ps,st)
# y =
[3, 4]
[1, 2]
model = ReverseSequence(2)
y,st_new = model(x,ps,st)
# y =
[2, 4]
[1, 3] Is my understanding correct? Are there any suggestions for high-performance implementation? |
I think ReverseSequence should be added to the output of the backward RNN layer, so that the output of the first time step of the forward RNN corresponds to the output of the last time step of the backward RNN:
And what |
Tensorflow allows you to choose fuse_ops that's why I want it to be user choice. For broadcasting just wrap the user input (default to vcat) with |
keras provides some default implementations ({"sum", "mul", "concat", "ave", None}), should we provide them too? Or provide a default implementation of concat and a custom |
I can't find |
Ok.. I figure out.. bvcat(a,b) = vcat.(a,b)
model = Parallel(bvcat,
Recurrence(GRUCell(3=>2),return_sequence = true),
Recurrence(GRUCell(3=>2),return_sequence = true)
) |
I have roughly figured out how to build a bidirectional RNN using Parallel layers. I will start implementing it once #698 is merged. |
This is basically what you wrote, just do |
Which one is better and easier to implement in Lux.jl? I'm willing to try to implement it and open a pr, but may need some guidance.
The text was updated successfully, but these errors were encountered: