Fixed issue #894 and #858 (aws_models issues)

[drawal1]

EnsembledProgram can now be loaded/saved
Inference of programs (forward function) is now concurrent
Added checks for size = 0 or size > number of programs

[okhat]

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

[drawal1]

@okhat @arnavsinghvi11 - This PR now also includes a fix for issue #824. The fix is in teleprompt/random_search.py

[drawal1]

@okhat - any progress on this review? I am continuing to work on intelligent ensemble routing and don't want to muddy this PR with commits related to that work

btw, I have been testing ensemble concurrency and so far it works like a charm

also, I can reverse the commit for issue #824 and submit a separate PR for that, if it would expedite this PR review

[drawal1]

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

I think I understand. I see this code in evaluate.py:

        def wrapped_program(example_idx, example):
            # NOTE: TODO: Won't work if threads create threads!
            thread_stacks = dspy.settings.stack_by_thread

So I need to test and verify the parallelization NOT in inference but as part of another optimization pipeline

[drawal1]

Fixed #894 in dsp/modules/aws_models.py, in class AWSMeta(AWSModel) by moving below functionality from init function:

        max_tokens = query_args.pop("max_tokens", None)
        if max_tokens:
            query_args["max_gen_len"] = max_tokens

to the create_body function

and copying kwargs to base_args by value

[drawal1]

@okhat - I have reverted the parallelization in EnsembledProgram.forward() based on your review and feedback.

The remaining fixes/improvements are targeted and should not have side-effects

[arnavsinghvi11]

why are we skipping over the zero-shot, labeled_fewshot and unshuffled_fewshot cases for all scenarios? I think even in the conditions above when a teacher is set, it is fine to run through these preliminary candidates and have it weighted in the scores.

[drawal1]

fair enough. reverted to previous behavior

[arnavsinghvi11]

can remove this based on decision above

[drawal1]

assert fails Ruff check (S101 Use of assert detected). Lets keep the raise exception here. its good practice

[arnavsinghvi11]

spelling typo

[drawal1]

fixed

[arnavsinghvi11]

what are the purpose of these changes? I think the ruff has been working correctly so far but just curious

[drawal1]

ERA001 because Ruff complains about commented out print() statements. I often find these useful for debugging

Re. S301, ensemble.py load_folder/save_folder use pickle for serialization. Ruff flags pickle as unsafe for load because it could load arbitrary objects. I suppose we could switch to https://jsonpickle.github.io/. If jsonpickle is acceptable, I can make the change and remove the S301.

[drawal1]

@arnavsinghvi11 - what's your opinion on jsonpickle? see my comment above...

[arnavsinghvi11]

Thanks @drawal1 for the PR! left a few comments. The changes to Ensemble look correct to me, but I think the ones for RandomSearch are not needed? or can be handled separately without impacting existing behavior. lmk what you think.

There's also a merge conflict to resolve and feel free to remove extraneous comments from the changes that do not help with following the code. Thanks!

tagging @okhat for reference on Ensemble/RandomSearch again (but I believe the parallelization changes are removed so should be good there).

[drawal1]

@arnavsinghvi11 - I have reverted ALL the complicated changes. Hopefully now this PR is acceptable

[arnavsinghvi11]

Hi @drawal1 , thanks for the changes. can you also revert the changes to BootstrapWithRandomSearch since those changes are not relevant to patching #858 .

[drawal1]

@arnavsinghvi11 - BootstrapWithRandomSearch changes are now reverted

[arnavsinghvi11]

Thanks @drawal1 !