Plan for removing extras

[ViralBShah]

We started using extras as a place to keep all sorts of code that was experimental, or domain specific. Now that we have a packaging system, it is time to remove extras. Some of the stuff should go into Base, and the rest into packages. Here's my first cut at doing this:

• Rmath.jl: Do we need this for Distributions? I would prefer to continue bundling libRmath, but I believe Rmath.jl is no longer needed. ( @johnmyleswhite ?)
• arpack.jl: Move to Base
• cpp.jl: CPPCall package
• dcm_dict.jl: A Dicom package
• dicom.jl: Dicom package (@JeffBezanson ?)
• enum.jl: enum support in base #3080
• gzip.jl: Compress package (@kmsquire ?)
• image.jl: Image package ( @timholy ? )
• julia_web.jl: WebRepl package (@loladiro ?)
• julia_web_base.jl: WebRepl package
• lru.jl: LRU package
• memoize.jl: Memoize package
• nearequal.jl: Include anything we want to retain in Base/test.jl
• ode.jl: ODE package (@pao ?)
• pkg.jl: Could be deleted now (@StefanKarpinski ?)
• poly.jl: Polynomials package (@vtjnash ?)
• strpack.jl: Already in a package. Delete? (@pao ?)
• suitesparse.jl: Move to Base (@dmbates ?)
• suitesparse_h.jl: Move to Base
• test.jl: Keep what we need in Base/test.jl
• time.jl: Move to base
• units.jl: Units package
• webrepl_msgtypes_h.jl: WebRepl package
• zlib.jl: Compress package
• zlib_h.jl: Compress package

Once all this is complete, extras/ can be removed from the repository, and from all the various load paths.

This issue should not be closed until the corresponding documentation is included in the packages.
20db526#commitcomment-2821097

[aviks]

Good plan. A couple of comments

lru.jl : I have a slight preference for lru to go into base, it seems widely useful. One of my plans was to use an lru cache, for example, to cache GMP structures for reuse.

time.jl: is, I think incomplete functionality, and shouldn't go into base at this time. It risks baking in. I would suggest moving it to @nolta 's Calendar package right now, and that whole package move to base eventually. Calendar seems to have the most momentum for date/time functionality.

[johnmyleswhite]

I think @dmbates needs the final say on Rmath.jl. I believe, but am not sure, that we only need libRmath, not Rmath.jl.

[pao]

strpack.jl removal is blocked on #2315.

ode.jl needs an owner who's smarter than me. The codes I contributed don't quite work right, and we need more codes with adaptive timesteps.

[dmbates]

Regarding Rmath.jl I have an example in a blog posting that uses it so users might encounter some of the functions. However, the performance is poor relative to the code in Distributions.jl so it may be best to simply remove it and respond to questions about it saying "use the Distributions package instead". I really don't know if others are using it.

[ViralBShah]

@dmbates Perhaps we can update your blog post examples with Distributions? At the very least, we can add an update there pointing to the Distributions package.

[ViralBShah]

In all cases, I would prefer if the original authors take ownership of the new repositories and add them as packages. I will make repositories for the rest of them under JuliaLang.

[timholy]

I'll try to get to Units.jl this weekend.

[StefanKarpinski]

I think time should be merged into Units. Will update the plan to reflect this.

[ihnorton]

I'll volunteer to package dicom (I have a lot of test data) and merge the image stuff into Image.jl w/ @JeffBezanson and @timholy consent/guidance.

[JeffBezanson]

That would be great. I'm willing to help maintain dicom.jl as well if you'd like to give me access.

[dcjones]

I have a more efficient wrapper to zlib in Codecs.jl. (It resizes as needed, rather than attempting decompression and starting over with a bigger array if it runs out of space.)

I will merge whatever is missing, which is trivial things like setting the compression level and strategy, then we could deprecate that.

[StefanKarpinski]

Excellent. Thanks for doing that, @dcjones. I think we can probably just remove extras/zlib.jl altogether rather than deprecating it.

[timholy]

Jeff and Isaiah, you both have push access to Image.jl

[timholy]

...and Stefan, when there's actually a Units.jl in existence, I'll do the same for you so you can merge your time stuff in.

[dmbates]

@ViralBShah I am continuing to develop extras/suitesparse.jl on the db/suitesparse branch. I think it is close to the point where it could be incorporated into the master branch. A lot of the changes create Julia methods corresponding to CHOLMOD functions, which I find quite convenient. However, most users will only want the basics of factorizing and solving systems so we may want to create modules for UMFPACK and CHOLMOD, similar to those for BLAS and LAPACK, with only the most important functionality exposed with the idea that advanced users will use fully qualified names for advanced functions. I think UMFPACK could be in a separate module because it is not tied to CHOLMOD structures the way that SPQR is.

[kmsquire]

As requested, I'll package up gzip (which, since it focuses on a file format, is probably fine as separate from zlib).

[ViralBShah]

@dmbates I am aware of your db/suitesparse branch, and will wait for that to be merged into master.

Also, in general, I do not think that anything in extras needs deprecation, since much of it was never documented.

[vtjnash]

Since I made ode depend upon poly, I can package both. Should this live under JuliaLang or my personal tag (vtjnash)?

[ViralBShah]

@vtjnash Go with your personal tag for now. Neither of those packages has any active development, and I do not expect those packages to have a large number of contributors in any case.

[kmsquire]

GZip.jl is now a package (https://github.com/kmsquire/GZip.jl) with docs (https://gzipjl.readthedocs.org/en/latest/) (and it's listed in METADATA).

I know at least @dmbates and @carlobaldassi are using extras/gzip.jl. It seems to work for me, but can you test the package to make sure everything still works?

You can also now use GZip.open() instead of gzopen() now; open isn't exported, so you need the GZip qualifier. I like this better personally, but gzopen is still available.

After I hear back that it works for others, I'll announce this on julia-dev and julia-users lists and wait a few days before removing the code from extras.

[kmsquire]

Oh, and @ViralBShah, I chose to go with GZip as the package name, rather than the more encompassing Compress. I think that Compress could be a metapackage eventually, if necessary, but I don't feel the need to control all future julia compression packages.

I also don't have write access to this issue, so if you could check off gzip, that would be great.

[kmsquire]

I also fixed a problem with memoize that I haven't submitted, and created a least-fixed-point iterator version, so I can package that up as well.

[ViralBShah]

Great. I checked off Gzip.

[pao]

gzip.jl is going to run into the same problem as strpack.jl--there are packages using it, so the packages will need to do something for their compatible-with-0.1 branch and something different for their compatible-with-0.2+ mainline, for which we really need #2315. @StefanKarpinski, I think I may have already asked this, but is there a plan to do this soon?

[ViralBShah]

#2315 is the only real issue at this point that is preventing us from announcing 0.1 on the julia blog. We need to address it and give package managers some time to update their packages and tie them to 0.1.

[vtjnash]

I checked off ODE and Polynomial. They can be deleted from extras (and test) now.

[ihnorton]

dicom can be checked off and removed: https://github.com/JuliaLang/METADATA.jl/tree/master/DICOM

[aviks]

@kmsquire If you add the url to the docs into the github project details, it will be linked from the package listing page. See StrPack or Gadfly for examples.

[dmbates]

@kmsquire Is there a consensus on what to do about old dependencies on, say, extras/gzip.jl for the 0.1 branch as opposed to the Gzip package for 0.1+? I use gzopen in the DataFrames package which itself is used by many other packages. If I switch to Gzip.open() do we need to tag the DataFrames package for 0.1 before doing so?

[kmsquire]

@aviks, sorry, I'm not quite following you. I have the GZip.jl URL for the ReadTheDocs at the end of the README.md for GZip.jl, and I can't see anywhere else StrPack or Gadfly have a link. What am I missing?

@dmbates, I'm not aware of any consensus.

Generally, #2315 needs to be addressed, but this doesn't directly affect the current version of GZip.jl (and probably won't any time soon), as it is compatible with 0.1, even with the version in extras. So I think you would be fine just depending on the new package.

(I'm recompiling julia 0.1 right now to make sure the GZip.jl module works, but since I just copied the version in extras with minimal changes, there shouldn't be any issues.)

[ViralBShah]

@timholy I saw a package for image processing. Should we remove image.jl here?

[timholy]

Thanks for the reminder. Done.

I notice that @StefanKarpinski is very actively developing pkg2 inside extras/, so we may want to keep extras around a while longer (depending on how that proceeds relative to the next release).

[ViralBShah]

@dcjones I am planning to remove zlib.jl. We do have some good zlib tests. Would you be able to incorporate them in Codecs?

Perhaps it would be better to rename Codecs.jl to Zlib.jl. I guess one can expect having Base64 encoding / decoding in a compression library.

[kmsquire]

Hi Viral, @dcjones already created a separate Zlib.jl package.

https://github.com/dcjones/Zlib.jl

Kevin

On Wednesday, April 24, 2013, Viral B. Shah wrote:

@dcjones https://github.com/dcjones I am planning to remove zlib.jl. We
do have some good zlib tests. Would you be able to incorporate them in
Codecs?

Perhaps it would be better to rename Codecs.jl to Zlib.jl. I guess one
can expect having Base64 encoding / decoding in a compression library.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2422#issuecomment-16925567
.

[ViralBShah]

Thanks. I just removed it from extras in 0d5af2c

[tomasaschan]

Is there a plan for what's going to happen to nearequal.jl? If it goes with test code, should it go in /test or with a test package such as FactCheck?

I'm asking because I'm preparing some modification suggestions for FactCheck and ODE, and I'd like to use this code there. But it would be nice to know from where it should be imported...

[ViralBShah]

@tlycken If possible, could you submit a pull request merging the useful parts of nearequal.jl into base/test.jl? The APIs may need revising, and a pull request will propel us in the right direction.

[tomasaschan]

@ViralBShah Sure, will do.

But the more I look at the functions in nearequal.jl, I feel they rather belong in base/floatfunc.jl or something. Basically, they're defining a substitute for the == operator that takes machine epsilon (or other, user-specified tolerance) into account, so it isn't necessarily associated to automated testing. Is base/test.jl still a good place for it?

[ViralBShah]

Now that you mention it, they do seem to be more appropriate for floatfuncs.jl. Could you add them to floatfuncs.jl and put the documentation in the usual place in doc?

Looking at the names, I actually feel they are fine.

[JeffBezanson]

I agree this can go in Base.

[StefanKarpinski]

Can we come up with a better name than nearequal? I've never cared for that. Maybe isapprox?

[ViralBShah]

nearequal.jl is just the filename. The functions in it are all called isapprox, isapproxn, and isequaln. Also, since these go into floatfuncs.jl, the filename will become irrelevant.

[StefanKarpinski]

Ah, ok.

[tomasaschan]

@ViralBShah @JeffBezanson @StefanKarpinski I'm working on putting this together, and will send you a pull request when I have something to show for it =)

[simonster]

I just finished tweaking memoize.jl to play well with keyword/default arguments. @kmsquire, do you want to make a package for it, or should I?

[kmsquire]

Hi @simonster, please feel free to make a package.

https://gist.github.com/kmsquire/5591523 includes a few modifications:

1. adding a random string to the "_unmemoized" function, so that different functions with the same name could be memoized separately
2. an unmemoize macro, which allows (or might be required) for functions with memoized versions to have unmemoized versions
3. a memofixed macro, which memoizes the fixed point of a self-recursive function

Feel free to include these, modifying as needed, or I can submit a pull request after you create the package.

[kmsquire]

Actually, I remember now. The unmemoize macro is needed so that mutually recursive functions with the same name can call one another, while allowing for some of them to be unmemoized.

[simonster]

@kmsquire, I made a Memoize package (https://github.com/simonster/Memoize.jl), taking a slightly different route to creating the cache so that it can hopefully be freed by GC if a file is reloaded. I have to admit I'm not entirely sure what why @unmemoize is useful, or what @memofixed is doing and how I should be testing it. I gave you commit access to the repository so you can make changes as you see fit.

It also occurred to me that type inference won't work for @memoize. I'll post more about this on julia-users.

[kmsquire]

Thanks. I'll look at your updates and post an example of where both @unmemoize and @memofixed are useful, and add them if that makes sense.

[ViralBShah]

Can we remove memoize.jl from extras? Does the Memoize package include lru.jl too?

In Makefile.jl:19, I would also like to remove extras from being installed. It also needs to be removed from being added to the search paths and such.

Does anyone still depend on anything in extras - @StefanKarpinski I see some pkg2 stuff.

I say that the remaining files can be safely removed - time.jl, test.jl

[pao]

Does LRU have other users? StrPack.jl doesn't use it anymore, but it is a generic structure and not specific to memoize. Maybe roll it into DataStructures (possibly built on its' deque?) @lindahua?

[JeffBezanson]

Maybe move time.jl to examples?