Add tokens_xptr object

[koheiw]

For #2208, I created a tokens_xptr object, a sub-class of tokens, whose underlying object is an external pointer instead of a list. All the underlying C++ code for tokens_* functions are also upgraded to accept the new object.

Users can switch between tokens and tokens_xptr objects via as.tokens() and as.tokens_xptr() whenever they want to. If they apply tokens_* functions to old objects, as.tokens_xptr() is applied to the input and as.tokens() to the output internally, so they do not even notice the changes in underlying functions.

Apart from the detail of the implementation, remaining design issues are when the tokens_xptr methods should return a deep copy of the input objects. Currently, a deep copy is returned after subsetting (e.g. [], tokens_subset() and tokens_sample()).

We can create tokens_xptr.corpus(), or tokens.corpus(xptr = TRUE), to get tokens_xptr objects straight away. tokens.tokens() is already a wrapper around tokens.tokens_xptr().

We should also add env to functions that perform NSE just like I did in tokens_group() and tokens_sample(). Otherwise, developers find it difficult to nest them in their functions.

For now, I kept the old code for dfm.tokens() and tokens_recompile.tokens() to compare results.

[kbenoit]

I like the addition of the xptr logical in tokens(). One thing about naming, it suggests we should name this new object tokensx, since our grammar is objectclass_verb, so that tokens_xptr suggests a function that takes in and returns a tokens object.

So the constructor and object class could just be called tokensx() and tokensx, although I like tokens(x, xptr = TRUE) even better.

What do you think?

[koheiw]

I like the addition of the xptr logical in tokens(). One thing about naming, it suggests we should name this new object tokensx, since our grammar is objectclass_verb, so that tokens_xptr suggests a function that takes in and returns a tokens object.

tokens_xptr is a sub-class of tokens, so there is no problem for tokens() to return the object.

> class(tokens("aa", xptr = TRUE))
[1] "tokens_xptr" "tokens"