Skip to content

andrewjpage/crawl2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

                                    +DMMMMMMMMMMMMMMMMMMMMMMMM$,                                    
                               .8MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM?..                              
                            ?MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN.                            
                        .?MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN.                         
                       NMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM?                       
                    ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMO                     
                  ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMO.                  
                 MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM?.                
              ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM8.              
             ZMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.             
            MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM?            
          ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMD.          
         =MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM          
        IMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM         
       +MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.       
      ~MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM       
     7MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM      
    .MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMN.    
    MMMD.                                                                                  ,MMM?    
   ZMMMMM                                                                                 ?MMMMM    
   MMMMMMMMMMMMMMMMMMM.  .MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM.  ZMMMMMMMMMMMMMMMMMM8   
  DMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMM.  
  MMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMD  
 $MMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMM  
 MMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMM$ 
~MMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM 
OMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM 
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM:
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM7
MMMMMMMMMMMMMMMMM.....    ................................................   .....MMMMMMMMMMMMMMMMMZ
MMMMMMMMMMMMMMMMM.                                                                MMMMMMMMMMMMMMMMMD
MMMMMMMMMMMMMMMMM?++++   .+++++++++++++++++++++++++++=:,.         .,=+?+++   ~++++MMMMMMMMMMMMMMMMMM
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMN:           ~DNMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMMD
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMM?            MMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMMZ
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMM.           .MMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM7
MMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMN$~   .     ~NMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM:
OMMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMM$      .,8MMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM 
~MMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMMM7          NMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMMM.
 MMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMMMMM$.          DMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMM$ 
 DMMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMMN~         ..INMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMM. 
  MMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMN$.         .ZNMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMMD  
  DMMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMM~..         7MMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMMM.  
   MMMMMMMMMMMMMMMMMMM   ,MMMMMMMMMN.           ?MMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMM8   
   ZMMMMMMMMMMMMMMMMMM   ,MMMMMMMMM.            NMMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMMM.   
    MMMMMMMMMMMMMMMMMM   ,MMMMMMMMMN            :MMMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMM?    
    .MMMMMMMMMMMMMMMMM   ,MMMMMMMMMMMO.          .NMMMMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMMN.    
     7MMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMM=.          7MMMMMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMMM      
      DMMMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMN~          .8MMMMMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMMM       
       +MMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMM:          . NMMMMMMMMMMMMMM   ZMMMMMMMMMMMMMM.       
        IMMMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMD.            ,NMMMMMMMMMMM   ZMMMMMMMMMMMMM.        
         =MMMMMMMMMMMM   ,MMMMMMMMMMMMMMMMMMMMMM.              .MMMMMMMMMM   ZMMMMMMMMMMMM          
          ,MMMMMMMMMMM?IIIMMMMMMMMMMMMMMMMMMMMM,                 NMMMMMMMMIII8MMMMMMMMMMN           
            MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM~.                   MMMMMMMMMMMMMMMMMMMMM?            
             ZMMMMMMMMMMMMMMMMMMMMMMMMMMMN+.                      7MMMMMMMMMMMMMMMMMMM.             
              ,MMMMMMMMMMMMMMMMMMMMMMMN,.                          MMMMMMMMMMMMMMMMM8.              
                7MMMMMMMMMMMMMMMMMMO                               NMMMMMMMMMMMMMMM                 
                  ,MMMMMMMMMMMMM:    .  .             ....  .      8MMMMMMMMMMMMO.                  
                    ,MMMMMMMMMM.   ~MMMMMMMMMM.     ZMMMMMMMMMO    MMMMMMMMMMMO.                    
                       NMMMMMMN.  $MMMMMMMMMMMM,   MMMMMMMMMMMMM. DMMMMMMMMM?                       
                         ?MMMMMM, .7MMMMMMMMMM~.   .NMMMMMMMMMM.,NMMMMMMMN.                         
                            ?MMMMMN?. . ,.               ,. .,DMMMMMMMN.                            
                               ,8MMMMMMMNN8$+~,,,,,:=IODNMMMMMMMMMM?                                
                                    +DMMMMMMMMMMMMMMMMMMMMMMMM$,                                    
                                        ...:?7Z8DMNDO$7=.
                                        
######################################################################################################
                                         Crawl, Number 2.
######################################################################################################


                                                                                           
More detailed documentation can be found in doc/index.html.

Crawl should run "out of the box", if you use the config files below, as these all point to public 
data or locally instantiated index clusters. Please note that all the example data used here are 
subject to the WSTI's data policies, see :

    https://www.sanger.ac.uk/legal/

The project is build with gradle, and includes a 
"gradlew" executable so you don't have to download and install gradle yourself. The first build will
take a while because it will download dependencies. OSX users please note that Java build tools like 
gradle need a properly set JAVA_HOME environmental variable. To build, do :

$ ./gradlew build

Also, as the build step involves downloading dependencies, and if you're behind a proxy, you may 
have to initially supply proxyHost and proxyPort Java settings, e.g. :

./gradlew build -Dhttp.proxyHost=wwwcache.sanger.ac.uk -Dhttp.proxyPort=3128


##############################################
# RUNNING OFF A CHADO DATABASE
##############################################

For a quick test run off the GeneDB public Chado database, try :

$ ./gradlew -Pconfig=resource-chado-public.properties jettyRunWar

and goto https://localhost:8080/services/index.html.

##############################################
# RUNNING OFF AN INDEXED GFF3 FILE
##############################################

If instead you want to try indexing a GFF file, try (assumes you're in bash) :

$ ./crawl gff2es -pe resource-elasticsearch-local.properties \
    -g  src/test/resources/data/Pf3D7_01.gff.gz \
    -o '{
    "ID":27, 
    "common_name":"Pfalciparum", 
    "genus":"Plasmodium", 
    "species":"falciparum", 
    "translation_table":11, 
    "taxonID":5833 
}' 
$ ./gradlew -Pconfig=resource-elasticsearch-local.properties jettyRunWar

and goto https://localhost:8080/services/index.html.

About

A spiritual successor to Crawl

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages