xKDR · ayushpatnaikgit · Oct 28, 2022 · Oct 28, 2022
diff --git a/README.md b/README.md
@@ -8,17 +8,29 @@
 
 National Oceanic and Atmospheric Administration (NOAA) releases nighttime lights images produced using the Visible Infrared Imaging Radiometer Suite (VIIRS) since April 2012. Nighttime lights data had emerged as a useful tool to measure economic activity. Many researchers have established a correlation between prosperity and the brightness of a region. In many situations, nighttime lights generates measures with accuracy, latency and geographical resolution that are superior to conventional methods of measurement, such as GDP.
 
-Using nighttime lights for economic analysis require cleaning of data and aggregating measurements of pixels over regions of interest. This is the first open source implementation of these procedures for nighttime lights.
+Using nighttime lights for economic analysis require cleaning of data and aggregating measurements of pixels over regions of interest. This package provides functions to clean nighttime lights data and well as some other niche functions used by reseachers in this field. 
 
-This package was a foundation for a research paper, "But clouds got in my way: bias and bias correction of VIIRS nighttime lights data in the presence of clouds" by Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas. This paper diagnoses a source of bias in the data and responds to this problem with a bias correction scheme. Along with other mainstream methods of data cleaning, this method is also implemented in the package. 
+This package was a foundation for a research paper, ["But clouds got in my way: bias and bias correction of VIIRS nighttime lights data in the presence of clouds"](https://xkdr.org/wp2021-07.html) by Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas. This paper diagnoses a source of bias in the data and responds to this problem with a bias correction scheme. Along with other mainstream methods of data cleaning, this method is also implemented in the package. 
 
-Landing page of the paper: https://xkdr.org/wp2021-07.html
+This package is build on top of [Rasters.jl](https://github.com/rafaqz/Rasters.jl/). Nighttime lights tif/nc files can be read using `Rasters.Raster()`. 
 
 ## To install: 
 ```Julia
  add NighttimeLights
 ```
 
-# Support
+## Basic usage
+
+```Julia
+using Rasters
+using NighttimeLights
+
+radiance = Raster("sample_ntl_radiance.nc")
+ncfobs = Raster("sample_ntl_ncfobs.nc")
+
+clean_complete(radiance, ncfobs)
+```
+
+## Support
 
 We gratefully acknowledge the JuliaLab at MIT for financial support for this project.
diff --git a/docs/make.jl b/docs/make.jl
@@ -16,10 +16,8 @@ makedocs(;
  pages=[
  "Home" => "index.md",
  "Basic concepts" => "concepts.md",
- "Polygons and shapefiles"=> "polygons.md",
  "Data Cleaning"=> "data_cleaning.md",
  "Tutorial" =>"tutorial.md",
- "Plots" => "plots.md"
  ],
 )
 

diff --git a/docs/src/chloropleth.png b/docs/src/chloropleth.png
diff --git a/docs/src/concepts.md b/docs/src/concepts.md
@@ -1,11 +1,15 @@
 
 # Basics of nighttime lights data
 
-## Data IO
+## Data structure
+
+NOAA provides tif files of the nightlights images. These are represented as 2D arrays with floating-point values. Tif files can be read using the [Rasters.jl](https://github.com/rafaqz/Rasters.jl/) package. While nighttime lights images are 2D arrays, reading them as Raster files creates an extra dimension for bands, which is irrelevant for nightlights images. We will ignore the extra dimension and refer to the image as 2D arrays instead of 3D. 
 
-NOAA provides tif files of the nightlights images. Images in the package are represented as 2D arrays with floating-point values. Images of different months are stacked together to form 3D arrays. Such 3D arrays are called datacubes. 
+Images of different months are stacked together to form 3D arrays. Such 3D arrays are called datacubes. Once again, there is an extra dimension for bands, which will be ignore and treat the 4D arrays as 3D. 
+
+## Data IO
 
-Raster data in Julia can be loaded using the [Rasters.jl](https://github.com/rafaqz/Rasters.jl/) package. While the package has a comprehensive [documentation](https://rafaqz.github.io/Rasters.jl/dev/). This page highlights the small set of tools required to study nighttime lights. It is recommended to study the full documentation of `Rasters.jl`. 
+While the Rasters.jl has a comprehensive [documentation](https://rafaqz.github.io/Rasters.jl/dev/). This page shows some concepts needed to be known to study nighttime lights. 
 
 The package can be used to load 2D matrices, saved as `.tif` files, and 3D matrices. saved as `.nc` files using the `Raster` functions. 
 
@@ -26,7 +30,6 @@ radiances = [Raster(i, lazy = true) for i in filelist]
 timestamps = collect(1:length(radiances))
 series = RasterSeries(radiances, Ti(timestamps))
 datacube = Rasters.combine(series, Ti)
-datacube = datacube[:,:,1,:] # needed to drop dimension regarding bands, since nighttime lights data is single band. 
 ```
 
 `write` function from Rasters can be used to write images into `.tif` files and datacubes into `.nc` files. 
@@ -38,12 +41,14 @@ write("image.tif", image)
 ```
 
 ## Indexing
-Images can be indexed like 2D arrays and datacubes can be indexed like 3D arrays. 
+The dimension regarding bands can be hidden using `view` as nighttime lights are single band images and then images can be indexed like 2D arrays and datacubes can be indexed like 3D arrays. 
 
 For example:
 
 ```julia
+image = view(image, Band(1))
 image[1, 2] # value of the image at location [1, 2]. 1st row and 2nd column 
+datacube = view(datacube, Band(1))
 datacube[:, :, 3] # Image of the 3rd month.
 datacube[1, 2, :] # Time series values of the pixel at location 1, 2
 datacube[1, 2, 3] # Value of the image at location [1, 2] of the 3rd month

diff --git a/docs/src/data_cleaning.md b/docs/src/data_cleaning.md
@@ -1,25 +1,14 @@
 Using nighttime lights for economic inference needs cleaning of the data. 
 
-There are negative observations in the data as a consequence of NOAA's cleaning procedures. Many researchers replace them with 0. It can be done using the following:
-```julia
-datacube = replace!(x -> x < 0 ? 0 : x, datacube) # replace all values below 0 with 0
-```
-These values can be replaced with missing and interpolated. 
-```julia
-datacube = replace!(x -> x < 0 ? missing : x, datacube) # replace all values below 0 with missing
-```
-
 ```@docs
-na_recode(radiance, clouds::Array{T, 2}) where T <: Any
-na_recode(radiance, clouds::Array{T, 3}) where T <: Any
+na_recode(radiance, clouds; replacement)
 na_interp_linear(timeseries)
 outlier_variance(datacube, mask)
 outlier_hampel(timeseries, window_size = 5, n_sigmas = 3)
 bgnoise_PSTT2021(radiance_datacube, clouds_datacube, th = 0.4)
-bias_PSTT2021(radiance, clouds::Array{T, 1}, smoothing_parameter=10.0) where T <: Any
-bias_PSTT2021(radiance_datacube, clouds_datacube::Array{T, 3}, mask=ones(Int8, (size(radiance_datacube)[1],size(radiance_datacube)[2]))) where T <: Any
+bias_PSTT2021(radiance, clouds, smoothing_parameter=10.0) 
+bias_PSTT2021(radiance_datacube, clouds_datacube, mask)
 PSTT2021_conventional(radiance_datacube, clouds_datacube)
 PSTT2021(radiance_datacube, clouds_datacube)
-
-
+clean_complete(radiance_datacube, clouds_datacube)
 ```
diff --git a/docs/src/index.md b/docs/src/index.md
@@ -14,28 +14,16 @@ While there are packages to do image processing in Julia, the assumptions about
 
 ![india lights](eog.png)
 
-This package was a foundation for a research paper, "But clouds got in my way: bias and bias correction of VIIRS nighttime lights data in the presence of clouds" by Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas. The paper diagnoses a source of bias in the data and responds to this problem with a bias correction scheme. Along with other mainstream methods of data cleaning, this method is also implemented in the package. The details are in the paper which can be accessed from its [landing page](https://xkdr.org/releases/PatnaikShahTayalThomas_2021_bias_correction_nighttime_lights.html). 
+This package was a foundation for a research paper, "[But clouds got in my way: bias and bias correction of VIIRS nighttime lights data in the presence of clouds]((https://xkdr.org/releases/PatnaikShahTayalThomas_2021_bias_correction_nighttime_lights.html))" by Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas. The paper diagnoses a source of bias in the data and responds to this problem with a bias correction scheme. Along with other mainstream methods of data cleaning, this method is also implemented in the package.
+
+The package is built on top of [Rasters.jl](https://github.com/rafaqz/Rasters.jl/), which provides cutting edge features to study raster data and perform routine tasks such as zonal statistics and plotting. 
 
 # Installation
 
 ```
 pkg> add NighttimeLights
 ```
 
-# Downloading data
-
-The monthly nighttime lights data is hosted by [Payne Institute](https://payneinstitute.mines.edu/eog/nighttime-lights/) at the [Earth Observation Group](https://eogdata.mines.edu/products/vnl/). 
-
-The data is stored in `.tif` files. These are opened as 2D matrices and for each pixel, there is a floating-point value representing the amount of light. Monthly composites are produced by taking the average of measurements produced on days free of clouds. The number of cloud-free observations used to generate those are also in stored in `.tif` files. 
-
-There are two types of annual images, those produced using daily images and those produced using monthly composites. 
-
-The `.tif` files covering the entire planet are large. NOAA also provides tiled data. The planet is divided into 6 tiles in the following manner: 
-
-![tile map](tile_map.png)
-
-If your region of interest is covered in one tile, then you may not need to download the full image of the planet. 
-
 # Getting help 
 
 To get inquire about the Julia package, you can join the Julia community on [slack](https://julialang.org/slack/) and post questions on the nighttime-lights channel. You can also email one of the authors of the package, [Ayush Patnaik](mailto:[email protected]). 

diff --git a/docs/src/plots.md b/docs/src/plots.md
diff --git a/docs/src/polygons.md b/docs/src/polygons.md
diff --git a/docs/src/tutorial.md b/docs/src/tutorial.md
@@ -53,18 +53,4 @@ If you haven't downloaded the data, and you want to run the remaining code, you
 A single function, `clean_complete`, can be used on `radiance_datacube` and `clouds_datacube` to generate a cleaned datacube of radiance. 
 ```julia
 cleaned_datacube = clean_complete(radiance_datacube, clouds_datacube) 
-```
-## 5. Zonal statistics
-
-1. Load the shape file for the districts of Mumbai. 
-You can download the [district level shapefile of Mumbai](https://github.com/xKDR/NighttimeLights.jl/tree/main/assets/mumbai_map) or load it using ```load_example()```. 
-
-```julia
-mumbai_map = Shapefile.Table("~Downloads/mumbai_map.shp")
-```
-These have been created from the Data{Meet}'s [district level shapefile](https://github.com/datameet/maps/tree/master/Districts/Census_2011) of India, based on census 2011. 
-
-2. Generate the time series of aggregate radiance for each district of Mumbai. 
-```julia
-mumbai_district_ntl = aggregate_dataframe(MUMBAI_COORDINATE_SYSTEM, cleaned_datacube, mumbai_map)
-``` 
+```
diff --git a/src/data_cleaning/full_procedures.jl b/src/data_cleaning/full_procedures.jl
@@ -64,7 +64,7 @@ end
 
 """
 The
-function `clean_complete()` will represent our views on an optimal set of steps for pre-
+function `clean_complete()` represents our views on an optimal set of steps for pre-
 processing in the future (for the period for which this package is actively maintained). As
 of today, it is identical to `PSTT2021()``
 """