Skip to content

Commit

Permalink
Tds 5.4 snapshot (#2)
Browse files Browse the repository at this point in the history 
* content migration and formatting

* style changes for further info callout

* finish jvm settings

* rename & move

* content migration

* added toc

* content reorg

* content dir content

* content migration

* content migration

* content reorg

* content migration

* content migration

* moved to quick start guide

* added overview section

* content migration

* Dependency upgrades

* Bump the following Bill of Materials versions:
  * netndf-java-bom (5.4.0-SNAPSHOT to 5.2.0-SNAPSHOT)
  * spring-framework-bom (5.3.1 to 5.3.4)
  * spring-security-bom (5.4.1 to 5.4.5)
  * chronicle-bom (2.20.93 to 2.21ea74)
* utilize the netcdf-java-testing-platform for testing dependency
  management
* Suppress OWAPS dependency check false positives
* Move to edal-java 1.5.0
  * this forces the minimum required version of Java from 8 to 11.

* Drop JDK 8 from GitHub Actions matrix.

* Upgrade to Gradle 6.8.3

* Migrate away from jcenter

* fixed typos

* supplemented content

* fixed broken links

* removed docs not needed in this guide

* content addition

* content reorganization

* updated security content section

* created separate section for customization

* catalog overview

* reorg of content

* clone userguide

* added tasks for new guides

* guide reconfig

* replace current -> version # for netcdf-java

* added docset_version

* replace current -> version # for tds

* changed sidebar to admin

* mv overview --> index.html

* removed netcdfjavatutorial

* add breadcrumbs include

* modify breadcrumbs layout

* breadcrumbs & side nav title css

* content change

* added all guides to docSets

* file restructure

* content reorg

* inject category titles into breadcrumbs

* breadcrumb styling

* elements for bc styling

* toc placeholder

* content migration

* rename and/or move

* added further info callout section

* table style changes

* content migration and formatting

* style changes for further info callout

* finish jvm settings

* rename & move

* content migration

* added toc

* content reorg

* content dir content

* content migration

* content migration

* content reorg

* content migration

* content migration

* moved to quick start guide

* added overview section

* content migration

* fixed typos

* supplemented content

* fixed broken links

* removed docs not needed in this guide

* content addition

* content reorganization

* updated security content section

* created separate section for customization

* catalog overview

* reorg of content

* content migration

* Remove private docs directory

Archived on google drive

* Migrate docs to new structure

* Remove buildSrc

Migrating to unidata nexus gradle plugin

* Update docset _config and sidebar files

* Add files shared between docsets

* Upgrade docs gradle build

Upgrade docs gradle build to use latest unidata jekyll plugin (and
theme), and begin using unidata nexus gradle plugin for publishing
documentation.

* Have all userguide pages use user_sidebar.yml

* Have all quickstart pages use quickstart_sidebar.yml

* Have all devguide pages use dev_sidebar.yml

* Use netCDF-Java builder interface by default.

* Migrate from NotTravis to NotPullRequest junit category

Also, add failing dap4 test to NotPullRequest - will still fail in
jenkins, but will allow us to get a baseline of passing tests on github.

* Update GitHub Actions

Split out dependency check and style check jobs into their own configs
Upgrade TDS test action to use thredds-test-action.

* Make javadoc happy with java 11.

* Use Unidata snapshot of edal-java 1.5.0 release.

* edal-java / ncWMS fixes.

* Fix thymeleaf warning about templateMode HTML5

* Chronicle-Map adjustments

* Remove chronicle-analytics transient dependency
* Implement custom seralization using BytesWriter<DatasetExt>, BytesReader<DatasetExt>, ReadResolvable<DatasetExtBytesMarshaller> instead of relying on Externalizable.

* Apply Aggregation settings to builder based Aggs

Make sure to set typical dataset mode and diskcache for
ucar.nc2.internal.ncml.Aggregation.

* Remove old documentation src.

* Migrate it subproject to tds test SourceSet

* Rework integration testing gradle infrastructure

* Don't startup H2 database instance with the TDS. Let wms handle that.

* Don't set datasetTracker object to null on close

Also, very minor cleanup in DatasetManager.

* Give epsg database time to cleanup

* Update README.txt template and create README.md for developers.

* Implement persistent caching for FMRC GridDatasetInv

Implement a InventoryCacheProvider using ChronicleMap to enable a
persistent cache for FMRC GridDatasetInv caching.

Resolves Unidata#19.

* Use cdm-test-utils over local test utils subproject.

* rename `testUtil` to `tds-test-utils`
* Use `cdm-test-utils` in place of local to see what needs to be kept in
  this repository.

* Only keep necessary classes in tds-test-utils

* Relocate package thredds to thredds.test.util

* Add ClientcatalogUtil to tds-test-utils

* Properly handle encoding when reading strings from netCDF-C

* Do not set system level file encoding

* Use correct resources path for dap4 servlet.

* Remove use of reflection to get netCDF-C library

* Set EpsgDatabasePath using URI to ensure tests work on Windows.

* Fix dap4 tests

* OPeNDAP html form changes from 4.6.x

Unidata#134, Unidata/thredds#1363

* Disable http credential caching for tests.

* Improve RadarServer tests

Improve tests to actually inspect content, not just check for 200. This
results in making the tests fail.

* Fix RadarServer initilization

Perform init logic in @eventlistener method, as afterPropertiesSet()
occurs before TdsInit.onApplicationEvent, which results in a partially
initilized service.

* Update RadarServer configs based on TdsConfig project.

* Use new netCDF-java API

No longer allow the use of the old API at runtime as a threddsConfig.xml
option or Java systems property. Also, fix a few modifier order issues.

* Clear HTTPSession credentials between tests

* Correctly handle NcML embedded inside TDS config catalogs.

* Handle netcdf element's addRecords attribute special

The new netcdf-java API does not really handle the addRecords attribute
of the netcdf element for NcML. Add a special case in DatasetManager to
deal with this.

New API IOSPs do not expose IOSP messages, so remove addRecords attribute.

* NcML addRecords attribute does not work for single dataset.

* Only use new API in tests.

* Enable use of wmsConfig.xml

During the move from ncWms 1.x to edal-java 2.x, use of the
wmsConfig.xml was not working and had to be reimplemented. Many of the
classes involved in making the custom defaults (by path, standard name,
or variable name) work was able to be reused, although porting to use
edal-java took a bit of time. A few new classes had to be created as
well.

* Update WMS documentation

Also add documentation specific to wmsConfig.xml.

* Remove CORS configuration from TDS

CORS should implemented at the servlet container or proxy server level.

* Publish tds war and tdm jar to downloads repo

* Remove the CdmrFeature service.

* Set setExpandEntities to false on SAXBuilder

See Unidata/netcdf-java#801.

* Bump spring framework and spring security deps

* Update cve suppression file to exclude those that are not applicable.

* Use netcdf-java 5.4.2 release.

* Fix integration tests that fail due to removal of CdmrFeature in Unidata#146

* Upgrade to gradle 6.9.1

* Temurin does not have java 14 binaries, so remove from github test matrix

* Fix typo

* Update and reshuffle some docs.

* Update the docs related to upgrading the TDS.

* Remove references to java 8 in favor of java 11

* Release 5.0.0-beta9

* Publish downloads under tds/version/ just like docs

* Resume work on 5.0.0-SNAPSHOT

* Fix download and doc publishing

* Publish docs under version/ and not project/version
* Increase heap space to account for memory needed for publishDownoads
task

* Remove server version info from catalog and serverInfo service.

* Upgrade to thredds-test-action@v2 and add zulu 14 to test matrix.

* Update palette selections in wmsConfig.xml to be valid ncWMS 2 names.

* Document old and new palette name mappings in release notes.

* Fix WmsDetailedConfigTest default palette check.

* update codeowners

* Fix service init sequence

Make sure the OPeNDAP and WCS init sequence happens after
threddsConfig.xml has been read. This issue was introduce after
migrating to a new version of spring and was just now noticed.

* Fetch TDS version info file from downloads.unidata.ucar.edu

* Fix topnav menu links.

* Release 5.0

* begin work on 5.1-SNAPSHOT

* update dependencies

* Fix missing OpenStreetMap attribution for NCSS maps

* fix testVersionRetrieval, which began failing after renaming latest versions to stable and maintenance

* removed link to admin guide

* Fix content-type returned by FileServer for JSON files (TDS GH Issue Unidata#161).

* Fix content-type testing (TDS GH Issue Unidata#161).

* do no wrap explicit line breaks on enum constants

* run spotless on ful project

* Minor content-type test clean-up and spotless fixes.

* Fix a bad merge and yet another spotless fix.

* Final, I hope, spotless fix.

* Spotless fix.

* Spotless fix.

* suppress new version of wire-ios CVE

* Change github action to refer to TDS, not netCDF-Java

* Change dtd location to use https.

* jdk version correction

* Turn off validation of wmsConfig.

* www --> schemas host migration

* Remove unused 4.6 wms palettes.

* spring framework upgrade

* Release 5.1

* begin work on 5.2-SNAPSHOT

* update docsets to working version, 5.2

* upgrade log4j to 2.15.0

* release 5.3

* begin work on 5.3-SNAPSHOT

* update threddsIso version

* upgrade log4j to 2.16.0

* supressing false positive owasp hit for h2-1.3.173.jar

* Upgrade to log4j 2.17.0

Use the log4j bom with enforcedPlatform in tds-platform project to allow
us to define the log4j version in one spot while controlling transitive
version resolution at the same time.

* release 5.3

* begin work on 5.4-SNAPSHOT

* bump threddsISO to snapshot

* add fixes for messed up release process (now netcdf-java 5.5.1)

* more log4j, forever log4j

* fix link to tds docs, used by thymeleaf footer

Co-authored-by: Jennifer Oxelson <[email protected]>
Co-authored-by: Sean Arms <[email protected]>
Co-authored-by: haileyajohnson <[email protected]>
Co-authored-by: Ryan May <[email protected]>
Co-authored-by: haileyajohnson <[email protected]>
Co-authored-by: geojs <[email protected]>
Co-authored-by: Ethan Davis <[email protected]>
Co-authored-by: Jennifer Oxelson <[email protected]>
  • Loading branch information
@greenteadigital @oxelson
@lesserwhirls @dopplershift @geojs @ethanrd
9 people committed Jan 20, 2022
1 parent 574b039 commit cd935fd
Show file tree
Hide file tree
Showing 1,688 changed files with 67,597 additions and 111,552 deletions.
18 changes: 3 additions & 15 deletions CODEOWNERS → .github/CODEOWNERS
View file
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -15,41 +15,29 @@
# users just like we do for commit author emails.
# docs/* [email protected]

* @haileyajohnson

#####################
# top level modules #
#####################

gradle/** @lesserwhirls
travis/* @lesserwhirls

###########################################
# classpath / not module specific matches #
###########################################

**/iosp/nids/** @dopplershift
**/thredds/server/radarServer2/** @dopplershift

**/opendap/** @DennisHeimbigner
**/dap4/** @DennisHeimbigner

**/webapp/godiva2/** @lesserwhirls
**/webapp/godiva2/** @lesserwhirls

######################
# general file types #
######################

*.gradle @lesserwhirls
gradle* @lesserwhirls

*.py @lesserwhirls @dopplershift
*.py @dopplershift

####################
# individual files #
####################

gradle/any/dependencies.gradle @lesserwhirls @DennisHeimbigner

docs/internal/release.md @lesserwhirls

tds/src/main/webapp/Godiva.html @lesserwhirls
313 changes: 313 additions & 0 deletions .github/CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,313 @@
![tds icon](https://www.unidata.ucar.edu/images/logos/thredds_tds-150x150.png)

# Welcome contributors!

First off, thank you for your interest in contributing to the THREDDS project!
This repository contains the code for the THREDDS Data Server.
The other projects held under the THREDDS umbrella are:
* [netCDF-Java](https://github.com/unidata/netcdf-java) (netCDF-Java)
* [Rosetta](https://github.com/unidata/rosetta)
* [Siphon](https://github.com/unidata/siphon) (a python data access library, which includes an interface to the TDS).

Please review the [UCAR code of conduct](https://github.com/Unidata/.github/blob/develop/CODE_OF_CONDUCT.md) before submitting a pull request.
Once your have submitted a pull requests, you will be asked to digitally sign the Unidata Contributor License Agreement.
For more information about the CLA, please visit https://www.unidata.ucar.edu/blogs/developer/entry/contributor-license-agreement-for-unidata.

## Process Overview

* [GitHub Setup](#gh-setup)
* [Join Github!](#gh-join)
* [Fork the Unidata TDS project](#gh-fork)
* [Pull down local copy of the Unidata TDS project](#gh-pull-ud-tds)
* [Add and pull down a local copy of your TDS project fork](#gh-pull-personal-tds)
* [Contribution workflow](#gh-contrib-workflow)
* [Make sure you have the latest changes from Unidata tds repository](#gh-sync-ud)
* [Make a new branch for your work and start hacking](#gh-branch)
* [Follow these Style Guidelines](#gh-style-guidelines)
* [Clean up your git commit history](#gh-history-cleanup)
* [Push changes to your fork to use for the pull request](#gh-final-commit-for-pr)
* [Make the pull request](#gh-pr)
* [Now what?](#gh-now-what)

## <a name="gh-setup"></a>GitHub Setup

### <a name="gh-join"></a>Join Github!
To get started contributing to a THREDDS project, the first thing you should do is [sign up for a free account on GitHub](https://github.com/join).

### <a name="gh-fork"></a>Fork the Unidata TDS project
Once you have an account, go ahead and [fork](https://github.com/unidata/tds#fork-destination-box) the TDS project.
By forking the project, you will have a complete copy of the TDS project, history and all, under your personal account.
This will allow you to make pull requests against the Unidata TDS repository, which is the primary mechanism used to add new code to the project (even Unidata developers do this!).

### <a name="gh-pull-ud-tds"></a>Pull down local copy of the Unidata TDS project
After cloning the Unidata repository, you can pull down the source code to your local machine by using git:

`git clone --origin unidata [email protected]:Unidata/tds.git` (for ssh)

or

`git clone --origin unidata https://github.com/Unidata/tds.git` (for http)

Note that these commands reference the Unidata repository.

Normally in git, the remote repository you clone from is automatically named `origin`.
To help with any confusion when making pull requests, this commands above rename the remote repository to `unidata`.

### <a name="gh-join"></a>Add and pull down a local copy of your TDS project fork

Next, move into the source directory git has created, and add your personal fork of the TDS code as a remote"

`git clone --origin me [email protected]:<my-github-user-name>/tds.git` (for ssh)

or

`git clone --origin me https://github.com/,my-github-user-name>/tds.git` (for https)

Now you are all set!

## <a name="#gh-contrib-workflow"></a>Contribution workflow

### <a name="#gh-sync-ud"></a> Make sure you have the latest changes from Unidata TDS repository
First, make sure you have the most recent changes to the TDS code by using git pull:

`git pull unidata develop`

All work on TDS is should branch from the `develop` branch.
Contributions will be backported to other versions as needed.

### <a name="#gh-branch"></a>Make a new branch for your work and start hacking
Next, make a new branch where you will actually do the hacking:

`git checkout -b mywork`

As of this point, the branch `mywork` is local.
To make this branch part of your personal GitHub Remote repository, use the following command:

`git push -u me mywork`

Now git (on your local machine) is setup to a point where you can start hacking on the code and commit changes to your personal GitHub repository.

At any point, you may add commits to your local copy of the repository by using:

`git commit`

If you would like these changes to be stored on your personal remote repository, simply use:

`git push me mywork`

Once you are satisfied with your work, there is one last step to complete before submitting the pull request - clean up the history.

### <a name="#gh-style-guidelines"></a>Follow these Style Guidelines

#### Java, Groovy, and Gradle
We are using the [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html) for Java, Groovy, and Gradle, with these exceptions and augmentations:

1. The recommended line width is 120, not 100.
Modern screens are much wider than tall, so having wider lines allows more code to fit on a screen.

2. As a rule, don't add empty javadoc that has no information that can't be seen directly in the code.
For example, do not do this:

~~~java
/**
* _more_
*
* @param nds _more_
* @param v _more_
* @return _more_
* @throws IOException
*/
protected Foo makeFoo(NetcdfDataset nds, VariableSimpleIF v) throws IOException
~~~

Better to not have any javadoc then to have empty javadoc.
Of course, best is to put actual, useful comments to help others understand your API.

To assist in following the style, we provide both IntelliJ and Eclipse style files.
These files can be found in the under the `project-files/code-styles/` directory of the repository.
If you are not using an IDE, you can check the style using:

~~~bash
./gradlew spotlessCheck
~~~

and you can apply the style using:

~~~bash
./gradlew spotlessApply
~~~

It can be easy to forget to run this command before pushing your changes to github.
For that, we have created git pre-commit hook scripts for you to use.
The pre-commit hook will run the spotlessApply task before the commit is made, ensuring that you have the proper code format.
To install the pre-commit hook in a *nix environment (linux, *BSD, MacOS, Cygwin, etc.), copy the file `hooks/pre-commit-nix` to `.git/hooks/pre-commit` and make it executable.
On windows, copy the file `hooks/pre-commit-windows` to `.git/hooks/pre-commit`.

#### Markdown

When writing markdown, we use one line per sentence.
At this time, markdown is not inspected by spotless.

### <a name="#gh-history-cleanup"></a>Clean up your git commit history

Commit history can often be full of temporary commit messages, and/or commits with code changes that ultimately didn't make the final cut.

To clean up your history, use the `git rebase -i` command, which will open an editor:

~~~bash
sarms@flip: [mywork]$ git rebase -i
pick 083508e first commit of my really cool feature or bug fix!
pick 9bcba01 Oops missed this one thing. This commit fixes that.

# Rebase 083508e..9bcba01 onto 083508e (2 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out
~~~

Based on my commit messages, you can see that commit `9bcba01` fixed a mistake from my first commit.

It would be nice to `squash` those changes into the first commit, so that the official history does not show my mistake..uhhh...this extra commit.

To do so, edit the text to change the second commits `pick` to `squash`:

~~~bash
pick 083508e first commit of my really cool feature or bug fix!
squash 9bcba01 Oops missed this one thing. This commit fixes that.

# Rebase 083508e..9bcba01 onto 083508e (2 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out
~~~

Once you have marked the commits to be squashed and exited the edit, you will prompted to change the commit message for the new, squashed, mega commit:

~~~bash
# This is a combination of 2 commits.
# The first commit's message is:

first commit of my really cool feature or bug fix!

# This is the 2nd commit message:

Oops missed this one thing. This commit fixes that.

#Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Date: Thu Oct 15 09:59:23 2015 -0600
#
# interactive rebase in progress; onto 083508e
# Last commands done (2 commands done):
# pick 09134d5 first commit of my really cool feature or bug fix!
# squash 9bcba01 Oops missed this one thing. This commit fixes that.
# No commands remaining.
# You are currently editing a commit while rebasing branch 'mywork' on '0835 08e'.
#
# Changes to be committed:
...
~~~

Edit the two commit messages into a single message that describes the overall change:

~~~
Once you have and exit, you will have a change to change the commit message for the new, squashed, mega commit:
~~~bash
Really cool feature or bug fix. Addresses the github issue Unidata/tds#1
#Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Date: Thu Oct 15 09:59:23 2015 -0600
#
# interactive rebase in progress; onto 083508e
# Last commands done (2 commands done):
# pick 09134d5 first commit of my really cool feature or bug fix!
# squash 9bcba01 Oops missed this one thing. This commit fixes that.
# No commands remaining.
# You are currently editing a commit while rebasing branch 'mywork' on '0835 08e'.
#
# Changes to be committed:
...
~~~

Now, when you look at your git commit logs, you will see:

~~~bash
commit 805b4723c4a2cbbed240354332cd7af57559a1b9
Author: Sean Arms <[email protected]>
Date: Thu Oct 15 09:59:23 2015 -0600

Really cool feature or bug fix. Addresses the github issue Unidata/tds#1

~~~

Note that the commit contains the text `Unidata/tds#1`.
This is a cool github trick that will allow you to reference GitHub issues within your commit messages.
When viewed on github.com, this will be turned into a hyperlink to the issue.
While not every contribution will address an issue, please use this feature if your contribution does!

### <a name="#gh-final-commit-for-pr"></a>Push changes to your fork to use for the pull request
Now that you have cleaned up the history, you will need to make a final push to your personal GitHub repository.
However, the rebase has changed the history of your local branch, which means you will need to use the `--force` flag in your push:

`git push --force me mywork`

### <a name="#gh-pr"></a>Make the pull request
Finally, go to your personal remote repository on github.com and switch to your `mywork` branch.
Once you are on your work branch, you will see a button that says "Pull request", which will allow you to make a pull request.

The github pull request page will allow you to select which repository and branch you would like to submit the pull request to (the `base fork`, which should be `Unidata/tds`, and `base`, which should be `develop`), as well as the `head fork` and `compare` (which should be `<github-user-name/tds>` and `mywork`, respectively).
Once this is setup, you can make the pull request.

## <a name="#gh-now-what"></a>Now what?

The Unidata TDS project uses GitHub Actions for automated testing for all pull requests.
The status of the tests can be seen on the pull request page.
For example, visit [Unidata/tds#355](https://github.com/Unidata/tds/pull/99) and select the [Checks](https://github.com/Unidata/tds/pull/99/checks) tab along the top.
GitHub Actions check the following:
* build and test TDS using AdoptOpenJDK 8, Zulu JDK 8 (*required*)
* build the documentation, if any documentation changes were made in the PR. (*required*)
* code style check using `spotlessCheck`

Pull requests whose changes cause one or more of the required checks to fail are generally not merged.
One of the Unidata TDS team members will work with you to make sure your work is ready for merging once the tests have passed.
Any changes to your pull request will trigger the GitHub Actions to re-run.

If your pull request addresses a bug, we kindly ask that you include a test in your pull request.
Learn more about testing within TDS on the netCDF-Java [wiki](https://github.com/Unidata/netcdf-java/wiki/Testing) (the TDS and netCDF-Java share testing infrastructure).
If you have test data that are larger than a few hundred kB, please let us know.
If you cannot reduce the size of the test file, we can add it to our [integration test datasets](https://github.com/unidata/thredds-test-data), as long as we have permission to redistribute the file.
If you do not know how to write tests in Java, we will be more than happy to work with you!
15 changes: 15 additions & 0 deletions .github/workflows/deps.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
on: [pull_request]

jobs:
check-dependency-availability:
name: Check fresh compile and assemble of TDS project
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up JDK 11
uses: actions/setup-java@v2
with:
distribution: 'adopt'
java-version: '11'
- name: Build with Gradle (refresh dependencies)
run: ./gradlew clean classes testClasses assemble --refresh-dependencies
Loading

0 comments on commit cd935fd

Please sign in to comment.