Added test coverage for CSVToAvro util #50
Conversation
ajayborra
commented
Aug 13, 2018
ajayborra
commented
- Added test coverage for CSVToAvro util
- Added tests to RichGenericRecord.scala
| @@ -0,0 +1,47 @@ | |||
| { | |||
There was a problem hiding this comment.
We have too many of the same or slighly different schema and data files already.
- Can we not reuse tthe existing ones in this test?
- If not, them them give them a better names than simply adding 2.
- Perhaps move these new “bad schema” files to utils/test/resources
There was a problem hiding this comment.
- 1:Re: To improve coverage, had to modify the data to cover negative cases. (Limited the data copies to 5 rows)
- 2:Re: Done, Renamed files, Check them out.
- 3:Re: Done, Moved the files to the resource folder.
|
|
||
| it should "convert GenericRecord to RichGenericRecord" in { | ||
| val res = RichGenericRecord(firstRow) |
There was a problem hiding this comment.
No need to explicitly convert it - its an implicit class.
There was a problem hiding this comment.
This is more of a coverage test, will remove it.
| val avroSchema: String = loadFile(avroSchemaPath) | ||
| val csvFile: String = s"$testDataDir/PassengerDataAllV2.csv" | ||
| val csvReader: CSVInOut = new CSVInOut(CSVOptions(header = true)) | ||
| val csvRDD: RDD[Seq[String]] = csvReader.readRDD(csvFile) |
There was a problem hiding this comment.
Make csvRDD and csvFileRecordCount lazy vals
|
|
||
| Spec(CSVToAvro.getClass) should "convert RDD[Seq[String]] to RDD[GenericRecord]" in { | ||
| val res = CSVToAvro.toAvro(csvRDD, avroSchema) | ||
| assert(res.isInstanceOf[RDD[GenericRecord]]) |
|
|
||
| it should "convert RDD[Seq[String]] to RDD[T]" in { | ||
| val res = CSVToAvro.toAvroTyped[Passenger](csvRDD, avroSchema) | ||
| assert(res.isInstanceOf[RDD[Passenger]]) |
Codecov Report
@@ Coverage Diff @@
## master #50 +/- ##
=========================================
+ Coverage 84.46% 84.77% +0.3%
=========================================
Files 298 298
Lines 9751 9752 +1
Branches 355 560 +205
=========================================
+ Hits 8236 8267 +31
+ Misses 1515 1485 -30
Continue to review full report at Codecov.
|
| /** | ||
| * Returns the resource directory path | ||
| */ | ||
| protected def resourceDir = _resourceDir |
There was a problem hiding this comment.
no need to create a private val _resourceDir = "src/test/resources" simply do:
protected def resourceDir: String ="src/test/resources"
