Fix unemployment age module

[ferblape]

This module got broken with a refactor a few months ago (the migration to D3 v5) because all the code that calculated the pct attribute is missing. I'm not sure why it wasn't failing before, but now that we have updated data it's totally broken

If you compare it with unemployment by sex you'll see the missing code in the method getData

You can check it in any site with the observatory module activated

[jorgeatgu]

I've fixed the charts, but the data I think is wrong.

The data comes without the under 25-year-olds. Is this okay? The unemployment_age.js have a function that refers to that age group

_getAgeRange(age) {
    switch (age) {
      case "<25":
        return I18n.t("gobierto_common.visualizations.less_25");
      case "25-44":
        return I18n.t("gobierto_common.visualizations.between_25_44");
      case ">=45":
        return I18n.t("gobierto_common.visualizations.more_44");
    }
  }

The percentage(pct) for 2020-2021 comes as Infinity/null and throws an error.

{
    "value": 5072,
    "location_id": "28065",
    "date": "2019-12-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-01-01/>=45",
    "pct": null
  },
  {
    "value": 4182,
    "location_id": "28065",
    "date": "2019-12-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-01-01/25-44",
    "pct": null
  },
  {
    "value": 5070,
    "location_id": "28065",
    "date": "2020-01-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-02-01/>=45",
    "pct": null
  },
  {
    "value": 4156,
    "location_id": "28065",
    "date": "2020-01-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-02-01/25-44",
    "pct": null
  },
  {
    "value": 4375,
    "location_id": "28065",
    "date": "2020-02-29T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-03-01/25-44",
    "pct": null
  },
  {
    "value": 5207,
    "location_id": "28065",
    "date": "2020-02-29T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-03-01/>=45",
    "pct": null
  },
  {
    "value": 5635,
    "location_id": "28065",
    "date": "2020-03-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-04-01/>=45",
    "pct": null
  },
  {
    "value": 5246,
    "location_id": "28065",
    "date": "2020-03-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-04-01/25-44",
    "pct": null
  },
  {
    "value": 5456,
    "location_id": "28065",
    "date": "2020-04-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-05-01/25-44",
    "pct": null
  },
  {
    "value": 5784,
    "location_id": "28065",
    "date": "2020-04-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-05-01/>=45",
    "pct": null
  },
  {
    "value": 5401,
    "location_id": "28065",
    "date": "2020-05-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-06-01/25-44",
    "pct": null
  },
  {
    "value": 5805,
    "location_id": "28065",
    "date": "2020-05-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-06-01/>=45",
    "pct": null
  },
  {
    "value": 5839,
    "location_id": "28065",
    "date": "2020-06-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-07-01/>=45",
    "pct": null
  },
  {
    "value": 5334,
    "location_id": "28065",
    "date": "2020-06-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-07-01/25-44",
    "pct": null
  },
  {
    "value": 5957,
    "location_id": "28065",
    "date": "2020-07-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-08-01/>=45",
    "pct": null
  },
  {
    "value": 5450,
    "location_id": "28065",
    "date": "2020-07-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-08-01/25-44",
    "pct": null
  },
  {
    "value": 5308,
    "location_id": "28065",
    "date": "2020-08-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-09-01/25-44",
    "pct": null
  },
  {
    "value": 5907,
    "location_id": "28065",
    "date": "2020-08-31T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-09-01/>=45",
    "pct": null
  },
  {
    "value": 5936,
    "location_id": "28065",
    "date": "2020-09-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-10-01/>=45",
    "pct": null
  },
  {
    "value": 5286,
    "location_id": "28065",
    "date": "2020-09-30T22:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-10-01/25-44",
    "pct": null
  },
  {
    "value": 5907,
    "location_id": "28065",
    "date": "2020-10-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-11-01/>=45",
    "pct": null
  },
  {
    "value": 5226,
    "location_id": "28065",
    "date": "2020-10-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-11-01/25-44",
    "pct": null
  },
  {
    "value": 6021,
    "location_id": "28065",
    "date": "2020-11-30T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2020-12-01/>=45",
    "pct": null
  },
  {
    "value": 5313,
    "location_id": "28065",
    "date": "2020-11-30T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2020-12-01/25-44",
    "pct": null
  },
  {
    "value": 6155,
    "location_id": "28065",
    "date": "2020-12-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": ">=45",
    "_id": "28065/2021-01-01/>=45",
    "pct": null
  },
  {
    "value": 5462,
    "location_id": "28065",
    "date": "2020-12-31T23:00:00.000Z",
    "province_id": 28,
    "autonomous_region_id": 13,
    "age_range": "25-44",
    "_id": "28065/2021-01-01/25-44",
    "pct": null
  }

[Crashillo]

So the charts were failing because they're not able to deal with null values, aren't they?

[jorgeatgu]

So the charts were failing because they're not able to deal with null values, aren't they?

Yes, in the case of unemployment_age, I've filtered out those values that are null.

In the unemployment_sex the problem is which pct value is less than 1(see the screenshot). For the moment, I've multiplied it by 100.

I had forgotten another question, what is the formula to obtain the pct of unemployment? Right now, the pct values are less than 1, that unemployment percentage we aren't calculating well.

[ferblape]

I think we had to remove <= 25 for some reason years ago but I don't remember why, maybe you can just comment that range and left the two others.

[jorgeatgu]

The percentage(pct) for 2020-2021 comes as Infinity/null and throws an error.

The error is because the age property in the 2020 data comes with age + "años", for example, age: "13 años", the rest of the years in the data comes only with the age. This is the reason why it wasn't failing before.

https://data.populate.tools/demo/datasets/ds-poblacion-municipal-edad.json

The data is grouped with d3.nest expecting only the age, not a string with "13 años". That's why it returns in 2020 infinity /null.

gobierto/app/javascript/gobierto_observatory/modules/classes/get_unemployment_age_data.js

Lines 36 to 56 in 68acc48

	v.filter(function(d) {
	return d.age >= 16 && d.age < 25;
	}),
	function(d) {
	return d.value;
	}
	),
	"25-44": sum(
	v.filter(function(d) {
	return d.age >= 25 && d.age < 45;
	}),
	function(d) {
	return d.value;
	}
	),
	">=45": sum(
	v.filter(function(d) {
	return d.age >= 45 && d.age < 65;
	}),
	function(d) {
	return d.value;

I've created a fix, removing "años" .replace(/años/g,''), but I think it is better that the data comes only with the age.

[Crashillo]

You can do also a simply parseInt(x), but I agree with you about the data issue must be fixed in DB/ETL

[ferblape]

I agree with you both, but we need to do an exception and control it here, we don't have too much time to fix it right now in the data. So please, clean it in javascript and I'll create an issue for the data issue.