Unable to read metadata (l3a,l3b) #28

smfcrts · 2019-11-09T19:53:54Z

Hello,
I run L3A composite through command line:
/usr/bin/composite_processing.py --syntdate 20190901 --input "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR " "/mnt/archive/maccs_def/morocco/l2a/S2B_MSIL2A_20190818T112119_N0208_R037_T29SMS_20190818T122949.SAFE/S2B_OPER_SSC_L2VALD_29SMS____20190818.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190823T112121_N0208_R037_T29SMS_20190823T114631.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190823.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190902T112111_N0208_R037_T29SMS_20190902T132637.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190902.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190912T112111_N0208_R037_T29SMS_20190912T114646.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190912.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2B_MSIL2A_20190917T112119_N0208_R037_T29SMS_20190917T132014.SAFE/S2B_OPER_SSC_L2VALD_29SMS____20190917.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190922T112111_N0208_R037_T29SMS_20190922T114652.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190922.HDR" --res 10 --outdir "/mnt/archive/temp/20190901/29SMS/" --bands /usr/share/sen2agri/bands_mapping_s2_L8.txt --scatteringcoe /usr/share/sen2agri/scattering_coeffs_10m.txt --synthalf 25
It gave this error:
The following error occurred during application execution : /home/cudroiu/sen2agri/sen2agri-processors/ProductReaders/MetadataHelpers/src/MetadataHelperFactory.cpp:43:
itk::ERROR: MetadataHelperFactory(0x16a8810): Unable to read metadata from /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR
OTB app finished in: 0:00:04.178794
OTB application error
Please how can i overcome this issue

smfcrts · 2019-11-13T09:11:25Z

@lnicola please could you give a solution to this issue

lnicola · 2019-11-15T08:17:15Z

@smfcrts sorry for the late reply. Can you tell us what version of the system you are running and maybe attach a copy of that HDR file?

smfcrts · 2019-11-15T13:57:57Z

Hello,
The version of the system is 2.0.1, and find below example of the HDR file(I changed here
S2A_OPER_SSC_L2VALD_29SMS____20190813.txt
the extension to .txt because i can't upload HDR file):
Uploading S2A_OPER_SSC_L2VALD_29SMS____20190813.txt…

smfcrts · 2019-11-18T09:24:57Z

@lnicola I sent a copy of attached HDR file,please have a look on it

lnicola · 2019-11-18T09:28:27Z

I'll think about it when I find some time, but I didn't see anything obviously wrong there.

smfcrts · 2019-11-18T09:32:01Z

Thank you @lnicola

smfcrts · 2020-01-09T14:33:31Z

@lnicola Hi, did you have a look on the HDR file, because i still have the same problem even if the L2A products are well generated.

lnicola · 2020-01-10T13:18:43Z

I couldn't see anything obviously wrong with the file. Can you upload the whole product somewhere (e.g. https://send.firefox.com) so I can pass it to my colleague who knows more about the L3A?

In the meanwhile, can you test with fewer input products in that command line? Even a single one, if it allows you. Also, what happens if you remove the first product from that list? Do you get an error on the second one as well?

smfcrts · 2020-01-10T13:40:23Z

I tested with only one product and it give the same error, and even if i remove the first one it give error on the second one. So which product should i upload?

lnicola · 2020-01-10T13:41:12Z

Either of them, I suppose.

smfcrts · 2020-01-10T13:45:21Z

Do you think it could be a problem of permission?

lnicola · 2020-01-10T13:46:28Z

It might be. Does cat /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR work from the account you're using to run the processor?

smfcrts · 2020-01-10T13:47:11Z

ok i'll try, what does cat do?

lnicola · 2020-01-10T13:47:44Z

It should display the contents of that file (it's an XML) to the console/screen. If you get a "Permission denied" error, it means it's not working.

smfcrts · 2020-01-10T13:48:57Z

yes it works.

lnicola · 2020-01-10T13:52:17Z

Can you also try this?

$ sudo yum -y install strace
$ strace -f /usr/bin/composite_processing.py --syntdate 20190901 --input [...] 2>&1 | tee log.txt # use the whole command line here

When it finishes, log.txt should contain the complete list of operating system calls done by the programs. If there's an OS error, it should show up in there.

PS: I just noticed that your original command line has this:

"/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR "

Note the extra space before the closing quote. Assuming the file doesn't have a space in its name, you'll get a file not found error on it.

On the other hand, you said you got the same error even after removing the first file, so...

smfcrts · 2020-01-10T13:55:47Z

the error mention that it couldn't read file, not the file doesn't exit.So I'll try right now what you suggest.

lnicola · 2020-01-10T13:56:44Z

the error mention that it couldn't read file, not the file doesn't exit.So I'll try right now what you suggest.

I'm pretty sure it's the same thing (it can't read a file that doesn't exist).

smfcrts · 2020-01-10T14:11:19Z

I couldn't find the log file, it show information on terminal

lnicola · 2020-01-10T14:12:15Z

There should be a log.txt file in the current directory (from the 2>&1 | tee log.txt part of the command.

smfcrts · 2020-01-10T14:19:20Z

PLease find attached the log file log.txt

lnicola · 2020-01-10T14:22:44Z

The file you've attached is truncated:

"log.txt" 2394L, 182930C

It looks like you've opened the file with a text editor (vi?), then copied the screen contents and pasted it to this file. The original file had 2394 lines and 180 KB, while the one above has 38 lines.

smfcrts · 2020-01-10T14:24:52Z

ah yes you're right

smfcrts · 2020-01-10T14:33:33Z

because i run the command on sen2agri-service user, so how can i get the log file without using vim.

smfcrts · 2020-01-10T14:36:08Z

Here is the file but i run the command line with user log.txt

lnicola · 2020-01-10T14:53:18Z

because i run the command on sen2agri-service user, so how can i get the log file without using vim.

You can copy it to your account with e.g. sudo cp ~sen2agri-service/log.txt ..

Here is the file but i run the command line with user log.txt

From the log: composite_processing.py: error: argument --synthalf is required

smfcrts · 2020-01-13T08:20:21Z

HI, I added synthalf to the command line.Please find attached the log.txt file
log.txt

lnicola · 2020-01-13T08:21:48Z

error: argument --outdir is required

Can you test it beforehand (without strace) to make sure that the command works?

smfcrts · 2020-01-13T08:26:36Z

here is the log file without strace log.txt

lnicola · 2020-01-13T08:27:49Z

All right, at least it's starting now. Can you try with strace? Maybe use strace -s256 -f composite_processing.py [...] this time, it will print the whole paths.

smfcrts · 2020-01-13T08:33:22Z

Here is the log file with strace -s256 -f log.txt

lnicola · 2020-01-13T08:41:58Z

So it's not a missing file nor a permission issue. Can you make an archive of the product and upload it somewhere like https://send.firefox.com?

smfcrts · 2020-01-13T09:10:21Z

I was not able to upload the zip file through the link you sent

lnicola · 2020-01-13T09:18:52Z

Is it larger than 1 GB? If so, you can make an account there, or try a different service.

smfcrts · 2020-01-13T09:29:30Z

I do, i created an account

smfcrts · 2020-01-13T12:00:21Z

https://send.firefox.com/download/411db81e23a5e49c/#wtTGt5QzxrbJ9Ay8Muc9WQ

smfcrts · 2020-01-16T13:51:38Z

HI @lnicola , please did you have a look on the link

lnicola · 2020-01-16T16:09:26Z

I was able to reproduce the failure, and it seems to be related to the format of the product metadata. Specifically, we're now using a version of MAJA that outputs a different product format. If you look at https://logiciels.cnes.fr/fr/node/57?type=desc, you'll see two versions, "TM" and "noTM". We use the former, but you might have the other one.

Your product is similar to MACCS (the predecessor of MAJA) products and we can actually open it, but we reject it because of an over-eager validation. You can test it by opening the product HDR file and changing

        <Source>
            <System>MAJA</System>
            <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
            <Creator_Version>3.2.2</Creator_Version>
            <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
        </Source>

to

        <Source>
            <System>MACCS</System>
            <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
            <Creator_Version>3.2.2</Creator_Version>
            <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
        </Source>

After making the change the processing started, at least. I couldn't let it finish because of unrelated reasons.

We will probably fix the validation in a future version, but it might be easiest for you to just change the string in the HDR files. This can be done in an automated way. I can guide you through the process, if you want to take that route.

smfcrts · 2020-01-16T21:26:50Z

Hi, what do you mean by the former one for maja output, it means you use the version TM or noTM. So i have to change MAJA by MACCS? That 's it Le jeu. 16 janv. 2020 à 17:09, Laurențiu Nicola <[email protected]> a écrit :

…

I was able to reproduce the failure, and it seems to be related to the format of the product metadata. Specifically, we're now using a version of MAJA that outputs a different product format. If you look at https://logiciels.cnes.fr/fr/node/57?type=desc, you'll see two versions, "TM" and "noTM". We use the former, but you might have the other one. Your product is similar to MACCS (the predecessor of MAJA) products and we can actually open it, but we reject it because of an over-eager validation. You can test it by opening the product HDR file and changing <Source> <System>MAJA</System> <Creator>MAJA_L2_NOMINAL_CHAIN</Creator> <Creator_Version>3.2.2</Creator_Version> <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date> </Source> to <Source> <System>MACCS</System> <Creator>MAJA_L2_NOMINAL_CHAIN</Creator> <Creator_Version>3.2.2</Creator_Version> <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date> </Source> After making the change, the processing start, at least. We will probably fix the validation in a future version, but it might be easiest for you to just change the string in the HDR files. This can be done in an automated way. I can guide you through the process, if you want to take that route. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ALRSPNN5FZ6MOZQEZTJWVC3Q6CBDPA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJETPEA#issuecomment-575223696>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRSPNOX4IBDVXHOVBROQBTQ6CBDPANCNFSM4JLJJRQQ> .

lnicola · 2020-01-16T21:29:30Z

Hi, what do you mean by the former one for maja output, it means you use
the version TM or noTM.

We use TM, I believe you have noTM.

So i have to change MAJA by MACCS? That 's it

Yes, in the System element. The Creator doesn't matter.

smfcrts · 2020-01-16T21:35:09Z

So i change maja to TM, it will solve the problem Le jeu. 16 janv. 2020 à 22:29, Laurențiu Nicola <[email protected]> a écrit :

…

Hi, what do you mean by the former one for maja output, it means you use the version TM or noTM. We use TM, I believe you have noTM. So i have to change MAJA by MACCS? That 's it Yes, in the System element. The Creator doesn't matter. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ALRSPNJSSQATUS7OCIYCIRLQ6DGTVA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFTWIA#issuecomment-575355680>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRSPNLWANNX3572KEINR33Q6DGTVANCNFSM4JLJJRQQ> .

lnicola · 2020-01-16T21:36:00Z

It should, but you might as well replace that string in the HDR files to keep the L2A products that you already have.

smfcrts · 2020-01-16T21:39:39Z

yes you're right, so how can i do it automatically Le jeu. 16 janv. 2020 à 22:36, Laurențiu Nicola <[email protected]> a écrit :

…

It should, but you might as well replace that string in the HDR files to keep the L2A products that you already have. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ALRSPNIIVCYRKNXUWVMJE53Q6DHMBA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFUJVQ#issuecomment-575358166>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRSPNM3ZVDFG3X7UDUQBB3Q6DHMBANCNFSM4JLJJRQQ> .

lnicola · 2020-01-16T21:44:20Z

Not really tested, but something like:

$ sudo su -l sen2agri-service
$ find /mnt/archive/maccs_def/morocco/l2a/ -maxdepth 1 -name "*.HDR" -print0 | xargs -0 sed -i 's|<System>MAJA</System>|<System>MACCS</System>|'

You might need -maxdepth 2 instead of 1.

smfcrts · 2020-01-16T21:46:16Z

ok i'll test it and i'll let you know, thank you Le jeu. 16 janv. 2020 à 22:44, Laurențiu Nicola <[email protected]> a écrit :

…

Not really tested, but something like: $ sudo su -l sen2agri-service $ find /mnt/archive/maccs_def/morocco/l2a/ -maxdepth 1 -name "*.HDR" -print0 | xargs -0 sed -i 's|<System>MAJA</System>|<System>MACCS</System>|' You might need -maxdepth 2 instead of 1. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ALRSPNLXNLCSAJWVIV4NZHLQ6DILLA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFVBPI#issuecomment-575361213>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALRSPNP7NTV5LA62P6Y4R6DQ6DILLANCNFSM4JLJJRQQ> .

smfcrts · 2020-01-17T15:04:17Z

I've got this error

cannot rename /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190614T112121_N0207_R037_T29SMS_20190614T114731.SAFE/sedI0PNHV: Permission denied

lnicola · 2020-01-18T11:00:28Z

I'm not sure what that would happen. You ran it under the sen2agri-service system account, right? Does it not have permissions on that directory?

smfcrts · 2020-01-29T10:07:41Z

yes i ran under sen2agri-service system account and it have permission on that directory

smfcrts · 2020-03-03T14:50:13Z

@lnicola I overcome the problem, I replace in the HDR files to keep the L2A products that I already have. But when i installed Maja3.2.2 TM and i downloaded the appropriate gipp file it give's me error that gip CMN00 not found.

Unable to read metadata (l3a,l3b) #28

Unable to read metadata (l3a,l3b) #28

Comments

smfcrts commented Nov 9, 2019

smfcrts commented Nov 13, 2019

lnicola commented Nov 15, 2019

smfcrts commented Nov 15, 2019

smfcrts commented Nov 18, 2019

lnicola commented Nov 18, 2019

smfcrts commented Nov 18, 2019

smfcrts commented Jan 9, 2020

lnicola commented Jan 10, 2020 • edited

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 10, 2020

smfcrts commented Jan 10, 2020

smfcrts commented Jan 10, 2020

lnicola commented Jan 10, 2020

smfcrts commented Jan 13, 2020

lnicola commented Jan 13, 2020

smfcrts commented Jan 13, 2020

lnicola commented Jan 13, 2020 • edited

smfcrts commented Jan 13, 2020

lnicola commented Jan 13, 2020

smfcrts commented Jan 13, 2020

lnicola commented Jan 13, 2020

smfcrts commented Jan 13, 2020

smfcrts commented Jan 13, 2020

smfcrts commented Jan 16, 2020

lnicola commented Jan 16, 2020 • edited

smfcrts commented Jan 16, 2020 via email

lnicola commented Jan 16, 2020

smfcrts commented Jan 16, 2020 via email

lnicola commented Jan 16, 2020

smfcrts commented Jan 16, 2020 via email

lnicola commented Jan 16, 2020

smfcrts commented Jan 16, 2020 via email

smfcrts commented Jan 17, 2020

lnicola commented Jan 18, 2020

smfcrts commented Jan 29, 2020

smfcrts commented Mar 3, 2020

lnicola commented Jan 10, 2020 •

edited

lnicola commented Jan 13, 2020 •

edited

lnicola commented Jan 16, 2020 •

edited