feat: Add translations

[revanced-bot]

Sync translations from crowdin.com/project/revanced

Issues to sort out

• Incorrect strings on Crowdin source
• Support for all YouTube supported languages
• Non-translatable parts in strings (add Crowdin project placeholder tag of <notranslate> to indicate do not translate) edit: not available with project plan, and using a random placeholder tag gives a different and undesired behavior
• Fix cron sync to run once on first of the month
• Add Crowdin quality check regular expression edit: With Crowdin identifying the strings as Android, Crowdin appears to handle quotations correctly
• Try adding update_as_unapproved to project config to preserve existing translations during GitHub -> Crowdin sync edit: this change will not be needed.
• Add Crowdin regex preprocessor to strip patches string enclosing xml elements
• Modify the sync script to copy or rename /he/Strings.xml to /iw/Strings.xml after pulling from Crowdin to GitHub. YT uses the older iw and Crowdin uses the newer he. If both old and new language files are on an phone installation then only one code might be used and Hebrew strings will be partially broken. Copying to both or using only iw fixes this. Edit: For now this is fixed by using a Crowding he -> iw mapping, and this works for Manager/YT/Twitch since none use any he resources. If a new app is added in the future that does use he then this may need to change, but for now this is simple and it works.
• Modify the sync script to move /ur-IN/Strings.xml to /ur/Strings.xml after pulling from Crowdin to GitHub. Then remove ur-PK from Crowdin (use one Urdu translation for all countries). Edit: a Crowdin ur-IN to ur mapping was added
• Optional: Add to Crowdin project, an API key for Google Translate or some other free machine translate service to compliment and provide for more languages than the existing Crowdin machine translate. Edit: Skipping this. Can upload other machine translated languages by hand if needed.
• Fix Crowdin pushing empty strings on sync Edit: fixed
• Change cron job to run on dev branch and not main branch
• Fix the last small issues with merging strings during patching

[LisoUseInAIKyrios]

Is the cron job setup to run every hour? I may need an adjustment.

[Ushie]

The cron job runs at the first of every month, as requested by osumatrix

[Ushie]

Unless GitHub interpreted it differently - Seems like it did

[LisoUseInAIKyrios]

I think it's one of the * values, where it's uploading every hour of the first day of the month.

If the cron isn't changed, it might stop syncing on April 2nd.

[Ushie]

Odd, GitHub suggests using Crontab.guru which we already have, yet it doesn't follow it correctly

[Ushie]

I think it's one of the * values, where it's uploading every hour of the first day of the month.

If the cron isn't changed, it might stop syncing on April 2nd.

Right, because the minute value resets every hour

Opened a PR #2967

[LisoUseInAIKyrios]

Any idea why some strings changed here are not syncing with Crowdin?

for example, here on GitHub dev a string is:
<string name="revanced_hide_keyword_toast_invalid_length">Invalid keyword. \'%1$s\' is less than %2$s characters</string>

But on Crowdin it still shows the old (incorrect) prior value:
<string name="revanced_hide_keyword_toast_invalid_length" formatted="false">Invalid keyword. \'%s\' is less than %s characters</string>

[Ushie]

Oh, just realized something, for the repo to be syncing to Crowdin, the Sync Crowdin needs to be run, we may need to add two different workflows if you want syncing down to the repo to continue being monthly - but this may be unrelated to your issue as the workflows have just run, going to check now

[oSumAtrIX]

@Ushie Shouldn't the same workflow push sources? I think that's a parameter of the action that is currently set to true.

[Ushie]

The way things currently are, the, source in crowdin will get updated monthly, it should be more frequent so that people can translate on the go

[oSumAtrIX]

That seems reasonable. In that case the workflow could skip over the step that pulls translations, but it seems unnecessary not to, which is why the workflow can run weekly.

[LisoUseInAIKyrios]

If there is a way to just push changes to Crowdin, that could be done daily.

Because if someone is translating old outdated strings they are wasting their time since their translations will be outdated when the GitHub -> Crowdin push later occurs. (This is going in right now, since Crowdin still has old strings).

It also would be nice if no strings were changed on Dev, then do not push to Crowdin. That would prevent the Crowdin activity page from showing meaningless "ReVanced Bot compiled but changed nothing" events.

[Ushie]

Syncing to crowdin does not have to be bound by a cron, it can be triggered by modifications to the source file, this also solves the problem you stated at the end of your message

[LisoUseInAIKyrios]

Yeah that is true.

Did we figure out why it's not pushing changed strings from GitHub -> Crowdin?

[Ushie]

I got carried away with something meaningless and forgot to look, I'm at work now, I'll see if I can RDP home and check

[LisoUseInAIKyrios]

Duplicate keys are a problem with Crowdin. Previously there was a debug string in multiple apps, and it causes Crowdin to get confused and every time strings were synced (it would randomly pick one of the duplicate strings, and delete the prior translations of the other strings).

This only happens when the same string path is used in multiple apps.

[LisoUseInAIKyrios]

There's some issues with the langauge codes and the merging of strings:

YT uses a mix of 2 digit locale, and some 2 digit + 3 digit region:

But AddResourcePatch only uses the 2 digit locale:
Locale.getISOLanguages().asSequence().map { "values-$it" }.forEach { addStringResources(it) } // 2 digit iso-639 code

How to handle this?

[LisoUseInAIKyrios]

For now, add a hard coded list of the 5 digit locales found in YT?

so pt-rPT and the others are added in?

[oSumAtrIX]

The add resources patch is not specific to YouTube so making YouTube specific changes to it doesn't sound good. Are these three digit country code standardized? If so, the add resources patch could be changed to support that.

[LisoUseInAIKyrios]

Yes the language + region 5 digit codes are standardized, but the issue is YT using a mix of 2 and 5 digit codes.

So when merging the files, AddResourcePatch has to somehow figure out how to sometimes go from 5 digit codes to 2 digit codes so it merges the right file. That is not very app agnostic either.

Just to get this working, I am going to try hard coding these few specific languages to a list of of optional 5 digit languages. this can improved later. This could be improved by allowing specific patched apps to provide their own mapping (similar to how Crowdin has language mappings). For now I think the priority is getting this working and worry about improving the design semantics later.

[oSumAtrIX]

I think a solid solution would for the add resources patch to collect all available translations in addition to the Java language codes. That was it will support any app. The only issue is, if an app is using a language code but has no translations for it, then the patch would never know that this translation exists.

[LisoUseInAIKyrios]

This is not quite as easy as using a hard coded list of regions to allow (or at least, I can't figure out an easy way to modify add resource patch).

I think the solution here is:

• export strings from Crowdin in 5 digit locale (this is already being done)
• during patching, check if the target app has a string file for a given locale.
• if the target app does not have the translations string file, then create an empty file in that place
• merge the translated strings (the target file will be either a new file just created, or it will be an existing file the target app already had).

@oSumAtrIX any suggestions?

For now, short term solution right now is to only use translations for languages with 2 digit codes (which is everything except Brazil Portuguese and all the Chinese translations)

[LisoUseInAIKyrios]

Actually, for now this can probably work by specifying a map of Crowdin -> Android locale names.

I'll try that as it's simple and will get this out the door. Can make this better later.

[LisoUseInAIKyrios]

Only one last thing:

"Change cron job to run on dev branch and not main branch"

Anyone want to give this a go? I'm not so familiar with what needs to be done here.

[LisoUseInAIKyrios]

I don't understand why, but if the Crowdin setting "do not export untranslated strings" is enabled, it fails to export a language unless all strings are translated. So it breaks translations if it's not 100% translated.

So I have that setting off for now.

[LisoUseInAIKyrios]

The automatic running of pushing strings needs to be fixed up so it happens anytime the dev strings changes (I think specifying the dev branch will fix that).

Other than that, I think this is good to go.

[oSumAtrIX]

Perhaps this is what should be done:

[oSumAtrIX]

Refer to e2a9b1e#r143615849

Are you sure the syntax is right?

[LisoUseInAIKyrios]

Refer to e2a9b1e#r143615849

Are you sure the syntax is right?

I have no idea what is the right syntax. I don't use yaml often.

Maybe change this to push to crowdin on every commit to dev? That's usually not more than once a day or so.

Everything else seems ok so could merge and fix the crowdin sync later

[oSumAtrIX]

That's the idea. It should sync every commit to dev but only if committed to a string path. I'll take a look at the workflows in a bit. In yaml you can create objects and lists and nest them in other objects:

list:
 - item1:
   propertyOfItem1: 1
 - item2:
   - item2_1
   

[oSumAtrIX]

I think it should work now

[welcome]

Thank you for contributing to ReVanced. Join us on Discord to receive a role for your contribution.

[kazimmt]

Yay 🎉🎉
Finally 💖