Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: generate only heavy vishraams in vishraam_first_letters column and strip line endings from first_letters #1776

Merged
merged 3 commits into from
Jun 4, 2020
Merged

build: generate only heavy vishraams in vishraam_first_letters column and strip line endings from first_letters #1776

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a0e6d8d
build: update to [email protected]
Harjot1Singh Jun 3, 2020
90cb86d
build: generate vishraam_first_letters with heavy first letters only
Harjot1Singh Jun 3, 2020
6f8e1d8
build: strip accents from first letters in database build
Harjot1Singh Jun 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions lib/build-database.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,16 @@ const memoize = require( 'memoizee' )
const { reduce, map, forEach } = require( 'fast.js' )
const { unload } = require( 'freshy' )
const { omit } = require( 'lodash' )
const { firstLetters, toEnglish, toHindi, toShahmukhi, toUnicode } = require( 'gurmukhi-utils' )
const {
firstLetters,
stripEndings,
stripVishraams,
stripAccentts,
bhajneet marked this conversation as resolved.
Show resolved Hide resolved
toEnglish,
toHindi,
toShahmukhi,
toUnicode,
} = require( 'gurmukhi-utils' )

const colors = require( './string-colors' )
const { findIndex } = require( './utils' )
Expand Down Expand Up @@ -316,17 +325,26 @@ const importLines = async trx => {
line_id: line.id,
source_id: sourceId,
gurmukhi,
first_letters: firstLetters( gurmukhi ),
vishraam_first_letters: firstLetters( gurmukhi, true, true ),
first_letters: [
stripEndings,
stripVishraams,
firstLetters,
].reduce( ( text, fn ) => fn( text ), gurmukhi ),
vishraam_first_letters: [
stripEndings,
// Retain heavy vishraams only
text => stripVishraams( text, { light: true, medium: true } ),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's odd that you have to choose what to strip (I'd expected all weights by default and code having to toggle heavy: false)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All weights are stripped by default, but if you supply options, then you choose which are (because I thought it'd more be confusing to specify which you'd exclude in a strip function)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes more sense to change parameters to affect the default. So if the default is light, medium, heavy true. You'd put in a parameter to make one of these false. But if you think this makes more sense to people, that's fine -- I am not up to date on these conventions lol

firstLetters,
].reduce( ( text, fn ) => fn( text ), gurmukhi ),
} )

transliterateAll( gurmukhi ).forEach( ( [ languageId, transliteration ] ) =>
transliterateAll( gurmukhi ).forEach( ( [ languageId, transliteration ] ) => (
transliterationData.push( {
line_id: id,
language_id: languageId,
source_id: sourceId,
transliteration,
} ) )
} ) ) )
} )
} )
) )
Expand Down
6 changes: 3 additions & 3 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
"commander": "^4.1.1",
"fast.js": "^0.1.1",
"freshy": "^1.0.4",
"gurmukhi-utils": "^2.3.1",
"gurmukhi-utils": "^3.0.0",
"is-ascii": "^1.0.0",
"knex": "^0.19.5",
"lodash": "^4.17.15",
Expand Down