Modern javascript localization with good

old gettext

https://c-3po.js.org/

Mostovenko Alexander

Company: EVO company

Twitter: @MostovenkoA

Github: github.com/AlexMostMe

About this talk

About this talk

Localization process in general and more about gettext format.

About this talk

Localization process in general and more about gettext format.

What is c-3po and how it helped us to improve existing process?

About this talk

Localization process in general and more about gettext format.

What is c-3po and how it helped us to improve existing process?

Continuous translations.

Our initial tasks

Our initial tasks

1. Localize frontend on several projects:

(cabinet, company sites)

Bigl.ua

others...

Our initial tasks

1. Localize frontend on several projects:

(cabinet, company sites)

Bigl.ua

others ...

2. Setup translation process when devs and translators can work independently.

Gettext (.po)

ICU

...Localization

formats

ICU - International Components for Unicodehttp://userguide.icu-project.org/

ICU - International Components for Unicodehttp://userguide.icu-project.org/

ICU features >>> gettext features

ICU is cool, but ...

ICU is cool, but ...

1. Poor tooling (editors for translators e.t.c).

ICU is cool, but ...

1. Poor tooling (editors for translators e.t.c).

2. Our backend already used gettext.

ICU - cool but ..."{gender_of_host, select, " "female {" "{num_guests, plural, offset:1 " "=0 {{host} does not give a party.}" "=1 {{host} invites {guest} to her party.}" "=2 {{host} invites {guest} and one other person to her party.}" "other {{host} invites {guest} and # other people to her party.}}}" "male {" "{num_guests, plural, offset:1 " "=0 {{host} does not give a party.}" "=1 {{host} invites {guest} to his party.}" "=2 {{host} invites {guest} and one other person to his party.}" "other {{host} invites {guest} and # other people to his party.}}}" "other {" "{num_guests, plural, offset:1 " "=0 {{host} does not give a party.}" "=1 {{host} invites {guest} to their party.}" "=2 {{host} invites {guest} and one other person to their party.}" "other {{host} invites {guest} and # other people to their party.}}}}"

GNU gettexthttps://www.gnu.org/software/gettext/

GNU gettexthttps://www.gnu.org/software/gettext/

1. Simple format.

GNU gettexthttps://www.gnu.org/software/gettext/

1. Simple format.

2. Has good ecosystem for translation process.

gettext idea

‘Hello world’

Source code

gettext idea

gettext(‘Hello world’)

Source code

gettext idea

gettext(‘Hello world’)

Source code

‘Hello world’

.pot file

gettext idea

gettext(‘Hello world’)

Source code

msgid: ‘Hello world’msgstr: ‘’

.po file

‘Hello world’

.pot file

gettext idea

gettext(‘Hello world’)

Source code

msgid: ‘Hello world’msgstr: ‘Здоровеньки були’

.po file

‘Hello world’

.pot file

gettext idea

gettext(‘Hello world’)

Source code

msgid: ‘Hello world’msgstr: ‘Здоровеньки були’

.po file

‘Hello world’

.pot file

gettext idea

gettext workflow

Code .pot Extracting translation strings from code (AST traverse)extract

gettext workflow

Code .pot Extracting translation strings from code (AST traverse)

.pot .po Merge new extracted translations from .pot file with existing. msgmerge

extract

merge .po.po

gettext workflow

Code .pot Extracting translation strings from code (AST traverse)

.pot .po Merge new extracted translations from .pot file with existing. msgmerge

extract

merge

Translator adds translationstranslation

.po.po

.po.po.po.po.po.po

gettext workflow

Code .pot Extracting translation strings from code (AST traverse)

.pot .po Merge new extracted translations from .pot file with existing. msgmerge

extract

merge

Translator adds translationstranslation

.po.po

.po.po.po.po.po.po

resolve CodePlacing translations back to the code (by locale).po.po.po

.po file structure

Example .po file

msgid ""

msgstr ""

"Project-Id-Version: c 3po-webpack-start\n"

"PO-Revision-Date: 2017-02-01 22:22+0200\n"

"Language-Team: Ukrainian\n"

"Language: uk\n"

"MIME-Version: 1.0\n"

"Content-Type: text/plain; charset=UTF-8\n"

"Content-Transfer-Encoding: 8bit\n"

"Plural-Forms: nplurals=2; plural=(n!=1);\n"

#: app.js:6

msgid "Choose locale"

msgstr "Оберіть локаль"

#: app.js:10

msgid "${ 0 } second"

msgid_plural "${ 0 } seconds"

msgstr[0] "${ 0 } секунда"

msgstr[1] "${ 0 } секунди"

msgstr[2] "${ 0 } секунд"

#: app.js:14

msgid "webpack with c-3po localization demo"

msgstr "Демо локалізації з c-3po та webpack"

msgid ""

msgstr ""

"Project-Id-Version: c 3po-webpack-start\n"

"PO-Revision-Date: 2017-02-01 22:22+0200\n"

"Language-Team: Ukrainian\n"

"Language: uk\n"

"MIME-Version: 1.0\n"

"Content-Type: text/plain; charset=UTF-8\n"

"Content-Transfer-Encoding: 8bit\n"

"Plural-Forms: nplurals=2; plural=(n!=1);\n"

#: app.js:6

msgid "Choose locale"

msgstr "Оберіть локаль"

#: app.js:10

msgid "${ 0 } second"

msgid_plural "${ 0 } seconds"

msgstr[0] "${ 0 } секунда"

msgstr[1] "${ 0 } секунди"

msgstr[2] "${ 0 } секунд"

#: app.js:14

msgid "webpack with c-3po localization demo"

msgstr "Демо локалізації з c-3po та webpack"

Headers

msgid ""

msgstr ""

"Project-Id-Version: c 3po-webpack-start\n"

"PO-Revision-Date: 2017-02-01 22:22+0200\n"

"Language-Team: Ukrainian\n"

"Language: uk\n"

"MIME-Version: 1.0\n"

"Content-Type: text/plain; charset=UTF-8\n"

"Content-Transfer-Encoding: 8bit\n"

"Plural-Forms: nplurals=2; plural=(n!=1);\n"

#: app.js:6

msgid "Choose locale"

msgstr "Оберіть локаль"

#: app.js:10

msgid "${ 0 } second"

msgid_plural "${ 0 } seconds"

msgstr[0] "${ 0 } секунда"

msgstr[1] "${ 0 } секунди"

msgstr[2] "${ 0 } секунд"

#: app.js:14

msgid "webpack with c-3po localization demo"

msgstr "Демо локалізації з c-3po та webpack"

Translations

"Plural-Forms:

nplurals=2;

plural=(n!=1);\n"

Plural-forms header

"Plural-Forms:

nplurals=2;

plural=(n!=1);\n"

Plural-forms header

Plurals count

"Plural-Forms:

nplurals=2;

plural=(n!=1);\n"

Plural-forms header

Plurals formula

#: app.js:6

msgid "Choose locale"

msgstr "Оберіть локаль"

1 to 1 translations

#: app.js:10

msgid "${ n } second"

msgid_plural "${ n } seconds"

msgstr[0] "${ n } секунда"

msgstr[1] "${ n } секунди"

msgstr[2] "${ n } секунд"Plural forms example

#. Notes for the translator

#: app.js:6

#, flag

msgid "Choose locale"

msgstr "Оберіть локаль"

Comments

msgctx "email"

msgid "Hello"

msgstr "Привіт"

msgctx "main page"

msgid "Hello"

msgstr "Вітаємо"

Context example

c-3pohttps://c-3po.js.org/

c-3po

Code .pot Extracting translation strings from code (AST traverse)extract

resolve CodePlacing translations back to the code (by locale).po.po.po

Library:

https://github.com/c-3po-org/c-3po

Core functionality.

Translation functions (tags):

t, ngettext, gettext, jt

Babel-plugin:

https://github.com/c-3po-org/babel-plugin-c-3po

Extract, Resolve, Validation

Key ideas

Key idea 1. Tagged template strings for formatting

c-3po translations are based on tagged template strings

import { t } from 'c-3po'

const name = 'Mike';

console.log(t`Hello ${name}`);

What is a template string?

What is template string?

console.log('Hello ' + name)

console.log(`Hello ${name}`)

What is a tagged template string?

const name = ‘Mike’

t`Hello ${name}`

function t(strs, ...exprs) {

// strs -> [‘Hello’, ‘’]

// exprs -> [‘Mike’]

}

Valid javascript

t `Hello ${name}`

t(5)`Hello ${name}`

t(5) `Hello ${name}`

Why just not simple functions but tags?

Why just not simple functions but tags?

c-3po must work without transpilation

Why just not simple functions but tags?

// Without babel transpile !!!

const name = ‘Mike’

t(`Hello ${name}`)

function t(str) {

// strs -> ‘Hello Mike’

// no chance to get ‘Hello ${name}’ msgid here :(

}

Existing solutions

Force you to use sprintf formatting(i18next, jed e.t.c)

sprintf

gettext("Hello %s", user)

gettext("%2$s %3$s a %1$s", "cracker", "Polly", "wants")

gettext("Current timestamp: %d", Date.now)

sprintf

● Alien formatting for js

sprintf

● Alien formatting for js● No reason to use es6 template strings

sprintf

● Alien formatting for js● No reason to use es6 template literals● Extra CPU work on the client (sprintf parsing)

Key idea 2. Precompile translation on a build step

Traditional translations resolve flow

Traditional translations resolve flow

Load assets to the client (browser)

Traditional translations resolve flow

Load assets to the client (browser)

Resolve locale (from cookie, params e.t.c)

Traditional translations resolve flow

Load assets to the client (browser)

Resolve locale (from cookie, params e.t.c)

Fetch locale data

Traditional translations resolve flow

Load assets to the client (browser)

Resolve locale (from cookie, params e.t.c)

Fetch locale data

Apply translations

Wanted translations resolve flow

Wanted translations resolve flowLoad assets to the client

(browser)

Wanted translations resolve flowLoad assets to the client

(browser)

That’s it, they are already localized !!!

Benefits of precompiled translations:

Benefits of precompiled translations:

● Smaller bundle size

Benefits of precompiled translations:

● Smaller bundle size● Less work on the client

Benefits of precompiled translations:

● Smaller bundle size● Less work on the client● No wait until translations are loaded and applied

Key idea 3.Works with and without babel

Demo on jsfiddle

Key idea 4. Can extract and resolve translations from

everything that works with babel (jsx)

Extract and resolve translations with simple config

Extract:

{ "extract": { "output": "extract.pot" } }

Resolve:

{ "resolve": { "translations": "uk.po" } }

https://c-3po.js.org/quick-start.html

Key idea 5. Efficient dev and prod setup

Requirements for the dev setup

Requirements for the dev setup● Faster builds

Requirements for the dev setup● Faster builds● Simple integration

Dev setup with c-3po

Dev setup with c-3po● Use whole c-3po lib (import c-3po).

Dev setup with c-3po

● Use whole c-3po lib (import c-3po).

● Babel plugin only for extraction translations

Requirements for the prod setup

Requirements for the prod setup● Smaller result assets

Requirements for the prod setup

● Smaller result assets

● Faster locale load

Prod setup with c-3po

Prod setup with c-3po● Use c-3po mock lib (alias in webpack e.t.c).

Prod setup with c-3po

● Use c-3po mock lib (alias in webpack e.t.c).

● Babel plugin for transformations.

Prod setup with c-3po

● Use c-3po mock lib (alias in webpack e.t.c).

● Babel plugin for transformations.

● Separate build for each locale.

Key idea 6.Validate translation strings

Validation problem

msgid "http://some.random.domain.link"msgstr ?

msgid ""msgstr ?

msgid "translate ${ test ? result1() : result2() }"msgstr ?

msgid "5"msgstr ?

c-3po validation (empty str)

t``

Module build failed: SyntaxError: Can not translate empty string

5 |

6 |

> 7 | t``

| ^

c-3po validation (only numbers)

t`3243243`

ERROR in Error: Module build failed: SyntaxError: Can not translate '3243243'

2 |

3 |

> 4 | t`3243243`

| ^

c-3po validation (programs inside translations)

t`Translate ${someVar ? callFn1() : callFn2()}`

ERROR in Error: Module build failed: SyntaxError: You can not use

ConditionalExpression '${someVar ? callFn1() : callFn2()}' in localized

strings

2 |

3 |

> 4 | t`Translate ${someVar ? callFn1() : callFn2()}`;

| ^

5 |

c-3po validation (only var wrap)

t`${someVar}`

ERROR in Error: Module build failed: SyntaxError: Can not translate '$

{ someVar }'

2 |

3 |

> 4 | t`${someVar}`;

| ^

msgid:’${someVar}’

Validate not translated strings

Validation (no translated str)

t`non translated str`

ERROR in Error: Module build failed: SyntaxError: No "non translated str" in

"uk.po" file

2 |

> 3 | t`non translated str`;

https://c-3po.js.org/configuration.html#configresolveunresolved-string-one-of-fail-warn-skip

Key idea 7.Can use any default locale in sources

(standard gettext uses only English locale)

The problem

Standard ngettext accepts only 2 plural forms

char * ngettext(const char * msgid1, const char * msgid2, unsigned long int n);

Ukrainian/Russian strings in sources

How to be with plurals?

Ukrainian/Russian strings in sources

How to be with plurals? Standard ngettext accepts only 2 plural forms?

Ukrainian/Russian strings in sources

How to be with plurals? Standard ngettext accepts only 2 plural forms? What if we have more or less than 2 plural forms?

Ukrainian/Russian strings in sources

How to be with plurals? Standard ngettext accepts only 2 plural forms? What if we have more or less than 2 plural forms?

ngettext for other locales ( plural forms > 2 )

ngettext for other locales ( plural forms > 2 )

Doc - https://c-3po.js.org/ngettext.html

Change defaultHeaders setting to:

'plural-forms':'nplurals=3; plural=(n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);'

ngettext for other locales ( plural forms > 2 )

Demo on jsfiddle

Doc - https://c-3po.js.org/ngettext.html

Change defaultHeaders setting to:

'plural-forms':'nplurals=3; plural=(n%10==1 && n%100!=11 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);'

ngettext(msgid`${n} банан`, `${n} банана`, `${n} бананів`, n)

c-3po tags and functions

t - tag gettext

Demo on jsfiddle

Doc - https://c-3po.js.org/tag-gettext--t-.html

t - tag gettext

t`Hello ${name}` < Usage

Demo on jsfiddle

Doc - https://c-3po.js.org/tag-gettext--t-.html

t - tag gettext

t`Hello ${name}`

msgid "Hello ${ name }"msgstr "Привіт ${ name }"

< Usage

< Extract

Demo on jsfiddle

Doc - https://c-3po.js.org/tag-gettext--t-.html

t - tag gettext

t`Hello ${name}`

msgid "Hello ${ name }"msgstr "Привіт ${ name }"

< Usage

< Extract

`Привіт ${name}` < Resolve

Demo on jsfiddle

Doc - https://c-3po.js.org/tag-gettext--t-.html

ngettext - plural forms

ngettext - plural forms

ngettext(msgid`${n} time clicked`, `${n} times clicked`, n) < Usage

ngettext - plural forms

ngettext(msgid`${n} time clicked`, `${n} times clicked`, n)

msgid "${ n } time clicked"msgid_plural "${ n } times clicked"msgstr[0] "${ n } time clicked [translated]"msgstr[1] "${ n } times clicked [translated]"

< Usage

< Extract

Forgot msgid ?

ERROR in Error: Module build failed: SyntaxError: First argument must be tagged

template expression. You should use 'msgid' tag

2 |

3 |

> 4 | ngettext(`${hours} hour`, `${hours} hours`, hours);

| ^

5 |

Demo on jsfiddle

Doc - https://c-3po.js.org/ngettext.html

ngettext - resolve

function _tag_ngettext(n, args) {

return args[+(n != 1)];

}

_tag_ngettext(n,

[n + " time clicked [translated]", n + " times clicked [translated]"]));

Demo on jsfiddle

Doc - https://c-3po.js.org/ngettext.html

First contribution !!! jt (jsx-gettext)

First contribution !!! jt (jsx-gettext)

jt - jsx gettext

jt - jsx gettext

const btn = <button key="btn">{ t`me`

}</button>;

<span>{jt`Click ${ btn }`}</span>

< Usage

jt - jsx gettext

const btn = <button key="btn">{ t`me`

}</button>;

<span>{jt`Click ${ btn }`}</span>

< Usage

< Extractmsgid "Click ${ btn }"

msgstr "Click ${ btn } [translated]"

jt - jsx gettext

const btn = <button key="btn">{ t`me`

}</button>;

<span>{jt`Click ${ btn }`}</span>

< Usage

< Extractmsgid "Click ${ btn }"

msgstr "Click ${ btn } [translated]"

['Click ', btn, ' [translated]'] < Resolve

gettext

gettext

gettext('simple gettext'); < Usage

gettext

gettext('simple gettext'); < Usage

< Extractmsgid "simple gettext"

msgstr "simple gettext [translated]"

gettext

gettext('simple gettext'); < Usage

< Extractmsgid "simple gettext"

msgstr "simple gettext [translated]"

'simple gettext [translated]' < Resolve

Why add gettext func if we have ‘t’ tag?

Legacy - *.coffee *.eco (100 K+ LOC)

1. Wrap string in gettext inside *.coffee *.eco

2. Add babel loader after coffee loader to the

webpack pipeline

3. Profit !!!

Even more features

Aliasinghttps://c-3po.js.org/aliasing.html

Babel plugin will extract aliased translation

import { t as i18n } from 'c-3po'

i18n`this translation will work`

Multilinehttps://c-3po.js.org/multiline-strings.html

import { t } from 'c-3po';

function test(name) { return t`multi line string with multiple line breaks and with formatting ${name}`}

Multiline (auto dedent)

Translator comments

Translator comments// translator: some description for the extracted string. t`some string`

Translator comments// translator: some description for the extracted string. t`some string`

#. some description for the extracted stringmsgid "some string"msgstr ""

Summary

Summary● Native js tagged template strings for

formatting

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step● Works with and without babel

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step● Works with and without babel● Extracts and resolves translations from

everything that works with babel (jsx)

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step● Works with and without babel● Extracts and resolves translations from

everything that works with babel (jsx)● Efficient dev and prod setup

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step● Works with and without babel● Extracts and resolves translations from

everything that works with babel (jsx)● Efficient dev and prod setup● Validation for translated strings

Summary● Native js tagged template strings for

formatting● Can precompile translation on a build step● Works with and without babel● Extracts and resolves translations from

everything that works with babel (jsx)● Efficient dev and prod setup● Validation for translated strings● Can use any default locale in sources (not

only English)

Continuous translations process

Devs VCSrepo

Continuous translation process

Translator

Devs VCSrepo

Continuous translation processpush

Translator

Devs VCSrepo

Continuous translation processpush notify

Translator

Devs VCSrepo

Continuous translation processpush

translate

notify

Translator

Devs VCSrepo

Continuous translation processpush

translate

notify

update Translator

Tools for translators

Poeditor - https://poeditor.com/

Transifex - https://www.transifex.com/

Weblate - https://weblate.org

Devs VCSrepo

Continuous translation processpush

push

pull

update Translator

notify

translate

Weblate example

Weblate example

Weblate - https://docs.weblate.org/en/latest/

Links

● c-3po doc - https://c-3po.js.org/

● c-3po quick start - https://c-3po.js.org/quick-start.html

● gettext - https://www.gnu.org/software/gettext/

● ICU - http://userguide.icu-project.org/

● Weblate - https://weblate.org

Thx :)