FAQs

Translators are very familiar with this format. It is the txt version (database version) of a Translation Memory. Every time a translator saves a segment, it is creating an equivalent of the source sentence in the target language. This is wonderful for machine learning as translators create parallel data. It is the basis of many developments at PangeaMT.

A neural network will find the relations between the sentences and similarities, to the syllable and letter level if necessary (this is a very useful feature in neural training called BPE). It is also responsible for neural machine translation’s success and higher acceptability than the previous n-gram based “Statistical Machine Translation”, that is still successful with short sentences because its higher “memorizing” capabilities, as explained in our first neural machine translation developments publications back in 2017. Our findings at the time proved that a short sentence with less than 9-10 words could probably be translated more accurate with a statistical system than with a neural system. As system have improved over the years, the gap between one and the other has shortened. However, it is true that when ecommerce websites only need to translate a couple of words, and those words have been part of the training data, a statistical system will recall them more quickly and efficiently. A neural system, however, will reconstruct the sentence with a more human fluency.

Therefore, if you ask if our machine translation is good with Czech language, the answer is YES! We have the team, technology and data to make your MT engine run smoothly and produce high quality translation, millions of words!

Companies cannot underestimate (and often only begin to understand) the effort required in data cleaning when they begin to export bilingual (parallel) data for machine learning. Due to CAT limitations and features, noise can enter in a sentence in the shape of unwanted code, but the concept of data cleaning goes beyond removing in-lines, as explained in Q14.Some typical examples of data cleaning which is necessary were presented at Japan Translation Federation 2011 as part of our Japanese Syntax-Based Machine Translation Hybrid.

Anybody who has been in the translation industry long enough has come across some kind of “bad” TM. This could come in many shapes, from simply being a bad translation to being terminologically inaccurate, etc. Fortunately for our users, this kind of data cleaning has become part of PangeaMT’s standard cleaning procedure.

Some of the basic cleaning cycles are described below. They take into account some procedures which have been automated to system owners so they can rest assured that

• their initial training data is clean before engine training in order to achieve the best results possible
• any future post-edited material also goes through a virtuous cleaning cycle in order to check any noise that may be introduced in the system and thus affect re-trainings.

Greatly. As part of its mission to organize the world’s information, Google takes on translation as an informative offering, very cleverly done, but also generalistic. Its translation application is actually state-of-the-art, but it attempts to a portal that can handle translation requests for every topic.  Just as we found during our initial efforts with MT, Google decided to ditch rule-based approached to machine translation and it embraced statistical methodologies for translation. This is not so rare as scientists at both organizations are strong SMT advocates and there has been a degree of academic collaboration beween PangeaMT’s core R&D team and some of Google’s lead researches.

While Google focuses in making as much general information available and has wide resources to gather trillions of words of data, PangeaMT’s approach is to build a custom application for your particular needs with your preferential terminology, expressions and word usage, i.e. a machine translation application that translates like you wish. Training data is typically provided by you and enhanced by PangeaMT. Additional language data may be added so there are sufficient lexical resources in the engine. A Language Model may be specifically built for you or adapted for your purposes. Furthermore, PangeaMT’s system is designed to fit and aid current TM-based systems by translating TMX or xliff formats, something that Google Translate cannot do (it only translates plain text).

By translating files and not plain text, PangeaMT plugs in directly and easily into any localization or Knowledge Base workflow. TMX or XLIFF files can be easily post-edited using most (if not all) of yesterday’s CAT tools as editing tools.

In short, PangeaMT’s developments fit in current translation environments and automate current processes, whereas Google Translate is an informative engine.

If you remember the translator’s resistance to use CAT tools in the late 90’s (I do, I freelanced in the UK those days), you will get an idea of how post-editing may be viewed in 2010 and onwards.

Every “new” technology (or technique) always faces resistance. There is nothing we love more than security, certainties. In the translation world, this means the relatively long (?) cycle of learning CAT tools. We do not mean the omnipresent tools which are marketed so well, but also the lesser known tools that can also do the job pretty well. Some have made a conscious effort to offer plug-ins to MT (Swordfish, from maxprograms.com) and, like PangeaMT are designed on open standards with a “no lock-in” mentality. Now you are telling your translators to “correct” machine output and at a lower fee. Back to the 90’s…

Indeed there may be some resistance from long-standing translators. Recent graduates are still trained in translation theory linked to computer-assisted tools.

However, now that end users can in certain contexts play with already built systems, even if not fully customized to their domain, the postediting stage may become a selection criterion. Before full deployment, corporations, organizations, industries and LSPs usually  run quantifiable evaluation pilots to become accustomed to post-editing tasks, identifying recurrent changes for automatic solutions, and base expectations about quality and pricing on objective data. This means that future post-editors, whether they are current translators or new recruits need to be involved at some stage prior to deployment.

Post-editing is still a nascent profession and experimentation with MT systems is required to gain a set of skills relative to each language. For example, if you are running an engine which lacks general “world” vocabulary, or very usual words. This may be annoying in large-scale systems and we run statistical dictionary modules to add words which were not in your training corpus. Nevertheless, post-editors in localization or documentation environments may think it is better to leave unknown terms in the source language so they can run “search & replace” and post-edit quickly. Thus, do expect the same resistance any new technology encounters but explain the benefits of it. Human translation cannot resolve the issues in terms of speed and cost in the digital content era. There are simply not enough qualified translators and even if there were, the logistics and costs of translating 50.000 words in a day or two would make project managers crazy. These pressures may also explain the high human “turn around” in the language industry. The truth is that with the advent of online translation services and desktop services and MT server engines, machines translate more words than humans already….

Our engines pass several tests (including post-editing trials) before delivery to ensure your investment is worth the money and the time from the first day. Implementation is smooth and can be virtualized, installed in a server in your organization, work on an intranet or customized to your specific needs.

Yes, your costs in translation (and more importantly time-to-market) will be reduced significantly. You will be able to notice that from the first week. Nevertheless, do not forget that engines will improve over time and that a few re-training exercises (at least 1 a year) are highly recommended. Post-edited material is a very good candidate for engine retraining since it reflects your day-to-day needs.

Market trends point to a rate 60% of full translation fee for post-editing good MT output… but this should be considered a guideline rather than a fixed rule. There are many cases both sides of that figure. We cannot tell what is the best post-editing fee in every circumstance and domain. Nevertheless, LSP and content writers are taking that figure as a reference and working on production improvement figures. We know post-editing is also being paid by the Kb, by the segment or by the time.

Any experienced MT (or at least reader or post-editor of MT) will tell you that Statistical MT flows much better than the traditional rule-based systems (RB). Anyone who has studied or implemented SMT will tell you implementation and development times are much shorter (thus ROI). RB is usually bought as a cheaper package once a company has done all the programming of rules and built in the syntactics. The package is closed and customization (or hybridizing) is a longer process. Statistical MT can improve by Coupling Reordering and Decoding, and by applying many many other mathematical and statistical formulas which will determine with certainty that a word (or series or combinations of words) happen together in comparison with other words. Read below if you need a comprehensive listing.

• SMT only needs to learn parallel corpus to generate a translation engine. In contrast, RBMT needs a great deal of knowledge external to the corpus that only linguistic experts can generate, e.g. superficial categorization, syntax and semantics of all the words of one language in addition to the transfer rules between languages. These latter rules are entirely dependent on language pair involved and are not generally as studied as the characterization of each separate language. Defining general transfer rules is not easy, and so multiple rules according to individual cases need to be defined, especially between languages with very different structures, and / or when the source language has greater flexibility for the management of structural objects in a sentence.

• An SMT system is developed rapidly if have the appropriate corpus is available, making it more profitable. A RBMT system, in turn, requires great development and customization costs until it reaches the desired quality threshold. Packaged RBMT systems have already been developed by the time the user purchases them:  most users approach MT by purchasing “out of the box” or “server ready” programs. The program works and will work in a certain way, but it is extremely difficult to reprogram models and equivalences. Above all, RBMT deployment generally is a much longer process involving more human resources. This is one key issue when companies calculate full implementation cost.

• SMT is adapted to automatically be retrained to situations not seen before (hitherto unknown words, new expressions that are translated differently from the way they were previously translated, etc.). RBMT is ‘re-trained’ by adding new rules and vocabulary among other things, which in turns means more time / increased handling by “expert humans”.

• SMT generates more fluent translations (fluency), although pure statistical systems may offer less consistency and less predictable results if the training corpus is too wide for the purpose. RBMT, however, may not have found the surface / syntactic information or words suitable for analyzing the source language, or does not know the word. This will prevent it from finding an appropriate rule.

• While statistical machine translation works well for translations in a specific domain, with the engine trained with bilingual corpus in that domain, RBMT may work better for more general domains.

• It is clear need for powerful computing in SMT in terms of hardware to train the models. Billions of calculations need to take place during the training of the engine and the hardware and computing knowledge required for it is highly specialized. However, training time can be reduced nowadays thanks to the wider availability of more powerful computers. RBMT requires a longer  deployment and compilation time by experts so that, in principle, building costs are also higher.

• SMT generates statistical patterns automatically, including a good learning of exceptions to rules. As regards to the rules governing the transfer of RBMT systems, certainly they can be seen as special cases of statistical standards. Nevertheless, they generalize too much and cannot handle exceptions.

• Finally, SMT systems can be upgraded with syntactic information, and even semantics, like the RBMT. But in this case, the statistical patterns that a SMT would learn can be seen as a more general type of transfer rules, although currently the inclusion of such information in current systems does not provide significant improvements.

• A SMT engine can generate improved translations if retrained or adapted again. In contrast, the RBMT generates very similar translations between different versions.

Most people will tell you that 2 million words is the bare minimum you can provide for a “bare bones” engine and some kind of automation within a domain – but do not expect great results if you are dealing with texts they may include a lot of new, unexpected words like economics, or journalism. If you are dealing with a highly controlled language and you little variation on your theme (technical manuals, set documentation packages, etc), try to pump up as much text as you can.

Typical PangeaMT developments within domains (software, electronics, automotive, engineering, tourism) have started at 5M words. There several ways to increase the number of words by gathering reliable parallel texts and PangeaMT offers consultancy and guidance so you can start an engines with as many words as possible. We call a engine with 15M or 20M words “mature” within a domain, because it is likely to have most of the terminology, vocabulary and expressions required for that language domain. Do not despair if you do not have so much data. The important thing is to get the engine started. You can add post-edited material and other materials that you gather with experience in later re-trainings.

There has been much argument about “unreasonable effectiveness of massive amounts of data” versus “smaller amounts of well-selected data”. Many people considering their first MT development are unsure as to whether put in as much text as possible (massive amounts of data) or to select the most accurate bilingual texts possible even if that means dealing with smaller sets of data. Our experience points in several directions

a) if you are trying to build a generalist type of engine, capable of translating the unexpected (from news articles to economics papers and literature), gather as much data as possible. You are trying to build a system to cater for sunny days and rainy days. No number will ever be enough. Sooner or later, you will need to build some kind of syntactical aids into it.

b) if you are trying to build an engine that will fit your particular language field and needs (or even if you want an engine that understand your products and services, but also some kind of financial information and legalities), you do not need trillions of literature. In this case, gathering as much data as possible from your organization (or similar) seems more reasonable and worth the effort.

Either way, do not underestimate the effort and teamwork required during the data-gathering stages. This is essential for the good training (and thus, the results) of the engine. It will be the beginning of the change in your adoption of MT technologies and a good chance to involve stakeholders in the process.