I uploaded two example pdf files of segmented files to my Dropbox. The first one is 3 short news pieces, in ascending order of difficulty (3 short news pieces). The second one is a summary that I wrote about Qingyi, a short novel about the Beijing opera (Qingyi - Bi Feiyu).
The news pieces are accompanied by a word and name list. The second file uses the same markup but does not have a glossary.
I would say that they are for intermediate level - someone who knows about 1000 characters should be able to understand most of it. I have set the threshold for characters that are marked up a little lower, as knowing 1000 characters does not necessarily mean that you know the most common ones.
Is there anything that you would like to see improved with this markup? I would appreciate any comments about the files and the markup.
Visar inlägg med etikett segmentation. Visa alla inlägg
Visar inlägg med etikett segmentation. Visa alla inlägg
tisdag 21 februari 2012
måndag 13 februari 2012
Segmenting news
I have been working a bit with presentation this week. I segmented 3 recent news articles, among them the one included below, and created a pdf file from it. Unfortunately, I don't have any good place to upload pdf files to for the moment so I only include an image and a text file with one of the articles - will try to work on getting that working this week.
This article is about the opening of a new North Korean restaurant in Amsterdam.
I have made a few adjustments to the final segmented file, so that all words are segmented the way I want them too. The program made about 5 errors on the above file.
I am pretty happy with the display settings I have already, so it looks pretty similar to before. Difficult words are marked in bold, and names are underlined. All characters above a certain threshold are marked with pinyin, and in names, some of the easier characters are marked with a tone but not a sound.
This is really useful for me, as it helps me when I read texts out aloud. I generally find the tones much harder to remember than the sounds. The names are not found automatically (yet), but the difficult words are selected by their frequency. One thing that's missing so far is to keep a closer track of the common characters that have several pronunciations. This is a tricky problem, so I will save it for a bit, but I think it is also interesting and important.
Some common characters to look out for are 地, 得, 中, 当, 为 and 了. Characters like 了 that basically only take another sound in composite words are a bit easier, as the dictionary can be used to find the pronunciations, but characters like 得 or 为 that can have multiple pronunciations even when they stand alone are really tricky.
Otherwise, the tool already works pretty well, and I do feel that the program manages to find most of the difficult words. The only problem I have noticed so far is that it will pick out some easy words as well - especially when it comes to composite words.
At the same time, I don't think it is possible to create a tool that will find and explain all hard words. For example, composites such as 中央银行 (中央(central) + 银行(bank)) or 贫困地区 (贫困(poor) + 地区(region)) may occur with a much lower frequency than their constituents. However, it is difficult to know exactly when a composite word can easily be understood from its constituents.
Therefore, I think of the work I am doing also as a way to create a workflow tool for a teacher to generate a text with a word list and comments in a simple way. Just reading through a text, a teacher may have an idea of which words will be hard for the student, but getting the aid of a frequency dictionary of words and characters to grade the difficulty of text will make it a lot faster and easier.
I managed to get the program to print a word list as well, with definitions but for now, this is only in the pdf file. Hopefully, it will be useful for me as well - I personally spend too much time in front of the computer, so being able to print texts with pre-generated word lists will be really useful.
Of course, in the end, presentation is just a matter of choice. The above could be printed in a book or be presented in HTML. But so far, the pdf version is closest to the way I like to read a text.
Do you have any ideas for how to make the presentation better, or other preferences for how to make the presentation? Let us know in the comments.
Or do you have any interesting texts that you are struggling with, and want to try to read in this format? You can send them by email to carljohanr@gmail.com, and I will send back a segmented pdf version for you.
荷兰 首家 朝鲜 餐馆 开业
据 韩联社 2月 5日 报道，1月 28日，首家 朝鲜 餐馆 在 荷兰 首都 阿姆斯特丹 开业。这家 餐馆 由 荷兰 企业家 同 朝方 合作 开办，正式 名称 为 “阿姆斯特丹 平壤 海棠花 餐馆”。餐馆 的 9名 工作人员 都 是 朝鲜人，包括 总经理 韩明姬 和 4名 厨师。
与 海外 其它 朝鲜 餐馆 一样，这里 也 有 几名 身穿 朝鲜 传统 服装 的 年轻 女服务员 为 顾客 提供 服务，还 会 演唱 朝鲜 歌曲。餐馆 里 悬挂着 几幅 朝鲜 画家 的 画作。
该 餐馆 提供 的 套餐 由 9种 料理 构成，价格 为 每 人 79欧元，与 当地 高级 餐馆 的 套餐 价格 水平 相当。
餐馆 总经理 韩明姬 2月 2日 在 接受 韩联社 采访 时 表示，原本 计划 开设 包括 餐馆 在内 的 文化中心，为 西方人 提供 了解 朝鲜 的 平台。这家 餐馆 可以 成为 朝鲜 与 世界 其它 国家 进行 交流 的 窗口。她 还 表示，餐馆 今后 将 通过 举办 演讲会、上映 电影、展览 美术 作品、宣传 朝鲜 旅游 商品 等 活动，起到 与 西方 国家 进行 沟通 的 桥梁 作用。
I have made a few adjustments to the final segmented file, so that all words are segmented the way I want them too. The program made about 5 errors on the above file.
I am pretty happy with the display settings I have already, so it looks pretty similar to before. Difficult words are marked in bold, and names are underlined. All characters above a certain threshold are marked with pinyin, and in names, some of the easier characters are marked with a tone but not a sound.
This is really useful for me, as it helps me when I read texts out aloud. I generally find the tones much harder to remember than the sounds. The names are not found automatically (yet), but the difficult words are selected by their frequency. One thing that's missing so far is to keep a closer track of the common characters that have several pronunciations. This is a tricky problem, so I will save it for a bit, but I think it is also interesting and important.
Some common characters to look out for are 地, 得, 中, 当, 为 and 了. Characters like 了 that basically only take another sound in composite words are a bit easier, as the dictionary can be used to find the pronunciations, but characters like 得 or 为 that can have multiple pronunciations even when they stand alone are really tricky.
Otherwise, the tool already works pretty well, and I do feel that the program manages to find most of the difficult words. The only problem I have noticed so far is that it will pick out some easy words as well - especially when it comes to composite words.
At the same time, I don't think it is possible to create a tool that will find and explain all hard words. For example, composites such as 中央银行 (中央(central) + 银行(bank)) or 贫困地区 (贫困(poor) + 地区(region)) may occur with a much lower frequency than their constituents. However, it is difficult to know exactly when a composite word can easily be understood from its constituents.
Therefore, I think of the work I am doing also as a way to create a workflow tool for a teacher to generate a text with a word list and comments in a simple way. Just reading through a text, a teacher may have an idea of which words will be hard for the student, but getting the aid of a frequency dictionary of words and characters to grade the difficulty of text will make it a lot faster and easier.
I managed to get the program to print a word list as well, with definitions but for now, this is only in the pdf file. Hopefully, it will be useful for me as well - I personally spend too much time in front of the computer, so being able to print texts with pre-generated word lists will be really useful.
Of course, in the end, presentation is just a matter of choice. The above could be printed in a book or be presented in HTML. But so far, the pdf version is closest to the way I like to read a text.
Do you have any ideas for how to make the presentation better, or other preferences for how to make the presentation? Let us know in the comments.
Or do you have any interesting texts that you are struggling with, and want to try to read in this format? You can send them by email to carljohanr@gmail.com, and I will send back a segmented pdf version for you.
荷兰 首家 朝鲜 餐馆 开业
据 韩联社 2月 5日 报道，1月 28日，首家 朝鲜 餐馆 在 荷兰 首都 阿姆斯特丹 开业。这家 餐馆 由 荷兰 企业家 同 朝方 合作 开办，正式 名称 为 “阿姆斯特丹 平壤 海棠花 餐馆”。餐馆 的 9名 工作人员 都 是 朝鲜人，包括 总经理 韩明姬 和 4名 厨师。
与 海外 其它 朝鲜 餐馆 一样，这里 也 有 几名 身穿 朝鲜 传统 服装 的 年轻 女服务员 为 顾客 提供 服务，还 会 演唱 朝鲜 歌曲。餐馆 里 悬挂着 几幅 朝鲜 画家 的 画作。
该 餐馆 提供 的 套餐 由 9种 料理 构成，价格 为 每 人 79欧元，与 当地 高级 餐馆 的 套餐 价格 水平 相当。
餐馆 总经理 韩明姬 2月 2日 在 接受 韩联社 采访 时 表示，原本 计划 开设 包括 餐馆 在内 的 文化中心，为 西方人 提供 了解 朝鲜 的 平台。这家 餐馆 可以 成为 朝鲜 与 世界 其它 国家 进行 交流 的 窗口。她 还 表示，餐馆 今后 将 通过 举办 演讲会、上映 电影、展览 美术 作品、宣传 朝鲜 旅游 商品 等 活动，起到 与 西方 国家 进行 沟通 的 桥梁 作用。
söndag 5 februari 2012
What is Project Dongcheng?
I have been interested in studying Chinese ever since I came to China in 2005 to study at Beiwai. And back then, I also met Erik, who was also studying at Beiwai at the time.
We have been living in different cities of Sweden for the last 3 years, but almost as by accident, we happened to move back to Dongcheng (东城, a part of Beijing), last year.
I had the idea to write a teaching tool for Chinese (basically a word segmenter) as I do a lot of reading in Chinese (very slowly, but I try to improve). After he spent some time to get a basic prototype of this working during our X-mas vacation, we agreed to put our efforts together to create a useful teaching tool for Chinese.
And that was the birth of Project Dongcheng!
So far it is mainly for ourselves, but that is hopefully something that will change as our tool gets better.
We have been living in different cities of Sweden for the last 3 years, but almost as by accident, we happened to move back to Dongcheng (东城, a part of Beijing), last year.
I had the idea to write a teaching tool for Chinese (basically a word segmenter) as I do a lot of reading in Chinese (very slowly, but I try to improve). After he spent some time to get a basic prototype of this working during our X-mas vacation, we agreed to put our efforts together to create a useful teaching tool for Chinese.
And that was the birth of Project Dongcheng!
So far it is mainly for ourselves, but that is hopefully something that will change as our tool gets better.
Prenumerera på:
Inlägg (Atom)