2020-04-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, codemeta.json, configure.ac: create a bug fix release

2020-04-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bumped versin after release

2020-04-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: News about upcoming release

2020-03-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fix

2020-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia2.ok: changed test result, after removeing
	auth="no" from Sentence in Quote

2020-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: replaced 'tokenise*' by
	'tokenize*' everywhere. That is correct English

2020-02-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: adapted to refactored
	AnnotationType::AnnotationType in libfolia

2020-02-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue71_b.xml, tests/testissue70.ok, tests/testissue71,
	tests/testissue71.ok: adapted/fixed test

2020-02-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: attempt to fix problems when processing FoLiA
	with some segmentation like paragraphs

2020-01-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small code refactoring. no change in outcome.

2020-01-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: make Clang static checking happy

2020-01-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx, tests/testcorrect: renamed option --correctwords to
	--allow-word-corrections

2020-01-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: made some functions
	more generic

2020-01-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia-correct-corrected.xml, tests/folia-correct.xml,
	tests/testcorrect.ok: updated tests

2020-01-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, tests/folia-correct.xml,
	tests/testcorrect.ok: improving on provenance of corrections

2020-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: less debugging

2020-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some trickery with END/BEGIN OFSENTENCE in
	correct_words. needs more thinking...

2020-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: return new tokens on
	correction()

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testcorrect.ok: updated test result

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: add tok_set parameter to correct_word()

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: Add correct set for corrections. Also handle
	paragraphs with only Words better

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia-correct-corrected.xml, tests/folia-correct.xml,
	tests/testcorrect.ok: updated correction test to folia 2.0 usage

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, tests/folia-correct-corrected.xml,
	tests/folia-correct.xml, tests/testcorrect, tests/testcorrect.ok: 
	add test for --correctwords option

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: removed restriction again

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall, tests/testcorrect, tests/testcorrect.ok: added test

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: forbid corrections to the same outputclass

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: save
	state

2020-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: start
	implementing Word corrections in FoLiA

2019-12-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/my_textcat.h, include/ucto/setting.h,
	include/ucto/tokenize.h, src/my_textcat.cxx, src/setting.cxx,
	src/tokenize.cxx, src/ucto.cxx: enter 2020

2019-11-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-11-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about release

2019-11-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a fix to solve
	https://github.com/LanguageMachines/frog/issues/84, which was
	triggered by: https://github.com/proycon/tscan/issues/14

2019-11-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some refactoring and comments improved

2019-11-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, tests/testissue71: fix for
	https://github.com/LanguageMachines/ucto/issues/76

2019-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: quick fix for frog issue
	https://github.com/LanguageMachines/frog/issues/83

2019-10-17  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: one line

2019-10-17  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 40be96ea9451af5880380aeca573ef7d67028f7c Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Thu Oct 17 16:01:58 2019
	+0200

2019-10-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fix for
	https://github.com/LanguageMachines/ucto/issues/74

2019-09-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-09-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about release

2019-09-09  Maarten van Gompel <proycon@anaproy.nl>

	* src/tokenize.cxx: more verbose error message

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: more travis fixes

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: more typo!

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: typo :{

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: mabye this makes TravisCI happy

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: MEH

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: debugging TravisCI again

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, tests/testissue72: small update

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue72, tests/testissue72.ok: fixed test

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, tests/issue72_a.xml: added a fix
	for issue72.

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue72: updated test

2019-09-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/small-tokconfig-nld, tests/testissue72,
	tests/testissue72.ok: added a test to demonstrate issue72

2019-08-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, tests/testissue70.ok, tests/testnbsp.ok: adapted to
	the newest libfolia

2019-08-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/testissue71.ok,
	tests/testissue72.ok: when passthru and an ucto processor is already
	defined, use that one

2019-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: updated dependency

2019-08-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: ow f*ck

2019-08-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to trigger buid. Hopefully on bionic

2019-08-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: small change

2019-08-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: removed boost dependency

2019-08-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue70_c.xml: added missing tetsfile

2019-08-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue70, tests/testissue70.ok, tests/testone: added test

2019-08-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed code to make sure that the 'ucto'
	processor is addec unconditionally

2019-08-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: made the set's for PARAGRAPH, SENTENCE and QUOTE
	explicit empty ("None")

2019-07-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-07-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about upcoming release

2019-07-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small style issue

2019-07-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue70_b.xml, tests/testissue70, tests/testissue70.ok: 
	added another (yet failing) test on issue70

2019-07-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue70.ok: missing file added

2019-07-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: start fixing issue 70

2019-07-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue70.xml, tests/testall, tests/testissue70: added a (yet
	failing!) test

2019-06-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/multilang2.txt, tests/testlang.ok: does it work on OSX now?

2019-06-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia-lang.xml, tests/testlang, tests/testlang.ok: addapted
	testlang test, because of
	https://github.com/LanguageMachines/ucto/issues/62 we need longer
	sentences!

2019-06-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: we want to see the errorlog

2019-06-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang: force a test error. for Travis/OSX debugging

2019-06-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: some cleanup in .travis.yml

2019-06-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: try this

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: boost already there?

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to cleanup brew stuff

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: still failing

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to add a cache for Homebrew bottles

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-06-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about release

2019-06-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testnbsp: next attempt to debug on TravisCI/MacOS

2019-06-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: disable OpenMP on TravisCI/clang

2019-06-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall: added tests

2019-06-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue71_b.xml, tests/issue71_c.xml, tests/issue71_d.xml,
	tests/issue71_e.xml, tests/issue72_a.xml, tests/testissue71,
	tests/testissue71.ok: added more tests

2019-06-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue72, tests/testissue72.ok: added a test

2019-06-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, tests/testone: next attempt to debug on travisCI

2019-06-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall: no longer accept 1 failure as ok

2019-06-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a fallback token-annotation for passthru

2019-06-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: attempted fix for folia input with --pasthru
	combined with a language

2019-06-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, tests/testone: debugging om mac failed for now.

2019-06-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, tests/testone: desparate attempt

2019-06-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, tests/testmac: attempt to use valgrind to search for
	problem on OSX

2019-06-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: argl, wrong filenames

2019-06-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: next attempt to find details about MacOS problem

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: why does it fail on TravisCI/MacOS?

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: avoid replacing a good declaration

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain.ok, tests/testissue66.ok, tests/testlang.ok,
	tests/testnbsp.ok, tests/testslash.ok, tests/testutt.ok: adapted
	tests. No more empty lines in the FoLiA

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: avoid empty last line in FoLiA output

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: make sure to use latest/greates libfolia

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testissue64.ok, tests/testissue66.ok, tests/testlang.ok,
	tests/testnbsp.ok, tests/testslash.ok, tests/testutt.ok: adapted
	tests to change in provenance naming and numbering

2019-06-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small change in provenance naming and numbering

2019-05-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-05-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about the release

2019-05-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/Makefile.am: Bumped library version

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 9877294a8ff932499990200f71fea72e612bec63 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue May 28 14:33:45 2019
	+0200

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed debugging lines

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed FoLiA output for empty sentences

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall: added test

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testnbsp, tests/testnbsp.ok: added 2 testfiles

2019-05-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit dcb79716077bbe95780e5e38895bb9edc64931fb Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue May 28 11:50:41 2019
	+0200

2019-05-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added more infromation in output

2019-05-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some fixes to handle running ucto on already
	uctoed FoLiA

2019-05-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: better message

2019-05-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed some debugging lines

2019-05-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added code to avoid
	retokenizing FoLiA already processed by ucto

2019-05-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 86723cc7b6a6dbb63550b63e63fef3ec8e309502 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed May 22 16:43:01 2019
	+0200

2019-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: make the clang analyser happy

2019-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 167fc3eb52aa3d3672bfa1a0231b3d8680fc3773 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed May 22 12:44:48 2019
	+0200

2019-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: adapted to changes in libfolia for folia2.0

2019-05-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testissue64: add --KANON to folialint call

2019-05-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testissue64.ok: adapted to small
	change in libfolia

2019-05-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bump version after release

2019-05-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: updated NEWS for the release

2019-05-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added debug line

2019-05-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: avoid leak

2019-05-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: fixed test.py for Valgrind results

2019-05-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: function renamed

2019-05-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: added finction to the API

2019-05-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testfolia.ok,
	tests/testfolia2.ok, tests/testfoliain.ok, tests/testissue64.ok,
	tests/testissue66.ok, tests/testlang.ok, tests/testslash.ok,
	tests/testutt.ok: don't add lang-annotation declaration when no lang
	annotation nodes are added

2019-05-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added comment

2019-05-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: adapted test result

2019-05-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: started code cleanup

2019-05-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed unneeded vector

2019-05-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: add begindatetime to provenance

2019-05-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: adapted to change in processor's "generator"
	handling

2019-05-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: allow an extra
	parameter to set provenance as a sub_processor

2019-05-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfolia.ok: added quote-annotation. for
	now outcommented code to add text-annotation. Don't think it is
	required.

2019-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testissue64.ok, tests/testutt.ok: updated test results

2019-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added code to add more
	provenance data

2019-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testlang.ok: updated
	tests

2019-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: added member

2019-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: also add sentence and paragraph annotation, when
	needed

2019-04-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: more language stuff
	improved

2019-04-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testfolia.ok, tests/testfolia2.ok: 
	simplified language handling

2019-04-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleanup some confusion in language handlin.
	still messy

2019-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: set default language if possible

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: make sure the default language is found

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: updated and simplified
	provenance stuff

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: save current state

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleaning up

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: more refactoring

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testlang.ok: refactoring step

2019-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: save working state

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed a space

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok, tests/testissue64.ok,
	tests/testissue66.ok, tests/testslash.ok, tests/testutt.ok: when a
	token annotation in the same set as an alias exists, it is replaced.
	Assuming tranfer to new style

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testlang.ok: adapted
	tests

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: avoid extra newline at the end of the FoLiA
	documents.

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testissue66.ok, tests/testlang.ok, tests/testslash.ok,
	tests/testutt.ok: adapted test results

2019-04-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: give the processor a more useful name

2019-04-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testone: fixed test scripts to get all green
	again

2019-04-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed parameters to call of text(). They are
	the default

2019-04-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added functions to
	extract useful information from the configuration

2019-04-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: adapted to Processor ==> Engine change

2019-04-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: adapted to change in Processorr class name

2019-04-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: add the
	command to provenance

2019-04-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok, tests/testissue66.ok,
	tests/testlang.ok, tests/testslash.ok, tests/testutt.ok: add more
	infor to the processors

2019-04-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testfolia.ok,
	tests/testfolia2.ok, tests/testlang.ok: adapted tests to newer
	provenance handling

2019-04-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testone: adapted test scrips to filter
	datetime stuff

2019-04-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testlang.ok: improved provenance
	assignment

2019-04-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok, tests/testissue66.ok,
	tests/testlang.ok, tests/testslash.ok, tests/testutt.ok: further
	improvement on provenance

2019-04-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, include/ucto/tokenize.h, src/tokenize.cxx,
	tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testlang.ok: improving on provenance info. adapted tests

2019-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testall, tests/testfolia.ok,
	tests/testfolia2.ok, tests/testfoliain.ok, tests/testlang.ok: 
	improving lang-annotation

2019-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok, tests/testissue66.ok,
	tests/testlang.ok, tests/testslash.ok, tests/testutt.ok: add more
	provenance information

2019-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.xml, tests/testfolia.ok,
	tests/testfolia2.ok, tests/testlang.ok: start adding provenance
	data. adapted tests..

2019-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: be sure to use modern libfolia

2019-04-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: update

2019-04-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small change, make it build with libfolia2.0 too

2019-03-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testissue64.ok, tests/testlang.ok, tests/testutt.ok: merged
	'processor' branch

2019-03-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added an 'upward'
	compatability function :)

2019-03-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testissue66.ok, tests/testlang.ok, tests/testslash.ok,
	tests/testutt.ok: adapted to change in libfolia (spurious empty
	lines at end of FoLiA)

2019-02-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 059ca0e4a8f9017862e2e2996c67a5d1633a1cb5 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Feb 28 10:45:59 2019
	+0100

2019-02-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: small fixes

2019-02-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: more cleanup and
	refactoring

2019-02-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: save working state.
	cleaning commences

2019-02-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit a469e42fb5f57026f1f92af5e8f9c0d6fc83dda4 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Feb 25 16:24:54 2019
	+0100

2019-02-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 539774218366cdebf4bb92e73447823b711ac42f Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Feb 25 11:09:37 2019
	+0100

2019-02-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue064c.xml, tests/testall, tests/testissue64,
	tests/testissue64.ok: expanded test again, and improved testall
	script

2019-02-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/{issue064.xml => issue064a.xml}, tests/issue064b.xml,
	tests/testissue64, tests/testissue64.ok: expanded test. (it SHOULD
	FAIL)

2019-02-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fixes

2019-02-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleared code

2019-02-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleared more redundant code

2019-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: save the day

2019-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: cleanup some flag
	stuff. Fun with Flags!

2019-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/Makefile.am: bump .so version, as we are shaking the tree a
	lot

2019-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: start protecting our private parts :)

2019-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed superfluous
	function

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: more cleaning up and refactoring

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: refactoring

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: more cleanup

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: get rid of internal,
	redundant function

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: renamed function

2019-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: start simplifying the
	cose. Remove redundant functionality

2019-02-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit ce761ce90cbc85ce7c59b7352f752872c47d0a9a Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Feb 11 16:55:08 2019
	+0100

2019-02-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 0da9df086ff4081e751273566549ae3027bd85cc Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Feb 7 14:49:17 2019
	+0100

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue066.xml: forgotten file

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: generate_id only works for tag id's. NOT for the
	document id.

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testissue66.ok: make sure all words get an
	ID

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall, tests/{testfoliabug => testissue64},
	tests/{testfoliabug.ok => testissue64.ok}, tests/testissue66,
	tests/testissue66.ok: added new test

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliabug.ok: when tokenizing FoLiA,
	descen into Sentences and Paragraphs too

2019-02-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue064.xml, tests/testall, tests/testfoliabug,
	tests/testfoliabug.ok: added a new test

2019-01-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit dc9e680dd57363d80d6269af55a34c451e0f52cd Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Jan 22 22:59:00 2019
	+0100

2019-01-15  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: Added latest DOI badge

2019-01-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: only emit "Initiating tokeniser..." when debug
	level > 0

2019-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/my_textcat.h, src/my_textcat.cxx: made LogStream
	optional. should fix backward compatibility problems

2019-01-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: only display rules info when debugging

2019-01-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit a96bcef90ab0080c27280a0767fd760f3f468387 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Jan 9 10:26:49 2019
	+0100

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/my_textcat.h: Next MacOSX attempt

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/my_textcat.cxx, src/tokenize.cxx: attempt to set a property
	(if only MacOS would accept that....)

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a textcat debug
	setter to the API

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: ok, leve long lines. yamllint rejects them, but alas

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: next .travis.yml fix

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: fixes in .travis.yml

2019-01-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/my_textcat.h, src/my_textcat.cxx, src/tokenize.cxx: 
	added debugging possibility to TextCat class

2019-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a log message about textcat

2019-01-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: typos in comment

2018-12-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 716ca36e30c9a2498e7d83d37c743f54fe0d6f77 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Dec 19 10:45:59 2018
	+0100

2018-12-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: update

2018-12-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: restore version number after bug fix
	release 0.14.1

2018-12-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: added NEWS entry

2018-12-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, codemeta.json, configure.ac: get ready for release 0.14.1

2018-12-02  Maarten van Gompel <proycon@anaproy.nl>

	* config/Makefile.am: copy instead of link otherwise we might link
	to a staged temporary directory (#36)

2018-12-02  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto.1: typo

2018-12-02  Maarten van Gompel <proycon@anaproy.nl>

	* config/Makefile.am: attempted fix for debian packaging issue #36

2018-11-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* codemeta.json, configure.ac: bumped version after release

2018-11-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, codemeta.json: updated NEWS for release

2018-11-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: back to travis:trusty, until we understand what fails
	with boost and clang.

2018-11-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, src/tokenize.cxx: small fix. Also build on
	travis:xenial now

2018-11-16  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: updated link to ucto documentation in README

2018-11-16  Iris Hendrickx <iris@i-hx.nl>

	* docs/source/index.rst: minor spelling edits to manual

2018-11-16  Iris Hendrickx <iris@i-hx.nl>

	* : commit 89cc9fcfc63f398bd192e74c6b35ee6262f1ba8d Author: Iris
	Hendrickx <iris@i-hx.nl> Date:   Fri Nov 16 10:31:56 2018 +0100

2018-11-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit a41d7965a9200d54c2677728b62cae4d2c5a2f7a Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Fri Nov 16 09:21:33 2018
	+0100

2018-11-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, include/ucto/tokenize.h, src/setting.cxx,
	src/tokenize.cxx: cleaning up icu namespace mess a bit.

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/source/index.rst: Update index.rst

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/source/index.rst: Update index.rst

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/source/index.rst: Update index.rst

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: updated usage(). Removed 'S' from allowed options.

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/ucto.1: add missing info to the man page

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/ucto.1: updated man page

2018-11-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: improved usage() layout

2018-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 4cc63b838c2f33bf29dce0657e41eccf1bd739d3 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Nov 1 15:00:03 2018
	+0100

2018-11-01  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/index.rst: [documentation] refer to new frog site
	instead of ILK site

2018-11-01  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/index.rst: processed ucto documentation: some fixes,
	spelling corrections, elaborations, and RST style improvements

2018-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit d988612bbfaf657fe72b9d357c41cdd94265ab7f Merge: a4f70cb
	6c83d1d Author: Iris Hendrickx <iris@i-hx.nl> Date:   Thu Oct 25
	14:58:16 2018 +0200

2018-10-25  Iris Hendrickx <iris@i-hx.nl>

	* docs/source/index.rst: adding license and citation info to ucto
	rest documentation

2018-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* m4/Makefile.am: cleanup

2018-10-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 2c77f9ca825f345ce5ba719feb18076a97cb58f1 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Oct 24 12:31:24 2018
	+0200

2018-10-24  Iris Hendrickx <iris@i-hx.nl>

	* docs/source/index.rst: ucto documentation -minor changes

2018-10-24  Iris Hendrickx <iris@i-hx.nl>

	* docs/source/index.rst: ucto documentation -maior update on
	index.rst file

2018-10-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: added test and adapted usage. --uselanguages is ONLY
	for FoLiA inpuy

2018-10-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: typo in .travis.yml

2018-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx: fixed problem: printing an
	unitialized filter would crash....

2018-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b2c258ca29029aa4d836bf46e7f142edb83c4a4c Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Oct 18 16:44:29 2018
	+0200

2018-10-11  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/conf.py: removed theme option

2018-10-11  Maarten van Gompel <proycon@anaproy.nl>

	* .gitignore: updated gitignore

2018-10-11  Maarten van Gompel <proycon@anaproy.nl>

	* docs/Makefile.sphinx: makefile fix

2018-10-10  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/conf.py: changed documentation theme

2018-10-05  Iris Hendrickx <iris@i-hx.nl>

	* : commit c144928a22da93eb5247526ff651225137b6f3da Author: Iris
	Hendrickx <iris@i-hx.nl> Date:   Fri Oct 5 16:50:42 2018 +0200

2018-10-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b9eae191b5f0cbe79b699448a8afc883400d5964 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Fri Oct 5 16:40:43 2018
	+0200

2018-10-04  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/index.rst: rst fix

2018-10-04  Maarten van Gompel <proycon@anaproy.nl>

	* docs/source/index.rst: copied the README contents as initial
	documentation #2

2018-10-04  Maarten van Gompel <proycon@anaproy.nl>

	* : commit a97100477c4bf76719ebdba345f60a38e617ca29 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Thu Oct 4 17:10:53 2018
	+0200

2018-08-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: make sure the correct textclass is added to <w>
	nodes

2018-08-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: closing in on the icu:: namespaces

2018-08-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: add more icu:: namepsaces

2018-08-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h: add icu:: namespace

2018-07-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: clarified comment

2018-07-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit ecdbef293e85ff9b93241026cd71ab2ba50ed6a6 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon May 28 17:45:48 2018
	+0200

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bump version after release

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: created a bug fix release

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am: no longer install default rule. uctodata is
	required now

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about bug fix release 0.13.1

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: reverted last edit.

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: removed debug lines from configure.ac

2018-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/setting.cxx: configure now tries to locate the
	uctodata files. We use that location. Should work on Mac too :{

2018-05-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bump version after release

2018-05-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: typos!

2018-05-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: get ready for release 0.13

2018-05-02  Maarten van Gompel <proycon@anaproy.nl>

	* codemeta.json, configure.ac: metadata update

2018-05-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed the experimental bracketing code. is
	solved in the configuration

2018-05-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue051.nl.tok.V, tests/issue051.nl.txt: modified example
	with ABBREVIATION inside brackets

2018-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added experimental code for bracket handling.
	(disabled for now)

2018-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue051.nl.tok.V, tests/issue051.nl.txt: for now, let the
	test pass. Fixing might be hard

2018-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue051.nl.tok.V, tests/issue051.nl.txt: added a (yet
	FAILING!) test for issue051

2018-04-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testoption-split, tests/testoption-split.ok: modified test

2018-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: spitOnly wasn't initialized...

2018-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: improved --split option

2018-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall, tests/testoption-split, tests/testoption-split.ok: 
	added a test for the --split option. It proves that quotes are very
	difficult!

2018-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/Makefile.am, src/tokenize.cxx,
	src/ucto.cxx: added a --split option

2018-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testoption-P, tests/testoption-P.ok: extended test

2018-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall, tests/testoption-P, tests/testoption-P.ok: added a
	test for the -P option (disable paragraph detection)

2018-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h: removed the -S option (disable
	sentence-detection) because is was only half implemented....

2018-04-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: removed the -S option (disable
	sentence-detection) because is was only half implemented....

2018-04-23  Maarten van Gompel <proycon@anaproy.nl>

	* codemeta.json: forgot to actually commit codemeta.json

2018-04-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue048.nl.tok.V, tests/issue048.nl.txt: added another
	test. (may run forever or not)

2018-04-18  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 2bff822b2353530e12aa6a74472bbe981c9293bf Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Wed Apr 18 20:39:05 2018
	+0200

2018-04-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 678df7646890df190d33f820f05de5f6a6ff5d83 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Apr 9 14:29:38 2018
	+0200

2018-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testtokens, tests/testtokens.ok, tests/tokens,
	tests/tokens.txt: added a test for the new --add-tokens option

2018-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, include/ucto/tokenize.h: add an extra
	--add-tokens option to add special tokens to settings of the default
	language

2018-04-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx, src/ucto.cxx: add an extra
	--add-tokens option to add special tokens to settings of the default
	language

2018-04-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx: add icu:: namespaces

2018-04-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/cell.xml: oesp, lost a file!

2018-04-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx, tests/cell.xml: some
	refactoring concerning namespace usage.

2018-04-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue046.nl.tok.V, tests/issue046.nl.txt: expanded test

2018-04-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue046.nl.tok.V, tests/issue046.nl.txt,
	tests/ocr.nl.tok.V, tests/test.nl.tok.V: added new test. adapted
	some others to newest uctodata

2018-03-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: updates .travis.yml

2018-03-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: and try to fix

2018-03-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: Try to brake build on travis

2018-03-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, m4/ac_check_old_textcat.m4, m4/ac_search_lm.m4: 
	attempt to clean up configure a bit

2018-03-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, m4/ac_search_lm.m4: next attempt to fix configure

2018-03-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 380e35eb977d8d9ea4f1ea68f412ea1935e3af74 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Mar 6 11:40:25 2018
	+0100

2018-02-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: let configure fail when no libtextcat is found.

2018-02-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: added experimental code to enable an alternative
	search path for the ucto data files.

2018-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bump version after release

2018-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/unicode.h,
	src/Makefile.am, src/unicode.cxx: removed old and unsused unicode
	stuff

2018-02-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, README.md: Updated README abit

2018-02-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/setting.cxx, src/tokenize.cxx,
	src/ucto.cxx: cleanup old ucto includes

2018-02-16  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/setting.h,
	include/ucto/tokenize.h, src/Makefile.am, src/ucto.cxx: use
	UniFilter from ticcutils now

2018-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added include

2018-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/unicode.h, src/unicode.cxx: removed unused includes

2018-02-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, include/ucto/setting.h, include/ucto/unicode.h,
	src/setting.cxx, src/unicode.cxx: moved unicode code to ticcutils (v
	0.18 or above)

2018-01-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: don't use 'diff -Z' mac OSX doesn't know it

2018-01-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/textcat_alt.cfg, config/{textcat_normal.cfg =>
	textcat_alt.cfg.in}, config/textcat_normal.cfg.in: oesp, wrong files
	in last commit

2018-01-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, configure.ac, include/ucto/my_textcat.h: 
	updated configuration

2018-01-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/textcat_alt.cfg, config/textcat_normal.cfg,
	m4/ac_check_old_textcat.m4, m4/ac_search_lm.m4: new files

2018-01-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, include/ucto/my_textcat.h: improved configuration
	for libtextcat (notably for MacOSX)

2018-01-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to fix runtime dependency

2018-01-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: move cppcheck to better location and DON'T fail,
	please

2018-01-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: enable OSX test again

2018-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, include/ucto/unicode.h, src/unicode.cxx: 
	use the TiccUtils UnicodeNormalizer now

2018-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: it works vor Clang!

2018-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: why doesn't is work? Or does it?

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: that didn;t work well...

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: next clang attempt

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: disable osx for now. focussing in CLANG

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, configure.ac, src/Makefile.am, src/{textcat.cxx =>
	my_textcat.cxx}: getting desperate

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, configure.ac: and the next round

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: and next round of errors

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: next OSX attempt

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: attempt to fix libtextcat stuff on OSX

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: the same oesp in another file

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: oesp. include files has a different name now

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, include/ucto/Makefile.am, include/ucto/{textcat.h
	=> my_textcat.h}, src/textcat.cxx, src/tokenize.cxx: attempt to
	simplify lib(ext)textcat stuff using PKG_CONFIG

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: added missing libtextcat dependency for OSX

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: ok, lets try OSX build

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: brainfarth corrected

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* m4/ac_osx_pkg.m4: forgotten a file

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to fix .travis.yml

2018-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, configure.ac: test new .travis config. No OSX yet

2018-01-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: another 2017 ==> 2018 transition

2018-01-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, include/ucto/textcat.h,
	include/ucto/tokenize.h, include/ucto/unicode.h, src/setting.cxx,
	src/textcat.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	2018

2018-01-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore, Makefile.am, README: also package README.md

2017-12-20  Kobus van der Sloot <sloot@mlp06.science.ru.nl>

	* : commit dedee22f879053ac64163de54aacdb0c4a094c7b Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Dec 4 14:51:54 2017
	+0100

2017-12-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS about release 0.11

2017-12-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed problem with text in cell (code get
	hackier every time....)

2017-12-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/cell.xml, tests/testfoliain, tests/testfoliain.ok: added a
	new test

2017-11-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-11-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: new NEWS prior to yar (yet another release)

2017-11-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 49f6aa8d553e988c92654a828f96be925d4cf52f Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Nov 6 17:24:11 2017
	+0100

2017-11-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac, src/Makefile.am: Minor fix: bumped the .so
	version to 3.0.0

2017-11-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, include/ucto/tokenize.h, src/tokenize.cxx: some
	refactoring

2017-10-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.de.tok.V: German test result is changed (improved)
	after fix in NOSPACE handling

2017-10-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed a problem with NOSPACE inside recursive
	rules

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/smileys.nl.tok.V: added a test

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/smileys.nl.txt: added detection of
	UBLOCK_MISCELLANEOUS_SYMBOLS_AND_PICTOGRAPHS

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: when adding 'extra' text, skip <br> nodes. They
	don't take <t>

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS

2017-10-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testutt, tests/testutt.ok, tests/utt2.xml: added anothrer
	utterance test.

2017-10-22  Maarten van Gompel <proycon@anaproy.nl>

	* src/tokenize.cxx: Attempted fix for utterance/sentence problem #37

2017-10-22  Maarten van Gompel <proycon@anaproy.nl>

	* src/tokenize.cxx: another related comment

2017-10-22  Maarten van Gompel <proycon@anaproy.nl>

	* src/tokenize.cxx: just added a comment/suggestion on detection
	structure elements

2017-10-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: small folia ==> FoLiA edit

2017-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: some typos in NEWS

2017-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: Updated NEWS with old news from 23-01-2017

2017-10-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: some news

2017-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok: fixed
	textredundancy="full". Now it adds text upto the highest level.

2017-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain, tests/testfoliain.ok, tests/textproblem.xml: 
	added and modified tests, after change in FoLiA parser

2017-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a
	setTextRedundancy member

2017-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testlang.ok, tests/testutt.ok: adapted tests to changed
	textredundancy level

2017-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: changed textredundancy default to
	'minimal'

2017-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain.ok: adapted test to changed <br/> handling

2017-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: for now, disable the <br/> handling. It is too
	complicated.

2017-10-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia2, tests/testfolia2.ok, tests/testfoliain.ok: fixed
	tests

2017-10-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/testfolia, tests/testfoliain, tests/testfoliain.ok: 
	implemented --textredundancy option (replaces --noredundanttext)

2017-10-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed an unused
	function. Give a warning when attempting to set language on metadata
	of non-native type

2017-10-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: re-instated --with-icu in configure.ac

2017-09-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added safeguards around set_metadata

2017-09-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: the default is doRedundantText == true

2017-09-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain: adapted test to check automagically detecting
	folia

2017-09-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: automatically switch to -F or -X when input or
	outputfile have '.xml' extension(s)

2017-09-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia2, tests/testfolia2.ok: modified test to also test
	-T option

2017-09-26  Maarten van Gompel <proycon@anaproy.nl>

	* src/ucto.cxx: added CLST, Nijmegen to --version

2017-09-26  Maarten van Gompel <proycon@anaproy.nl>

	* src/ucto.cxx: Added shortcut option for --noredundanttext (-T) and
	changed help text a bit #31

2017-09-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok: add updated file, missing from previous commit

2017-09-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/testfolia, tests/testfoliain, tests/testfoliain.ok: 
	implemented an --noredundanttext option. and added tests

2017-09-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: be sure to use recent libfolia

2017-09-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testfoliain.ok: set textclass on <w> when
	outputclass != inputclass

2017-09-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: use C++!

2017-09-05  Kobus van der Sloot <sloot@fancypants.science.ru.nl>

	* : commit 04f894f0f0cd6913430ede43f2be96cd6b6623b0 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Aug 30 16:49:43 2017
	+0200

2017-08-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 5ee40601de62c8612f4660a7748151fee7ea9929 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Aug 30 16:24:06 2017
	+0200

2017-08-30  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: typo fix (and automatic trailing space
	stuff)

2017-08-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia9a.xml, tests/folia9b.xml, tests/testfoliain,
	tests/testfoliain.ok: added test documents with embedded tabs,
	newlines and multiple spaces.

2017-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia8.xml: new file

2017-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, tests/testfoliain, tests/testfoliain.ok: added a
	test wikt xml comment inside a <t>

2017-08-17  Maarten van Gompel <proycon@anaproy.nl>

	* src/tokenize.cxx, src/ucto.cxx: language fix

2017-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added some more debug lines

2017-08-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: try to generate id's based on the parents ID or
	there parents ID.

2017-07-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: add libtar-dev too

2017-07-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 00c3b9e94e36331b756f67110c0fc940ff83075d Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Jul 25 10:45:38 2017
	+0200

2017-07-20  Maarten van Gompel <proycon@anaproy.nl>

	* tests/testall: use python2 explicitly

2017-07-20  Maarten van Gompel <proycon@anaproy.nl>

	* tests/test.py: use python 2 explicitly

2017-07-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testutt.ok: fixed utterance handling
	(quite hacky)

2017-07-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testall, tests/testutt, tests/utt.xml: added a (yet failing)
	test

2017-07-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to fix clang test on travis

2017-07-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: disable filtering in XML files in more cases

2017-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to fix build

2017-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain.ok: adaped test, now newline handling is fixed

2017-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added code to handle
	embedded newlines in FoLiA documents.

2017-06-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: adapted to changed libfolis

2017-06-01  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 2037878fff5e9bb47911c1a0c54b9c79291754fc Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Thu Jun 1 21:30:05 2017
	+0200

2017-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx, src/ucto.cxx,
	tests/testfiles2.ok, tests/testfoliain.ok, tests/testlang.ok,
	tests/testoption2.ok, tests/testslash.ok: sorted out logging and
	such a bit.

2017-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfoliain.ok, tests/testlang.ok, tests/testslash.ok: 
	adaptes tests

2017-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: No longer SILENTLY set --filter=NO for FoLiA with
	equal input ans output class

2017-05-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx, tests/testnormalisation: added an --filter option.
	superseeds -f (that could only switch filtering OFF)

2017-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia1.xml, tests/testfoliain, tests/testfoliain.ok: 
	enhanced and extended folia testing

2017-05-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, tests/testfoliain.ok: Disable
	filtering of characters on FoLiA input with same inputclass and
	outputclass

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/filter.xml, tests/testfoliain.ok, tests/testtext,
	tests/testtext.ok: added a test, and adapted to changes results

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: now we adapt text on <s> and <p> to the lower
	layers

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: simplified configuration

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: added IRC notification

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: adepted test after fix in libfolia

2017-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/ucto.1, src/ucto.cxx: update manpage. Fixed typo.

2017-05-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* Makefile.am, configure.ac, ucto.pc.in: more configuration cleanup.

2017-05-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* bootstrap.sh, configure.ac: modernized build system

2017-05-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: still a leak was left. plugging...

2017-05-03  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx: fixed a memory leak

2017-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added some comment

2017-04-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: better debug output

2017-04-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia7.xml, tests/testfolia, tests/testfoliain,
	tests/testfoliain.ok: added a test

2017-04-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: revert back to default g++

2017-03-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: numb edits

2017-03-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/folia-lang-2.xml, tests/testlang: started implementing
	language detection in FoLiA input too. Not done, nothing broke (yet)

2017-03-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed a problem with log token detection

2017-03-14  Maarten van Gompel <proycon@anaproy.nl>

	* : Merge pull request #17 from sanmai-NL/speed_up_CI_build Limit network transfers, add `ccache`

2017-03-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: Oops. A function got lost... :{

2017-02-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: removed redundant mentioning of configfile. (is
	empty > 90% of time)

2017-02-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: in case of problems in
	tokenizeLine(), we display the offending line numner OR the FoLiA
	element ID.

2017-02-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: for extreme long 'words' display a part of the
	offensive intput. Also typo corrected.

2017-02-23  Kobus van der Sloot <sloot@fancypants.science.ru.nl>

	* : commit 91dabbf7c73f5328f575e589346e1230c5b2732f Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Feb 21 11:56:42 2017
	+0100

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: updated usage()

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* docs/ucto.1: updated ucto man page

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: another final attempt :{

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: final attempt

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: getting closer?

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: wow wat lastig

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: next try

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: another attempt

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to fix

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: modernized Travis config

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: added dependency for travis

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: Warn about use of unsupported languages. Don't use
	'generic' by default.

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: check specified languages against the installed ones

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, src/setting.cxx, src/ucto.cxx: use a set
	to store resulte, not a vector

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, src/setting.cxx, src/ucto.cxx: added a
	function to search for installed languages

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: typo corrected

2017-02-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: choke on words from 2500 characters ore more

2017-02-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: some more repait
	considering outputclass

2017-02-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: when using the --textclass option.
	make sure --inputclass and --outputclass are not used.

2017-02-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/Makefile.am, src/tokenize.cxx: 
	attempt to speed up some stuff

2017-02-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/Makefile.am, src/tokenize.cxx: minor changes

2017-01-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/textcat.h, src/Makefile.am, src/setting.cxx,
	src/textcat.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	some refactoring to satisfy static checkers

2017-01-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: rely on uctodata 0.4

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* : citation fix

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* : layout fix rendered

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: layout fix

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: no need for ILK affiliation anymore in
	documentation #2

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto_manual.tex: Updates to ucto manual for 0.9.6
	(installation procedure, new language codes, contact address). Text
	still need to be revised/expanded sometime. #2

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto.1: man page update

2017-01-23  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: updated the readme for the new language codes

2017-01-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, src/Makefile.am: unstall and look for
	datafiles in $PREFIX/share/ucto

2017-01-18  Sander Maijers <S.N.Maijers@gmail.com>

	* .travis.yml: Speed up CI builds

2017-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added more DATE testcases
	and the desired result

2017-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : Merge pull request #15 from sanmai-NL/static_analysis Check code quality during CI

2017-01-15  Sander Maijers <S.N.Maijers@gmail.com>

	* .travis.yml: Check code quality during CI Use CppCheck and the Clang Static Analyzer and fail on them by
	default.

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small refactoring to avoid potential 0 pointer

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: removed unused variable

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: minor refactoring

2017-01-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: removed dead code

2017-01-16  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: Temper expectations about ucto's simple date/time
	recognising abilities (#16)

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/Makefile.am, tests/testoption2, tests/testtwitter: fixed
	test scripts to correctly find the uctodata

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/es.abr, config/exotic-eos.eos, config/exotic-quotes.quote,
	config/ligatures.filter, config/nl_afk.abr, config/pt.abr,
	config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-it,
	config/tokconfig-nl, config/tokconfig-nl-sonarchat,
	config/tokconfig-nl-twitter, config/tokconfig-nl-withplaceholder,
	config/tokconfig-pt, config/tokconfig-ru, config/tokconfig-sv,
	config/tokconfig-tr: removed files that were moved to uctodata long
	ago

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: typo

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: updated NEWS for the release

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: We do not longer require the uctodata package to be
	installed. But issue a notice! If present we check for a recent and
	decent version.

2017-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 3bb3f7b6fba6a1d1ce566591cba65b606bbf738b Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Fri Jan 6 13:10:35 2017
	+0100

2017-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-generic: Updated tokconfig-generic with version
	information

2017-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2017-01-05  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: updatede NEWS for upcoming release

2016-12-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/unicode.cxx: some refactoring, generally use
	back() to get last element of a collection

2016-12-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/setting.h, include/ucto/textcat.h,
	include/ucto/tokenize.h, include/ucto/unicode.h, src/setting.cxx,
	src/textcat.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	also entering 2017 now

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: added libexttextcat to the travis dependencies

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fix merge problem

2016-11-21  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b9af1264b2d4ad9f96ce59afd82ab4ca8c8e29c0 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Mon Nov 21 10:42:32 2016
	+0100

2016-11-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: ok, no more exit() calls

2016-11-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: better error messages (including filename)

2016-11-14  Maarten van Gompel <proycon@anaproy.nl>

	* docs/ucto.1: fixed typo in manpage

2016-11-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: fiddling with configure to test libexttextcat in the
	searchpath

2016-11-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: merges fix from language-detect branch. (clearer
	warning)

2016-11-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx: better warninin message iin case off missing
	version number

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a 'real' set url for language

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixes for language set/class

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testlang.ok: small fix.We should sort out
	"none" vs. "default"

2016-11-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/folia-lang.xml,
	tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testlang, tests/testslash.ok: added code to add correct
	language annotations. Stila bit clumsy.  Needs cleaning and
	thinking...

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/empty_line.txt, tests/testpassthru,
	tests/testpassthru.ok: merged the passthru fix from language-detect
	branch

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/{empty_line.nl.txt => empty_line.txt}, tests/testpassthru: 
	some file renaming in test

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/empty_line.nl.txt: added testfile

2016-11-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testpassthru, tests/testpassthru.ok: fixed
	issue 11 (https://github.com/LanguageMachines/ucto/issues/11). added
	a test.

2016-11-09  Kobus van der Sloot <sloot@applejack.science.ru.nl>

	* : commit f686580db7b8f66354dbb0608946dd92937bc1c3 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Nov 8 12:05:43 2016
	+0100

2016-11-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/textcat.h, include/ucto/tokenize.h, src/textcat.cxx,
	src/tokenize.cxx: added const qualifiers. made sure config.h is NOT
	included in other .h files.

2016-11-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/textcat.h,
	include/ucto/tokenize.h, src/Makefile.am, src/textcat.cxx,
	src/tokenize.cxx, src/ucto.cxx: put all textcat related stuff in own
	files

2016-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: 3 typos corrected in usage()

2016-11-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: 3 typos in usage() corrected

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/textcat.cfg, tests/multilang.txt: new files for testing

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: updated test

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fix detection

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/testlang: add langauage detection for NON folia input. still
	buggy

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: changed testresult

2016-10-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, include/ucto/setting.h,
	include/ucto/tokenize.h, src/setting.cxx, src/tokenize.cxx,
	tests/testlang: next step in language detection. still rough edges

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: started incorporationg
	textcat stuff. Now working/used yet

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: added textcat check to configure

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fix

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a stub for
	upcoming language support

2016-10-27  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fiddle with the (default) language settings

2016-10-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: working
	on language detection.

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/setting.cxx, src/tokenize.cxx: more cleanup

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/setting.cxx, src/tokenize.cxx: 
	cleanup

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* bootstrap.sh, include/ucto/setting.h, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/setting.cxx, src/tokenize.cxx,
	src/unicode.cxx: more code reshuffling. keep everything in one
	namespace for now

2016-10-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/Makefile.am, include/ucto/setting.h,
	include/ucto/tokenize.h, include/ucto/unicode.h, src/Makefile.am,
	src/setting.cxx, src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 
	some refactoring. splitup of huge tokenize.cxx file

2016-10-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: some
	small refactoring

2016-10-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: save
	current work. all tests still work

2016-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: save
	current state. lot of new but unused code :)

2016-10-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: started implementing language detection

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfiles1.ok: adapted another test

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx, tests/testlang.ok: added backward compatability for
	ISO 639-1. Adapted test

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/folia-lang.xml: converted to ISO 639-3

2016-10-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/ucto.cxx: simplified language parameter parsing.

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: fix some language isssues

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testlang.ok: needed for testing

2016-10-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	tests/folia-lang.xml, tests/testall, tests/testlang: added language
	support for folia documents. Some ISO-639 hackery to support older
	files too

2016-10-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testinclude.ok, tests/tst.cfg: fixed
	stupid bug. 'make check' now works agaiin. And tests/testinclude now
	gives correct result

2016-09-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bump version after release

2016-09-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: NEWS for the release

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: now request at least ICU 5.2

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: modified test.py so that ucto finds the data
	automagicly, not using -c.

2016-09-08  Kobus van der Sloot <sloot@applejack.science.ru.nl>

	* tests/test.py: let ucto find the data itself (not using -c)

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: attempt to let travis run Trusty

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testall: fix exit status stuff

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testall: make sure that the python script
	returns an exit code on failure

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: one more down :{

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: ok, try icu 4.9 because Travis uses old versions

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/tokenize.cxx: ask for ICU 5.0 or more. assuming
	the (?i) bug is fixed then

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/small.cfg, tests/testinclude.ok: fixed test now we check
	versions

2016-09-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: check version info if
	present.

2016-09-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 39ef5df2ab8f6b20eb5e43141f2897d0326b81ee Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Aug 23 16:52:28 2016
	+0200

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.en.tok.V, tests/test.fr.tok.V: adapted to newer
	insigths

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some refactoring

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/ligaturen.nl.tok.V, tests/normalisation.nl.tok.V,
	tests/normalisationUTF16.nl.tok.V, tests/nu.nl.tok.V,
	tests/ocr.nl.tok.V, tests/testfolia.ok, tests/testslash.ok: adapted
	tests to newest insights

2016-08-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: introduced type_symbol

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added some currency tests

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added code to parse [CURRENCY]. That entry was
	silently ignored!

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0054.nl.tok.V, tests/issue01.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/nu.nl.tok.V, tests/ocr.nl.tok.V,
	tests/smileys.nl.tok.V, tests/test.es.tok.V, tests/test.fr.tok.V,
	tests/testquotes.ok, tests/testquotes2.ok, tests/testslash.ok: 
	reflect advances in NOSPACE handling

2016-08-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed NOSPACE in sequences

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testone: merge good idea from 'recursive'
	branch

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctuation.ok: progress

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/ligaturen.nl.tok.V: another improvement

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testencoding2.ok: a moving target...

2016-08-19  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/nu.nl.tok.V, tests/test.nl.tok.V, tests/test.py,
	tests/testencoding2.ok, tests/testone: save latest and greatest
	results

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0083.nl.tok.V, tests/testencoding2.ok: results adapted to
	newest state of affairs

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: some fixes. getting closer to 'old' ucto

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: improved error message
	about invalid regex

2016-08-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/abbreviations.nl.tok.V, tests/test.nl.tok.V,
	tests/testpunctfilter.ok, tests/testpunctuation.ok: adapted test
	results to improved tokconfig-nl

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/test.en.tok.V, tests/test.fr.tok.V,
	tests/testpunctfilter.ok, tests/testpunctuation.ok: save current
	state. 1 problem left...

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx,
	tests/abbreviations.nl.tok.V, tests/bug0083.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/normalisation.nl.tok.V,
	tests/normalisationUTF16.nl.tok.V, tests/nu.nl.tok.V,
	tests/ocr.nl.tok.V, tests/test.nl.tok.V, tests/testencoding2.ok,
	tests/testfolia.ok, tests/testslash.ok: save current state of
	affairs. still not quite good

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: code layout

2016-08-17  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 6c1ad9e94acad2452ee089a9fdd14bb4ba6822b0 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Aug 16 17:59:12 2016
	+0200

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: test recursive application of rules

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: since we have the config in uctodata, we better
	check that!

2016-08-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: ignore more

2016-08-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: calling tokenizeWord() with a wrong parameter
	only accidentaly worked :{

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added heavy debug option

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: hmm adding -wB to diff somehow defeated tokenizing

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctfilter.ok, tests/testquotes.ok,
	tests/testquotes2.ok: adapted results

2016-08-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testpunctuation.ok, tests/testpunctuation.txt: added more
	tests

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: version bump after release

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: README update

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: README update

2016-07-30  Maarten van Gompel <proycon@anaproy.nl>

	* configure.ac: autoconf update to facilitate debian packaging

2016-07-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-07-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: bug fix announcement

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: moved uctodata test after other tests, so that
	PKG_CONFIG_PATH is set correctly

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-07-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: NEW, we release v0.9

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: huh?

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: typo

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: fix travis

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: improved errormessage.

2016-06-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/Makefile.am, src/ucto.cxx: per default, only install
	tokconfig-generic and its includes. Use 'generic' when no language
	is specified.

2016-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: interactive prompt is only shown when runnning
	from a TTY now input redirection still works as expected

2016-06-28  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac, src/tokenize.cxx: add libreadline support when
	running from stdin

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-de, config/tokconfig-es, config/tokconfig-fr,
	config/tokconfig-fy, config/tokconfig-generic, config/tokconfig-it,
	config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter,
	config/tokconfig-nl-withplaceholder, config/tokconfig-pt,
	config/tokconfig-ru, config/tokconfig-sv, config/tokconfig-tr: added
	'magical-micorsoft-quotes' to all languages (not all tested!)

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: as said: added a rule ...

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/dimin.nl.tok.V, tests/dimin.nl.txt: added a rule for
	diminutives like CD'tje en baby'tjes. Also added tests

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: added more suffix
	tests.

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: acute accents are used as quote's too :{

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: added an extra 'quote' to the rules
	concerning ' and the like

2016-06-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: Don't check BOM on interactive input!

2016-06-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/ucto.cxx: added --help and --version

2016-06-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testencoding2, tests/testencoding2.ok: added an extra test.
	very weird and inconvenient Windows file

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: better handling of DOS files with CR

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testencoding2.ok: fixed UTF16BE (for the 1
	testcase we have...)

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: improved UTF16 support BE still not working

2016-06-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/normalisationUTF16BE.nl.tok.V,
	tests/testencoding2, tests/testlanguage.ok: for now, we throw an
	encoding error on UTF16BE it is hard...

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/normalisationUTF16BE.nl.tok.V: added one more test, for BIG
	Endian. It FAILS!

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added comment

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: more refactoring

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed unused
	checkBOM() variant. some refactoring doen

2016-05-31  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/W1258.nl,
	tests/normalisationUTF16.nl.tok.V, tests/testencoding2,
	tests/testencoding2.ok: some code refactoring: check BOM only once.
	removed allatonce from tokenizeStream(( added poor-mans solution for
	UTF16. Should be tested on more files/platforms

2016-05-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small refactoring

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: started some
	refactoring

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: only check the BOM for files, and only for the
	first line

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testconf1.ok: code cleanup. Fixed minor
	change in test result.

2016-05-26  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: more fancy file loolup. fallback to default
	configdir.

2016-05-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: the tests now survive changes in libfolia version
	numbers

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/testall: removed debug line. Make sure new
	test is realy run.

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testclassnormalization, tests/testclassnormalization.ok: 
	added a test

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added a
	--normalize option. This allows for every word in a certain
	TokenClass to be mapped to that class' name

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: NEWS, again a new release

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: ignore README

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-nl-sonarchat,
	config/tokconfig-nl-twitter, config/tokconfig-nl-withplaceholder,
	config/tokconfig-pt, config/tokconfig-ru, config/tokconfig-sv,
	config/tokconfig-tr: added adapted ABBREVIATION rule to alls
	languages

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/{issue04.nl.txt => issue04.nl.txt.FAILS}: 
	added extra debug lines.  move issue04 out of the tests. It is known
	to fail AND WONTFIX

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue04.nl.tok.V, tests/issue04.nl.txt: added a new test

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue03.nl.tok.V, tests/issue03.nl.txt: added test for
	issue03 (fixed already)

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0065.nl.tok.V, tests/bug0065b.nl.tok.V,
	tests/buggie01.nl.tok.V, tests/buggie01.nl.txt, tests/ocr.nl.tok.V,
	tests/quotetest_folgert.nl.tok.V,
	tests/quotetest_folgert2.nl.tok.V,
	tests/quotetest_folgert3.nl.tok.V,
	tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert5.nl.tok.V,
	tests/quotetest_folgert6.nl.tok.V,
	tests/quotetest_folgert7.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested2.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.de.tok.V,
	tests/test.en.tok.V, tests/test.es.tok.V, tests/test.fr.tok.V,
	tests/test.nl.tok.V: adaped to change in verbose output

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: modified ABBREVIATION rule. A ,: or ; may
	follow directly

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: small fix in verbose output: no newline at
	ENDOFSENTENCE when NOSPACE too

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/issue01.nl.tok.V, tests/issue01.nl.txt: more usefulle name

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS, configure.ac: bumped version after release. added NEWS

2016-04-25  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/buggie01.nl.tok.V, tests/nu.nl.tok.V: 
	merged bug fixes from 'generic'

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-en, src/tokenize.cxx: reimplemented regexp stuff,
	but no improvement yet

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/buggie01.nl.tok.V: added a "good" result

2016-04-20  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit b25b97c646df98715b93c6b2440de41d58130962 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Apr 19 16:27:57 2016
	+0200

2016-04-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/buggie01.nl.txt: added a new test

2016-04-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: addapted testresults to recent libfolia

2016-03-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed problem with passThru on an empty line

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: removed debug line, now the test is fixed

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: tru another testmethod

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testone: added -B option to diff

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: output 1 of the files that seem to fail on travis....

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: bumped version after release

2016-03-10  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* NEWS: news!

2016-03-09  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 45c9ddfa67de37eefa775c0594153215351f255b Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Mar 8 16:54:51 2016
	+0100

2016-03-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted tests to libfolia version change

2016-03-08  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: this branch needs libfolia 1.0

2016-03-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, tests/folia1.xml, tests/folia2.xml,
	tests/folia3.xml, tests/slashes.xml: replaced all div-annotation by
	division-annotation

2016-02-29  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: use KWargs to instantiate folia elements

2016-02-23  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/testfolia.ok, tests/testfolia2.ok: small diff in results
	because of newer folia version.

2016-02-22  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/Makefile.am, src/tokenize.cxx: Rule
	Of Three!

2016-02-19  Maarten van Gompel <proycon@anaproy.nl>

	* README.md: oops, ucto had wrong lamabadge

2016-02-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit 9f4405001213e0a31c05c677780d3e2fab8af4d9 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Thu Feb 18 16:49:26 2016
	+0100

2016-02-17  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 736d144b0a4d8c8da9626554a1dac9df53b443a3 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Wed Feb 17 14:03:39 2016
	+0100

2016-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: oesp

2016-02-15  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit cbc8840d0d48d301e22d5b0c0638e9d90d720589 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Feb 3 12:08:13 2016
	+0100

2016-02-02  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml, include/ucto/tokenize.h, src/tokenize.cxx,
	src/ucto.cxx, src/unicode.cxx: a 'generic' branch to wotrk with the
	new 'generic' libfolia

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: test only the master

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .travis.yml: test only master

2016-02-01  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx: added more includes for the
	'generic' branches

2016-01-20  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-pt, config/tokconfig-ru,
	config/tokconfig-sv, config/tokconfig-tr: added meta rules to all
	language configs

2016-01-20  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-it: aggiunto abbreviazioni italiani e regoli
	metà

2016-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testslash.ok: adapted tests to newer
	libfolia version

2016-01-18  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* configure.ac: Bumped version after release

2016-01-14  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore, Makefile.am, NEWS, configure.ac: from SVN to Git

2016-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: 
	changed rules added two more tests

2016-01-13  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: cleaner code

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: added
	a new rule and test

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: we now support %include from different locations
	(real paths) lot of code reformatting. generally use {} in if
	statements.

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx, tests/test.nl.tok.V,
	tests/test.nl.txt: added another rule, and a test, of course

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/nl_afk.abr, config/tokconfig-nl, tests/test.nl.tok.V,
	tests/test.nl.txt: added some more abbrev's added a
	ABBREVIATION-COMPOUND rule and a testcase

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/nl_afk.abr, config/url.rule, tests/urls.nl.tok.V,
	tests/urls.nl.txt: added wnd to dutch abbreviations fixed url rule
	and added testcase

2016-01-12  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added default: to switch

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: rules are stored in a
	map now.  Ehen multiple rules with the same name occur, the last one
	is kept.

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed logging

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: new style config file for dutch.  with
	meta-rules

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: added a configurable split token to meta-rules
	default is % can be changed per rule

2016-01-11  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* include/ucto/tokenize.h, src/tokenize.cxx: continued work on
	META-RULES

2016-01-08  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 14e0a858a5a02c7e5633d246e5c31730690d21d8 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Fri Jan 8 09:40:07 2016
	+0100

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx: added code to parse an use
	META-RULES.  Works for 1 rule yet.  So don't use it!

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx: adapted 2 rules.
	introduces a dutchism in the code (good for at least English too, i
	think) More generic solution needed

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/url.rule: added a top level and cleaned up too

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/url.rule, tests/urls.nl.tok.V, tests/urls.nl.txt: fixed url
	rules.  added tests

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, src/tokenize.cxx, tests/suffix.nl.tok.V,
	tests/suffix.nl.txt, tests/urls.nl.tok.V, tests/urls.nl.txt: some
	more fixes. Also added more tests

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/encoding.nl.tok.V: WORD-COMPOUND may be
	all numbers too now.  Moved DATE detection in front of WORD-COMPOUND
	This triggered 1 change in 1 test.

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: WORD-COMPOUND may also start with numers now.
	(16-jarige, 12-en-een-half) Als moved WORD-COMPOUND to hit before
	NUMBER-STRING

2016-01-07  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: fixed NUMBER-ORDINAL rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V: adapted to strickter rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: strickter rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: typo

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/ocr.nl.tok.V, tests/test.nl.tok.V: all
	tests green

2016-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* : commit 637ccc620f231e928c58f8c668c38e50e4579501 Author: Maarten
	van Gompel <proycon@anaproy.nl> Date:   Wed Jan 6 13:41:25 2016
	+0100

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V: updated testresult

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit dd7e11259e76ea9f38837e9125c7e85581688f7a Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Wed Jan 6 13:32:05 2016
	+0100

2016-01-06  Maarten van Gompel <proycon@anaproy.nl>

	* config/tokconfig-nl: always keep hyphens in words (hyphen is not a
	separator)

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl: added a NUMBER-COMPOUND rule

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* config/tokconfig-nl, tests/test.nl.tok.V, tests/test.nl.txt: added
	more tests (some failing yet)

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt: added more word-compound
	tests

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: added -Z to the diff command.  Don't care about
	trailing whitespace

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.py: added -b to diff command. otherwise trailing whit

2016-01-06  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/test.nl.tok.V, tests/test.nl.txt, tests/test.py: added a
	(failing!) test

2016-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* .gitignore: added

2016-01-04  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* README, configure.ac, fink/ucto.info, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: updated copyright/disclaimer

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: fixed includes

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* src/tokenize.cxx: handle leading spaces in passthru mode correctly

2015-11-30  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* tests/passthru2.txt: added a leading space to test that it is
	skipped

2015-11-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* README: Small edits

2015-11-24  Ko van der Sloot <K.vanderSloot@let.ru.nl>

	* : commit d6c2fb1388ccf36462e8c7c0d92bf7fd4ca9f238 Author: Ko van
	der Sloot <K.vanderSloot@let.ru.nl> Date:   Tue Nov 24 12:13:55 2015
	+0100

2015-11-24  Maarten van Gompel <proycon@anaproy.nl>

	* README: README update

2015-11-24  Maarten van Gompel <proycon@anaproy.nl>

	* README: README update

2015-06-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml,
	tests/testfolia.ok, tests/testfolia2.ok: adapted tests to libfolia
	version 0.12 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18477
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml, src/tokenize.cxx: attempted fix for now non-existant
	FoLiAElement::deeptext().. will a normal text() do? (it's supposed
	to be deep by default now)? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18474
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-withplaceholder: Added
	variant of dutch tokenizer that includes placeholders (variables in
	the form %identifier% that will be preserved as-is, useful for
	subtitutions after tokenisation) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18443
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest2_folia.nl.xml, tests/partest_folia.nl.xml: tests
	have changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18421
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok: tests have changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18420
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: smal fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18416
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-19  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* README: updated instruction after flogging git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18414
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-05-18  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* README: added installation instructions git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18410
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-04-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: updated, modernized git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: adapted usage() git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18301
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: safeguard git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18300
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18299
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to fix punctuation filter for FoLiA
	input git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18298
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18297
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added experimental code to allow -l and -u on
	FoLiA input.  needs more testing!!!! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18296
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: fall back to c++0x git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18266
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-03-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: small improvement git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18135
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18069
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: modernized. using
	C++11 features now! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18056
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: C++11 aware git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18035
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed useless const.  typos in comments
	corrected.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18019
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-02-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: implemented smarter
	tokenizeSentenceStream() function.  Needed to fix flushSentences()
	to get it to work! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18017
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added another
	tokenize() function. (reading tokenized strings froma steram) We
	realy need to weed these out! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18014
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18012
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: get ready for release 0.8 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18009
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: 2015 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@18006
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: allatonce enabled by default for tokenize() to
	folia doc git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17995
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h: 2015 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17993
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added new
	tokenize(string,string) meta-function for the API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17959
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: added FRACNUMBER too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17958
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-fy, config/tokconfig-generic,
	config/tokconfig-it, config/tokconfig-nl,
	config/tokconfig-nl-twitter, config/tokconfig-pt,
	config/tokconfig-ru, config/tokconfig-sv, config/tokconfig-tr: 
	fixing date rules and adding FRACNUMBER git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17957
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: <Flo> zou je ook nog een variant
	dd/mm toe kunnen voegen? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17956
	12f355fe-0486-481a-ad91-c297ab22b4e3

2015-01-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: handle dates with slashes in twitter:
	< proycon> doen wij datums niet met hyphens officieel in het
	Nederlads?     <Flo>| heeft twitter maling aan ;) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17955
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-ru: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17934
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-ru: added Russian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17933
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: fixed hyphens according to
	/usr/share/doc/groff-base/README.Debian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17908
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-12-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: fixed hyphens according to
	/usr/share/doc/groff-base/README.Debian git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17907
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed terrible bug, also some code-cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17873
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17872
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/tokenize.cxx, tests/testfolia.ok,
	tests/testfolia2.ok: made ucto Part aware.  2 tests fail atm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17871
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, tests/partest.nl.tok,
	tests/partest.nl.txt, tests/partest2.nl.tok, tests/partest2.nl.txt,
	tests/partest2_folia.nl.txt, tests/partest2_folia.nl.xml,
	tests/partest_folia.nl.txt, tests/partest_folia.nl.xml,
	tests/test.py: Re-added memory saving code: tokens of entire input
	not retained in memory prior to output   +    added extra tests  +
	updated test.py framework to deal with FoLiA tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17865
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: yet another fix for travis git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17857
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: (travis-ci) show some logs after failure git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17856
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17854
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testone: tests return proper return codes now git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17852
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: travis-ci: forgot libfolia as dependency git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17851
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: fix for travis git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17850
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: travis update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17846
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: update for travis, output tests to stderr instead of
	stdout git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V, tests/test.de.txt: fixed german test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: another update for travis-ci git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: fix for travis-ci build git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17842
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* .travis.yml: added travis-ci git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17841
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: shut up! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17840
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added
	experimental punctuation filter.  Filters most punctuation. but NOT
	when rules like ABBREVIATION SUFFIX etc are applicable.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17839
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testpunctfilter, tests/testpunctfilter.ok: 
	added a test for the experimantal punctuation filter git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17838
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{quotetest_folgert8.nl.txt =>
	quotetest_folgert8.nl.txt.disabled}: disable this one too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17837
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{munten.nl.txt => munten.nl.txt.disabled},
	tests/{units.nl.txt => units.nl.txt.disabled}: disable 2 tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17836
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Reverted the pevious 2
	patches.  The fist was wrong. The second didn't fix it.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17835
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testpunctuation.ok: accept suboptimal solution for now git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17834
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: more libfolia bumps git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17833
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-11-20  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok: bumped libfolia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17832
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: attempted fix for FoLiA
	output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17756
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Reducing the
	unnecessarily high memory usage, especially when no FoLiA output is
	requested git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17755
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: macro fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17746
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-10-19  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: added compiler fallback git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17737
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac, src/tokenize.cxx: bumping git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17695
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17687
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: clearer option handling, better checks, and fixed
	non-working -S option (nobody seems to use it) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17686
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: no more nasty segfault for ucto -h git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17684
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: small edit to satisfy clang git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17682
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17674
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: removed ugly hack git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17662
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: added a useful function to the API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17654
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed declared, but undefined function git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17651
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-09-13  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: little alternative
	function working with pointers, needed for python-frog git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/ucto.cxx: use ticcutils 0.6 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17549
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-23  antalb <antalb@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk.abr: added RK git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17546
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: switched to TiCC CommandLine stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17544
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-14  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapt to newer libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17513
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: generally switched to
	UChar32 and Unicode codepoints.  removed some dead code too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17484
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added EMOTICON recognision. Needed some rework
	vor 2 codepoint long characters git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17483
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/smileys.nl.tok.V, tests/smileys.nl.txt, tests/testfolia.ok,
	tests/testfolia2.ok: added Emoticons to the smiley test.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17482
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some code moved around git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17453
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V, tests/testfolia.ok, tests/testfolia2.ok: some
	tests are changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17451
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-07-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: handle Note like Caption git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17418
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapted to new libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17393
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/pt.abr, config/tokconfig-pt: Adicionei
	regras para tokenização portuguesa.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17387
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: avoid crash on empty inputfile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17357
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tst.sh: hmm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac, include/ucto/tokenize.h, src/tst.sh: NEWS
	fixed 'make distcheck' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17332
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: and reinstated git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17245
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: removed soft link git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17244
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: made getSentence() public git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17237
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: moved folia include to header git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17236
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: removed "config.h" git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17235
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-05-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, include/ucto/tokenize.h, src/tokenize.cxx: added to
	facilitate secret project ;) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17234
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: add git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17181
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17180
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/Makefile.am, src/tst.sh: fixed 'make check' (but
	only for automake 1.13 and up) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17177
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-04-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, Makefile.am: fixed 'make dist' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@17175
	12f355fe-0486-481a-ad91-c297ab22b4e3

2014-01-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/ligatures.filter, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: Happy Newyear! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16823
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-12-16  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-tr: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16771
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/unicode.cxx: compiler satisfaction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16652
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* bootstrap => bootstrap.sh, configure.ac: modernized git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16634
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.es.tok.V, tests/test.fr.tok.V: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16574
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-fr, config/tokconfig-it,
	src/tokenize.cxx: fixes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16573
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-it, config/tokconfig-nl,
	config/tokconfig-nl-twitter, config/tokconfig-sv, src/tokenize.cxx,
	tests/test.pt.txt.disabled: Case insensitivity does not work, libicu
	too buggy, made rules explicitly case insensitive git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16572
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Do NOT replace existing <t> content.  Added
	Caption to exceptions for addign <p> nodes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16564
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: test has changed after
	decision about embedden stuff in <t> nodes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16563
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed ListItem stuff.  Also DON'T add a
	paragraph to Head elements git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16562
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: keep track of line
	numbers in verbose output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16560
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-generic: generic update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16525
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es, tests/test.es.tok.V,
	tests/{test.es.txt.disabled => test.es.txt}, tests/test.fr.tok.V: 
	spanish update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16524
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16523
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr, tests/{test.fr.txt.disabled => test.fr.txt}: 
	changes in french tokenisation (suffixes) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16522
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: oops git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16521
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: test result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16520
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-fr,
	tests/test.en.txt: updating configurations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16519
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-09-17  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/tokenize.cxx: readded (?i) , test still to be
	added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16518
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-08-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: added a reset() member.
	Enables a fresh start.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16418
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-08-01  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-generic: added generic
	language-independent tokeniser configuration, for unsupported
	languages git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16407
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: hmpfff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16331
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok, tests/testslash.ok: modified tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16330
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: small change in testresult git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16307
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed problem with empty text git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16304
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/empty.xml, tests/testfoliain, tests/testfoliain.ok: added a
	test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16303
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: re-re-refixed folia stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16297
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia6.xml, tests/testfoliain, tests/testfoliain.ok,
	tests/testslash.ok: modified and extended test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16296
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: oesp git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16295
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: adapted test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16294
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/ucto.cxx: sorted out textclass stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16293
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/ocr.nl.tok.V, tests/ocr.nl.txt: added a test with some
	terrible stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16292
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: changed tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16291
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-07-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: folia handling improved: - separate --inputclass and --outputclass options - better <t> text added on <p> and  <s> git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16290
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain, tests/testfoliain.ok: added a test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16281
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed outputclass stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16280
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testoption1.ok, tests/testslash.ok, tests/testusage.ok: 
	modified tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16279
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: replaced the --textclass opion.
	Use --inputclass ans --outputclass! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16278
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testpunctuation, tests/testpunctuation.ok,
	tests/testpunctuation.txt: added another test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16231
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: keep fiddling with EOS stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16230
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: desired result is changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16226
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/testencoding2.ok: hacking the eosmarkers
	again git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16225
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/standard-eos.eos, include/ucto/tokenize.h,
	src/tokenize.cxx: added some ugly hacks to improve EOS detection
	when quotes are in sight.  A code cleanup is very desirable! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16224
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/testall, tests/testquotes.ok,
	tests/testquotes2, tests/testquotes2.ok: added more tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: removed comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16201
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V: modified test result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16186
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: solve the Alex' and Frans' problem with an
	extra RULE (only match words with an Uppercase) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16185
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: modified test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16175
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V: hmm..  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16173
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: added more SUFFIX-es git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16172
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/suffix.nl.tok.V, tests/suffix.nl.txt: new test added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16171
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to make more silly-quote proof git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16168
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-06-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/unicode.h, src/tokenize.cxx, src/unicode.cxx,
	tests/encoding.nl.tok.V: updated unicodefilter API git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16163
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-05-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fix ENDOFSENTENCE problems in combination with
	punctuation.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16140
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-05-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testquotes.ok, tests/testslash.ok: 
	adaptes tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@16139
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15945
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: wrong date in NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15943
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am: increment current library version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15925
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.nl.tok.V: adapt to new tokenisation rules git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15918
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: changed folia include path git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15910
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-03  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15905
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: we need the most recent ticcutile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15885
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx,
	src/unicode.cxx: modified folia include path git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15875
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-04-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* : tags git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15851
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: more quote hacking git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15809
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes.ok, tests/testslash.ok: some rule neams have
	changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15807
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: small edits git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15806
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx, tests/testquotes.ok,
	tests/testslash.ok: some rexexp patches for words like A50,
	vanessa23 etc.  adaptes tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15805
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: more quotes for '40 etc.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15803
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: small fix for prime's/quotes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15802
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added small hack to appen 'dangling' quotes to
	the previous sentence.  This avoids sentences with only a ' or a ".
	(extend to exotic quotest too?) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15797
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: be sure to take "current" as the default class for
	text searching git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15774
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: oesp.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15773
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, config/tokconfig-en, config/tokconfig-es,
	config/tokconfig-fr, config/tokconfig-it: fixed ABBREVIATION rule
	for other languages git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15769
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: Updated.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15766
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: added a
	--textclass option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15765
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-03-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia5.xml, tests/testfoliain, tests/testfoliain.ok,
	tests/testoption1.ok, tests/testusage.ok: added test for the new
	--textclass option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15764
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added the passthru fix for FoLia docs also
	elsewhere git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15693
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: passtru mode was handled incorrectly for FoLiA
	input git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15683
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: make sure that we tokenize existing FoLiA in our
	own set! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15682
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-14  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: added to makefile git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15675
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-14  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/es.abr, config/tokconfig-de, config/tokconfig-en,
	config/tokconfig-es, config/tokconfig-fr: updated foreign
	tokenisation configurations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15674
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some debugging lines silenced git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bracket.nl.tok.V, tests/bracket.nl.txt: added a test for the
	EOS in brackets problem.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15646
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-02-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/test.py: added some trickery to transfer
	EOS markers to next token if it is a closing bracket. EXPERIMENTAL git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15645
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testone: improved test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/utf8bom.nl: we need this files too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15586
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: now we handle (and use)
	BOM markers.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testencoding2, tests/testencoding2.ok: added a test for the
	BOM git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15584
	12f355fe-0486-481a-ad91-c297ab22b4e3

2013-01-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h,
	src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: Bump year git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15571
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-06  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1: wrong man git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15532
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk.abr: removed 'al' beacuse it isn't an abrivation,
	ans it IS a word! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15529
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/testfolia.ok, tests/testfolia2.ok,
	tests/testfoliain.ok, tests/testslash.ok: use datetime='now()' in
	declarations. And test that too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15528
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: add datetime attribute to token-annotation
	declaration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15522
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: only add a set default definition when there is
	something to do git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15520
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: removed odd typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15519
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-12-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/{nl_afk => nl_afk.abr}: fixed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15518
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: added to config data git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15490
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/nl_afk, config/tokconfig-nl-twitter: adding forgotten file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15489
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter: use
	the nl_afk abbreviations file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15480
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, include/ucto/tokenize.h, src/tokenize.cxx: 
	implemented %include for ABBREVIATIONS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15479
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: docupdate git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15478
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: updated after comments git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15472
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.bib, docs/ucto_manual.tex: Added documentation
	(finally) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15468
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/reverse-smiley.nl.tok.V, tests/reverse-smiley.nl.txt,
	tests/smileys.nl.tok.V, tests/smileys.nl.txt: added a test for
	reversed smileys.  addapted the smiley testbtoo.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15397
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule: attempt to support 'repetitive' smileys like
	this :))))) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15396
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-sonarchat, config/tokconfig-nl-twitter: added
	REVERSE-SMILEY rule git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15395
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule, config/tokconfig-nl: attempt to define a rule
	for reverse smileys {-8.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15394
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-11-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl-twitter: get in line with standard
	tokconfig-nl git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15387
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testtwitter, tests/testtwitter.ok,
	tests/twitter.txt: added a test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15386
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* bootstrap: bootstrap fix (matched failed on automake 1.11.6 with
	Ubuntu 12.10) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15361
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: adapted to changed FoLiA stylesheet handling git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15355
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfolia2.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted to improved libfolia output (explicit
	UTF-8 encoding) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15278
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-10-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h: safeguarded the
	safeguards git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15275
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-09-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: unneeded check git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15158
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-08  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, configure.ac, include/ucto/tokenize.h, src/tokenize.cxx: 
	use ticcutils in TiCC namespace git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15062
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{test.es.txt => test.es.txt.disabled}, tests/{test.fr.txt
	=> test.fr.txt.disabled}, tests/{test.pt.txt =>
	test.pt.txt.disabled}: teporary disabled until Proycon does his job git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15027
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en: improved english tokenization rules git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15026
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.tok.V: accaptable result git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15025
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de: V.S. en d.h. gaan nu goed. a.d.Donau
	misschien niet git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15024
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V: ik denk dat dit dichter tegen de waarheid
	zit.  Nu nog zorgen dat het er uit komt :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15023
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.tok.V: new git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15022
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-de, src/tokenize.cxx: attempt to fix .de git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15021
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: keep it private git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15020
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-08-01  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: save intermediate
	refactored state git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15019
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15016
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: fixed eos problems.
	'-s' option seems to work now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15015
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall: test the new test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15014
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/eos.txt, tests/testoption-s, tests/testoption-s.ok,
	tests/utt.txt: added a new test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15013
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: better info.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15012
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: typo, and some refactoring.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15011
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: small cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15010
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: further refactoring git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15009
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: re-re-re-factored :P git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15005
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: one small step… git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15004
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: more cleanup.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15003
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: some more refactoring,
	and added some comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15001
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: ok, more next week git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@15000
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: next refactor step git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14999
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: hmm. progress… git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14998
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: save intermediate result during cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14997
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: refactoring: removed
	parameters without any visable changes in behaviour.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14996
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert8.nl.tok.V: Correct the test case.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14995
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-18  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* AUTHORS: Honour Folgert git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14994
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: indentation and such git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14993
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: some renaming for my
	own mental health git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14992
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: remove unused functionm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14991
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: started some
	refactoring git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14990
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia2.txt, tests/testall, tests/testfolia2,
	tests/testfolia2.ok: added a complex folia test. asume the result is
	OK for now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14989
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: I think this more in line with wat we
	want.  still doubtfull quotation git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14988
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to fix bug94 (fixes quotetest_folgert7
	too??) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14987
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert7.nl.tok.V: I think the correct file wasn't
	correct at all. Folgert?? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14986
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed debugging lines git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14972
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14962
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-05  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed folia id's git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14961
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: partial revert git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14947
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14946
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-07-03  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fy: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14945
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-13  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes: properties git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14861
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-13  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0094.nl.tok.V, tests/bug0094.nl.txt: new bug found git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14860
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0065b.nl.tok.V, tests/quotetest_folgert.nl.tok.V,
	tests/quotetest_folgert2.nl.tok.V,
	tests/quotetest_folgert3.nl.tok.V,
	tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert5.nl.tok.V,
	tests/quotetest_folgert6.nl.tok.V,
	tests/quotetest_folgert7.nl.tok.V,
	tests/quotetest_folgert8.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested2.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.nl.tok.V,
	tests/testquotes.ok: addapted to changed verbose output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14847
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: ok, we are more happy now, but there is
	still a BEGINOFSENTENCE missing I think git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14846
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed indentation of the source also always put
	an extra newline after ENDOFSENNTENCE in verbose mode git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/nu.nl.tok.V: dit is al meer wat we willen git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.py: reversed parameters to diff.  now we do 'diff <what
	we got> <what we want>" This is more in line with other tests git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-06-04  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* COPYING, config/tokconfig-nl, src/tokenize.cxx,
	tests/quotetest_folgert5.nl.tok.V, tests/quotetest_folgert5.nl.txt,
	tests/quotetest_folgert6.nl.tok.V, tests/quotetest_folgert6.nl.txt,
	tests/quotetest_folgert7.nl.tok.V, tests/quotetest_folgert7.nl.txt,
	tests/quotetest_folgert8.nl.tok.V, tests/quotetest_folgert8.nl.txt: 
	Fix nested quoted sentences; added testcases; small additions to
	tokconfig-nl git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14833
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: update header, u_isquote git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14826
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: getting used to svn...  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14825
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-31  fkarsdorp <fkarsdorp@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quotetest_folgert4.nl.tok.V,
	tests/quotetest_folgert4.nl.txt: small fix in quote detection,
	testcase added.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14824
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, src/Makefile.am: exterminated AC_DEFINE_DIR here too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14823
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14820
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/bug0065b.nl.tok.V,
	tests/quotetest_folgert.nl.tok.V, tests/quotetest_folgert.nl.txt,
	tests/quotetest_folgert2.nl.tok.V, tests/quotetest_folgert2.nl.txt,
	tests/quotetest_folgert3.nl.tok.V, tests/quotetest_folgert3.nl.txt,
	tests/test.py: Folgert fixed quote detection, extra tests added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14819
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: handle ' just like " in quote-detection git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14777
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/standard-quotes.quote: updated comment git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14776
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testquotes.ok: let's call this OK, for now.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14775
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/qtest.nl, tests/qtest.spec, tests/testall, tests/testquotes: 
	added tests for quotes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14774
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: attempt to fix 'time' problem git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14721
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/smiley.rule, tests/smileys.nl.tok.V: fixed smiley git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14720
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-05-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/datetime.nl.tok.V, tests/datetime.nl.txt,
	tests/smileys.nl.tok.V, tests/smileys.nl.txt, tests/testall,
	tests/testconf2.ok: added tests (still fail!) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14719
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-04-26  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/slashes.xml, tests/testfolia.ok, tests/testfoliain.ok,
	tests/testslash.ok: adapted to new folia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14703
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bump version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: get ready for release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: numb change git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14472
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok, tests/testslash.ok: 
	newer libfolia version git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14471
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-03-09  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-fy: added frisian
	tokenisation (untested yet) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14432
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-29  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14356
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: News! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14342
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-27  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: bit of cleanup git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14336
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: formatting git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14322
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/Makefile.am, tests/slashes.xml, tests/testall,
	tests/testfoliain, tests/testslash, tests/testslash.ok: added a test
	for wierd \\\ problems git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14304
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: replaced all ?i stuff in
	regular expressions.  in an attempt to avoid ICU bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14302
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-sonarchat: added for sonar
	chat tokenisation git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14292
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: maarten mangled the test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14287
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml,
	tests/folia4.xml, tests/testfoliain, tests/testfoliain.ok: test
	update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14286
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex, include/ucto/tokenize.h, src/tokenize.cxx: 
	folia input fixes (regarding event tag) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14285
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok: adapted to folia 0.8 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14284
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-08  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: typo git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-02-08  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto_manual.tex: added beginnings of ucto manual git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14222
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/ucto.cxx, tests/testoption1.ok, tests/testusage.ok: 
	added a -X and an --id= option. this is more in line with frog.  -x
	<id> still works git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14159
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: added 'escape' possibility
	to config files now we can read: [ATTACHEDSUFFIXES] \['`’]s
	\['`’]t \['`’]n which means: [ATTACHEDSUFFIXES] ['`’]s ['`’]t ['`’]n git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14145
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml,
	tests/testfolia.ok, tests/testfoliain.ok: fixed tests. libfolia is
	strict on <div> nodes (because FoliA.pdf demands it) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14144
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr, config/tokconfig-nl,
	config/tokconfig-nl-twitter: support for unicode apostrope 0x2019 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14134
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: Make debugging output
	go to theErrlog generally git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@14109
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-12  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* ucto.spec: was moved to Timbl-dev/rpm git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13991
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: cleanup configure mess git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13944
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after Release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13934
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, config/Makefile.am, config/smiley.rule,
	config/tokconfig-nl, config/tokconfig-nl-twitter: put SMILEY in a
	separate .rule file.  It is language independent.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13918
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl: added a much more powerfull SMILEY rule.
	Which also works with older ICU cversions! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13917
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Even better error message git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13916
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf2.ok, tests/testfolia.ok, tests/testfoliain.ok: 
	adapted to small changes in output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13915
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: slighly better error message git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13914
	12f355fe-0486-481a-ad91-c297ab22b4e3

2012-01-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, include/ucto/unicode.h,
	src/tokenize.cxx, src/ucto.cxx, src/unicode.cxx: next year git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13842
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13762
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-21  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, configure.ac: bumped version somehow 0.4.8 was already
	released git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13756
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-14  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: be more informative git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13698
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: libfolia API is a moving target git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13652
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac, include/ucto/tokenize.h, src/tokenize.cxx: adapted
	to major change in libfolia: AbstractElement ==> FoliaElement git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13648
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-12-06  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfolia.ok, tests/testfoliain.ok: adapted to new folia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13647
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testfoliain.ok: test results changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13613
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml: stray " git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13612
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia1.xml, tests/folia2.xml, tests/folia3.xml: replaced \n
	wilt real newline character &#xA; git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13611
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: attempt to fix -F. still uncertain git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13610
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{folia.xml => folia1.xml}, tests/folia2.xml,
	tests/folia3.xml, tests/testfoliain, tests/testfoliain.ok: added and
	changed -F tests Still not sure if this is wahat we want git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13609
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl-twitter: Added specicial
	tokeniser configuration for Twitter (dutch) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13608
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, config/url.rule, tests/smileys.nl.tok.V,
	tests/smileys.nl.txt, tests/urls.nl.tok.V, tests/urls.nl.txt: Added
	URL-DOMAIN and SMILEY detection (enabled only in tokconfig-nl for
	now) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13607
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia.xml, tests/testall, tests/testfoliain,
	tests/testfoliain.ok: added simple test for the _F FoLiA XML input
	option git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13605
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* docs/ucto.1, src/ucto.cxx: small change in usag() Enhanced man
	page git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13601
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testoption1.ok, tests/testusage.ok: again output was changed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13600
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13599
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13598
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testoption1.ok, tests/testusage.ok: output has changed, so
	adapt test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13597
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-28  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: removed 2 compiler warnings 2 others still
	there: Proycon, what were you thinking? git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13596
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: completed FoLiA input
	support (needs more testing though) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13587
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-26  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: tokenisation of sentences in existing FoLiA now
	works git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13586
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixes for FoLiA input support git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13585
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-25  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: 
	Attempting to add FoLiA read support to ucto.. not entirely finished
	and tested yet! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13583
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-09  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: added some hacks to avoid a nasty ICU bug.  see
	http://bugs.icu-project.org/trac/ticket/8824 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13536
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13511
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/Makefile.am, src/tst.sh: more lost stuff git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13510
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, tests/Makefile.am, tests/bug0083.nl.tok.V,
	tests/bug0083.nl.txt, tests/test.de.txt, tests/test.en.txt,
	tests/test.es.txt, tests/test.fr.txt, tests/test.pt.txt,
	tests/tst.cfg, tests/tst.ok, tests/tst.txt: hmm lost a lot of stuff
	in the merge :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13509
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-11-02  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, NEWS, config/Makefile.am, config/tokconfig-en,
	config/tokconfig-es, config/tokconfig-fr, config/tokconfig-generic,
	configure.ac, docs/ucto.1, include/ucto/tokenize.h,
	include/ucto/unicode.h, src/Makefile.am, src/tokenize.cxx,
	src/tst.sh, src/ucto.cxx, src/unicode.cxx, tests/Makefile.am,
	tests/bug0083.nl.tok.V, tests/bug0083.nl.txt,
	tests/quoteproblem.txt, tests/test.de.txt, tests/test.en.txt,
	tests/test.es.txt, tests/test.fr.txt, tests/test.pt.txt,
	tests/test.py, tests/testencoding2, tests/testfolia,
	tests/testfolia.ok, tests/testinclude, tests/testnormalisation,
	tests/testone, tests/testoption1.ok, tests/testoption2,
	tests/testpassthru, tests/testusage.ok, tests/tst.cfg,
	tests/tst.ok, tests/tst.txt, ucto.pc.in: Merged the folia branch
	into ucto.  ucto now always requires libfolia git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13508
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: some layout changed and slack removed git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13462
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-25  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx: small changes git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13459
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: made compiler happy.  Removed dead code git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13449
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: support for spanish ¿ and ¡ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13366
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-10-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13365
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am: update git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13349
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-30  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-generic: generic tokeniser configuration, not for
	any particular language  (not recommended) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13348
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en: English tokenisation: wo n't  , do n't git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13347
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-es: Added spanish configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13346
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: Added french abbreviations git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13345
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-fr: updates to french configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13344
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-en, tests/test.en.txt: update English
	configuration git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13342
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13341
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13340
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.de.txt: generic german test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13339
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: correction git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13336
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: corrections git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13335
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.pt.txt: Generic portuguese test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13334
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.es.txt: Generic Spanish test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13333
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.fr.txt: more french git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13332
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.fr.txt: Initial generic french test.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13331
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-29  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.en.txt: generic English test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13330
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/Makefile.am: last small oesp git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13189
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-09-15  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* Makefile.am, configure.ac, src/Makefile.am, src/tst.sh,
	tests/Makefile.am, tests/tst.cfg, tests/tst.ok, tests/tst.txt: 
	attempt to provide 'make check' and 'make distcheck' git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13188
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: Bumped version after Release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@13043
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V: attempt for more precise desired version\ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12956
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0083.nl.tok.V, tests/bug0083.nl.txt: added an other
	quoteing problem and the disered output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12955
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12950
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: release! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12949
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: the API should have the same defaults ofcourse git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12942
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/test.py, tests/testfolia, tests/testoption1.ok,
	tests/testusage.ok: adapted test to changed QuoteDetection regime.
	hardcoded -Q in test.py, arghh.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12940
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, docs/ucto.1, src/tokenize.cxx, src/ucto.cxx: Reversed
	working of -Q Updated man more NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12938
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx, src/ucto.cxx: applied
	same changes to readsettings an init as in the libfolia branch git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12931
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-22  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok, tests/testconf2.ok, tests/testfiles1.ok,
	tests/testinclude.ok, tests/testlanguage.ok: outpu is changed a
	little bit git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12930
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-08-18  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx, tests/quoteproblem.txt,
	tests/quotetest_nested2.nl.tok.V, tests/quotetest_nested2.nl.txt: 
	Fixed nested quote issue (unbalanced quote/sentence issue) and build
	in extra protections. Incl. test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12909
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-07-28  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/abbreviations.nl.tok.V, tests/bug0048.nl.tok.V,
	tests/bug0051.nl.tok.V, tests/bug0052.nl.tok.V,
	tests/bug0054.nl.tok.V, tests/bug0063.nl.tok.V,
	tests/bug0065.nl.tok.V, tests/bug0065b.nl.tok.V,
	tests/bug0065c.nl.tok.V, tests/bug0067.nl.tok.V,
	tests/bug0075.nl.tok.V, tests/datetime.nl.tok.V,
	tests/encoding.nl.tok.V, tests/genitive.nl.tok.V,
	tests/ligaturen.nl.tok.V, tests/multisentenceoneline.nl.tok.V,
	tests/normalisation.nl.tok.V, tests/nu.nl.tok.V,
	tests/onesentencemultiline.nl.tok.V, tests/parentheses.nl.tok.V,
	tests/quotetest_multiline.nl.tok.V,
	tests/quotetest_multisentence.nl.tok.V,
	tests/quotetest_nested.nl.tok.V,
	tests/quotetest_nosentence.nl.tok.V,
	tests/quotetest_onesentence.nl.tok.V, tests/test.nl.tok.V,
	tests/urls.nl.tok.V: fixed ucto tests \o/ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@12749
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-06-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/quoteproblem.txt: added git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10447
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10238
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testall, tests/testfolia.ok, tests/testone,
	tests/testoption1.ok, tests/testpassthru, tests/testpassthru.ok,
	tests/testusage.ok: added and improved tests for passthru updated a
	few .ok files.  I hope proycon fixes all .V.tok files git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10237
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: fixed encoding problem with
	passthru mode.  Added --passthru option to usage() git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10236
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: attempt to handle Windows™ files with \r\n.
	Just strip the last \r git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10223
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest_crlf.nl.txt: converted the file to CRLF and set the
	svn:eol-style property to CRLF git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10220
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-24  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/partest_crlf.nl.tok.V, tests/partest_crlf.nl.txt: added test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10214
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-23  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: I think I fixed bug77 (the NOSPACE bug), but now
	all tests fail (understandably), they will have to be checked
	thoroughly checked and updated.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10209
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-23  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/folia.txt, tests/testall, tests/testfolia,
	tests/testfolia.ok: added a simple FoLiA test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10156
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-20  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx: Fix: proper XML entities in FoLiA output git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10121
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10015
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-17  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: set date for release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@10013
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS too git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9980
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-16  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: avoid crash when no new tokens are found because
	of emptyp words (this might happen on Mac and DOS files with \r line
	separators ) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9979
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: ok, this is the ugly code :{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9871
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testpassthru.ok: sort of fixed <utt> handling in passthru
	code gets ugglier every day ;{ git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9867
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/passthru2.txt, tests/testpassthru, tests/testpassthru.ok: 
	added a better passthru test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9866
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: more News git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9863
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, src/ucto.cxx: small change in configfile
	handling.  Now the messages to stdout are correct.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9862
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok, tests/testconf2.ok, tests/testinclude.ok: 
	results ar slighlty different after small code reshuffle git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9861
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/invoer.rule, tests/small.cfg, tests/small.txt,
	tests/testall, tests/testinclude, tests/testinclude.ok: added a test
	for inclusion git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9855
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-11  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/testconf1.ok: small change after bug fix git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9854
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: fixed include file problem. I think. :) When a
	configfile with a (absolute/relative) path is provided (-c option)
	the include files will be searched in the same dir as that config
	file git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9853
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/{bug0072.nl.tok.V => bug0075.nl.tok.V},
	tests/{bug0072.nl.txt => bug0075.nl.txt}: wrong name for this bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9845
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: KNOWN Abbreviations are case sensitive now.
	Fixes bug 72 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9844
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-05-10  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0072.nl.tok.V, tests/bug0072.nl.txt: added bug72 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9843
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: version bump after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9732
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9730
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-27  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Implemented sentenceperlineinput support
	directly in DetectSentenceBounds and PassTru (previously
	unsupported) git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9729
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: also %include for
	[EOSMARKERS] git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9591
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/exotic-eos.eos,
	config/exotic-quotes.quote, config/ligatures.filter,
	config/standard-eos.eos, config/standard-quotes.quote,
	config/tokconfig-de, config/tokconfig-nl: added common files for
	inclusion in config git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9590
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/Makefile.am, config/tokconfig-nl, include/ucto/tokenize.h,
	src/tokenize.cxx: now [FILTER] and [QUOTES] also honour %include
	requests in the config For [FILTER] the file extension MUST be
	.filter For [QUOTES] the file extension MUST be .quote git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9589
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* config/tokconfig-nl, src/tokenize.cxx: fixed bug 67.  Now it is
	possible to add an empty replacement in the [FILTER} block git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9582
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-19  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0067.nl.tok.V, tests/bug0067.nl.txt: added a new bug test git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9581
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: no schemalocation for FoLiA git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9350
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h, src/tokenize.cxx: arghh git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9313
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* include/ucto/tokenize.h: hmpff.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9311
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, include/ucto/tokenize.h, src/ucto.cxx: added functions to
	display version info.  Updated NEWS git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9310
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-07  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/ucto.cxx: return conforming exit status.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9309
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* configure.ac: bumped version after release git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9271
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS: news! git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9270
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx, tests/bug0065.nl.tok.V,
	tests/bug0065b.nl.tok.V, tests/bug0065b.nl.txt,
	tests/bug0065c.nl.tok.V, tests/bug0065c.nl.txt: fixed bug0065 git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9268
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-04-04  mvgompel <mvgompel@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* tests/bug0065.nl.txt: new bug git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9267
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* NEWS, src/tokenize.cxx, src/unicode.cxx: Ok, now ALL unicode
	strings are output as UTF8.  Just to be sure.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9261
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* src/tokenize.cxx: Attempt to fix output encoding problems. We want
	it to be UTF8.  git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9258
	12f355fe-0486-481a-ad91-c297ab22b4e3

2011-03-31  sloot <sloot@12f355fe-0486-481a-ad91-c297ab22b4e3>

	* moved ucto to its final destination git-svn-id: https://ilk.uvt.nl/svn/sources/ucto/trunk@9240
	12f355fe-0486-481a-ad91-c297ab22b4e3

