Thrift module: structure

Enumeration: TokenizationKind

Enumerated types of Tokenizations

Struct: Constituent

A single parse constituent (or "phrase").

Struct: Parse

A theory about the syntactic parse of a sentence.

\note If we add support for parse forests in the future, then it
will most likely be done by adding a new field (e.g.
"forest_root") that uses a new struct type to encode the
forest. A "kind" field might also be added (analogous to
Tokenization.kind) to indicate whether a parse is encoded
using a simple tree or a parse forest.

Struct: ConstituentRef

A reference to a Constituent within a Parse.

Struct: Dependency

A syntactic edge between two tokens in a tokenized sentence.

Struct: DependencyParseStructure

Information about the structure of a dependency parse.
This information is computable from the list of dependencies,
but this allows the consumer to make (verified) assumptions
about the dependencies being processed.

Struct: DependencyParse

Represents a dependency parse with typed edges.

Struct: Token

A single token (typically a word) in a communication. The exact
definition of what counts as a token is left up to the tools that
generate token sequences.

Usually, each token will include at least a text string.

Struct: TokenRefSequence

A list of pointers to tokens that all belong to the same
tokenization.

Struct: TaggedToken

Struct: TokenTagging

A theory about some token-level annotation.
The TokenTagging consists of a mapping from tokens
(using token ids) to string tags (e.g. part-of-speech tags or lemmas).

The mapping defined by a TokenTagging may be partial --
i.e., some tokens may not be assigned any part of speech tags.

For lattice tokenizations, you may need to create multiple
part-of-speech taggings (for different paths through the lattice),
since the appropriate tag for a given token may depend on the path
taken. For example, you might define a separate
TokenTagging for each of the top K paths, which leaves all
tokens that are not part of the path unlabeled.

Currently, we use strings to encode annotations. In
the future, we may add fields for encoding specific tag sets
(eg treebank tags), or for adding compound tags.

Struct: LatticePath

Struct: Arc

Type for arcs. For epsilon edges, leave 'token' blank.

Struct: TokenLattice

A lattice structure that assigns scores to a set of token
sequences.  The lattice is encoded as an FSA, where states are
identified by integers, and each arc is annotated with an
optional tokens and a weight.  (Arcs with no tokens are
"epsilon" arcs.)  The lattice has a single start state and a
single end state.  (You can use epsilon edges to simulate
multiple start states or multiple end states, if desired.)

The score of a path through the lattice is the sum of the weights
of the arcs that make up that path.  A path with a lower score
is considered "better" than a path with a higher score.

If possible, path scores should be negative log likelihoods
(with base e -- e.g. if P=1, then weight=0; and if P=0.5, then
weight=0.693).  Furthermore, if possible, the path scores should
be globally normalized (i.e., they should encode probabilities).
This will allow for them to be combined with other information
in a reasonable way when determining confidences for system
outputs.

TokenLattices should never contain any paths with cycles.  Every
arc in the lattice should be included in some path from the start
state to the end state.

Struct: TokenList

A wrapper around a list of tokens.

Struct: SpanLink

A collection of tokens that represent a link to another resource.
This resource might be another Concrete object (e.g., another
Concrete Communication), represented with the 'concreteTarget'
field, or it could link to a resource outside of Concrete via the
'externalTarget' field.

Struct: Tokenization

A theory (or set of alternative theories) about the sequence of
tokens that make up a sentence.

This message type is used to record the output of not just for
tokenizers, but also for a wide variety of other tools, including
machine translation systems, text normalizers, part-of-speech
taggers, and stemmers.

Each Tokenization is encoded using either a TokenList
or a TokenLattice. (If you want to encode an n-best list, then
you should store it as n separate Tokenization objects.) The
"kind" field is used to indicate whether this Tokenization contains
a list of tokens or a TokenLattice.

The confidence value for each sequence is determined by combining
the confidence from the "metadata" field with confidence
information from individual token sequences as follows:


 For n-best lists:
metadata.confidence 
 For lattices:
metadata.confidence * exp(-sum(arc.weight)) 


Note: in some cases (such as the output of a machine translation
tool), the order of the tokens in a token sequence may not
correspond with the order of their original text span offsets.

Struct: Sentence

A single sentence or utterance in a communication.

Struct: Section

A single "section" of a communication, such as a paragraph. Each
section is defined using a text or audio span, and can optionally
contain a list of sentences.