Overview

Story Understanding
Characters
Trait Learner
Mental Model
Action Predictor
Reader
Reader Emotion Predictor
Author Intention Detector
Story Comparer
Question Answerer
Learner
Rule Learner
Database of How-To Procedures
WikiHow
HTN Learner
Story Pattern Learner
.
.
.
Problem Solving
Simulation
2D Blocks World
3D Unreal Engine
Table-top Robot
Planners
HTN Planner
Regression Planner
Memory
HTN Generator
HTN Selector
Story Telling
Storytellers
Story Summarizer
Instructive Storyteller
Persuaive Storyteller
Character-aware Storyteller
Explanation
Viewpoint Explainer
3D Scene Describer
Plan Explainer
Interactive Planner
Interaction
Interactive Task Learner
Interactive Concept Learner
Commonsense Rules
Database of Lexical and Commonsense Knowledge
ATOMIC
ConceptNet
WordNet
Word Similarity
Classification Thread
Linguistic Triples
Parser
CoreNLP
START
Inference Makers
.
Translator
Generator
Matcher
Frame Experts
Trajectory
Causality
Belief
Transition
Story Aligner
Concept Patterns
Mental Model
Memory
Substitutor
Frame Manipulation

Projects and models

Here is a summary of the computational theories developed by the Genesis group and systems built based on those theories.

  • Topic: The human abiity that the researcher wants to model.
  • Vision: The behavior that the research wants a system to do.
  • Theory: The main idea behind the research's computational model of the human behavior.
  • Contributions: The systems built and the behavior they demonstrate.
  • Author: The name of the researcher and the link to a paper or thesis that documents the project.
Story Understanding

The Genesis group believes that story understanding is essential to artificial general intelligence. The “story” here broadly refers to any related sequence of events; a traditional narrative structure is not necessary.

Using the Genesis system as a substrate, story-understanding goes beyond generating binary classification or relying on combinations of keywords. Instead, we carry out a full, systematic interpretation of the story elements and we are able to explain every step.

Topic Vision Theory Contributions Author
Humor detection Develop a system that understands and explains humor in stories. To detect humor, a program must model the listener's mental trajectory using background knowledge, commonsense rules, and methods of describing expectations on the story elements. Humor expert, a program that detects humor in stories based on error identification and resolution. The program consists of a system of experts: some flags tracks characters' states and unexpected events, some explains and resolves these potential anomalies—get the joke. Ada Taylor (2018, MS)
Hypothetical reasoning Develop a system that reasons hypothetically about stories and characters. To judge a character’s moral actions, a program can reason hypothetically about the alternative strategies the character could have used—but didn’t. Concept patterns as moral constraints, a new use-case which links narrative patterns and moral codes. (1) Hypothetical expert, a program that answers What-If questions about alternative stories. It discovers the pattern of spiteful violence instead of self-defense when asked "what would happen if Alex doesn't brandish a knife?" It discovers Teaching a lesson instead of aggression of a bully when asked "what would happen if I am not from Estonia?" (2) PERSONATE, a program that predicts what a character would do in a novel situation based on his methods, motives, and moral constraints inferred from previous actions. Dylan Holmes (2017, MS)
Commonsense knowledge base Incorporate ConceptNet commonsense knowledge into Genesis. By incorporating commonsense knowledge bases, story understanding systems can understand more causal relations between story events and amplify each rule to be applied in many more contexts that are similar to the specified one. (1) CSERM, a method that allows user-defined rules to be applied to story events involving similar verbs to those in the rules, amplifies the applicability of a rule by a factor of ten to one hundred. (2) ASPIRE, a system for inferring characters’ goals from the actions they take within the story. (3) Connecting ConceptNet to Genesis, Bryan's work also contributed the equivalent of 30,000 rules to Genesis that can be used to identify causal links between story elements. Bryan Williams (2017, MEng)
Who knows what Develop a system that understands a story from multiple characters’ perspectives. Each character should have their own mental model, containing everything they know or believe about how the world works. Characters observe events only while onstage, and their observations populate mental models. Perspectives Expert, a program that forms detailed mental models of characters in a story, thus able to retell the stories from different character's perspectives, answer reading-comprehension questions to demonstrate understanding, and explaining opposing viewpoints to reconciliate conflict. The program retells the stories of Inspector Javert and Jean Valjean in Les Miserables, along with their goals and interpretation of events that they observe, showing the reader why they think and act differently. Jessica Noss (2017, MEng)
Infer reader & character emotions Develop a system that empathizes with human emotions while reading stories. To infer the emotions of characters, it's not enough to look at the words, but the patterns that happen from the perspectives of the characters. Isabella, a program that determines the emotions evoked by individual words, a sentence, or a story. (1) it has a vector space model trained on multiple corpus that assigns each word a six-dimensional vector, representing Ekman’s six basic emotions: anger, sadness, joy, surprise, disgust, and fear. (2) To infer the emotions of multiple characters in the story, Isabella aligns the story to story patterns in a memory bank that label the typical emotions of those characters. Ryan Alexander (2016, MEng)
Commonsense knowledge Develop a system that understands stories from the Crow culture. By tesing a story understanding system with a new corpus of stories, we can identify areas that challenge the system and then develop mechanism to improve it. (1) Added mechnism for inserting elements into stories as the consequences of concept patterns. (2) Created a commonsense knowledge base for understanding stories from the Crow literature. Wolfgang Yarlott (2014, MEng)
Cultural mental models Develop a system that understands the same story from the perspectives of the eastern culture and the western culture. A story can be augmented by a question to simulate alternative viewpoints. (1) A program that answers "Did Lu kill Shan because America is individualistic" by inserting the prior belief of "I believe America is individualistic" into the story and finding the causal chains. (2) Enabled Genesis to associate a personality trait with the reader, thus able to read the story using different sets of rules and concept patterns that are signature in different personality models. Hiba Awad (2013, MEng)
Personality traits Develop a system that understands character traits. Applying personality traits can help a story understanding system explain and predict a character's behavior, as well as verifying whether his actions are unexpected given his predispositions. (1) Encoded ten example personality traits, such as "kind" and "mean," accompanied by action patterns of those traits. (2) Developed concept-pattern based trait representation; designed the processes of trait learning, inference, and application (explain and predict). Susan Song (2012, MS)
Story similarity Develop a system that retrieves similar stories from memory when given a new story and measures the similarity between stories. When judging the similarity of stories, a program that uses structural similarity performs more similar to humans than one that uses keyword similarity. The Similarity Module, a program that retrives and compares stories. It uses two methods to measure similarity: (1) Vector Angle, which calculates the similarity between vectors of concept patterns (order doesn't matter); (2) In-Order comparison, which is the ratio of the aligned patterns to the total number of patterns. Given 15 conflict stories to compare, when the program uses Genesis discovered concept patterns and vector matching, its result best predicts human results. Caryn Krakauer (2012, MS)
Story alignment Develop a system that finds the common sequence in stories efficiently. Story comparison can be used to draw on old experiences to understand and imagine new ideas. (1) Adapted the Needleman-Wunsch sequence alignment algorithm for aligning stories, which can be used to predict future events. (2) The simultaneous matching and alignment algorithm is able to rapidly prune and search the exponential search space that results from the goal to keep the associations between entities of the two plots consistent. The story alignment program is used to fill in missing gaps in descriptions of videos by comparing the video to an existing video corpus. Matthew Fay (2012, MS)
Story matching Develop a system that makes analogies between stories. For proper reasoning and inference of stories, the ability to make analogies is also crucial. A subsystem that makes analogies between stories according to plot units. Shown that the current Structure Mapping Engine algorithm still needs to be improved for analogizing between stories in Innerese form Jesse Dunietz (2011, UG)
Plot units Develop a system that processes and searches plot units. Plot units are the common set of structures found in human narrative. An idiom for describing plot units in English and a program to do plot-unit searches on stories. The program finds 10 plot units spread over Macbeth, Hamlet, and the Estonia-Russian Cyber Conflict. David Nackoul (2010, MEng)
Memory system Develop a system that reasons with commonsense knowledge. A story understanding system needs a memory for storing commonsense knowledge and generalize from previous experiences. A memory system that generalizes from related examples, predicts using the generalization, and answers questions based on commonsense knowledge learned. LLMerger clusters frames into chains for generalization. Sam Glidden (2009, MEng)
Story Telling

balabala

Topic Vision Theory Contributions Author
Story modulation Develop a system that understands communicator’s intentions from wording and explain them in a coherent argument. Story modulation: a storyteller uses rhetorical device to shape the inner-story of his audience. RASHI, a program that reads short news-like stories and identifies rhetorical indicators using a system of "experts", such as The Passive Voice Expert and The Repetition Expert. The program explains author's intentions along three dimensions--sympathy, agency, and doubt, after reading story Regime slaughters freedom fighter. Suri Bandler (2018, MEng)
Story Summary Develop a system that summarizes stories People read summaries mainly to acquire useful knowledge in minimal time, thus a summary program should focus on conveying knowledge useful as precedent and exclude obvious inferences. A story summarizer that uses a collection of method to reduce elements of stories: eliminating disconnected story elements, delivering only story elements connected to conceptual patterns, and focusing on particular concepts of interest. It summarizes an 85-element synllege of Shakespeare’s Macbeth with 14-element concise story about Pyrrhic victory. Across stories, the summarizer can compress stories by 60-74%. Patrick Winston (2015)
Character-aware storytelling Develop a system that generates stories with rich character models. Character trait can guide a storytelling system how to organize its knowledge. A program that generates stories given the traits of characters. It generates by adapting the plots associated with previous characters with those traits to the current character, then efficiently weaving the plots of differet characters together to make them compatible and consistent in time. The program can generate stories given character with evolving traits, character with cross-genre traits, and characters with conflicting goals. In one generated story, a character ends up falling in love with his lost daughter and they get married--this is unexpected use of knowledge gathered from the romances in Shakespearean comedies. ⭐Matthew Paul Fay (2014, PhD)
Audience-aware storytelling Develop a system that retells stories to instruct and persuade. Storytelling is a question of goal-driven social interaction between the storyteller and the audience. Among storytelling goals are instruction and persuasion. (1) Instruction expert, a program that simulates a teacher using stories to feed the student facts, or explanations of the facts, or knowledge used in the explanation. (2) Persuasion expert, a program that retells a story to emphasize a character trait by removing story events that show the opposite trait of the character and removing story events that show other characters having that trait. Sila Sayan (2014, MEng)
Learning

balabala

Topic Vision Theory Contributions Author
Learn moral of stories Develop a system that extracts generalizable moral of a story. Aligning stories by emotional states abstracts away story-specific details. A story can be both a motivational story and a cautinary tale depending which character's perspectives you take. MAXIM, a program that aligns two stories by the emotional states of characters and explains the moral using the common causes of the emotions. The program learns "It’s ok if you are different because Valuable Rudolph is different" after reading Rudolph the Red-Nosed Reindeer and The Math Aficionado. Zhaozheng Alice Jin (2019, MEng)
Learn recipes Develop a system that learns recipes by asking questions and reading stories. To learn the relevant steps for achieving a high-level goal, a program can find the common goals achieved by the steps in two texts. Genesis Novice Learners that acquire hierarchical procedural knowledge by asking questions and by aligning successful problem-solving stories. One learner learns the steps to make a real-world fruit salad by conversing with a human. Another learner learns to replace a phone battery from two 80-word stories that have much irrelevant detail and nothing expressed in exactly the same way. Zhutian Yang (2018)
Learn concepts Develop a system that acquires concepts by a few example, as opposed to learning by explicit definition. Humans can build up concepts by first seeing positive examples and then negative examples. STUDENT, an interactive learning system that takes in a small series of positive and negative examples of concepts and builds an internal model for these concepts, including the elements that is required, forbidden, and optional. Jake Barnwell (2018, MEng)
Learn character traits Develop a system that discovers character traits and associated plots from a corpus of stories. Topic modeling is a scalable method for clustering traits on unlabeled data. Methods for automaticly learning character traits in both an unsupervised way using topic modeling (Latent Semantic Analysis) and a semi-supervised way using a new method of alignment-based trait learning. The model unexpectedly discovered the trait of doomed royalty, with which a character is both to be king and to be killed ⭐Matthew Paul Fay (2014, PhD)
Detect high-level concept Develop a system that detects high-level story elements such as drinking from low-level data that can be produced from videos. To build high-level story elements from low-level data, a system can filter and group the data, then label the groups using regular supervised methods. A program that groups series of low-level events in the transition space into clusters of high-level story events, such as flying and rinking. Erik Speed (2012, MEng)
Learn story patterns Develop a system that learns narrative structure from annotated folktales. Narrative structure opens the door to understand of cultural influences on cognition and cultural differences captured in stories. Analogical Story Merging, a new machine learning algorithm that can extract culturally-relevant plot patterns from sets of annotated folktales. Mark Finlayson (2011, PhD)
Learn parsing rules Develop a system that learns the mapping from language to semantics from examples. Learning how to parse instead of hand-coding it can provide a system with motr robust and extensible parser. A learning semantic parser that is able to read 43 parsing examples and generate 18 rules, which was originally implemented with 1000 lines of Java code. Harold Cooper (2009, MEng)
Learn rules Develop a system that understands natural language using knowledge learned from a few examples, instead of hard-coded. To learn/generalize from just a few examples, we need a positive e example to seed the description and a negative example to specify it. (1) Lattice-Learning, a concept-learning algorithm that can learn Thread descriptions and rules from very few examples. For example, “Robins, bees and helicopters can fly, but cats, worms and boats cannot,” is enough for Understand to narrow in on our concept of flying things: birds, insects and aircraft. (2) Understand, a program that learns to transform parsed English into semantic frames using Lattice-Learning. It takes only 8 positive and 4 negative examples to teach Understand how to interpret sentences as complicated as “The cat ran from the yard because a dog appeared. Michael Klein (2008, MEng)
Learn typical events Develop a system that learns typical events in the world through observation and explanation. By using clusters of similarity as an implicit representation, the self-organizing event maps mimics natural memory systems and do not suffer from being tied to the limitations of a specific explicit representation of regularity. The self-organizing event map, a data structure that stores a plane of event models that are continually updated and organized according to events that are observed by the system. Seth Tardiff (2004, MEng)
Learn about objects Develop a system that is capable of broad purpose learning and reasoning of commonsense knowledge. For a system to set goals for itself and carry it through, it needs a reflection thread Squirrel, an architecture that learns commonsense knowledge about the properties and behavior of objects in the world. Oana Stamatoiu (2004, MEng)
Language

balabala

Topic Vision Theory Contributions Author
Robust parsing Develop a system that parses ungrammatical and complex sentences. A program can understand ungrammatical sentences by leveraging the power of expectation. Unified Plausibility Parser (UPP), a tool that fixes complex and unparsable sentences by generating plausible interpretations to them and selecting the one that best fits the examples previously read. To parse "The pigeon the fox eats," UPP assigns a greater plausibility to “The fox eats the pigeon” than “The pigeon eats the fox” because UPP has read "The cat likes the dove" before. Emanuele Ceccarelli (2017, MEng)
Word disambiguation Develop a system that disambiguates word-senses. Word disambiguation requires lexical database. CLARIFY, a program that disambiguates words based on WordNet threads. Martin Marcel Couturier (2011, MEng)
Word disambiguation Develop a system that disambiguates word meanings with minimal starting information. It is important to explore how a computer reading a text can go from a word to its meaning. A disambiguator System that uses Lattice Learning to capture the concept of contexts. Elyi Stickgold (2011, MEng)
Thread memory structure ... .... .... Saba Gul (2009, MEng)
Trajectory frame Understand how human language enables the uniquely broad scope of our reasoning ability. In order for a sentence to convey a transition, it must contain a verb that means either “change,” “appear,” or “disappear.” Lance, a program that learns how to extract semantic descriptions from language by analyzing example pairs of phrases and descriptions. Adam Kraft (2007, MEng)
Symbol grounding problem Understand how meanings can be learned from the regularity of sensory experiences. Humans learn symbols not through instruction but finding regularity. A representation can be built from low-level sensory information. Implemented and experimented the three levels of representation: cell map, cluster, and association for the Symbol Grounding Problems. Stephen Larson (2003, ME)
Disambiguation with vision Build a Cognitively Complete System with vision and language faculties. A reconciliatory exchange makes the best use of two sources of information, each has ambiguities and missing information. A system that solves ambiguity by reconciling between linguistic and non-linguistic semantic information. Keith Bonawitz (2003, MEng)
Vision

balabala

Topic Vision Theory Contributions Author
Visual alignment Build vision systems that are versatile, flexible, and introspective like humans. Alignment hypothesis: robust perception requires pervasive alignment of partial information throughout a multimodal network. A vision system that can track people as they move and infer geometric information about the scene Adam Kraft (2018, PhD)
Recognize actions Develop a system that recognizes simulated "worm" action. Humans represent actions internally with concrete sensory data: touch, proprioception, muscle tension. EMPATH, a system that recognizes the actions of a simulated worm through sensory information learned by watching the creature free-play. Robert McIntyre (2014, MEng)
Recognize human actions Develop a system that recognizes human actions from videos. Recognize human actions based on pose classification of a single frame and a model of transitions between poses. A vision system that recognizes basic human actions with 91% recall. Nikola Otasevic (2013, MEng)
Recognize visual actions Develop a system that recognizes transition-based visuospatial patterns. An important competence of artificial intelligence is transition-based visuospatial pattern recognition. A vision system for learning and recognizing visual actions based on state transitions. Telmo Correa (2011, MEng)
Trajectory simulation Develop a system that initiates visual representations from natural language The basic representations of Is-A, Is-JJ, Is-Superlative, Is-Possessive, Trajectory, Relation, and Transition is sufficient to reason about the world. Span, a system that can generate trajectory descriptions from natural language to imagine a 3D inner world. Mark Seifter (2007, MEng)
Represent path To understand how human language and vision connect, we need to model how human describe events and visualize them. Visualization by incremental composition. (1) Implemented Ray Jackendo 's Lexical-Conceptual Semantics for representing path, (2) VISUALIZE, a program that maps a sentence to an image or a movie, and DESCRIBE for gener5ating language from visual predicates. John Bender (2001, MEng)
Human Intelligence

balabala

Topic Vision Theory Contributions Author
Memory augmentation Understand how human memories are encoded, stored and retrieved. Humans can trigger episodic memory by using interfaces to passively stimulate the hippocampus, the entorhinal cortex, and the neocortex. NeverMind, an augmented reality (AR) interface designed to help people memorize effectively, tripling the long-term memory recall accuracy of sequences of items compared to paper-based memorization tasks. Oscar Rosello (2017, MS)
Human attention Understand human attention and peripheral vision. The medium for action recognition are videos and the basic driver of attention should be movement. Conducted experiments on human attention and peripheral vision, unveiling computational heuristics for vision systems. Sam Sinai (2014, MEng)
Human eye-tracking Understand how humans visually construct a space. To inquire into patterns of visual attention in 3D space, we need to develop new instruments and new methods of representation. PUPIL, a custom set of hardware and software instruments, that capture the subject’s eye movements. Moritz Kassner and William Patera (2012, MEng)
Human analogical reasoning Understand human analogical reasoning ability on moral stories. The Moral Expert Hypothesis, people are experts in the morality of their own culture. Proved that people see the shared higher-order moral similarities in their own culture and only superficial similarities in others Emilie Kim (2009, MEng)
Human trust Understand how people learn to trust each other when they have a conversation. Trust can be built up by catering to another person’s world-view. Modeled how humans learn to trust each other, where the “characteristics” of a person are specified by a map and some constants in two equations. Catherine Miller (2006, MS)
Human decision-making Understand human decision-making. States, Actions, and Goals as the basic units of decisions. Augmented Decision Rationale Language, a representation of understanding decision rationales. Arian Shahdadi (2003, MEng)
Human acquisition of inflection Understand the acquisition of inflection in natural language. Generalization postulates rules that change the word from one form into the other, Sifting tackle conflicts by assigning weights to the rules. Generalize and Sift, a learning model of how children learn inflections Raymond Molnar (2001, MEng)