CSLI CALO Y3 Test Question Analysis

PQ0002

PQ String: Is |io:%Person| responsible for |io:%ActionItem|?

General Strategy & Learning: see CaloY3ActionItemTestQuestions

2) Resource Level: Certainly, by end May

3) English Interpretation: Given an ActionItem and a Person in the KB (or presumably, their IDs), we want to know whether Person is one of the Owners of ActionItem.

5) Specific Answer Strategy:

1. (Choose an ActionItem from the list returned by csliGetActionItems)
2. Call csliGetActionItemOwners to get responsible parties for the action item
3. Check whether Person is in that list

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemOwners(User, ActionItem, OwnerList)

6) Sensitivity to Parameter Instantiations: Person and ActionItem must exist in the KB. To be able to measure a learning delta, ActionItem must have been produced automatically by the CSLI ActionItemIdentifier (not manually by SmartNotes or the MeetingBrowser).

PQ0025

PQ String: List the action items generated at the |io:%Meeting| and persons responsible for each.

General Strategy & Learning: see CaloY3ActionItemTestQuestions

2) Resource Level: Certainly, by end May

3) English Interpretation: Given a Meeting, return the list of ActionItems proposed in that Meeting, together with the values of their Description and Owner properties. (NB for the Y3 test we can't really distinguish between action items 'proposed', 'discussed' or 'generated').

5) Specific Answer Strategy:

1. Call csliGetActionItems to get the list of action items
2. For each result
   1. Call csliGetActionItemDescriptions to get its hypothesized task description
   2. Call csliGetActionItemOwners to get its list of owners

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemDescriptions(User, ActionItem, Meeting, DescriptionList)
  • csliGetActionItemOwners(User, ActionItem, Meeting, OwnerList)

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0199

PQ String: Given an action item from |io:%Meeting|, generate a ranked list of discussions from previous MEETINGS that are relevant to the action item.(PS).

General Strategy & Learning: see CaloY3TopicTestQuestions and CaloY3ActionItemTestQuestions

2) Resource Level: Highly likely, by mid June

3) English Interpretation:

5) Specific Answer Strategy:

1. (Choose an ActionItem from the list returned by csliGetActionItems)
2. Find the segment of discussion for the action item:
   • For SmartNotes-provided action items, we will have to look up the time when the note was made - CMU may be able to advise on this;
   • For CSLI-hypothesized action items, call csliGetActionItemDiscussion
3. Pass the segment (a start and end time) to csliExtractTopic, which produces a topic
4. Pass this topic to csliSearchTopic

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemDiscussion(Person, ActionItem, Meeting, [StartTime, EndTime])
  • csliExtractTopic(Meeting, StartTime, EndTime, Topic)
  • csliSearchTopic(Topic, DiscussionList)

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0220

PQ String: Who was present at |io:%Meeting| [instance:MeetingOrConference] while |io:%Topic| was discussed.

2) Resource Level: Highly likely, by mid June

General Strategy & Learning: see CaloY3TopicTestQuestions

2) English Interpretation: List of people whose presence was overlapping, even if only a little bit, with the segment where the topic was discussed.

5) Specific Answer Strategy:

1. (Choose a topic from this meeting using the browser or csliGetTopics)
2. Locate the relevant region of the meeting (time period) using csliLocateTopic
3. An MOKB query can then pull out people present during the returned time period

• Relevant OAA Solvables
  • csliGetTopics(Meeting, Topics)
  • csliLocateTopic(Meeting, Topic, [StartTime, EndTime])

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0299

PQ String: Based on observed behavior, list all the people who participated in discussions about |io:%Topic|.

General Strategy & Learning: see CaloY3TopicTestQuestions

2) Resource Level: Highly likely, by mid June

3) English Interpretation: List of people who participated, even if only a little bit, in the segment where the topic was discussed. Participation includes speaking, typing, writing and drawing.

5) Specific Answer Strategy:

1. (Choose a topic using the browser or csliGetTopics)
2. For each meeting:
   1. Pass the Topic to csliLocateTopic which will produce start and end time
   2. A MOKB query can then pull out all people who were agents of a speaking/typing/writing/drawing event during that time period.

• Relevant OAA Solvables
  • csliGetTopics(Meeting, Topics)
  • csliLocateTopic(Meeting, Topic, [StartTime, EndTime])

6) Sensitivity to Parameter Instantiations:

PQ0301

PQ String: Which of the following set of topics |{|io:%SubjectCategory|, |io:%SubjectCategory, . . . io:%SubjectCategory|}| were discussed at |io:%Meeting|.

General Strategy & Learning: see CaloY3TopicTestQuestions

2) Resource Level: Certainly, by mid June

3) English Interpretation:

5) Specific Answer Strategy:

1. For each SubjectCategory, find the closest related named Topic:
   1. If already specified as a topic name/ID, just use that;
   2. If specified as raw text, find the most relevant Topic, either:
      1. using csliGetRelevantTopic
      2. using the browser
      3. importantly, whichever method you use, do this over all meeting data, rather than just the given Meeting (as that would always return a discussed topic, of course)
2. For the related topic, determine whether it was discussed in Meeting using csliWasDiscussed

• Relevant OAA Solvables
  • csliGetRelevantTopic(String, Topic)
  • csliWasDiscussed(Meeting, Topic, TrueOrFalse)

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0302

PQ String: Order the topics |{|io:%SubjectCategory|, |io:%SubjectCategory, . . . io:%SubjectCategory|}| in terms of time spent in discussion during |io:%Meeting|.

General Strategy & Learning: see CaloY3TopicTestQuestions

2) Resource Level: Highly likely, by mid June

3) English Interpretation:

5) Specific Answer Strategy:

1. If |SubjectCategory|s are being specified as topics: choose set of Topics for this Meeting using the browser or csliGetTopics
2. If |SubjectCategory|s are being specified as text: choose a named Topic for each SubjectCategory using the browser or csliGetRelevantTopic
3. For each Topic, find the time period of discussion in Meeting via csliLocateTopic
4. Order the time periods by length

• Relevant OAA Solvables
  • csliGetRelevantTopic(String, Meeting, Topic)
  • csliLocateTopic(Meeting, Topic, [StartTime, EndTime])

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0302a

PQ String: When were each of the following topics |{|io:%SubjectCategory|, |io:%SubjectCategory, ... io:%SubjectCategory|}| discussed at |io:%Meeting|?

General Strategy & Learning: see CaloY3TopicTestQuestions

2) Resource Level: Highly likely, by mid June

3) English Interpretation:

5) Specific Answer Strategy:

1. If |SubjectCategory|s are being specified as topics: choose set of topics for this Meeting using the browser or csliGetTopics
2. If |SubjectCategory|s are being specified as text: choose a named Topic for each SubjectCategory using the browser or csliGetRelevantTopic
3. For each Topic, find the time period of discussion in Meeting via csliLocateTopic
4. (optionally take the start/midpoint of each period?)

• Relevant OAA Solvables
  • csliLocateTopic(Meeting, Topic, [StartTime, EndTime])

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0451

PQ String: What were the action items discussed in |io:Meeting|?.

General Strategy & Learning: see CaloY3ActionItemTestQuestions

2) Resource Level: Certainly, by end May

3) English Interpretation: Given a Meeting, return the list of ActionItems discussed in that Meeting. (NB for the Y3 test we can't really distinguish between action items 'proposed', 'discussed' or 'generated').

5) Specific Answer Strategy:

1. Call csliGetActionItems to get the list of action items
2. For each action item
   1. Call csliGetActionItemDescriptions to get its task description

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemDescriptions(User, ActionItem, Meeting, DescriptionList)

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.

PQ0581

PQ String: When is |io:%ActionItem| due?

General Strategy & Learning: see CaloY3ActionItemTestQuestions

2) Resource Level: Highly likely, by end May

3) English Interpretation: Given an ActionItem in the KB, when is it due (can be an absolute date or time, or relative to some other event).

5) Specific Answer Strategy:

1. (Choose an ActionItem from the list returned by csliGetActionItems)
2. Call csliGetActionItemTimeframes to get a list of the timeframe description strings

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemTimeframes(User, ActionItem, TimeframeList)

6) Sensitivity to Parameter Instantiations: ActionItem must exist in the KB. To be able to measure a learning delta, ActionItem must have been produced automatically by the CSLI ActionItemIdentifier (not manually by SmartNotes or the MeetingBrowser).

PQ0582

PQ String: What action items proposed during |io:%Meeting| remain unassigned?

General Strategy & Learning: see CaloY3ActionItemTestQuestions

2) Resource Level: Highly likely, by end May

3) English Interpretation: Given a Meeting, we want a list of ActionItems which were generated in that Meeting and which do not appear to have had an Owner specified.

5) Specific Answer Strategy:

1. Call csliGetActionItems to get the list of ActionItem IDs from Meeting
2. For each action item
   1. Call csliGetActionItemOwners and keep the ActionItem only if the resulting Owner list is empty
   2. Call csliGetActionItemDescriptions to give task descriptions for the result

• Relevant OAA Solvables
  • csliGetActionItems(User, Meeting, ActionItemList)
  • csliGetActionItemOwners(User, ActionItem, Meeting, OwnerList)

6) Sensitivity to Parameter Instantiations: Meeting must exist in the KB and have been processed by the MA suite.