Background for Action Item-Based Test Questions

This gives general information common to the action item-based Y3 test questions: PQ0002, PQ0025, PQ0199, PQ0451, PQ0581 and PQ0582 (see CaloY3TestQuestions for the individual questions).

1) Background Info:

• The MA suite provides SmartNotes for people to manually record action items during meetings. Action items are recorded with a task description, a responsible party (owner) and a due date (timeframe) (all optional).
• The CSLI action item identifier will also attempt to automatically produce action item hypotheses, including their task descriptions, owners and timeframes.
• This identifier is based on a supervised classifier which uses speech recognition output to produce hypotheses. It is a complex hierarchical classifier, consisting of simple sub-classifiers which detect description, owner and timeframe assignment and the presence of explicit agreement (see the MLMI and ACTS 2006 papers on this page).
• We also provide a user browser which displays both manual and automatic action item hypotheses for a meeting, and allows users to edit, delete or confirm them.
• This user feedback (editing/deletion/confirmation etc.) is then stored and used as implicit supervision: the feedback is interpreted to produce more training data for the action item classifier (deletion -> negative instances; confirmation -> positive instances, etc.), and its model learns accordingly.

2) Resource Level:

• See individual questions

3) English Interpretation:

• See individual questions

4) Description of Learning:

• BCALO (baseline) answers are supposed to be those that CALO can produce given no information extracted from natural interaction (and therefore only that information explicitly provided by e.g. typing it in).
• BCALO (baseline) answers are therefore supplied from SmartNotes-generated action items only; or, if no SmartNotes are taken during test meetings, no answers can be provided and the baseline is zero.
• LCALO (learning) answers are supplied from all action items, whether generated by SmartNotes (if available) or hypothesized by the CSLI action item detector.
• Any & all correct automatic hypotheses will therefore count towards the learning gain.
• In addition, the automatic detection algorithms will improve over time and use, via implicit supervised learning from user feedback provided from the Meeting Browser.

5) Answer Strategy:

• ActionItems hypothesized by the CSLI action item detector will get inserted into the CSLI KB, along with their Task, Owner and Timeframe properties (possibly multiple hypotheses about these property).
• ActionItems recorded manually using SmartNotes get interpreted by the CSLI detector and inserted into the CSLI KB, along with their Task, Owner and Timeframe properties
• There is one central instance of the CSLI KB on the server; one instance of the detector; and one single set of detector-generated hypotheses. SmartNotes generates one shared set of public notes, but also allows users to generate their own private notes. Feedback on automatic hypotheses is also given by each user individually. The questions therefore need to be asked (and answered) from the point of view of an individual user.
• The CSLI action item detector provides a Java API for accessing information about the ActionItems stored in the CSLI KB, including listing action items and getting their properties.
• CSLI will provide OAA interfaces to these methods for direct use by CATS scripts.

2) Sensitivity to Parameter Instantiations:

• See individual questions

7) Answer Key:

• The test team will have to manually establish a list of action items by listening to/watching the meeting recording (there's no way round this).
• These action items should be noted along with:
  • task description (sentence/phrase)
  • owner (person responsible)
  • timeframe (either a concrete date, or a more general expression)
• This could be done with SmartNotes, which has fields for these properties, or with NOMOS, or just by hand.
• TODO

8) Scoring Method:

For action items:

• 0-4 points credit based on the relevance/correctness of the "task description" property text.
• Note that the system may produce an action item which the test team didn't notice (this often happens with tasks which do define something to be done, but a human annotator might not consider something important enough to list as an action item). In this case it seems unfair to award 0 points, but perhaps a maximum of 2 should be allowed?

For assigned owners:

• 4 points for the correct KB entity
• 0-3 points based on the relevance of the "owner description" property text.

For assigned timeframes:

• 0-4 points based on the relevance of the "timeframe" property text.

9) Design of LCALO-to-BCALO Transform:

• We can ablate to a BCALO simply by removing all CSLI-generated layers from the CSLI KB, thus leaving only SmartNotes data - either via a script that removes the data, or a KB handler that ignores layers generated by CSLI components.

10) Critical Learning Period (CLP) Conditions:

• We require a series of meetings throughout the CLP (ideally more than 10).
• The meetings should ideally be related to each other (and thus likely to discuss the same or related action items more than once).
• The meetings should ideally involve (some of) the same people.
• Between meetings, users must review the recorded/hypothesized action items in their meeting browser, and give their feedback (see below).
• The identifier model on the server will then be updated between each meeting with the available user feedback.

11) User Actions to Drive Learning:

• Between learning period meetings, users must review the recorded/hypothesized action items in their meeting browser. Feedback should include:
  • deleting those that aren't really action items
  • saving the real ones they are interested in, in their action item list
  • marking real ones that they aren't interested in, so they get removed from their list
  • adjusting any incorrect properties of the real action items
• (Someone running the central MA server will then have to inform the action item identifier (so that it re-trains on the new feedback) before the next meeting.)
• In the meeting being tested, it's important that some action items are discussed.
• Similarly, to get a learning delta, it's important that not everyone notes down all the test meeting action items manually in SmartNotes - otherwise there is no learning improvement possible.

12) Query Period Conditions:

• All relevant meetings must have been fully processed (including inter-meeting feedback and learning).

13) Parameter Instantiation Guide:

• Firstly, the action item and/or meeting in question must have been processed by the MA system.
• For questions in which an action item is specified, there will be no chance for any learning delta if it is one which has been produced manually (and had all of its properties specified) by SmartNotes.
• Therefore it's best to choose one which has been produced by the CSLI detector. For owner/timeframe questions ("who is responsible/when is it due?"), it's just as good to choose one produced manually but which did not have the relevant property specified.
• Best of all would be an action item discussed in the test meeting which has also been discussed in a previous meeting (perhaps in a slightly different form, e.g. with a different owner or due date), and which users have had a chance to give feedback on the associated hypotheses.

14) Specification of the CATS-CALO Interface:

• CATS will call Query Manager, which will access the information in the CSLI KB via CSLI's OAA solvables.
• (CSLI are importing SmartNotes output into their KB, so this covers both automatically-generated hypotheses, and SmartNotes versions.)
• The data passed/returned will be in the form of text strings (descriptive strings for task descriptions and timeframes, login names for owners, and unique IDs for meetings and action items).
• See individual questions for OAA solvable details.

15) CALO Implementation of PQ Support:

• Providing action item hypotheses: complete (SmartNotes and CSLI).
• Reporting required properties of action items (including OAA solvables): complete.
• Browser display, feedback and re-learning: not yet complete, but due by end May.

16) Data Sets:

• We have an annotated dataset for the Y2 G series of meetings, which can be used for initial testing.
• We are currently annotating Y3 data produced by SRI; this should be available for testing by mid-June.

-- MatthewPurver - 04 May 2006