# Query Log

The query logs provide you with direct access to specific user queries and sessions. User queries are logged and scrubbed of any extra data (i.e., log-system data, business logic payloads, etc.) before they are passed to the query response payload. They are stored inside log-stash and retrieved using OpenSearch.

The Query Log page has numerous features that you can utilize to keep improving your AI model’s quality with real end-user data. You can learn about these features in the following sections:

Filters

Assign Status and Add Notes

Correct Classification

Export Queries as Test Cases

# Filters

Queries can be filtered by the following criteria:

Date and time: Sort the queries by choosing a specific time frame. The time is consistent with the system timezone that you configured in Settings.
Competencies: Choose to view queries from a specific competency.
Platforms: Sort by platforms such as web, mobile, or smart devices.
Sentiment: Choose to show only positive or negative user queries.
Classification Probability: Sort queries by confidence scores.
Keywords: Search queries with specific keywords.
Custom Field: Any field in the query response JSON payload. For example you can filter by intent, and the value can be clean_hello_start.
Custom Value: The value of the field of your choice.
Status: Filter queries by one of the following statuses: New, Misclassified, Out of Scope, Flagged or Correct.
Status Marked By: Sort queries by the platform user who set the query status.

# Assign Status and Add Notes

This feature is designed so you can annotate the query log data and generate new test cases.

assign status

The available types of status are as follow:

New: The default status of when a new query is added to query logs. It cannot be set by users.
Misclassified: This status is used to label queries that are not classified correctly.
Reclassified: A query is set to “Reclassified“ automatically when you use the Correct Classification feature. Users cannot set it without using Correct Classification.
Out of Scope: Used to label queries that failed to be classified.
Flagged: Used to label queries that need to be revisited.
Correct: Used to label queries that are classified correctly.

You can also make notes to the queries by clicking the chat bubble next to each query.

# Correct Classification

If an utterance is misclassified, don't panic! It can mean that the utterance will serve as an excellent guide for AI model quality improvement. The Correct Classification is the tool to accomplish the goal.

To correct the intent of the query,

Head over to the next to the query, select Correct Classification.

correct classification

In the intent/competency dropdown menu, select the correct intent and competency.

reclassify classification

Note: You will be able to see the comparison between the original intent and reclassified intent, the platform user who modified it, and the timestamp.

Now it's ready to be exported as a new test case.

# Export Queries as Test Cases

Select the queries you wish to export as test cases.

export as test cases

Head over to the Testing tab, open the test suite designated to the AI version, and import the JSON file exported earlier.

import test cases

# Export Query Logs

On the export logs page, you can choose the start and end date of the query logs that you wish to export as well as the export file format (JSON or CSV).

export_logs

# Data Format

# JSON

{
    "_index": "logstash-finie-2019.12.02",
    "_type": "doc",
    "_id": "2c7304ab-6858-4393-8d43-f998c82a87c1",
    "_score": null,
    "_source": {
        "host": "3dca8f95906e",
        "tags": [
            "querylog"
        ],
        "@version": "1",
        "extra": {
            "interpreter_version": "3.6.8",
            "thread_name": "Dummy-1",
            "line": 246,
            "logger_name": "logstash_querylog",
            "interpreter": "/usr/local/bin/python3",
            "logstash_async_version": "1.4.1",
            "path": "/usr/local/lucida-clinc/uservices/logstash/client.py",
            "process_name": "MainProcess",
            "func_name": "store_query_response"
        },
        "pid": 601,
        "program": "mod_wsgi",
        "type": "python-logstash",
        "@timestamp": "2019-12-02T21:00:03.352Z",
        "logsource": "3dca8f95906e",

Log system specific data.

        "doc": {
            "TIMESTAMP": "2019-12-02T21:00:00.033Z",
            "INPUT": {
                "qid": "2c7304ab-6858-4393-8d43-f998c82a87c1",
                "lon": 0,
                "latency": 3.3747273557819426,
                "personality_id": "f2328332-ed27-4c4d-8470-8e568c252d5a",
                "device": "default",
                "time_offset": 0,
                "competency_id": "71a397fb-b5f1-4c8f-9db4-efcbd881170d",
                "classifier": "stateful-qsr_example-states-state",
                "platform": "web",
                "__RAW__slot_data": {
                    "_FOOD_MENU_": {
                        "type": "string",
                        "values": [
                            {
                                "value": "hamburger",
                                "resolved": 1,
                                "food_menu_dest": "hamburger",
                                "tokens": "hamburger"
                            }
                        ]
                    },
                    "remaining_slots": []
                },
                "institution_id": "1d8a75fd-aa9f-46f6-b957-ff286fe5b24f",
                "classifier_state": "food_order",

Everything in doc list are query specific data. Please reference Query Response for what each field stands for.

                "bl_resp": {
                    "query": "I want a burger",
                    "qid": "2c7304ab-6858-4393-8d43-f998c82a87c1",
                    "slots": {
                        "_FOOD_MENU_": {
                            "values": [
                                {
                                    "value": "hamburger",
                                    "food_menu_dest": "hamburger",
                                    "tokens": "hamburger",
                                    "resolved": 1
                                }
                            ],
                            "type": "string"
                        }
                    },
                    "lon": 0,
                    "state": "food_order",
                    "intent_probability": 0.9498837367690086,
                    "session_id": "3329bf2f4223412d9dba08cc115ae449",
                    "device": "default",
                    "dialog": "qnBKoS2AjVPAyXalsYvAlto+VEOwt+bj",
                    "time_offset": 0,
                    "intent": "food_order_start",
                    "ai_version": "292fdc79-d01d-4db7-ae71-800584fbd118",
                    "sentiment": 0,
                    "lat": 0,
                    "external_user_id": "1"
                },

Reference The Response Body section on the Business Logic page for what each field stands for.

                "sentiment": 0,
                "lat": 0,
                "slot_probabilities": {
                    "_FOOD_MENU_": [
                        0.9999535083770752
                    ]
                },
                "response_slots": null,
                "slot_data": "{\"_FOOD_MENU_\": {\"type\": \"string\", \"values\": [{\"value\": \"hamburger\", \"resolved\": #, \"food_menu_dest\": \"hamburger\", \"tokens\": \"hamburger\"}]}, \"remaining_slots\": []}",
                "query": "I want a burger",
                "intent_raw": "food_order_start",
                "visual_payload": null,
                "intent_probability": 0.9498837367690086,
                "competency": "food_order",
                "bl_url": "https://business-logic-qsr.clinc.com",
                "today": "2019-12-02T21:00:00.033Z",
                "session_id": "3329bf2f4223412d9dba08cc115ae449",

Please reference Query Response for what each field stands for.

                "bl_req": {
                    "query": "I want a burger",
                    "qid": "2c7304ab-6858-4393-8d43-f998c82a87c1",
                    "slots": {
                        "_FOOD_MENU_": {
                            "type": "string",
                            "values": [
                                {
                                    "food_menu_dest": "hamburger",
                                    "tokens": "burger",
                                    "resolved": -1
                                }
                            ]
                        }
                    },
                    "lon": 0,
                    "state": "food_order",
                    "intent_probability": 0.9498837367690086,
                    "session_id": "3329bf2f4223412d9dba08cc115ae449",
                    "device": "default",
                    "dialog": "qnBKoS2AjVPAyXalsYvAlto+VEOwt+bj",
                    "time_offset": 0,
                    "intent": "food_order_start",
                    "ai_version": "292fdc79-d01d-4db7-ae71-800584fbd118",
                    "sentiment": 0,
                    "lat": 0,
                    "external_user_id": "1"
                },

Reference The Request Body section on the Business Logic page for what each field stands for.

                "is_repeat": false,
                "dialog": "ZPqY5/a8qRq2t9Fktd6h7hJ5pjBnFNY6",
                "latencies": {
                    "webhook_total": 0.786525,
                    "webhook_latency": {},
                    "total": 3.3747273557819426,
                    "external_api": null
                },
                "version": "release-v0.8.21",
                "intent": "food_order_start",
                "is_correction": false,
                "ai_version": "292fdc79-d01d-4db7-ae71-800584fbd118",
                "browser": "Other",
                "slot_token_ranges": {
                    "_FOOD_MENU_": [
                        {
                            "end": 3,
                            "start": 3
                        }
                    ]
                },
                "__RAW__query": "I want a burger",
                "certainty": true,
                "user": "978fbedc-12f0-a2b4-8afd-5c740d46b3de",
                "external_user_id": "1"
            },
            "OUTPUT": {
                "__RAW__speakableResponse": "Okay, so you would like hamburger. Is that all you'd like?",
                "speakableResponse": "Okay, so you would like hamburger. Is that all you'd like?",
                "formattedResponse": "Okay, so you would like hamburger. Is that all you'd like?",
                "__RAW__formattedResponse": "Okay, so you would like hamburger. Is that all you'd like?"
            }
        },