The Semantic Telemetry Project spearheaded by Microsoft Research has provided new insights into user interactions with Microsoft Copilot, revealing patterns of engagement, skill levels, and satisfaction among users of Artificial Intelligence technologies. Using advanced data science techniques and large language model-generated classifiers, the project categorizes user interactions into topics and task complexities, offering a comprehensive look at how users interact with Copilot for real-world task completion.
Key findings indicate that users who engage with Copilot for complex, professional tasks tend to interact more frequently and show sustained usage over time. Novice users, initially engaging in simpler tasks, are gradually shifting towards more complex activities, demonstrating an evolving understanding and adaptation to the tool. Moreover, user satisfaction is predominantly influenced by the alignment of expertise between the user and the AI; experts are satisfied when the AI exhibits comparable knowledge, while novices report lower satisfaction universally.
The project also categorizes users based on their activity levels, dividing them into light, medium, and heavy engagement categories, with findings suggesting that heavy users perform a significant amount of high-complexity tasks, particularly in knowledge work areas. The research suggests that analyzing these usage patterns and satisfaction levels can guide the enhancement of AI systems to better cater to diverse user needs, ultimately improving task execution and user experience. Future exploration will delve into the specifics of LLM-generated classification engineering.
