M365 Copilot: Revolutionizing the Workplace with Generative AI?!

accilium’s 100-day reality check

According to a study by Microsoft, 70% of M365 Copilot users reported that their productivity was increased through the use of M365 Copilot, with 68% noting an improvement in their work quality. Additionally, 77% would not want to remove M365 Copilot from their daily office routine.¹

accilium, as an early adopter, put these study results to the test in a practical examination lasting 100 days. Selected consultants trialed M365 Copilot in their daily work and rigorously tested it. During the testing phase, accilium evaluated various functions for their utility, functionality, and user-friendliness. You can find impressions from our reality check on the following pages, as well as the resulting action guidelines derived from it.

Our use cases from our self-test

In the following section, we would like to briefly introduce those use cases that have excited us the most, as well as those that, at the present time, have fallen short of our expectations.

TOP: Microsoft Excel formel guidance

M365 Copilot was able to perform analyses of the referenced data quickly by processing the entered instructions (prompts). By automatically adding rows and columns, it saves users the sometimes tedious tasks of researching and testing formulas, offering significant potential for time savings.

💡 Good to know: M365 Copilot in Excel can only work within tables.

TOP: summaries in Word & Outlook

Similar time-saving potentials are possible in the processing and summarization of communication elements (Outlook, Teams) and in text documents. M365 Copilot can precisely outline entire communication channels or mailboxes in seconds, significantly speeding up information retrieval. Similarly, M365 Copilot also offers the ability to summarize lengthy Word documents, such as project reports, into concise bullet points within seconds.

💡Good to know: small nuances in the prompts can have a significant impact on the search results.

FLOP: creation of PowerPoint presentations

In the creation of PowerPoint presentations, the use of M365 Copilot was (currently) of lesser benefit for our needs. While the functionality allows for the creation of slides on any topic with a prompt input in a very short time, these slides are both content-wise and visually very generic.

💡Good to know: M365 Copilot is not yet capable of consistently working with company-specific slide masters. Content can be automatically generated, but the time-consuming visual adjustment of the slides to the corporate design still needs to be done manually.

FLOP: automated draft creation in Word

The automatic creation of text documents with M365 Copilot in Word can occur based on input documents or through a prompt request. In both cases, the generated content, much like in PowerPoint, tends to be very generic and prone to errors. As a result, extensive editing was necessary to meet our quality requirements (5%-10% variation).

💡Good to know: approaching the document chapter by chapter led to the best results. This method allows for more controlled and precise adjustments to each section, helping to improve the overall quality and relevance of the document.

Just a Copilot, not a pilot: role distribution in the daily work routine

Another frequent issue encountered with the use of M365 Copilot is hallucination (e.g., when summarizing email inboxes). Hallucinations are pieces of information created by M365 Copilot that are completely fabricated. It has been observed that M365 Copilot occasionally refers to communication elements (chats, emails) that did not occur. This requires the author to always have a critical eye to validate the results.

This highlights the importance of understanding that while M365 Copilot can significantly aid in automating and speeding up tasks, it cannot fully replace human oversight. Users must review and refine the outputs to ensure accuracy and relevance, effectively making M365 Copilot a supportive tool rather than a standalone solution.

Download the complete results of our M365 Copilot trial now

Important homework related to M365 Copilot

Investments made in data governance measures and a robust data culture are areas that will now particularly unfold their potential with M365 Copilot.

A crucial element for the successful deployment of Microsoft M365 Copilot is the effective management of the data lifecycle, which includes the processes and policies regarding how data is created, stored, used, and ultimately deleted within a company. Inefficient handling of these processes can lead to outdated or irrelevant data being processed by M365 Copilot systems, which can result in inaccurate analyses or recommendations.

Another significant challenge in the context of M365 Copilot is the correct implementation of data sensitivity classifications to ensure the security of sensitive information. These tools classify data based on its sensitivity to ensure that its integrity is protected. Without appropriate classification and access controls, M365 Copilot might process or disclose sensitive data. If personal information is not properly classified, M365 Copilot will be able to access and process this data.

Crucial competitive advantage for the future

Digital transformation is no longer an option but a necessity for companies that want to continue to succeed in the future. M365 Copilot and similar technologies require organizations to have done their ‘homework’ in terms of digitization, such as data governance and information security measures. Those who have postponed these issues in the past will find it difficult to keep up with the dynamics of the market. A solid digital data base and sufficient training measures for employees are key to effectively utilizing these powerful tools.

Our experiences from the 100-day reality check demonstrate how essential a well-thought-out strategy is for the use of advanced AI technologies. A proactive adaptation to digital requirements is not only an indispensable investment in the future but can also make the difference between leading and following in our fast-paced economy. Companies that master these challenges not only strengthen their market position but also lay the foundation for a secure and innovative future.

¹ What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? (microsoft.com)