The challenge
In December 2024, Ofgem made a significant announcement on energy debt and standing charges, supported by a nationwide campaign, which received BBC News coverage. Energy tariffs affect millions of UK consumers and Ofgem’s campaign generated widespread media attention.
The team had to therefore evaluate responses to their announcements at speed. They identified Assist as a solution due to its ability to quickly provide sentiment and thematic analysis of a large amount of information.
How Assist helped
To facilitate effective evaluation, the team gathered all relevant content, copying raw text from articles into a single Word document. This document was then uploaded to Assist, alongside their communication plan and multi-year strategy.
To guide Assist’s analysis, the team used the following prompt:
“Please analyse the attached stakeholder reactions and feedback document from stakeholders, relating to an announcement Ofgem made on 12 December 2024 (comms plan document also attached for reference).
Using the Government Communications Evaluation Cycle as a guide for your structure, please provide a 200-word bullet-pointed evaluation of the feedback, assessed against the priorities and principles of Ofgem’s multi-year strategy.
Please focus on general sentiment, strongest reactions, key successes and positives, areas for improvement and any general notable comments or recommendations for future communications work.”
The analysis was run twice, days apart, to include later-emerging information.
Outcomes
Assist rapidly delivered an accurate summary, identifying key issues and sentiment, and connecting these insights to the team’s goals and the organisation’s wider mission. This valuable evaluation was widely used across the department, informing future work and supporting senior leaders.
For other communication teams, the benefits are clear:
- Timesaving: Using Assist cut evaluation time by 75%
- Estimated 30 articles analysed over two days, alongside stakeholder reaction and social media analysis – approximately 40 pages or 20,000 words of text!
- Three people reviewed the outputs for accuracy and ‘hallucinations’ (nonsensical or inaccurate information outputs generated by generative AI/large language models).
- Ensures consistency: Assist strictly follows instructions, providing consistent analysis regardless of context.
- Connects projects to the bigger picture: By including Ofgem’s strategy into the prompt, Assist helped identify how responses to the campaign achieved their organisational aims.
Why this matters
Effective evaluation is key to identifying the impact of the work of government communications, as highlighted by our Evaluation Cycle. Many campaign teams face challenges in conducting rapid evaluation of online responses to campaign content, particularly from multiple sources. Assist was indispensable in addressing this challenge.