Test 1 - summarising text

This page has 0 threads | Add post

Blogs

Finding our blogs useful? Please consider helping us to share more!

Road-testing 4 different AI tools, so you don't have to! Part four of a nine-part series of blogs
In this blog we'll compare OpenAI's Chat GPT 4, Google's Gemini, Anthropic's Claude 3.5 and Microsoft's Copilot to see which AI tool gives the best results for different types of queries. Road-testing 4 different AI tools, so you don't have to The AI tools we've chosen to test - and why The 5 tests run for each AI tool Test 1 - summarising text (this blog) Test 2 - programming support Test 3 - presenting arguments Test 4 - creating images Test 5 - creativity and playing roles Conclusion and recommendations

Road-testing 4 different AI tools, so you don't have to!

Part four of a nine-part series of blogs

In this blog we'll compare OpenAI's Chat GPT 4, Google's Gemini, Anthropic's Claude 3.5 and Microsoft's Copilot to see which AI tool gives the best results for different types of queries.

Road-testing 4 different AI tools, so you don't have to
The AI tools we've chosen to test - and why
The 5 tests run for each AI tool
Test 1 - summarising text (this blog)
Test 2 - programming support
Test 3 - presenting arguments
Test 4 - creating images
Test 5 - creativity and playing roles
Conclusion and recommendations

Posted by Andy Brown on 15 July 2024

In this blog

ChatGPT 4 (21 seconds)
Gemini (almost instant)
Claude 3.5 (10 seconds)
Copilot (14 seconds)

So for my first test I decided to ask each of our AI tools under review this question:

You are an experienced book reviewer. You write concisely and without jargon or filler words. Create a table listing out the main characters in Pride and Prejudice by Jane Austen with these 2 columns: “Character name”, “Summary of personality”. You should limit the description of each character’s personality to no more than 30 words. Present the list of characters in order of importance (so with the most important character first), then underneath the table finish by explaining which character you like best, and why.

Most of this should be reasonably straightforward, but summarising the personality of the different characters in a book should test the tools. Let's see how they get on!

ChatGPT 4 (21 seconds)

This is seriously impressive. The characters are presented in the right order, the thumbnail sketches are spot-on and the choice of Elizabeth Bennett as the favourite character is well-defended:

ChatGPT does "Pride and Prejudice"

I honestly don't think I can faullt this!

Gemini (almost instant)

Here's another outstanding offer from Gemini:

Gemini does "Pride and Prejudice"

What I particularly like about this answer is the fact that it's so concise (something I'd specifically asked my AI tool to be).

Claude 3.5 (10 seconds)

Here's what Claude had to say - again, it's near-perfect:

Claude does "Pride and Prejudice"

It's hard to separate these first 3 contenders. They have all (as is typical of AI tools) done a much better job at this task than the vast majority of humans would do.

Copilot (14 seconds)

Finally, here's what Copilot had to say:

Copilot does "Pride and Prejudice"

Another near-perfect summary, presenting the characters in the right order and giving a well-argued defence of Elizabeth Bennet as the book's favourite character at the end.

I've got to be honest, I was expecting to have to explain how much better ChatGPT 4 was than the other tools, but it turns out that they are all as good as you could possibly hope them to be. Given how equal the answers are, perhaps we should choose the winner on speed, in which case Gemini is a clear winner and ChatGPT a clear loser.

Parts of this blog
Road-testing 4 different AI tools, so you don't have to The AI tools we've chosen to test - and why The 5 tests run for each AI tool Test 1 - summarising text (this blog) Test 2 - programming support Test 3 - presenting arguments Test 4 - creating images Test 5 - creativity and playing roles Conclusion and recommendations