How far is AI from passing a maths exam?

[StephenG1CMZ]

It was interesting seeing this thread about ChatGPT
https://www.hpmuseum.org/forum/thread-19351.html

It has me wondering, it such an AI were trained specifically on a particular maths syllabus instead of being quite general, how far are we from having it pass an exam?

Without necessarily understanding everything, and assuming you don't need 100% correct to pass.

From what I see of the general chatbots, it seems unlikely this decade... But are there more specialised and successful ones?

(Of course, examples such as Wolfram Alpha are good at answering questions - but do they cover a syllabus?)

[pascal_meheut]

(12-31-2022 12:00 AM)StephenG1CMZ Wrote:  It has me wondering, it such an AI were trained specifically on a particular maths syllabus instead of being quite general, how far are we from having it pass an exam?

Which level? middle high-school, SAT or master degree? Not the same.

(12-31-2022 12:00 AM)StephenG1CMZ Wrote:  From what I see of the general chatbots, it seems unlikely this decade...

If you can predict where we will in AI be at the end of the decade, you could make money.

(12-31-2022 12:00 AM)StephenG1CMZ Wrote:  But are there more specialised and successful ones?

Yes: https://arxiv.org/abs/2112.15594

[StephenG1CMZ]

Any exam...and your link covers a University-level exam.
So I was unduly pessimistic.

[Luigi Vampa]

I asked ChatGPT to solve some problems about physics, that I asked my students to solve in a first course of Physics (undergraduate level). The answers looked really well written and confident, but most were wrong.

[pascal_meheut]

(12-31-2022 11:20 AM)Luigi Vampa Wrote:  I asked ChatGPT to solve some problems about physics, that I asked my students to solve in a first course of Physics (undergraduate level). The answers looked really well written and confident, but most were wrong.

It was not designed to do so and undergraduate level is aiming too high for such a tool. It can solves simpler problems but as it does not "think" the way we expect our students to do, results are kind of random.

[pier4r]

I can only tell that in adventofcode.com chatGPT (that is NOT optimized for code), passed many of the initial 10 days (part 1 and part2), that in itself is quite impressive. Now adventofcode is very specific domain of problems, but it gives an idea.

[EdS2]

From what I've seen, the prompt you give ChatAI makes all the difference. One might try starting with
"You are an A-grade student who always works very carefully and shows all their working"

I've read that getting it to proceed step-by-step improves the outcome of any non-trivial problem. Counting the legs of mixtures of animals, for example, is a non-trivial problem:

Quote:Q: If I have a duck and a dog how many legs do they have in total?

and for more advanced testing:

Quote:Q: I have 2 cats and 1 horse. How many legs do they have in total?

(From this interesting and sometimes amusing discussion https://news.ycombinator.com/item?id=33990567)

[BruceH]

A friend recently asked a chat bot:

Quote:Tell me what is interesting about the number 2023?

2023 is an interesting number because it is the start of a new decade. It also has several interesting mathematical properties; for example, it is the sum of four consecutive prime numbers (2011 + 2017 + 2019 + 2021). It is also the smallest number that can be expressed as the sum of three squares in three different ways (1^2 + 20^2 + 22^2, 9^2 + 16^2 + 22^2, and 11^2 + 12^2 + 20^2).

Not much sign of being able to pass an exam in there.

[pascal_meheut]

(01-05-2023 05:12 PM)BruceH Wrote:  Not much sign of being able to pass an exam in there.

Maybe because AIs such as ChatGTP are not designed to do so.
But yet it scores 1060 when passing the SAT which is not bad for something that does not think nor count.

[Luigi Vampa]

A counterexample to the Fundamental Theorem of Algebra?... by ChatGPT:

https://gruposinvestigacion-unir-net.tra...r_pto=wapp