thread repost: Discussion on Parser/UI Options

[Tim Wessman]

[This is Manolo's post, recaptured from a saved copy]
========================================
I said that I sincerely don't want to write anything more about the Prime, but you drag me to it! In any case, this is very interesting stuff and I thank you (I hope all here will do as well) for your earnest effort to explain the decisions you've made.

Quote:I agree completely!!! At least when the CAS is behaving consistently, not refusing to solve something for an unclear reason, and when the results are returned in and expected manner. Until that time though, seeing why the CAS refused to solve, or what it was doing can be very helpful. In short, one things are acting nicely, it is completely unhelpful. I don't think it will be much longer long though before it would be possible to be hidden away and only if you select to tap the little icon or similar will it bring up some additional useful info.

Excellent news, and the most interesting single piece of information about the Prime posted here since it appeared. New users will be glad. Current users will be puzzled because that's quite a spectacular change of viewpoint. If that was the plan maybe saying that the CAS environment was the CAS debugging environment would have been appropriate. Anyway, that would solve the main problems with the current Prime.

Quote:In what way does it make sense? I've seen you cite that it makes sense, but other then "coherent UI" (in what way? How will that make it such?) and that "others have done it", I haven't seen much in the way of details.

Building it around Giac would have meant having in the end just one parser and one typesystem. In a calculator with an input text line, your bread & butter UI is your parser. Not that it's the only way to get there, but if we analyse how it was done before we see that that way got results.

For instance, and I'm telling this again for the unaware folks. TI wanted a CAS calculator, in the end that was the 92, and they weren't building a CAS from scratch. So they hired Soft Warehouse. The problem was that Derive was a PC program written in Lisp (mulisp), maybe too complex for an embedded device with a non standard resolution. I don't know why they chose the 68K but it the end they not only had to pick just a subset of Derive, but to rewrite it in C (probably there's a fair amount of assembler there too).

The good thing is that the syntax and functional capabilities of the 92 are that of Derive (that is the most loved thing about the 89) and you can share a good amount of Derive expressions with it. Sadly, TI wasn't really interested in taking advantage of this: the last version of Derive could export/import to your handheld. Had they not killed it, it could have evolved into a very interesting convergent environment.

That's what I meant with building around Giac. Keep the algorithms you can, keep the syntax and the overall philosophy but have the capabilities of your device in mind while rewriting everything else.

Quote:Having spent a very, very long time thinking about this, and working with the giac codebase though, I can give about 20 reasons in an instant why it doesn't make sense, and only 1-2 that do make sense. Addressing your point #1 regarding UI:

(By UI I assume you are referring to the current parser schizophrenia)

The giac parser is done with lexx/yak. That lets you add things quickly, but when you get right down to it there is not a whole lot of flexibility in attempting to do anything outside the conceived original design of how a parser should work. You can easily recognize *static* known words and patterns, but if you attempt to do more intelligent parsing that changes behavior slightly depending on the context it gets ridiculously difficult and complex to implement VERY quickly.

In addition to not having a large amount of control to do selective recognition, there really is no mechanism in place for working with sub parts of the expressions. You either have an all or nothing parsing. You can't stop in the middle to handle errors, identify problems and prompt the user to resolve them, or do any similar things that would be helpful to the end user. This is part of the reason why any sort of "parse" error in the CAS returns a big error string, while in the home screen you are stopped and asked to fix it with the cursor right at the point of problem.

Yes, that's a problem. You can't use the Giac parser, I agree. Probably you have to write your own parser.

Quote:So assuming that in order to get the UI and experience to the point that I'd like it to be, there really are two choices with regards to parsing. Either you spend the time and effort to modify and extend the existing lex/yakk parser to handle EVERY potential plan and idea we have for future development, or you modify the home/numeric parser to support what is needed in order to shift all parsing in the system under the same roof. Either of those tasks is very, very large. One of them however is much, much, much shorter.

If that's the case, then the parser is so tightly integrated with the Giac interpreter that you really have no choice. I suspected that. If refactoring the Giac code is not possible, you have to go with your own.

Quote:When geogebra implemented a CAS system in geogebra 5.0, they had a similar choice. There was an already existing parser that fit extremely well with the system design and met all the desires and needs in the way that made sense for that piece of software. They could have used the CAS parser directly... yet they didn't. They modified their parser to be able to pass only specific items to giac for evaluation, and interpret the results back into the existing format.

That's exactly the way to solve the ugliest Prime problems without changing the underlying design.

Quote:One of giac's weaknesses is that it does not have very tight memory control. This is primarily due to it being written from the assumption of having exceptions (which are very often not supported well or buggy on embedded system compilers) and that it will be running on a desktop. It uses tons of small memory allocations and pointers everywhere - the base data structure is quite complicated and it thrashes the memory. Again, this isn't an incorrect decision for a desktop application, but for a limited memory embedded device with potentially questionable OS support it does make problems.

These are facts. Architectures are so different that they have to be essentially two different programs.

Quote:Now if we move on to the wider idea (excluding the technical issues) of building the system around giac, we immediately run into vision conflicts. I am not even going to get into the "business" considerations and concerns a PHB type person might have, but there are *seriously* many of those as well.

That's why TI eventually bought Soft Warehouse, I understand that.

Quote:1. Assume that the system would be built around giac. Exactly what does that mean? (I'd be interested to know that because depending on how you are interpreting that statement we might have much more common ground then you thing) For now, lets assume that all parts of the system use giac for parsing end evaluation.

Not for parsing (see above), but for evaluation. And yes, we're getting close, or at least I'm getting close to your options.

Quote:a) You have just lost the ability to disable parts of the calculator. Giac is a monolithic blob and for many places in the world the calculator now has no future because it can't be controlled in an exam. While nobody here would consider that a problem, not making money on a device is about the quickest way to ensure that device doesn't stay around long.

Many people here don't like this. I get that you have to do this and you're right, you need at least a parser independent from the CAS engine if you want to disable the CAS capabilities.

Quote:b) you now have very slow programs. Giac is a great CAS from a capability point, but it has as a CORE ASSUMPTION that everything is exact and symbolic. Remember how on the 50g you can get radically different times for programs based on approx or exact? Now assume that EVERYTHING will always be exact first and there is basically nothing you can do about it. That is pretty much the situation. Throw on top of that the higher memory requirements, less tight control of loops and recursion, etc.

How did they solve that with the 92/89? My guess is that they are running everything symbolic and the approximate mode/results are something having to do with the APPROX Derive function. That is an option.

Quote:c) high speed graphing is no longer there. It will go much, much slower then the current basically instant graphing.

Agreed.

Quote:d) there can't be any sort of interval based graphing or calculations due to complete incompatibility between a symbolic engine and what is required for interval graphing.

I don't understand this very well, but you'd know better than I.

Quote:e) memory and objects now have a completely flat "session" interface. All calculations and objects are saved into a single flat blob that is only possible to load completely. No folders. No access from outside. (didn't people want folders?) I haven't even gotten to talk about the whole app thing yet and people would already be angry.

As it's been discussed, just porting the Giac code as it is really makes no sense.

Quote:f) any hp designed features are not going to be accepted into the "mainline" because in many cases they aren't desired for the users of a pure symbolic package as a whole. For example, having more then 8 colors. Yes, that is a limit inherent to the underlying data structure. Binary/hex/oct/dec simialr to the 48 series? Not in there. An integer is essentially a different thing and used for a different purpose. Spreadsheet access such as A1:B4 to get a region... not going to be there. What should sin(pi) be equal to? What is the meaning of accurate to n places? These are fundamental question and differences in assumptions that depending on your view can radically alter everything.

Yes, it's a whole new world. I agree, you have to fill whatever is lacking anyway.

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: So let's assume that we now rework giac/xcas to do everything exactly as we'd like, and to meet our plans. First, you've probably about doubled or tripled the length of development. Second, as stated before a lot of what HP users would like is not going to meld well, or be desired by xcas users or someone like geogebra. So in essence we've just forked it and made something that serves us and nobody else.

That means that all that time and effort ended up with us getting something which would NOT be compatible with the HP calculator vision either past or future, and not compatible with the purely symbolic system that exists, and in short wouldn't have made anything better! The problem just would have shifted to a much more difficult to resolve location.

Yes, but then you end up with a platform of your own, fully under your control, and the next calculator will probably be easier to develop. But if we're talking about just this single calculator I fully agree, it's not worth the trouble and you can get it done without.

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: Trust me, I do understand why people have said "well just use that cas engine", or "just buy one", or "why didn't you just use the 48 code base". Every single possible direction was considered and analyzed before the decision to move in this technical direction was taken. I am not trying to dismiss what people say, or tell them they are wrong, or make anyone angry. Rather, there WAS no good technical solution that would cleanly work in the first step.

I'm sorry about the abrasive reactions. You know better than anybody else the constraints you have for your product, that's for sure. I don't like people judging lightly the nuances of a tricky situation while having no idea about what's going on either, so I apologise if I've done just that. But there was a lot of BS regarding the HOME/CAS issue. The situation with two unrelated systems in the same device that do not really get along looks a lot like a desperate move to ship something early. When you read CAS.function(), you can tell that you are just calling the Giac library and that's it. Then you press the CAS key, the Giac console just pops out and the first thing you think is "you have to be kidding me". It is a substandard integration, because there is no integration. And then there are these sycophantic pieces that start with "I think that the purpose of the HOME/CAS separation is (amazing juggling skills)" they really wear me out. You have nothing to do with that and you deserve a big kudos for not falling for it.

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: Prime is doing very well. There are people getting into HP calculators that never have been before. Doors are opening that with a 50g, or a 51g, or a 52g, would never be open. For the vast majority of users who primarily use the calculator to do some math, draw some graphs, and take some tests - this calculator is amazing to them! I've seen people with no interest in math start playing with it, exploring the calculator, and getting excited about MATH! That just does not happen with other calcs in my experience.

I'm glad about it. The Prime can become a great calculator. The Nspires for instance, can't, I'd rather prefer people buying Primes instead. According to your stated intentions to "hide away" Giac it might happen (becoming a great calculator, and more people buying it).

Many folks here don't get that HP has to look more to the TI 89 series than to the 48s if we're talking about getting people to buy school calculators, so I fully agree with your opinion regarding focusing on the 50g series. In any case, I think that HP as a company should consider all those users that have been using RPL for 20 years as well. Unluckily the 48s' system is too tied to a dead processor and emulation is the only feasible path. Sucks, but that's life.

(And now that we're on it, please get a HW1 TI89 just to play with Lars Frederiksen RPN. I can't think of a better introduction to RPN for the masses, it would be very nice having that kind of stack with performed operations/values on this device.)

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: This isn't a children's game where the first time something wrong happens they throw the pieces away, kick someone, and then run and hide under the bed. (just happened today... )

Just remember that big companies are a lot like big ships. They take a long time to get moving and a long time to turn. Once they are moving though, they either run into a rock and sink, or else are hard to stop.

Let's hope so, I really wish you the best.

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: End note - it may seem like I was picking on giac/xcas, or Bernard, or whatever. That is not the case. I think that giac has amazing capability. I have seen an impressive array of cases where it will solve something not possible in either casio/ti units, and even in some professional desktop or powerful open source cas systems (eg: maxima).

Yeah, having Parisse around is a golden asset. You should convince him that sometimes having lookup tables is not really a sin, unless you aim for automated theorem proving . Giac's integer and polynomial arithmetic is just impressive.

Quote:(Yesterday 09:23 AM)Tim Wessman Wrote: If I was to sum up the weaknesses, I would say that it makes some simple things more difficult then needed at times, and a lack of polish generated by restricted usage. That can be corrected though by feedback from a wide range of users and sources. Just in the past 6 months as there has been work with both HP and Geogebra I think it has made significant advances and is hugely better then it was before. I am quite confident that it will just continue to improve.

Good luck Tim (and the rest of the team), I know we can be hard to deal with but it is for the greater good.

--

Manolo Sobrino