version 2.4.7 not loading all AIML

[slow putzo]

It has been awhile since I did an update but a rather unusual problem I discovered in version 2.4.6 has caused me to update to the latest versison to see if the same issue is still in the current code.

Unfortunatly I have spent most of the day trying to get the new release working because of all the basic changes that have been made.

The one that is currently concerning me is that in version 2.4.6 my AIML file set that is loaded produces rather significant different results.

Here is what I get on version 2.4.6 from loading the zip file AIML.ZIP
The chatbot "Root Bot" has 29,867 AIML categories in the database.

After clearing the AIML and loading it on version 2.4.7 I get this:
The chatbot "Root Bot" has 28,105 AIML categories in the database.

In both cases I did the following steps:
Clear ALL AIML categories (Purge database)
Then load the zip file AIML.ZIP

It is very strange that a program would ignore something and not report that it did so and why.

Why would the number of AIML categories be so different from the exact same AIML files?

[Alekz]

Do you have <topic>s in your AIML? All files with <topic> tag get ignored.

[slow putzo]

Yes, I have many <topic> tags. Has <topic> been depreciated? What has taken it's place?

[Alekz]

See this thread: http://forum.program-o.com/viewtopic.php?f=7&t=676

[slow putzo]

I looked at that post and I do not think that is why some 1,762 categories are missing from my AIML load into this new version.

I have around 15 categories that are in a "topic" section. If you mean any category that does a "set topic" is ignored, then yes I would think I might have about 10,000 categories that would be ignored.

I believe it is something else that is being ignored and not loaded into the database.

If you know of some magical way to find an example of what is missing other than trying to compare two databases against one another using phpmyadmin I am all ears.

Ignoring any AIML without giving some kind of warning is a very poor thing for program-o to be doing.

To knowingly release code that does this, or, to not revert the changes made to the program that caused this is not a very comforting feeling either.

The topic issue was reported back in January, that is a long time ago. I suspect that particular issue has been resolved. If it is not maybe the code should be reverted back until the solution is found.

I believe my issue is something different. I just don't know what it might be.

[slow putzo]

I decided I had to get to the bottom of this discrepancy, so I carefully counted every category I have in the 46 files that make up my total AIML set. Those 46 files are all placed in a ZIP file and loaded into the chat bot.

I did not actually count each category, I used notepad++ editor to tell me how many categories were in each file two ways. One with a special sort macro I have that places every category as the start of each line allowing me to sort them, and then by searching for a <category<>pattern> string. The counts came out to be exactly the same for each file.

I added them all together twice as well as using notepad++ to scan all open files, all 46 of them to report the number of occurrences of <category<>pattern> string. Again a perfect match.

I have 28,125 categories.

On load release 2.4.7 reports 28,119 loaded.

To find out why the difference, I decided to load each file individually and check how many categories were loaded.

Only two files showed a difference. The files containing all "a" and "I" categories were both off by -1.

That accounts for two of the missing 6.

Without more searching, I am making the assumption the 6 missing categories are actually duplicate categories so they were not loaded into the database.

That may be a bad assumption, but I really do not want to spend more time looking for 6 lost categories.

I did look to see if by change a log file was generated indicating a duplicate was found, but on load of AIML absolutely no log files are generated by my server.

What this says is that the older version is horrible at telling you how many categories you have loaded. It actually created more than you loaded I guess.

My only suggestion would be to put a log file out if indeed it is catching duplicates so that you can find them and correct them.

As for the <topic> question, I do have a single file with all topics in it. There are 43 <topic> entries and all of them are loaded into the database.

So the comment that program-o is ignoring those files is not accurate.

[Alekz]

It's not the categories that get ignored, but the whole files. So, if you have a file with 100 categories, but only 1 category is in the <topic>, the whole file (read 100 categories) will be ignored.

[slow putzo]

You must have a different version of program-o than I do because the version I downloaded on 2015-10-13 does not ignore files with topics, and all of my topics are in the database after load.