How can PK$save be stopped from adding an extra byte 'FF'?

[Lostgallifreyan]

How can the service PK$SAVE be stopped from adding an extra byte 'FF'?

It does this, as confirmed on XP (LA) v3.3, and LZ, v4.6. It must be possible to prevent it, or a pack could never be filled because PK$save would try to force an 'FF' beyond the end of the pack!

For anyone wanting to experiment, I suggest using a test ODB-type file on a RAM pack, getting the address of a record in it with FL$frec, then using the address as argument for PK$sadd, and using the record as a buffer, writing length-1 bytes as maximum, to avoid the extra 'FF' corrupting the next record, or causing truncation of the pack. Sticking to this test method will allow all three pack extraction methods (UNMAKE, SendPack, DumpPack) to yield a safe result that you can inspect.

There is no bounty for an answer, but if you find one, you'll have beaten me and Jaap Scherphuis, so gain extensive satisfaction...

EDIT: A FEW POSTS DOWN, THERE IS A PACK IMAGE IN A ZIP FILE. (For repeatable and easy testing). I should have done that for this first post, but better late than never.. The rest of this post will describe what happens so you know in advance of adding unknown pack images to your stuff!

Start with a blank RAM pack, with a MAIN file containing one record, with one field, with a 14 char string in it (guaranteed to be big enough).
Add the following program, translated on a PC to OB3 if building a pack for easily repeatable testing.

Code: Select all

A:
LOCAL M$(12),C%(9)
M$="PK$save Test"
OPEN "B:MAIN",A,A$ :FIRST :CLOSE :rem saves a lot of machine code if you do this first.

C%(1)=$3F2D :rem SWI FL$FREC
C%(2)=$DC41 :rem LDAB 0m
C%(3)=$3F60 :rem SWI PK$SADD
C%(4)=$01CC :rem NOP,LDD Byte count
C%(5)=12    :rem BYTE COUNT
C%(6)=$01CE :rem NOP,LDX Source string
C%(7)=ADDR(M$)+1
C%(8)=$3F61 :rem SWI PK$SAVE
C%(9)=$3900

USR(ADDR(C%()),1)

[Zenerdiode]

Is it that once a pack is SIZED; the OS knows the size of the pack (by reading the pack ID string) therefore preventing PK$SAVE from attempting the undesirable write?

[Lostgallifreyan]

Is it that once a pack is SIZED; the OS knows the size of the pack (by reading the pack ID string) therefore preventing PK$SAVE from attempting the undesirable write?

No, because the write is earlier than the end of the pack. It could even be later, but you'd then need Boris Cornet's DumpPack to see it because UNMAKE would gauge space by looking for end of a valid file, and SendPack would not go any further than the content size declared in the pack header.

I edited the post since you replied, this will help clarify the simplest way I can think of to test this thing.

[MartinP]

I tried your code in Jape (LZ ROM 4.6) and didn't get any FF bytes when the record was overwritten, maybe you could explain the problem a bit more? I like your idea to create faster data access this way.
Martin P.

[Lostgallifreyan]

I tried your code in Jape (LZ ROM 4.6) and didn't get any FF bytes when the record was overwritten, maybe you could explain the problem a bit more? I like your idea to create faster data access this way.
Martin P.

Thankyou. It would make a lot of things possible, having that much storage available for random access. I don't know why this wasn't arranged decades ago. Even a 32K RAM pack offers a lot more than can be loaded into internal RAM.

I just tried Jape, with XP, ROM v3.3, and got no occurrence of the extra 'FF' byte either.

Bear in mind that Jaap told me he didn't know about this extra 'FF' byte. For whatever reason, this means the test should be done in hardware, for the same reason that even the best SPICE model can never replace raw IC's and transistors in electronics tests.

It's interesting that it did not happen in an emulator, it suggests that logically it should not happen (given the same ROM code), but it DOES, in two different types of Organiser II using the exact same code, the same pack image. I did move the same RAM pack between Organisers though. I doubt the pack is defective but I will try this test with a different one now that I have reason to try.


EDIT:
I just tried two different Organisers (both XP, v3.3 and v3.6, and a different, low power RAM pack).
The 'FF' byte was printed over the test string in each case. I started with one file record in B:MAIN, one field in it: "ZOODLEWURDLE14".
This gets replaced by "PK$save Test#4" (where the '#' is actually a full block character as defined for CHR$($FF) on a Psion Organiser. I can't print that on a browser page for obvious reasons...).

[Lostgallifreyan]

I've attached a zipped OPK file here, so exactly repeatable conditions can be used for anyone, in any test. Put the image on a 32K RAM pack, in slot B, before trying any variants of the test. The first need is to replicate the experiment and its results, to see where and how it may be different. (Ideally using an unpowered Organiser initialised by cold boot after supplying power to it, but I doubt the test will either harm existing Organiser content or be affected by it.)

[MartinP]

I tried it with the hardware (my LZ64 ROM v.4.4 and XP(LA) ROM v.3.6) and I got the FF byte. So that's confirmation, but I've no idea why it happens.
Martin P.

[Lostgallifreyan]

I tried it with the hardware (my LZ64 ROM v.4.4 and XP(LA) ROM v.3.6) and I got the FF byte.

Thanks for confirming that. I looked at Jape again and looking at a saved page and figuring out how to get its ROM (33-la.rom in this case), I tested it against the ROM in my Organiser, as sent by ROMSND32, and it's a perfect match, so it's not the code!

I don't think Amenjet is looking at this thread but he probably should, because it's now clear that there is something about the hardware that causes this, and it was not known by Jaap Scherphuis, and anyone making a hardware emulation of the Organiser needs to not only explore this, but to understand it.

[amenjet]

I tried it with the hardware (my LZ64 ROM v.4.4 and XP(LA) ROM v.3.6) and I got the FF byte.

Thanks for confirming that. I looked at Jape again and looking at a saved page and figuring out how to get its ROM (33-la.rom in this case), I tested it against the ROM in my Organiser, as sent by ROMSND32, and it's a perfect match, so it's not the code!

I don't think Amenjet is looking at this thread but he probably should, because it's now clear that there is something about the hardware that causes this, and it was not known by Jaap Scherphuis, and anyone making a hardware emulation of the Organiser needs to not only explore this, but to understand it.

I have been scanning this thread. Datapack support on the re-create organiser is not good (see other thread about the recreation), so I haven't seen this. All I have done so far is read small EPROM packs. I'm wondering if the recreation will support datapacks going forwards, again, see the other thread.

Looking at PK$SAVE in Jaap's documentation, it says it leaves the counter pointing to the byte after the last one written. That might be a clue. If the control signals to the datapack aren't handled correctly then you could get a spurious write pulse and if the data lines are driven to all 1s...
I'd check carefully how the slot is deselected.

Andrew

[Lostgallifreyan]

Looking at PK$SAVE in Jaap's documentation, it says it leaves the counter pointing to the byte after the last one written. That might be a clue. If the control signals to the datapack aren't handled correctly then you could get a spurious write pulse and if the data lines are driven to all 1s...
I'd check carefully how the slot is deselected.

Andrew

I agree, that's how it looks, something using the existing pointer as incidental. The act may even be intentional. The fact that the way I intend to use those services (to extend RAM available to a program by using a file on a RAM pack as as an array), appears to have never been applied in 45 years, suggests that the PK$save service was intended only to write at the end, in the first free space, and the hardware was expected (intended?) to force an 'FF' at the end of its write just in case the one that should have been there wasn't! After all, it is either intended, or is a bug.

Anyway, rather than trying to stop it, which is not possible for all those Organisers out there, the most practical answer is to force a write of that one byte, after writing to length-1, so the remaining byte in the buffer is written as wanted by some other method (not available as a system service), but if you're right about pack deselection being a cause, then nothing can ever work except to provide that extra byte in the buffer, in each record of the file, just so there is a safe place for it to happen. The use of the file as an array is safe, it's just that the waste of space in adding the extra byte is ugly... Even so, the resulting speed of accessing records for writing in place is worth all of this.

One thought occurs to me: If you, or anyone else, is building large RAM packs, the most likely thing to use this method to best effect, is there any chance that the packs could be designed to prevent whatever action the Organiser is inflicting on that last byte?

Yet another thought occurs to me: Setting PK$sadd to point at the same spare location immediately after writing! That could be a safe dump for the 'FF', and likely faster than any other trick I can think of. I can try this myself.. If it works I'll post it because that's the main aim of my thread, to get a desirable outcome, whether or not we ever get a full understanding of the observed state so far. :)