How can PK$save be stopped from adding an extra byte 'FF'?

[amenjet]

The trace comes directly out of the recreation hardware over USB. The asm listing is too big to put in the Pico, but maybe it could be combined with the trace on the PC. Maybe.

It did look like it might, there were addresses shared, and the gaps in the trace record at one point matched those in the ASM listing, based on instruction lengths. Given enough info between the two outputs, I think I can merge them in TextPad with a macro, then reduce major and obvious loops later by hand.

Maybe that routine is working with a concept like the OPL records? They are manipulated then written later, I think. The actual write must be done in some other call.

I think so too, the services might well match the order of events like EDIT and UPDATE, to do otherwise might be too inefficient once Psion decided on a method.

My aim is to use OPL where it's most efficient, e.g. it's great to OPEN a file by name in OPL, saves a lot of service calls, just doing that: sets pack, sets file, makes sure pack is switched on... Once I have code that can handle the port access to write one byte to a RAM pack at the current address without affecting any other byte, I'm in business. I'd arrange my own buffering, in fact, I'd just point to an OPL string as in the example I prepared near the start of this thread. That way I can optimise it. The hardest thing is always trying to understand complex inbuilt behaviour when we need to circumvent unwanted action. Defining it as new, MUCH easier...

EDIT: Well, it will be easier, once I know how to write that byte to a RAM pack. I never understood the hardware access, because things like timing to avoid trouble with NMI's and such made it difficult. At that level, the data is extremely dynamic, it never sits still, probably because so much hardware stuff is multiplexed constantly. We can imagine four 'packs' on an Organiser at once, but the Organiser can't.

I think you will be fine accessing RAM packs directly with your own code as there's no timing problems. EPROM and flash packs have timing constraints when being written to, the RAM packs don't, so stretching a read or write cycle won't be a problem. Your only problem will be leaving the bus in a state where the next piece of code that wants ot use it is happy with it, and managing the data on the pack.

[amenjet]

I've added flags to th etrace:

Code: Select all

0111:824C      A:FF B:00 X:012C SP:7FE0 FLAGS:D8 (_IN___)
0112:87C6      A:00 B:01 X:2401 SP:7FE0 FLAGS:C0 (______)
0113:87C5      A:00 B:09 X:23F3 SP:7FE0 FLAGS:C0 (______)
0114:8567      A:C4 B:00 X:012C SP:7FE2 FLAGS:C4 (___Z__)
0115:87C8      A:00 B:00 X:2403 SP:7FE0 FLAGS:C4 (___Z__)
0116:87C6      A:00 B:07 X:23F5 SP:7FE0 FLAGS:C0 (______)
0117:8246      A:C8 B:00 X:8242 SP:7FDC FLAGS:D8 (_IN___)
0118:87C1      A:00 B:02 X:23FF SP:7FE0 FLAGS:C4 (___Z__)
0119:87BF      A:00 B:09 X:23F1 SP:7FE0 FLAGS:C0 (______)
011A:854A      A:C4 B:00 X:012C SP:7FE2 FLAGS:C4 (___Z__)
011B:87C2      A:00 B:01 X:2402 SP:7FE0 FLAGS:C0 (______)
011C:87C1      A:00 B:08 X:23F3 SP:7FE0 FLAGS:C4 (___Z__)
011D:8291      A:FF B:00 X:012C SP:7FDB FLAGS:D8 (_IN___)
011E:87C4      A:00 B:03 X:23FE SP:7FE0 FLAGS:C4 (___Z__)
011F:87B8      A:FF B:12 X:23F1 SP:7FE0 FLAGS:C0 (______)
0120:853E      A:13 B:00 X:012C SP:7FE3 FLAGS:C4 (___Z__)
0121:87C5      A:00 B:02 X:2401 SP:7FE0 FLAGS:C0 (______)
0122:87C4      A:00 B:09 X:23F2 SP:7FE0 FLAGS:C4 (___Z__)
0123:8550      A:C4 B:00 X:012C SP:7FE2 FLAGS:C4 (___Z__)
0124:87C6      A:00 B:03 X:23FD SP:7FE0 FLAGS:C0 (______)
0125:D84F      A:FF B:00 X:012C SP:7FE0 FLAGS:C8 (__N___)
0126:8534      A:13 B:00 X:012C SP:7FE7 FLAGS:C4 (___Z__)
0127:87BF      A:00 B:02 X:23FF SP:7FE0 FLAGS:C0 (______)
0128:87C6      A:00 B:09 X:23F1 SP:7FE0 FLAGS:C0 (______)
0129:8542      A:C4 B:00 X:012C SP:7FE2 FLAGS:C4 (___Z__)
012A:87C1      A:00 B:04 X:23FB SP:7FE0 FLAGS:C4 (___Z__)
012B:8268      A:FF B:00 X:012C SP:7FDE FLAGS:C8 (__N___)

[Lostgallifreyan]

I think you will be fine accessing RAM packs directly with your own code as there's no timing problems. EPROM and flash packs have timing constraints when being written to, the RAM packs don't, so stretching a read or write cycle won't be a problem. Your only problem will be leaving the bus in a state where the next piece of code that wants ot use it is happy with it, and managing the data on the pack.

That's good to know (no timing trouble). I had an idea last thing last night, and only just now had time to test it. I assumed that writing a byte to a slot must use a very limited instruction, either 97 (STA A) or D7 (STA B) immediately followed by a zero-page address, specifically 03, for slot buss. It turns out that of these two possibilities, 33-LA.rom only contains one, and that only once: $9703 at ROM offset $D9B9. I've not looked further yet, I just wanted to let you know ASAP because either it's irrelevant, or it nails the location very precisely, at least it may give a chance to look outward from a central point rather than trying to capture it by stepping through larger code first.

[amenjet]

The technical manual has the algorithm for ram packs here:

Code: Select all

9.2.9  32k, 64k AND 128k RAMPACKS

All 32k and 64k rampacks are paged.

All 128k rampacks are segmented and paged.

The following conditions are required to read from rampacks
         SCLK Don't care
         SPGM_B Don't care
         SVPP Don't care (5v recommended)
         SMR low
         SOE_B low
         SS_B low


     The following conditions are required to write to rampacks
         SCLK Don't care.  (either high or low)
         SPGM_B Don't care
         SVPP Don't care (5V recommended)
         SMR low
         SOE_B high
         SS_B low

There's example code here:

Code: Select all

9.3.3  EXAMPLE

The following code powers up the slots and then reads the first 20 bytes of
a  pack in slot 1.  It contains no checking to see if a pack is in the slot
or if it is, what sort of pack it might be.

It is recommended that the operating system be used to power  up  and  down
the slots and the second example shows how this is done.  An explanation of
the operating system calls is given in the next section.

An example of how to do this using just operating system calls is given  in
section 9.5.11.


; Save interrupt mask and disable interrupts
        TPA
        PSH     A
        SEI
; Initialise ports
; 4 must be stored in RMCR to allow bit 2 port 2 to be an output.
        LDA     A,#$4
        STA     A,POB_RMCR:
; Make port 6 an input. This powers down the slots and deselects them.
        CLR     POB_DDR6
; Make port 2 inputs. Port 2 (pack data bus) should always be left inputs.
        CLR     POB_DDR2
; Initialise port 6 so it has the correct data when it is made an output.
; Make SS1_B=SS2_B=SS3_B=1,SMR=SCLK=PACON_B=SOE_B=0,SPGM_B=1
        LDA     A,#$74
        STA     A,POB_PORT6:
; Make PACON_B of port 6 an output, powers up slots
        LDA     A,#$80
        STA     A,POB_DDR6:
; Wait for power to settle
        LDX     #11520                  ; 50 ms delay
1$:
        DEX
        BNE     1$
; Make rest of port 6 outputs now the slots are on
        LDA     A,#$FF
        STA     A,POB_DDR6:
; Reset pack counters with master reset
        BSET    SMR,POB_PORT6:          ; Master reset high
; Store 0 in segment register in case we have 128k pack
        BSET    SOE_B,POB_PORT6:
        LDA     A,#$FF
        STA     A,POB_DDR2:             ; make port 2 output
        CLR     A
        STA     A,POB_PORT2:            ; 0 on data bus
        BCLR    SPGM_B,POB_PORT6:
        BCLR    SS1_B,POB_PORT6:         ; select slot 1 and latch data
        BSET    SS1_B,POB_PORT6:         ; deselect slot
        BCLR    SOE_B,POB_PORT6:
        BCLR    SMR,POB_PORT6:          ; Master reset low
        BSET    SPGM_B,POB_PORT6:
        CLR     POB_DDR2                ; make port 2 inputs again
        BCLR    SS1_B,POB_PORT6:        ; select slot 1
        LDX     #DATA                   ; where to save data
LOOP:
        LDA     B,POB_PORT2:            ; read data
        STA     B,0,X                   ; store data
        BTGL    SCLK,POB_PORT6:         ; increment counter on EPROM
        INX
        CPX     #DATA+20                ; Have we reached the end yet ?
        BNE     LOOP                    ; If not loop
        BSET    SS1_B,POB_PORT6:         ; deselect slot
        CLR     POB_DDR6                ; Turn off all slots
        PUL     A
        TAP                             ; restore interrupt mask


The following example does the same as previous one but uses operating
system calls for powering up and down the slots. If the user has to write
their own pack reading routine then this is the preferred method.

; Save interrupt mask and disable interrupts
        TPA
        PSH     A
        SEI
; Initialise ports, power up, select required slot and reset pack counters
        CLR     A
        LDA     B,#PAKB
        OS      PK$SETP
        BCS     ERROR                   ; optional error checking
        LDX     #DATA                   ; where to save data
LOOP:
        LDA     B,POB_PORT2:            ; read data
        STA     B,0,X                   ; store data
        BTGL    SCLK,POB_PORT6:         ; increment counter on EPROM
        INX
        CPX     #DATA+20                ; Have we reached the end yet
        BNE     LOOP                    ; If not loop
; turn off slots
        OS      PK$PKOF
        PUL     A

TAP ; restore interrupt mask

[Zenerdiode]

I too saw those sections of the TRM, but, frustratingly, they’re being very ‘woolly’ when it comes to helping Lostgallifreyan. They probably don’t want us switching on the pumps Willy-Nilly:

Code: Select all

9.3.2.7  Writing to a Datapack

The EPROMs are written to using  a  variation  of  the  INTEL  quick  pulse
programming algorithm.
Users are advised not to attempt to write their own EPROM programming code,
but to use the supplied operating system calls.

No mention of writing to a RAMPAK, but I think Lostgallifreyan is on to something when finding the Accumulator write to $03 (Port 2).

[Lostgallifreyan]

Thanks Zenerdiode for the comment, I think it really might be a valid location not least because it turns out that $D955 which Jaap mentioned, is very close to it.

Thanks amenjet for the code examples. It's not obvious from things like 'BSET SMR,POB_PORT6' not being a direct ASM instruction, but it looks like 'OIM #OE,POB_PORT6' is closer to raw syntax, as an example. I looked at Tech04.htm from Jaap's site and tried to use the same idea I had before, to narrow the range of possible instructions that did anything to zero-page address $04, the DDR for port 2, but found nothing obvious, and the possibilities for AIM, OIM, EIM, etc, gave too many possibilities to narrow it down the same way. Even scanning the code by eye, looking for anything I could understand, fails.

It seems to me that a large number of very small functions is used, and a lot of raw port handling, and pack paging and segmentation handling on the kinds of RAM packs big enough to need this method, and the kind of extreme reverse engineering needed to do this is far beyond anything I can handle. Inconsistencies in notation between different Psion docs isn't helping that at all.. They advise staying with service calls, with very good reason.

I don't mind if there is never an answer to this, because the aim is to use a data file on a RAM pack as an array, and despite the ugly writing of FF after every string of raw bytes written by PK$save, the easiest solution, one anyone with basic OPL skills can grasp, is that we must allocate a record size big enough to contain the FF as well as the wanted data. Preparing packs in advance with BLDPACK after preparing data files with a text editor makes fast and easy work of this.

If avoiding this extra FF needs the skill needed not only to build an Organiser that can step through code and report all flags and registers, but a total reverse engineering of most of its operating system, then this is a fusion bomb to crack a nut.

It would be great if it were solved, but I can accept if it is not solved. This is a rabbit hole that runs right through the Organiser's entire world. Diminishing returns, says I, except in the context of a full and perfect emulation, where maybe, at some point someone (likely amenjet) will probably discover the answer while looking for something else. :)

If there is a solution based on extremely low level code, I wonder how many people would ever use it. Most could not understand it, so may not trust it, so limiting to minimal service calls and doing as much as possible with OPL seems to be the best compromise to bridge the gap. It works, and it's fairly easy to understand, and the documentation is clear, consistent and relatively nontechnical.

[Lostgallifreyan]

This is the result of an hour's wandering through the code around the single write to port 2. The subroutine branches and jumps make the Spaghetti Junction look like a short straight cable! I did not look into those except to establish that some elegant but opaque stuff is going on to minimise duplication of larger sequences of instructions, much of it having to do with port switch settings.

One point of interest is this:

Code: Select all

D970:   86 FF                   LDAA #
D972:   B7 24 07                STAA mm

It's the only immediately local use of 'FF' as a value, other than a few settings of all bits to 1 (using OIM) in some other locations like port switches. This one looks like an actual data value, and the storage location is not listed in the system variables document, and it is out of bounds for OPL too..

There is a lot of use of the UTW_Sn scratch registers. All six of them. Given that we can't access the stack safely, and Psion want us to use service calls, it may be that the scratch registers are intended (in part) to take user data to cause routing changes in all that branching, thus avoid things like that unwanted 'FF' byte, and there seems to be no other mechanism available for doing this.

There are too many permutations for randomly shoving values into scratch registers to provoke changes in the behaviour of PK$save, but I am putting the results of my short exploration here in case someone finds it useful. I suspect it won't be so I'll stop there.

[amenjet]

Yes, I think working from scratch with new code is a more time efficient method of working. What is missing from the organiser II world, though, is a fully commented disassembly of the ROMs, or even just one ROM. That has been done for other platforms like the ZX Spectrum and it makes other projects based on the platform so much more time efficient. It's a lot of effort to disassemble and comment a ROM, though, and it would have ot be a multi-person effort. Ghidra is a good tool and it may have collaboration tools in it, I'm not sure. It can disassemble a ROM automatically in a couple of minutes.

Of course, if things are moving towards new code then knowing what the old does is not as useful, maybe.

[Lostgallifreyan]

Yes, I think working from scratch with new code is a more time efficient method of working. What is missing from the organiser II world, though, is a fully commented disassembly of the ROMs, or even just one ROM. That has been done for other platforms like the ZX Spectrum and it makes other projects based on the platform so much more time efficient. It's a lot of effort to disassemble and comment a ROM, though, and it would have ot be a multi-person effort. Ghidra is a good tool and it may have collaboration tools in it, I'm not sure. It can disassemble a ROM automatically in a couple of minutes.

Of course, if things are moving towards new code then knowing what the old does is not as useful, maybe.

I think the problem with new code to cover all the needs to handle pack segmentation, paging, port switching, is itself a huge burden. Psion warn almost anyone against trying this, at this level, but the main concern might be the use of RAM for this code. It might be very consumptive, defeating the point of making more of it available to a program, or alternatively, forcing an even greater need for the method I describe in the first few posts. :)

Pondering this and waiting for insights various might be the most productive way through. Knowing that only one location in the ROM can put data on port 2 was a useful find. Others are harder to narrow down usefully, reducing the number of things to test, but the one idea that keeps haunting me is this:

Psion provided no user means to write a single byte, so what would they have done if they wanted to do it? I do not know the answer, I just assume they had a way to do it that they did not reveal. If this is true, then there must be a way to communicate with the relevant subroutines and services. The stack is out, because arbitrary external manipulation is too risky. That leaves the global variables, and from what I have seen in my short look at the code around the single Port2 write instruction, it looks like UTW_Sn scratch registers are a likely method. Some are used as counters and other internal widgets, but some may be used to control program flow.

It was Jaap who pointed out location $2408 to me in email, and I discovered uses of $2407 last night. Again a downward off-by-one in an address given by Jaap, odd, but I don't know what tools he's using.. Jaap suggested setting 0 in $2048, but in my case it already was, and no other pre-poking changed the outcome, I always get the 'FF'. For the record, I tried another thing last night: writing NO bytes, as in attempt, with 0 as argument, just in case it was a special case. :) I'd hoped it might print the first from the source, and NOT add the 'FF' afterwards. Wishful thinking, and it didn't do anything at all, but it was worth a try... I often find that silly tests like these can provoke outlying behaviours in code, and they can yield a way in when little else had up to that point.

Anyway, I think the quickest way to get into the code, or even shortcut any need to comment it all, might be to look for 'hidden' ways to communicate with it. They may be easy to find, and if there are none, then using existing code to solve the 'FF' problem would be proven impossible because the code would be effectively deterministic after launch. A proof like that could save a lot of work.

[Lostgallifreyan]

Hi, I went back to look at Jaap's suggestion again. He'd suggested POKEB $2408,0 before my service calls, which on testing did not work. It turns out he was right, given small variants on how to apply the change.

First, early events after OPL's USR call cause that address's WORD content to be cleared, so I set up a test to store 0 to clear the WORD at $2407 immediately before ASM's PK$save call, which as expected, did nothing, but I assumed Psion might test for 'FF' as -1, true, to indicate a different action from a default. Setting '$00FF' at address $2407 works, when done immediately before calling PK$save. Setting or clearing the bits in the first byte made no detected difference to the outcome of setting or clearing those in the second byte.

The new test code is below. First, create a MAIN on a RAM pack in B: with one record, and one field as before, with a 14 character string in it. (I used "0123456789ABCD" for my latest tests.) The program below should change this to "PK$save TestCD" instead of putting a filled block where the C should be. If anyone tests this on hardware, please let me know what you find.

Code: Select all

A:
LOCAL M$(12),C%(13)
M$="PK$save Test"
OPEN "B:MAIN",A,A$ :FIRST :CLOSE :rem saves a lot of machine code if you do this first.

C%(1)=$3F2D :rem SWI FL$frec
C%(2)=$DC41 :rem LDAB 0m
C%(3)=$3F60 :rem SWI PK$sadd

C%(4)=$01CC :rem NOP,LDD Test bytes
C%(5)=$00FF :rem Test bytes
C%(6)=$01FD :rem NOP,STD Test bytes
C%(7)=$2407 :rem Test bytes address

C%(8)=$01CC :rem NOP,LDD Byte count
C%(9)=12    :rem BYTE COUNT
C%(10)=$01CE :rem NOP,LDX Source string
C%(11)=ADDR(M$)+1
C%(12)=$3F61 :rem SWI PK$save
C%(13)=$3900

USR(ADDR(C%()),1)

EDIT: It works on an XP with ROM 33-LA but not on the LZ64 I tried just now. There are three instances of code 7D2408 (TST $2408) in that ROM, not sure how many in others, but I think examination of those tests will eventually get a way to make this work on all versions of series II Organiser.