History

When I started out writing programs there wasn't much in the way of 'source control'. Very early on much of the stuff that I wrote was version controlled by just copying the sources to a new file, with a different suffix on the file name. !LongFiles helped greatly back then, in the RISC OS 3.1 days. The differences between the different files were generally unclear, and I took note of some of the PD software of the time which included lists of the changes between the different releases. When I was working on !FormEdExt I began keeping some notes of what I'd done and how each differed. But they still ended up being too vague to identify anything useful.

The !Help files on PD programs were also enlightening, mostly because they told you stuff about the program (there were many that didn't), but also because some digressed into the author's views, and complaints about other things. That, plus the Authors Notes on Piers Anthony books (which could be 10 pages or so), was some of the influence for my own rambling 'style'. Looking back on some of the !Help files there's a naivete that's quite funny. One program I wrote had a !Help file (sadly I've forgotten which program it was) which talked about how I'd been frustrated by not having a feature and wanted to do something about it... but on the way wrote this completely different program instead.

Looking back I wonder why important details were missed out of the documentation and other random things made it in to them instead. But I'm pretty sure that it was a product of the way I was thinking. I certainly cared more about writing clever things, and showing that I could do those things, but didn't really understand that the way I was doing it - describing the complicated technical details - was only going to work with certain people.

I'm still like that in some ways, but when you're doing Software Engineering full time, most people around you are clever and you don't need to prove you are too; you need to do things that are useful and make sure that people understand. Having something that's really neat but no bugger can work is actually less useful than not having it at all - because people waste time.

Rambling about generalities aside, documentation is vital and early on, reading the Doggysoft WimpSWIve documentation was very useful. It set out the problem (that you couldn't have multiple people patching the Wimp without them breaking the patches that the others made), their solution (the WimpSWIve module, which allowed claimants of Wimp SWI calls in a controlled manner), and the way in which you used it (the SWI calls, some caveats and the like).

It set a quite a good initial standard for me in how to write things. Later, I discovered that part of the style came from the PRMs - which I still hold up as an example of the gold-standard of programmers documentation. Anyhow, later the SysLog module had similarly clear well documented help. Good documentation isn't the be all and end all of usability, but it goes a good way. Good documentation on a good tool can mean that the uptake is high. Bad documentation on a good tool can mean that it's only used by people who invest the time in it, and therefore uptake will be lower initially (assuming that it does something that other people could find useful but don't because they can't pass that high barrier to initially use it). There's also factors like User Interface polish and actual functionality which have big impact, obviously.

These days 'how to use it' documentation is generally considered bad - User Interfaces should be obvious - and the amount of free software that's not given any sort of useful documentation because the code's right there and the author has explained the most esoteric thing they could in the README, is quite astounding. But for those early days it mattered more.

To digress further, I'd guess that the Internet has helped in that - instead of reading documentation to say how to do things, you search for the problem you have and the solution that others came to. It leads to a certain amount of laziness, though.

Anyhow... we started using version control a little at University when we did some team work. Only RCS, but still, it was something, and we needed to work together on things - just writing a simple database tool, I think it was - when we kept different hours. As is usual, a bit of thought and we could have rattled through the project in about 4 or 5 hours if we were all in one room... but that never happened, of course. Part of the project was about defining your interfaces properly first, reviewing them and then implementing to them - even if you didn't have other people's actual implementations, you could stub something that worked according to the API they designed.

But, the documentation - in the headers - had to be very clear in order to do that. We had to submit our team's APIs as header printouts (oh how archaic!) for review by the lecturer. When it came to mine I remember quite fondly being told that my APIs were many times larger than the largest submitted by the other teams. I explained - as they must have known - that the bulk of the printout was commentary about what the module's purpose was and how it fit into the other modules, and what the functions would do and their parameter usage, with the prototype lines taking up a tiny proportion of the paper.

You might argue that it was a little over the top (and I can maybe accept that), but I felt (and still do) that you need this level of communication and you cannot assume that the next person along will have a clue how things fit together. There are some who view this kind of documentation to be a part of an older style of development, rather than some of the more modern high turn around development styles. That might work sometimes, but I guess the fact that I've still got records of what was done and how things were meant to work shows that it serves me quite well.

StrongHelp

Later I came to maintain the 'OS' !StrongHelp manual, and the related manuals. The focus in those manuals was for correctness, with information about problems thrown in, and brevity. The !StrongHelp manuals weren't good for overviews, but great for reference. Many of the SWI usages that were in there were initially from experimentation and usages seen elsewhere. Later they were updated to correct things from the PRMs (without just lifting content), and the other released documents on BBSs, in programs and from people by email, or over IRC.

There were still a whole load of 'unreleasable' things which I kept back because they'd been supplied by individuals or companies and were embargoed. Some things became public, when they were mentioned in newsgroups, or a proper statement was made, but there was always something of a dance to make sure that I never released any information that might annoy people.

There were always sources like OSLib headers which had lots of useful little details in which hadn't appeared anywhere else, and often made its way into the manuals (as OSLib was public), but with little explanation. In some pages (I think some of the SWI and Service call indices) viewing the source for the page would give some extra hints to calls that hadn't been made public yet. Some of the API pages had little conversations in the comments between myself and other contributors about the problems we'd had with them.

Spin on a little way to the documentation that came out of Acorn for the Nested Wimp and the like. These were good, well structured documents that described the changes and had interesting things like version control on the documents themselves, information about change impact, memory footprint and file formats. All in HTML, which was the way to go if you weren't writing just plain text files. Someone had clearly spent the time making it look professional

RISCOS Ltd documentation

A little later, we come to RISCOS Ltd. We needed a way to issue update information - Tech notes, I think we called them - on the issues that we'd found and how they affected software. I remember David Thomas wrote a few on the problems he encountered which really shouldn't have been there - and would have bitten other people. We stuck to text, with a structured format, to try to make them easily accessible to people. For other parts of our internal documentation, I had a HSC template that we could use for our documentation which produced documents similar to those produced by Acorn. I'm still using HSC to this day - it provides the templates for my gerph.org website, and the diary. It's quite capable, but it is a bit esoteric.

PRMs

Along side this, there were the PRMs. The original source was in FrameMaker, and we couldn't guarantee we could a) keep it working, or b) that it was even sensible to do so. So these were converted to HTML on a per chapter basis. I believe that Acorn had a technical publications department at one time, which controlled the documentation and therefore made our lives easier. It was regularly formatted so the output was pretty reasonable, and we could modify it quite easily, even in the HTML produced. I say 'we'. I mean David Thomas. He did the real work on taking the base HTML files, correcting the broken formatting, tidying up the pages, correcting entities, testing it on RISC OS browsers of the time, and generally making it usable to people. Hat off to him for that - it was a bit dull, but he did a great job, and it helped me lots later.

We were a bit bad on documenting what was changed for RISC OS 4, partly because we mostly fixed things that weren't quite working for the 3.8 version and there's not a lot you can write about for that. Partly because there wasn't that much that was added. Those bits that were, were documented but it was a bit patchy.

Commit messages

After I went to Pace, things were interesting. Being a larger company they had better control over things than we'd had, and much more structure. Source commits were always done in a useful way, in particular (this is a particular bugbear of mine). I don't remember the exact form, but it became my 'standard' for subsequent source control messages for my own things and for RISC OS things. Taking the example from my own code, commit messages take the form:

Summary:
  (overview of changes, generally one line each, intended to
  be publicly readable)
Detail:
  (bulletted list of features that have changed, rationale,
  side effects, and problems encountered)
Admin:
  (highlight level of testing that has taken place)
  (bugfix number if appropriate)
Tag:
  (automatically generated for this commit)
The general idea is that:
  • The format is standardised so can be automatically processed if necessary.
  • Commit messages can be mailed to everyone and you only need to read the top section to get a general idea about what's going on, and whether you need to take any action.
  • The summary contains nothing that might identify customers, or divulge internal or restricted information.
  • The summary is therefore publicly releasable, even if it includes silly mistakes in it.
  • The detail can be long if necessary - and sometimes explaining why you're making a change can (and should) take some time.
  • The administrative information is useful to anyone who is reviewing the change. Incomplete work, and any other testing that's necessary should be talked about here, so that a reviewer can decide what's been done and what may still need to be.

Change summaries

If it's done properly, you should have sufficient general information to produce release notes which explain the changes. At the worst, you've got a whole lot of information that can help you determine where changes happened. The 'Summary' sections were released as part of Select 2 in text form - I believe every component had a little text file detailing its changes since RISCOS 4. With Select 3, the Change information was delivered with the ROM (I believe) and the !Boot.Softload.Changes directory updated with the change details for every module in the ROM in a file with the same name as the ROM.

Looking at my RiscPC boot sequence backup, it looks like the change files contained a lot of redundant info, like the release dates repeated for each component.

There's not much more in the way of description that these lines give - the proper documentation on the component was expected to give specific information about the changes. That wasn't always the case as many 'minor' bug fixes would not warrant more than the line that they got in the Changes file - despite many of them being complex to fix internally.

The change information was also a spur for talking points. Sometimes (frustratingly regularly) it was a 'why hasn't x been fixed', 'it has, look here', 'how can I tell that that line means it was fixed' exchange, which was quite fair. In most cases, though, it was never possible to document every single thing that was done, its rationale, and the other issues it causes. There just wasn't the available time, or justification, to do so. Very occasionally there were some interesting discussions sparked off 'what does x mean?', coming from the changes.

Although the main reason for releasing it was these summaries was that it may help people, and it had always been intended to be released, there were other factors. Some developers had privately commented that it was often hard to know what changed between versions - we could always reproduce any given version of a module, but only on request. Some developers moaned on and on in the newsgroups about how they couldn't do anything because the system and interfaces were changing, and they weren't told how they were changed. It was quite wearying, and I knew at the time that there was a lot we'd like to do but didn't - because I was always being pulled in different directions by other people's requirements. Sometimes the people that shout loudest do get what they want, even to the detriment of other things (which isn't to say that they're not right to do so, but it does have an impact).

Prior to Select 4, we produced a page of the changes which were made which was available to all.

I'd been keen on doing this for quite some time, and had previously released the same information in the form of a database to Marutan, which was being accepted by some people as an reference for module versions - but also didn't include any information about what changed between versions. The database version of the information wasn't really used that much - I have a vague recollection of someone at Acorn (Stewart Brodie, maybe?) having put all their component version information into an LDAP database for their own use, which seemed quite cute.

Modern documentation

Aside from the changes, there was real documentation. Much of the documentation for Select came in the form of text files. Mainly this was because text files are significantly easier to write. They take a lot less time than markup, and don't need to be well structured to get something useful out of them. On the downside, they don't present so well except in a text editor.

Most of the documentation tried to follow the spirit of the PRMs. Start out with a general overview of how what the component does and how it fits into the systems as a whole. Move on to some description of how you use it, and what it will do for you. Describe the APIs that it uses, and their relationship to one another. Give any examples and references to other documentation.

It doesn't work as well when the document is explaining a change in behaviour, but the general principle works ok. There also tended to be a few documents that just tied things together - some that explained where a stack of components, or a wider overview of the way that multiple components would interact with one another, referring to the new (and possibly old) documentation to do so.

There are some components which to this day I lament the quality of. The documents were reformatted by Aaron Timbrell and whilst I was initially quite dubious about it, he did a good job. The text version is still canon, but the reformatted HTML is a lot easier to read in a browser. I'm not sure if anyone actually ever read them - certainly I've had people say to me (in those rare times that people do talk to me about RISC OS stuff these days) that they never knew about them <sigh>.

RISC OS Select documentation is up on the RISCOS Ltd website. I still don't call it a 'PRM' because that's an insult to the people that actually produced the PRMs - my documentation is not to that level of quality - and they should only be considered as software documentation. Maybe that's just me, though. Call them what you will.

However, it brings me back to lamenting about the poor quality - PNG was never documented properly. It's a full PNG library, and the SDK would let you use all its features just as if it were directly linked, but there's no information on how to do that if you're not using the SDK - eg if you wanted to do it from BASIC.

Actually, I think the best bet if you wanted to find out how to use it was to look at the !ChangeFSI application, which has an example of how to create files. Oh, I've just looked. It's not good. It's got the definitions of the reason codes but that's all, and the calls to the SWIs are completely uncommented (this is !ChangeFSI, after all - comments happen to other components). Oh well.

So, PNG documentation isn't up to scratch. I guess I can also justify this by saying that we didn't really want people to call the SWI interface - the library stubs would just make it work. Bad reasoning, I think, though.

XML documentation

The plan to produce documentation with HSC didn't really work out that well. There are many reasons for this - the processor's a bit specialised, developer interest in it was low whenever it was discussed, and it has a limited focus in that it only produces HTML. These, coupled with a gap of a year or so in the middle of development where no work went into the documentation due to work at Pace and Picsel, meant that I needed something else to work with. The PRMs having been produced in HTML, and it being the primary medium (other than PDF) for documentation, meant that whatever solution I came up with needed to be able to be used for both. Additionally it would be nice if the format we used was readable in editors. !StrongHelp was considered as a primary form, but this too was rather specialised, and the !StrongHelp-to-HTML converters I'd written (or that others had written) had shown up its limitations.

At the time I was looking for possible ways of writing the documentation, Wikis were becoming common, with varying degrees of functionality. Their simple, text based, approach appealed to me but I believed that using a Wiki for documentation was the wrong route. They are not easily indexable, harder to parse semantically and generally had little structure. Additionally, I knew from my own experience of writing 'simple' parsers for "just what you wanted to present" that you often found yourself tied to less expressive methods because the format itself couldn't be easily extended and maintain its primary feature - simplicity.

I would happily sacrifice a little simplicity for structure. I wanted to be able to pull out details from a documentation file for reuse elsewhere easily, and to be able to index the contents in different ways. The PRMs provided a useful index, which included indexes of all the SWI calls, SWI OS_Byte, SWI OS_Word, and SWI OS_ServiceCall entries, and they were very useful to find things. Similarly, each section was cross referenced by providing related calls. Whatever I used had to be able to provide these features.

I had briefly investigated DocBook XML and found it to be a bit too 'booky' for what I wanted. I went for a homegrown XML solution, making up a DTD and some initial documents to see how the format felt. After mocking up a few documents, I began to write an XSLT to convert the format to HTML.

XML / XSLT commands

Back when I was writing the SVG converter I needed an XML parser. I chose to use expat because it was small and provided a logical interface that I could understand - and because it had also been chosen by Mozilla for their parser. It served me quite well, despite how much I butchered its interfaces to change it from callback based to tree based results.

Once I understood XML a lot more, I wanted to use it as part of the document processing for what was to become the !PRMinXML work. What I needed was a way to take data from an easy to write language to a presentation format without having to reinvent the wheel. XSLT seemed like a good way to do that, having read around it and played with a couple of tools.

Initially I tried some basic manipulation using TransformiiX, the XSLT parser in Mozilla's browser, based upon the expat library. I had real difficulties making it do anything useful, and that was after getting a command line version to build - which was not simple in its own right.

TransformiiX didn't seem to be going anywhere useful, and the difficulty I had experienced meant that it was more fragile than I liked. So I looked to other solutions. The libxml2 library provides a very comprehensive XML parser, and a some very useful tools to go with it. It supports XML catalogs for storing DTD files in a standard way, as well, which meant that maintenance across platforms would be easier. As an aside, its HTML parsing mode became the input parser for the !NetSurf browser, although I'm not sure if its still used as such.

More importantly, it came with an XSLT processor, in the form of xsltproc. The processor was proficient, reliable and robust. Additionally it came with extensions in a few forms, particularly some of the exslt collections. These made some of the common XSLT operations a lot simpler and allowed a lot of the operations that would otherwise be tedious (or impossible) to be implemented directly.

I began to write some simple documentation using the format, at first taking some of the old documentation I'd written and converting it to the new format - and writing the XSLT at the same time. XSLT transformations take a bit of getting used to. It's a quite different way of thinking to the normal procedural programming languages. XPath required a quite steep learning curve as well. But the time it took paid off when the documents that were produced looked professional and the input looked maintainable.

XPath truly makes manipulating the document simple. Ok, not simple, but it is far less difficult than playing with the DOM in HTML. Being able to specify a search and match criteria in a single string makes it both very compact and (sometimes even) obvious. Being able to use '//swi[@number = 12345]' to return a nodeset containing all the SWI declarations for a given number, is pretty nice. At least I believe so.

It is quite surprising that I've not really used it since. CSS selectors are quite ugly by comparison and not half as expressive as XPath can be, albeit they are targeted at different things.

I wrote a few main transform - a rudimentary !StrongHelp XSLT transform, which could produce simple help pages from the documentation, and a text presentation were the most working ones, and neither were really production ready. The HTML transform worked very well, though.

There were some support programs, in particular a rough-and-ready Perl script that would take the HTML PRM documents and converted them into partially complete !PRMinXML format. Some elements weren't closed and the layout wasn't great, but the main content was there. It just took a little time to finish the conversion to get the text into the right shape. Sometimes it took a lot of time, even after the automatic processing. Mainly nested tables of memory locations and bit fields.

Application icon for !Catalogs
!Catalogs
I wrote a small application to sit around the processor, to make it easier to convert a single document. And the catalogs resource which contained the root documents were collected together into the !Catalogs.

Once there were enough documents converted - and some written from scratch - I began to indexing them. There's a few ways to provide useful references to the content. The primary one is to list all the related details together in an order, which we usually call an index. That was pretty simple really. Read all the documents into memory and perform a search across all of them for particular tags. The tags themselves were stored inside the XSLT file as a data set - you can access the document you are currently executing and extract XML data from it just like any other document.

Another way to reference the data is the table of contents. This is very similar, except that instead of producing one page per class of information, you present a single page with the document (and section details) ordered by their position in the final production.

Both methods are reasonably simple in XSLT. I say this now that the code all works and I don't have to debug it. It's a bit mind wrenching to switch back to working in XSLT after these years (it was bad enough at the time). The contents were particularly interested, because you could select the reference type you were interested in. So, instead of just looking for the section headings to present as links, you could also have the SWI numbers inline in the contents page. Depending on what you are looking for, both of these methods are very useful.

At the end of the day, I was really pleased with the XML processor and the output. I'd got the libxml2 and libxslt source building down to a fine art - you downloaded the new release, extracted it to a directory, copied a few Obey files to the place you'd extracted it and made a few changes to indicate the version number and the old version that it was to upgrade from, and run one of the Obey file. The old version would be run through diff, applied with patch to the new source, and a report made of the failures. This was repeated for every file, and if no problems were found, it would start building the new executables.

Sometimes the patching failed, if the RISC OS specific changes were not compatible with the latest work that had gone on, but mostly it worked great. Which a minimal amount of actual work, I could get an upgrade up and running in about an hour - most of which was build time.

Converting to XML

Many of the documents were converted to XML, with much of the processing being done automatically by script from the HTML and then worked on by hand to tidy up the bits that the script couldn't work out. This was made possible by the HTML conversion from FrameMaker plus David Thomas' changes being very consistent. Documents that would have needed to be reworked a lot could be converted to XML in a couple of hours. Of course some chapters of the PRM were more complicated, or had odd differences that had to be allowed for in the XML. Only a limited number of presentation formats were provided in the XML form, and to deviate meant either changing the DTD and XSLT, or reworking the presentation to match the remaining style.

The XML format was based around the fields that the PRM used, and could be presented in a similar way. Additionally it was (in my opinion) quite readable and could be updated relatively easily. An example from the AMPlayer documentation:

<swi-definition name="AMPlayer_Stop"
                number="52E01"
                description="Stops playback"
                irqs="undefined"
                fiqs="enabled"
                processor-mode="SVC"
                re-entrant="no">

<entry>
 <register-use number="0">flags :
  <bitfield-table>
   <bit number="0" name="Cut">
    Playback continues with the queued file if any. If no file is queued,
    playback will stop.</bit>
   <bit number="1-30">Reserved, must be 0.</bit>
   <bit number="31">R8 contains the instance handle to which this call
                    should be directed.</bit>
  </bitfield-table>
 </register-use>
 <register-use number="8">if bit 31 of R0 set:<br />
                          instance handle to direct at, or 0 for
                          the base</register-use>
</entry>

<use>
<p>This SWI is used to stop playback, or start the skip to the queued
file.</p>

</use>

<related>
<reference type="command" name="AMStop" />
<reference type="swi" name="AMPlayer_Play" />
<reference type="swi" name="AMPlayer_Pause" />
</related>

</swi-definition>

These XML definitions could be collected together to produce indices and pages of contents automatically. An index like the PRM's index volume could be produced which would have direct links by section, SWI, module, service, events, or whatever.

Application icon for !PRMinXML
!PRMinXML
The !PRMinXML project produced a nearly complete PRM in the XML format, from the HTML documentation, with extract documents I'd produced along the way. Andrew Hill dealt with some of the Window Manager chapters (which was quite a big bit of it), but the rest I pretty much did 'when I had time'. That was usually late in evenings, or when my head wasn't really up to anything complicated.

The Toolbox documents were reasonably easy, as the style was quite standardised by then. The Style Guide was probably the hardest because it was itself quite different throughout its chapters.

I think I had done a really quite decent job on those documents - the plan had always been that the text change documents be discontinued once we had the entire manual up and we could fit the new text documents in directly. Instead of starting a note with an empty text file, I'd just fill in a few bits on a template XML document and we'd have something very presentable.

Responses

Whether the documentation that was released satisfied the people who complained about things not being documented, I'm not sure. In general, during RISC OS development, it was common to respond to people's queries with a reference to where the answer could be found - usually somewhere public (or at the least a reference in !StrongHelp). People from Acorn had, over the years, posted various things on newsgroups that had been useful. People like Robin Watts used to keep archives of these on their site. The old support network posts on BBSs were sometimes useful, albeit very old.

Then there was the Developer area on the Acorn site, where you could get more information, and download things if you were a registered developer. In some places this was a bit scarce, but Dave Walker had been improving some bits of that support - he was someone who really made developing things fun. He was always helpful, even if he couldn't say too much because the things they were working on weren't for public consumption. He was always responsive to the bug reports, especially during the Ursula testing period - I remember emailing quite a few little comments and bug reports during that period. I'm sure he must have had lots of other people doing the same, as Ursula had been released to the entire registered developer community. I have a vague recollection that he said that he didn't get too much back from people though. Might just be my memory playing up.

The PRMs CD, with HTML and PDF versions of the PRMs was, as far as I can tell, relatively well received by people. I had, a couple of years earlier, produced !StrongHelp conversions of the !Librarian documentation (!Librarian being another DrawFile based file format!), but I don't think they went out with that CD. Which is a pity as they were really useful (and linked directly from the OS !StrongHelp manual, too <smile>). It's possible that that might have helped some people, but I don't remember if I ever did release that - or if I did what response it got.

The component change information never seemed to illicit much praise or criticism really. A few remarks, as has been said, that the changes were unclear, but not much else. I remember it being similar for the main documentation, too. There was also the whole problem that we'd document our extensions and then find out later (sometimes much later, when users complained about problems) that Castle had extended interfaces in incompatible ways, usually for Kernel operations. I think we had a couple of collisions that could have been solved very simply with a little communication, but instead we had to back off.

When I extended interfaces, I had made sure that I informed them of the details, but only on one occasion did I get a similar communication from them. Invariably emails to say that there was a collision were met with no response, so I changed the calls to move out of their way - usually using sub-reasons that hadn't been used. I'm pretty sure there's a couple in the SWI OS_ReadSysInfo and SWI Wimp_Extend that had to be modified to avoid collisions.

Making the documentation public was one way we hoped to avoid this sort of problem. If Castle were going to ignore private emails, at least if the information was public, there was something we can point to and say that what was what. There's a little bit of 'we're using it, and it's documented as such' about that, but also the fact that that's what many people were asking for - they wouldn't pay money to upgrade, but they wanted to know what changed. Sort of reasonable, and you can't fault that.

In general, though, I don't remember too many products that produce much in the way of API documentation and the like. Many of the products are fixed use, so don't really have much 'API' to them. !SparkFS did, and some of the Computer Concepts things. I will always remember fondly trying to get my head around Euclid documentation and failing.

I don't really know whether people have continued to keep the documentation up to date - I know there are PRM bits up on the ROOL website as they turn up in some searches. Maybe they have lots of bits up there that just don't turn up. I was highly amused at being asked, in the middle of 2012 if I had any way of getting information out of FrameMaker because Castle had continued using it, and now they didn't have a way of accessing the documents. Made me smile anyhow.