Part two (of our three-part series explaining how we used our CMS for technical documentation) focuses on parsing code for the JS and Python client reference materials.
For our technical documentation migration from GitBook to TerminusCMS we needed to parse code documentation as well static content. In our use case, we needed a solution for the technical documentation for the Python and JavaScript clients that have documentation that is automatically generated when a new version comes out.
Creating a Schema for the Code
Since we needed to parse both the Python and JavaScript documentation, a generic schema was needed to accommodate both. One option was to create separate data products with independent schemas for each, but we chose a more abstract approach: to make a schema that could accommodate multiple languages.
Here’s the schema we used:
[
{
"@base":"terminusdb:///documentation/data/",
"@schema":"terminusdb:///documentation/schema#",
"@type":"@context"
},
{
"@id":"Returns",
"@inherits":"Documented",
"@type":"Class",
"type":"xsd:string"
},
{
"@abstract":[
],
"@id":"Documented",
"@inherits":[
"Named",
"Summarized"
],
"@type":"Class"
},
{
"@id":"Parameter",
"@inherits":"Documented",
"@type":"Class",
"default":{
"@class":"xsd:string",
"@type":"Optional"
},
"type":"xsd:string"
},
{
"@abstract":[
],
"@id":"Summarized",
"@type":"Class",
"summary":{
"@class":"xsd:string",
"@type":"Optional"
}
},
{
"@id":"Definition",
"@inherits":"Documented",
"@oneOf":[
{
"parameters":{
"@class":"Parameter",
"@type":"List"
},
"receives":{
"@class":"Parameter",
"@type":"List"
}
},
{
"returns":"Returns",
"returns_multiple":{
"@class":"Returns",
"@type":"List"
},
"yields":"Returns"
}
],
"@type":"Class",
"examples":{
"@class":"xsd:string",
"@dimensions":1,
"@type":"Array"
},
"extendedSummary":{
"@class":"xsd:string",
"@type":"Optional"
},
"index":{
"@class":"xsd:integer",
"@type":"Optional"
},
"notes":{
"@class":"xsd:string",
"@type":"Optional"
},
"raises":{
"@class":"Exception",
"@type":"Set"
},
"references":{
"@class":"xsd:string",
"@type":"Optional"
},
"section":{
"@class":"xsd:string",
"@type":"Optional"
},
"seeAlso":{
"@class":"Definition",
"@type":"Set"
},
"signature":{
"@class":"xsd:string",
"@type":"Optional"
}
},
{
"@id":"Language",
"@type":"Enum",
"@value":[
"Python",
"Javascript"
]
},
{
"@id":"Module",
"@inherits":"Documented",
"@type":"Class",
"classes":{
"@class":"Class",
"@type":"Set"
},
"definitions":{
"@class":"Definition",
"@type":"Set"
}
},
{
"@id":"Class",
"@inherits":"Documented",
"@type":"Class",
"memberFunctions":{
"@class":"Definition",
"@type":"Set"
},
"memberVariables":{
"@class":"Parameter",
"@type":"Set"
},
"name":"xsd:string"
},
{
"@abstract":[
],
"@id":"Named",
"@type":"Class",
"name":"xsd:string"
},
{
"@id":"Exception",
"@inherits":"Documented",
"@type":"Class",
"parameters":{
"@class":"xsd:string",
"@type":"List"
}
},
{
"@id":"Application",
"@inherits":"Documented",
"@type":"Class",
"language":"Language",
"license":"xsd:string",
"modules":{
"@class":"Module",
"@type":"Set"
},
"version":"xsd:string"
}
]
It contains everything we need to parse the JavaScript and Python client documentation. The Application class contains the name of the application and the language. It has a module set that contains the different modules.
In the Python example, this would be terminusdb_client for instance. A lot of the different classes in the schema inherit from Documented, which gives it the Named and Summarized properties, as most of the code documentation properties have a name and summary.
Each class object contains a set of memberFunctions and memberVariables. The memberFunctions are from the class Definition and are functions in Python and JavaScript. The functions contain a list of parameters and a return value. It also inherits the Documented class which gives it a summary and a name attribute. Some languages like Go allow for multiple return values, therefore the return values can also be a list in the returns_multiple property.
We inserted the schema using the TerminusCMS JSON schema editor. The following sections will show how to parse the JS and Python code to insert the code documentation data.
Parsing JavaScript Code
In order to render the JavaScript documentation, we first have to parse it. To do this, we created a custom JSDoc template – https://github.com/terminusdb-labs/jsdoc-terminusdb-template.
It currently only works with terminusdb-client-js as it required customization. For instance, we needed separate JSON in which we save the documentation sections. We run a GitHub Action each time we push a new version of the client. In that action, we run:
npx jsdoc --template ../jsdoc-terminusdb-template --recurse ./lib/ > docs.json
Which creates the appropriate document in a JSON file. Afterwards we push it to TerminusCMS using cURL.
curl -H 'Authorization: Token ${{ secrets.TERMINUSCMS_TOKEN}}' \
-H 'Content-Type: application/json' \
'${{ secrets.TERMINUSCMS_URL}}' -d @./docs.json
This uploads the new version of the JSDocs in TerminusCMS.
Parsing Python Code
Our Python client (terminusdb-client-python) uses the NumPy style convention for code documentation.
To parse the Python code we had to use the numpydoc library. We made a simple Python script that imports the TerminusDB client module, iterates through all the classes we want to document and then transforms it to the JSON document we need.
Just like we did with the JavaScript client, we have a CI process in GitHub Actions that uploads the document after the code documentation has been parsed and transformed. After running the script:
poetry run python .github/insert_docs.py > docs.json
It will upload it with cURL, just as we did in the JavaScript client:
curl -H 'Authorization: Token ${{ secrets.TERMINUSCMS_TOKEN}}' \
-H 'Content-Type: application/json' \
'${{ secrets.TERMINUSCMS_URL}}' -d @./docs.json
Conclusion and Future Possibilities
In the end, we have beautiful parsed code documentation in the TerminusCMS database. We use it to generate static documentation for our NextJS documentation website. But there are also future possibilities. A graph-like structure for code could be used for static analysis and linting as well. Dogfooding your own product is a surefire way to generate ideas for the future.
The next and final part of the tech doc series looks at the NextJS build and how we made it super fast.
