- Created by Robert Reiner, last modified on 29. Apr 2022
projectdoc Toolbox
Information for administrators on how to maintain projectdoc documents or to build them from scratch.
The projectdoc Toolbox extracts information from Confluence pages to form projecdoc documents. These documents are stored in index structures to ensure a quick lookup of document property information.
While the projectdoc Toolbox uses the Lucene Index to run searches, it also needs its own lookup tables as a cache to render pages with projectdoc documents quickly.
This document provides information on the use cases of the maintenance processes.
Prerequisites
The maintenance processes are available by the Rebuild Service since version 5.0 of the projectdoc Toolbox. Prior to this version the projectdoc Toolbox provided the Indexing Service to run on a different traversal strategy. The new processes of the Rebuild Service allow to usually perform more quickly.
We will use the term 'maintenance' to generally refer to either upgrade, update, rebuild, refresh, revalidate, repair, and repopulate processes. We reserve the term reindex for the content indexing with Lucene. The term indexing refers to the update processes being available prior to version 5.0.
Kinds of Maintenance
The projectdoc Toolbox knows seven kinds of maintenance processes to be performed for projectdoc documents. Two of them, update and repopulate, are currently not implemented and reserved for further versions of this service.
If everything works fine with the projectdoc Toolbox and no upgrade is required due to a version change then none of these services are required. Typically all lookup information is up-to-date. An update is processed whenever a projectdoc document is updated by a user.
Central Use Cases
For maintenance reasons an admin may need to do a refresh of the documents or to build all documents from scratch with a rebuild.
If an upgrade is required, it is part of an update of the projectdoc Toolbox app.
| Maintenance | Description |
|---|---|
| Refresh | Recalculate document properties of all projectdoc documents. Refresh is the fastest update task. Only confluence documents which are known to be projectdoc documents will be handled. |
| Rebuild | Clears the projectdoc document tables and iterates over all pages on the Confluence server. This should never be done without proper reason because it takes an enormous amount of processing time. If you think your Lucene reindex takes a lot of time, try this one. It is probably 10 times slower. |
| Upgrade | Starts the installation of a new version of the projectdoc Toolbox that includes a special kind of rebuild. Upgrade is always part of an update of the projectdoc Toolbox. |
Refresh
One use case for refresh may appear after the installation of an projectdoc Toolbox update. The new version may add new artificial properties or update the information stored in artificial properties. In this case a refresh is required to update the information for all existing projectdoc documents. New documents or existing documents that have been manually updated will provide the additional or updated document properties.
If the update of a document failed, because events did not get properly process, either due to system problems or issues with the software, a refresh may also be necessary to bring the documents back to the up-to-date state.
Rebuild
All projectdoc document information is derived from the source code of the corresponding Confluence page. So if the lookup tables are cleared this information may be recalculated.
Unfortunately this may be a very time consuming process. Not only does the process need to recalculate each projectdoc document, it may also need to recalculate a document multiple times. This is because a property value may depend one or more property values of other documents which in turn may depend on property values of other documents and so forth. The more complex the web of information is, the longer the process will take.
It is highly recommended to check this use case on a test installation with the production data in advance. This way the administrators may communicate the duration of the rebuild to the users.
Upgrade
The upgrade is part of the update of the projectdoc Toolbox
Guided by the tables of the last version, the upgrade process will populate the new tables. This is different to a rebuild that starts on completely empty tables. If the tables are empty, the process needs to traverse all pages of all spaces in its search for projectdoc documents. Since this is not the case for the upgrade, the upgrade process is typically quicker than the rebuild process.
For more information see section Upgrade Task in this document.
Special Cases
Special cases that are rarely used. If an incident happend, the usual process to run would be refresh or a rebuild. The processes revalidate and repair may be useful if other maintenance processes take advantage of the invalid flag or the version number.
| Maintenance | Description |
|---|---|
| Revalidate | Recalculates document properties for all invalid projectdoc documents. This is rarely necessary in the context of maintenance since invalid pages get revalidated on page access. |
| Repair | Recalculates document properties for all projectdoc documents with a lower version number than their Confluence pages. This should be rarely necessary in the context of maintenance because the document properties are updated on an update of the Confluence page. This process helps to find and correct information of pages where the update failed. |
Revalidate
When a document needs to be updated, the projectdoc Toolbox may decide to only mark the document as out-of-date and proceed with the current process. The update will be postponed to a later time when the updated version of the document is actually required.
In typical use cases the updated document will immediately be read for rendering to the user who has updated the Confluence page. So this is usually a use case if the Confluence document was updated programmatically.
The advantage of this process is that administrators may decide the point in time at which a larger set of invalid documents should be revalidated so that this process is not executed just-in-time when a user requests the projectdoc document.
Administrators may have a look at the invalid flag on a projectdoc document to check the amount of invalid documents.
Repair
Repair is required if a projectdoc document is not in-sync with its Confluence page. Being not in-sync is determined by checking the version information, which is an integer value. If the version number of the projectdoc document is smaller than that of the Confluence page then this projectdoc document is deemed not in-sync.
Future Use Cases
The following use cases are not yet available. The are reserved for special use case the projectdoc Toolbox may support in the future.
| Maintenance | Description |
|---|---|
| Update | Traverses the persistence structure to add new information. This is typically used to add new artificial properties, but leaves existing properties untouched. This process is currently not available and defaults to Refresh. |
| Repopulate | Used if the source for document information is provided externally. This process is currently not available and reserved as a term for future versions of the projectdoc Toolbox. |
Kinds of Iteration
The duration of an update process is also dependent on how the documents to process are found.
As long as the projectdoc documents are all in tables, the process may simply iterate over the document table and update each document accordingly.
If projectdoc documents are missing, then the Confluence pages must be traversed, which will take more time.
Administrators may decide to restrict the search for candidates to spaces. This is helpful for use cases where Administrators know in which spaces the projectdoc Toolbox is actually used.
Services may also be instructed to search only in spaces with a projectdoc homepage. This is helpful if users will either use projectdoc documents in a space or not and if they use them, all Confluence pages are projectdoc documents.
Admin User Interface
The Admin User Interface is a simplified interface to cover the typical use cases with preconfigured parameters.
If background processes start one of the update services, then this will be shown in the UI. Administrators may also access the underlying long-running task via the Confluence REST interface.
The UI is part of the General Configuration screen of Confluence for Administrators: Rebuild & Refresh
This UI provides access to two use case: Refresh and Rebuild.
In case administrators need to control the parameters, they need to use the underlying web services via their REST interface directly. The UI uses the Site Resource of the Rebuild Service.
Web Service Interface
The Rebuild Service is available via REST services.
All services are located under the following URL:
projectdoc-internal/1/rebuilder/
Administrator privileges are required to run these services.
This is an internal API for the projectdoc Toolbox and subject to change without notice.
Rebuilder Services
The rebuilder services provide a REST interface to the services. The same services are used by the administration UI, but the REST interface provides more options.
The services answer always with a 202. This indicates that the request has been accepted for processing, but the processing has not been completed. The body contains links to the long running tasks that have been started.
All services support POST.
| Service | Description |
|---|---|
site | Rebuild projectdoc documents infrastructure for the whole site. Parameters
|
projectdoc-spaces | Rebuild projectdoc documents infrastructure for spaces with at least one projectdoc document. All other spaces are skipped. Parameters
|
spaces | Processes the projectdoc documents for the specified spaces. At least one space must be specified. Note that the order of the specified spaces keys takes only a minor significance: space keys are processed first, then the sets determined by their space categories. If a strict order is requested, please run this service for each set separately. Note that exclusions are removed from the set of space keys calculated. Therefore a space explicitly specified is removed if its space category matches with one that is excluded. Parameters
|
Tooling Services
The following services allow to access lists of documents or analyze spaces. Information retrieved from these services support use cases with the rebuilder services where a defined set of pages or spaces should be processed.
All services support GET.
| Service | Description |
|---|---|
space-key-list | Allows to fetch the list of space keys to traverse for a rebuild. Only spaces that are part of the provided list are processed. It won't add delegate spaces. Typically this parameter is not provided and then will return the order based on all spaces in the site. Parameters
|
document-id-list | Allows to fetch the list of document identifiers to traverse for a rebuild. Parameters
|
taskId | Allows to find the task identifier to check the current status of the rebuilder task. Only one task must execute at any given time. The returned task may already have been completed. Clients need to check the status separately with the Long Running Task API. This service returns the URL to the long running task if there is one. Otherwise the HTTP status code of the response is 404. Parameters
|
test-space-specification | Allows to test the resolution of a space specification. This service may be used to select the space keys to pass to one of the rebuilder services explicitly. Parameters
|
Use the Atlassian REST API Browser to access the documentation for these services.
Indexer Service is deprecated
The old Indexer Service of version 4 is still available (see Troubleshooting Reindexer for projectdoc Documents), but considered deprecated.
Upgrade Task
Upgrade tasks apply changes to a projectdoc Toolbox installation, especially to the Active Objects (AO) database tables. These changes are necessary to improve the execution quality of projectdoc tools.
The upgrade task is applied automatically.
Automatic Execution of Upgrade
The system property de.smartics.projectdoc.upgrade.skip-auto allows to control the execution of the upgrade task. If this property is set to "true" then upgrade is not automatic. Members of the administration group would need to run the upgrade manually.
Per default this property is not set (same as a value of "false").
From pre 5 to 5
The upgrade task "AO Table Migration 4->5" migrates the AO tables from its old structure version 4 to the new structure version 5. The name of the tables reflect this by adding a number after the initial "PD" (and the initial identifier of the plugin).
The description of the task is "Migrates the AO tables of version 4 of the projectdoc Toolbox to version 5."
The old tables (identified by "PD_" in the table name) will stay untouched and the new structures will be stored in new tables (identified by "PD5_"). Even if the old data will still be available we highly recommend to have access to a current backup of your database.
PD5 Tables
AO_63F547_PD5_DOCAO_63F547_PD5_NAME_REFSAO_63F547_PD5_PAGE_REFSAO_63F547_PD5_PG-FRGMNTAO_63F547_PD5_PROP
When the automatic upgrade task starts it will check that the correct versions of additional projectdoc add-ons are installed. Only installed add-ons with older versions will be reported. The upgrade task will start once all add-ons are updated. You may disable the add-on if it is no longer used on your projectdoc pages.
Resources
More information on this topic is available by the following resources.
- Cache Refresh Actions
- projectdoc documents are stored in caches for quick access. Users may want to clear one of these caches.
- Troubleshooting Reindexer for projectdoc Documents
- In order to completely restore the projectdoc document index, not only must the Lucene index be re-created and the projectdoc document cache cleared, but the internal projectdoc structures must also be rebuilt.
