projectdoc Rebuild Manual

...

Document Properties Marker

doctype	topic
override	false

Short Description

Information for administrators on how to rebuild maintain projectdoc documents or to build them from scratch or start a reindex process.

Name

projectdoc Rebuild and ReindexManual

hide

Short Name

hide

Parent

Parent Property

property-name	Name

hide

Audience

Name List

doctype	role
render-no-hits-as-blank	true
property-restrict-value-range	true
property	Audience
empty-as-none	true

Subject

Name List

doctype	subject
property	Subject

Sponsors

Name List

doctype	stakeholder,organization,person,role
render-no-hits-as-blank	true
property	Sponsors

Sort Key

hide

...

Section

show-title	false
title	Description

The projectdoc Toolbox extracts information from Confluence pages to form projecdoc documents. These documents are stored in index structures to ensure a quick lookup of document property information.

While the projectdoc Toolbox uses the Lucene Index to make lookupsrun searches, it also needs its own lookup tables to be up-to-dateas a cache to render pages with projectdoc documents quickly.

This document provides information on the use cases of the rebuild and refresh maintenance processes.

Section

title	Summary

...

Section

title	Prerequisites

The rebuild maintenance processes are available by the Rebuild Service since version 5.0 of the projectdoc Toolbox. Prior to this version the projectdoc Toolbox provided indexing services that the Indexing Service to run on a different traversal strategy. The new processes of the Rebuild Service allow to usually perform more quickly.

We will use the term 'updatemaintenance' to generally refer to either reindex upgrade, update, rebuild, refresh, revalidate, repair, and repair repopulate processes. We reserve the term reindex for the content indexing with Lucene. The term indexing refers to the update processes being available prior to version 5.0. The terms reindex, rebuild, refresh, revalidate, and repair are explained in this document.

Recalculates document properties for all invalid projectdoc documents.

This is rarely necessary since invalid pages get revalidated on page access

Section

title	Kinds of UpdatesMaintenance

The projectdoc Toolbox knows four seven kinds of updates maintenance processes to be performed for projectdoc documents. Two of them, update and repopulate, are currently not implemented and reserved for further versions of this service.

If everything works fine with the projectdoc Toolbox and no upgrade is required due to a version change then none of these services are required. Typically all lookup information is up-to-date. An update is processed whenever a projectdoc document is updated by a user.

Section

title	Central Use Cases

For maintenance reasons an admin may need to do a refresh of the documents or to build all documents from scratch with a rebuild.

If an upgrade is required, it is part of an update of the projectdoc Toolbox app.

Update

Maintenance

Description

Refresh

Recalculate document properties of all projectdoc documents.

Refresh is the fastest update task. Only confluence documents which are known to be projectdoc documents will be handled

.

Revalidate

.

Repair

Recalculates document properties for all projectdoc documents with a lower version number than their Confluence pages.

This should be rarely necessary because the document properties are updated on an update of the Confluence page. This process helps to find and correct information of pages where the update failed.

Rebuild

Clears the projectdoc document tables and iterates over all pages on the Confluence server.

This should never be done without proper reason because it takes an enormous amount of processing time. If you think your Lucene reindex takes a lot of time, try this one. It is probably 10 times slower.

If everything works fine with

Upgrade

Starts the installation of a new version of the projectdoc Toolbox

none of these services are required. Typically all lookup information is up-to-date. An update is made whenever a projectdoc document is updated by a user

that includes a special kind of rebuild.

Upgrade is always part of an update of the projectdoc Toolbox.

Section

title	Refresh

One use case for refresh may appear after the installation of an projectdoc Toolbox update. The new version may add new artificial properties or update the information stored in artificial properties. In this case a refresh is required to update the information for all existing projectdoc documents. New documents or existing documents that have been manually updated will provide the additional or updated document properties.

If the update of a document failed, because events did not get properly process, either due to system problems or issues with the software, a refresh may also be necessary to bring the documents back to the up-to-date state.

Section

title	Rebuild

All projectdoc document information is derived from the source code of the corresponding Confluence page. So if the lookup tables are cleared this information may be recalculated.

Unfortunately this may be a very time consuming process. Not only does the process need to recalculate each projectdoc document, it may also need to recalculate a document multiple times. This is because a property value may depend one or more property values of other documents which in turn may depend on property values of other documents and so forth. The more complex the web of information is, the longer the process will take.

It is highly recommended to check this use case on a test installation with the production data in advance. This way the administrators may communicate the duration of the rebuild to the users.

Section

title	Upgrade

The upgrade is part of the update of the projectdoc Toolbox

Guided by the tables of the last version, the upgrade process will populate the new tables. This is different to a rebuild that starts on completely empty tables. If the tables are empty, the process needs to traverse all pages of all spaces in its search for projectdoc documents. Since this is not the case for the upgrade, the upgrade process is typically quicker than the rebuild process.

For more information see section

In-Document Link

anchor	Upgrade Task

in this document.

Section

title	Special Cases

Special cases that are rarely used. If an incident happend, the usual process to run would be refresh or a rebuild. The processes revalidate and repair may be useful if other maintenance processes take advantage of the invalid flag or the version number.

Maintenance

Description

Revalidate

Recalculates document properties for all invalid projectdoc documents.

This is rarely necessary in the context of maintenance since invalid pages get revalidated on page access.

Repair

Recalculates document properties for all projectdoc documents with a lower version number than their Confluence pages.

This should be rarely necessary in the context of maintenance because the document properties are updated on an update of the Confluence page. This process helps to find and correct information of pages where the update failed.

Section

title	Revalidate

When a document needs to be updated, the projectdoc Toolbox may decide to only mark the document as out-of-date and proceed with the current process. The update will be postponed to a later time when the updated version of the document is actually required.

In typical use cases the updated document will immediately be read for rendering to the user who has updated the Confluence page. So this is usually a use case if the Confluence document was updated programmatically.

The advantage of this process is that administrators may decide the point in time at which a larger set of invalid documents should be revalidated so that this process is not executed just-in-time when a user requests the projectdoc document.

Administrators may have a look at the invalid flag on a projectdoc document to check the amount of invalid documents.

Section

title	Repair

Repair is required if a projectdoc document is not in-sync with its Confluence page. Being not in-sync is determined by checking the version information, which is an integer value. If the version number of the projectdoc document is smaller than that of the Confluence page then this projectdoc document is deemed not in-sync.

Section

title	Rebuild	Future Use Cases

The following use cases are not yet available. The are reserved for special use case the projectdoc Toolbox may support in the future.

Maintenance

Description

Update

Traverses the persistence structure to add new information. This is typically used to add new artificial properties, but leaves existing properties untouched.

This process is currently not available and defaults to Refresh.

Repopulate

Used if the source for document information is provided externally.

This process is currently not available and reserved as a term for future versions of the projectdoc Toolbox

All projectdoc document information is derived from the source code of the corresponding Confluence page. So if the lookup tables are cleared this information may be recalculated.

Unfortunately this may be a very time consuming process. Not only does the process need to recalculate each projectdoc document, it may also need to recalculate a document multiple times. This is because a property value may depend one or more property values of other documents which in turn may depend on property values of orther documents and so forth. The more complex the web of information is, the longer the process will take.

It is highly recommended to check this use case on a test installation with the production data in advance. This way the administrators may communicate the duration of the rebuild to the users

.

Section

title	Kinds of Iteration

The duration of an update process is also dependent on how the documents to process are found.

As long as the projectdoc documents are all in the tables, the process may simply iterate over the document table and update each document accordingly.

If projectdoc documents are missing, then the Confluence pages must be traversed, which will take more time.

Administrators may decide to restrict the search for candidates to process by spaces. This is helpful for use cases where Administrators know in which spaces the projectdoc Toolbox is actually used.

Services may also be instructed to search only in spaces with a projectdoc homepage. This is helpful if users will either use projectdoc documents in a space or not and if they use them, all Confluence pages are projectdoc documents.

...

Section

title	Admin User Interface

The Admin User Interface is a simplified interface to cover the typical use cases with preconfigured parameters.

If background processes start one of the update services, then this will be shown in the UI. Administrators may also access the underlying long-running task via the Confluence REST interface.

The UI is part of the General Configuration screen of Confluence for Administrators: Rebuild & Refresh

Image RemovedImage Added

This UI provides access to all four two use case: Refresh , Revalidate, Repair, and Rebuild.

Image Added

In case administrators need to control the parameters, they need to use the underlying web services via their REST interface directly. The UI uses the Site Resource of the Rebuild Service.

Rebuild

The Rebuild Service is available via REST services.

All services are located under the following URL:

projectdoc-internal

-1-rebuilder-migrate

projectdoc-internal-1-rebuilder-projectdoc-spaces

projectdoc-internal-1-rebuilder-rebuild

projectdoc-internal-1-rebuilder-refresh

projectdoc-internal-1-rebuilder-refresh-spaces

projectdoc-internal-1-rebuilder-repair

projectdoc-internal-1-rebuilder-repair-spaces

projectdoc-internal-1-rebuilder-revalidate

projectdoc-internal-1-rebuilder-site

projectdoc-internal-1-rebuilder-spaces

projectdoc-internal-1-rebuilder-taskid

Section

title	Web Service Interface

Pending Box

Section

title	Related Tools

/1/rebuilder/

Administrator privileges are required to run these services.

This is an internal API for the projectdoc Toolbox and subject to change without notice.

Section

title	Rebuilder Services

The rebuilder services provide a REST interface to the services. The same services are used by the administration UI, but the REST interface provides more options.

The services answer always with a 202. This indicates that the request has been accepted for processing, but the processing has not been completed. The body contains links to the long running tasks that have been started.

All services support POST.

Service Description

site

Rebuild projectdoc documents infrastructure for the whole site.

Parameters

mode - Controls the mode in which the rebuilder is executed. Supported modes are upgrade, update, rebuild, refresh, revalidate, repair, and repopulate.
name - The optional name of the task to allow users to identify this instance easily. Must not be longer than 32 characters. The user is responsible to select a name that is appropriately unique.

projectdoc-spaces

Rebuild projectdoc documents infrastructure for spaces with at least one projectdoc document. All other spaces are skipped.

Parameters

mode - Controls the mode in which the rebuilder is executed. Supported modes are upgrade, update, rebuild, refresh, revalidate, repair, and repopulate.
name - The optional name of the task to allow users to identify this instance easily. Must not be longer than 32 characters. The user is responsible to select a name that is appropriately unique.

spaces

Processes the projectdoc documents for the specified spaces.

At least one space must be specified. Note that the order of the specified spaces keys takes only a minor significance: space keys are processed first, then the sets determined by their space categories. If a strict order is requested, please run this service for each set separately. Note that exclusions are removed from the set of space keys calculated. Therefore a space explicitly specified is removed if its space category matches with one that is excluded.

Parameters

space-keys - The keys of the spaces to rebuild. Space categories are allowed (#, for instance #project) and spaced may be excluded (!, for instance !#project, see
Static Document Link
document Search Space
for details on the space key specification).
mode - Controls the mode in which the rebuilder is executed. Supported modes are upgrade, update, rebuild, refresh, revalidate, repair, and repopulate.
name - The optional name of the task to allow users to identify this instance easily. Must not be longer than 32 characters. The user is responsible to select a name that is appropriately unique.

Section

title	Tooling Services

The following services allow to access lists of documents or analyze spaces. Information retrieved from these services support use cases with the rebuilder services where a defined set of pages or spaces should be processed.

All services support GET.

Service Description

space-key-list

Allows to fetch the list of space keys to traverse for a rebuild.

Only spaces that are part of the provided list are processed. It won't add

Static Document Link

document	PDAC:Delegate Space
label	delegate spaces

. Typically this parameter is not provided and then will return the order based on all spaces in the site.

Parameters

space-keys - The keys of the spaces to traverse. If not specified or empty then all spaces of the site are traversed (this is the main use case). Space categories are allowed (#, for instance #project) and spaced may be excluded (!, for instance !#project, see
Static Document Link
document Search Space
for details on the space key specification).
verbose - The flag allows to print the space order descriptor (true) instead of the space key (false, default).

document-id-list

Allows to fetch the list of document identifiers to traverse for a rebuild.

Parameters

mode - The mode in which the traversal should take place. The following modes are valid:
- hierarchy (default): traverse the Confluence page hierarchy.
- ao-hierarchy: Traverse the pages found in the projectdoc document table, ordered by level.
- ao-spaced: Traverse the pages found in the projectdoc document table, ordered by level within spaces.
- upgrade4-5: Traverse the pages found according to the upgrade from version 4 to version 5.
space-keys - The keys of the spaces to traverse. If not specified or empty then all spaces of the site are traversed. Space categories are allowed (#, for instance #project) and spaced may be excluded (!, for instance !#project, see
Static Document Link
document Search Space
for details on the space key specification).
verbose - If set to true then not only the page identifiers are written, but also the space key and title of the pages. If not specified or false (default), then only the page identifiers are rendered. Each page is always on a new line.

taskId

Allows to find the task identifier to check the current status of the rebuilder task. Only one task must execute at any given time. The returned task may already have been completed. Clients need to check the status separately with the Long Running Task API.

This service returns the URL to the long running task if there is one. Otherwise the HTTP status code of the response is 404.

Parameters

name - The optional name of the task to check its current status. Must not be longer than 32 characters. If not specified, the current task is returned regardless of its name.

test-space-specification

Allows to test the resolution of a space specification.

This service may be used to select the space keys to pass to one of the rebuilder services explicitly.

Parameters

space-keys - The keys of the spaces for the test. Space categories are allowed (#, for instance #project) and spaced may be excluded (!, for instance !#project, see
Static Document Link
document Search Space
for details on the space key specification).

Use the

Static Document Link

document	Atlassian REST API Browser

to access the documentation for these services.

Deprecated Box

title	Indexer Service is deprecated

The old Indexer Service of version 4 is still available (see

Static Document Link

document	Troubleshooting Reindexer for projectdoc Documents

), but considered deprecated.

Section

title	Upgrade Task

Upgrade tasks apply changes to a projectdoc Toolbox installation, especially to the Active Objects (AO) database tables. These changes are necessary to improve the execution quality of projectdoc tools.

The upgrade task is applied automatically.

Note Box

title	Automatic Execution of Upgrade

The system property de.smartics.projectdoc.upgrade.skip-auto allows to control the execution of the upgrade task. If this property is set to "true" then upgrade is not automatic. Members of the administration group would need to run the upgrade manually.

Per default this property is not set (same as a value of "false").

Section

title	From pre 5 to 5

The upgrade task "AO Table Migration 4->5" migrates the AO tables from its old structure version 4 to the new structure version 5. The name of the tables reflect this by adding a number after the initial "PD" (and the initial identifier of the plugin).

The description of the task is "Migrates the AO tables of version 4 of the projectdoc Toolbox to version 5."

The old tables (identified by "PD_" in the table name) will stay untouched and the new structures will be stored in new tables (identified by "PD5_"). Even if the old data will still be available we highly recommend to have access to a current backup of your database.

Example Box

title	PD5 Tables

AO_63F547_PD5_DOC
AO_63F547_PD5_NAME_REFS
AO_63F547_PD5_PAGE_REFS
AO_63F547_PD5_PG-FRGMNT
AO_63F547_PD5_PROP

When the automatic upgrade task starts it will check that the correct versions of additional projectdoc add-ons are installed. Only installed add-ons with older versions will be reported. The upgrade task will start once all add-ons are updated. You may disable the add-on if it is no longer used on your projectdoc pages.

projectdoc-

Pending Box

Indexer REST services

projectdoc-internal-1-indexer-find
projectdoc-internal-1-indexer-pages
projectdoc-internal-1-indexer-projectdoc-spaces
projectdoc-internal-1-indexer-site
projectdoc-internal-1-indexer-spaces
see Troubleshooting Reindexer for projectdoc Documents

Lucene Service

projectdoc-internal-1-lucene-mode

User UI Actions

Cache Refresh Actions

Automatic Upgrade

...

Space shortcuts

Page tree

Docs

Support

Free Blueprints

Tooling for Blueprints

Free Extensions

Spaces

Plugins for Maven

Incubator

Libraries for Java

Incubator

Add-ons for Confluence

Tools

Support

Team

Legal

Versions Compared

Old Version 4

New Version Current

Key

About

Keep up-to-date!

Powered by