Configuring Insights

This chapter discusses the following configuration tasks:

Configuring your PostgreSQL database

By default, an installation of PostgreSQL usually requires a valid operating system account to connect to a database. Because the Insights database connection comes from a Tomcat web application, Insights cannot access the database using a system account, but must instead connect using username/password authentication. If this has not been enabled already, you must configure the database to enable it.

If you are running Ubuntu and your installation of PostgreSQL is on the same machine as Insights, you can skip this configuration. For all other configurations (Ubuntu with PostgreSQL on a remote host; Red Hat or CentOS, whether PostgreSQL is local or remote), you must ensure that the database can be accessed with a username and password. In addition, if you do not want to provide remote administrator access to the database, you must create the role and database manually.

Important

The PostgreSQL database must be configured to use UTF-8. For information about character set configuration in PostgreSQL, see http://www.postgresql.org/docs/9.0/static/multibyte.html

Enabling remote access and user/password authentication

If you are running Insights and PostgreSQL on different machines, you must enable user/password authentication for remote access to the PostgreSQL database. You must also enable user/password authentication when PostgreSQL is local if you are running Red Hat or CentOS.

For more information about setting up PostgreSQL authentication, see http://www.postgresql.org/docs/8.4/static/auth-methods.html.

To enable remote access and password authentication:

  1. If you are running Insights and PostgreSQL on different machines, configure the database to allow access from remote clients.

    For CentOS or Red Hat, edit the following file:

    /var/lib/pgsql/data/postgresql.conf

    For Ubuntu, edit the following file (Ubuntu 12.04 uses PostgreSQL 9.1):

    /etc/postgresql/9.1/main/postgresql.conf

    Configure listen_addresses to allow access from the Insights application server. The following configuration example opens up the database to clients from any host, but you may want to restrict the configuration to a narrower definition:

    listen_addresses = '*'

  2. Enable password authentication.

    For CentOS or Red Hat, edit the following file:

    /var/lib/pgsql/data/pg_hba.conf

    For Ubuntu, edit the following file (Ubuntu 12.04 uses PostgreSQL 9.1):

    /etc/postgresql/9.1/main/pg_hba.conf

    At the bottom of this file are a number of entries ending with ident (in PostgreSQL 9.1, these entries may end with peer). The default PostgreSQL configuration looks something like this:

    # TYPE  DATABASE    USER       CIDR-ADDRESS          METHOD
    # "local" is for Unix domain socket connections only
    local   all        all                               ident
    # IPv4 local connections:
    host    all         all        127.0.0.1/32          ident
    # IPv6 local connections:
    host    all         all        ::1/128               ident

    The simplest solution is to change each instance of ident (or peer) to md5.

    You may also want to open up the access rules for one or both of the host lines (depending on whether your network infrastructure uses IPv4 or IPv6). For example, to allow password access over all network types and to open up host access from all IPv4 addresses, you can use a configuration like the following:

    # TYPE  DATABASE    USER       CIDR-ADDRESS          METHOD
    # "local" is for Unix domain socket connections only
    local   all         all                              md5
    # IPv4 local connections:
    host    all         all        0.0.0.0/0             md5
    # IPv6 local connections:
    host    all         all        ::1/128               md5

    If you want to limit access further, consult the PostgreSQL documentation.

    Note that by default, the postgres user does not have a password set, and you cannot log in as this user with a password until you have enabled password authentication. Until you do, you can still log in by using the postgres user account:

    sudo -u postgres psql

    The configure-insights.sh script uses this technique if no database admin password is provided when you are configuring the installation.

Creating the database and role manually

If the database host is remote and you do not want to give remote administrator access to the database, then the database administrator must create the role and database manually:

  1. Create a role and give it a password.

    Insights expects the role name insights by default, but you can use any role name you like and tell Insights during configuration.

  2. Create a database.

    Insights expects the name insightsdb with owner insights, but you can use any name you like and tell Insights during configuration.

  3. Run the installed SQL files against the new database.

    The perforce-insights Deb and RPM package installs the SQL files into /opt/perforce/usr/share/insights/db. If you are installing manually from the .zip or .tgz file, the SQL files are in the sql directory.

    1. These files create tables with owner insights. Rename the script owner if you created a different role name.

    2. Run the create_pa_tables.sql script

    3. Run the create_template_user.sql script

When you run the configure-insights.sh script on the Insights application server, you will be asked if you want to create a database. Say no and enter the role name and password you set up here.

Testing database access

To test access to the database, run the following:

psql -W -U insights -h localhost -d
          insightsdb -c "SELECT 1"

Replace insights and insightsdb with your role and database name. Replace localhost with the hostname of the server if the database is remote. This command should return a single row, if access is enabled for the user insights against the database insightsdb.

Configuring Insights using the online administrative interface

Insights provides an administrative interface that you can use to import a Perforce database checkpoint, update configuration options, set up projects, and manage users.

  1. Log in to http://hostname:8080/insights.

    The first time you log in, you can use insightsadmin with a password of P4insightsadmin.

    This is the default Insights administrative account, which allows you to configure the system.

  2. Go to the drop-down menu on the top right.

    There are options for Setup, Projects, Import, and Users. Each of these options is described below.

Import page: importing a Perforce checkpoint and enabling continuous import

The first time you install and set up Insights, you must import a Perforce database checkpoint.

  1. Obtain a recent checkpoint from your Perforce Server, or create a new one.

    If you do not know how to create a checkpoint, see the Perforce System Administrator's Guide, "Supporting Perforce: Backup and Recovery."

  2. Upload the checkpoint to the Insights server.

    Go the Insights web interface, log in as an Insights administrator (the default is insightsadmin/P4insightsadmin), and select Import from the drop-down menu on the top right.

    On the Import page, browse for the checkpoint file, select it, and click Upload Checkpoint.

    Note

    Alternatively, you can copy the checkpoint file directly to the Insights server using a mechanism like ftp, scp, or network shares. You must copy the file to the directory defined in the configuration option insights.upload.path. The default value for this option depends on your operating system. For more information about setting configuration options, see Setup page: setting configuration options.

  3. On the Import page, select the checkpoint file from the Checkpoint Filename dropdown.

    Click Import.

    Verify that the checkpoint character set encoding is correct. If your Perforce service runs in Unicode-mode, Insights detects the character set and you can ignore this setting.

    Click Import.

  4. Enable continuous import.

    Continuous import reads changes from the Perforce service as they happen in real time, keeping your Insights data up to date.

    Insights needs to know where to stream changes from, so before this can happen you should configure the insights.p4port, insights.user and insights.password configuration options on the Setup page (see below).

    The Import process reads any changes to the Perforce database that occur between the time you generate the checkpoint and the point at which continuous import begins, as long as there are journals available for the period.

    Important

    Do not use the -z option with p4d -jc or p4d -jj to compress checkpoints and journals. Instead use the -Z option so that journals remain uncompressed. Insights uses the p4 export command to extract data from the Perforce database journal. In some cases the journal can be rotated before p4 export has finished extracting the data. It is important therefore that any rotated journal is left uncompressed to enable p4 export to continue.

    The location of the rotated journal must be known to the Perforce service. The p4 export command expects the journal to be in the P4D server root (P4ROOT), unless you specify a directory with the Perforce Server configurable journalPrefix. If your Perforce Server configuration stores journals outside of the P4ROOT directory (which is recommended practice), then you must use the journalPrefix server configurable. If your journals are being moved to a different location after being rotated, or if a journal prefix is being specificed on the command line, p4 export will not be able to find them.

    For more information about configuring journal compression, see the Perforce System Administrator's Guide, "Supporting Perforce: Backup and Recovery".

The Import page also displays current import status, with the progress bar showing how far through the checkpoint import the process has gone, along with the journal number and offset reached. You can enable more detailed DBImport logging using insight-config.properties. For more information, see Setting up logging.

Setup page: setting configuration options

The Setup page displays a list of all of the configuration options for Insights. Not all of them are described here - most can be left unchanged.

  • To edit an option, click the option name and select Edit.

  • To delete an option, click the option name and select Delete.

  • To create a new option, click New Configuration.

Note

Initially, the values created in pa_insights_configuration are based on properties (for example /opt/perforce/usr/share/insights/insights-config.properties). Once a value is stored in the database on startup, modifying it in the properties file has no effect, because the application uses the values in the database table. Therefore, after initial startup you should use the Setup page to add, update, or remove configuration values.

Perforce Server options

You must define the Perforce Server instance that will be used for login authentication and continuous import. Until you set the Perforce Server information, non-administrative users will be unable to log into Insights using their Perforce user name and password.

Important

You must restart the Tomcat application server after changing any of these options.

  • insights.p4port: P4PORT setting for the Perforce server, such as ssl:myserver:1666.

  • insights.user: The username used to authenticate against the Perforce Server. This must be a super user. We recommend that it be a service user.

  • insights.password: the password or ticket for the Perforce user. For security reasons, we recommend that you define a long-lived ticket for this user and enter the ticket value here rather than the password.

    Instead of entering the ticket value, you can place a ticket file in the home directory of the account running Tomcat. Typically for Tomcat 6 on Linux, this is /usr/share/tomcat6. The ticket file is called .p4tickets and is created in your home directory when you log into the Perforce service.

    For more information about creating long-lived tickets for service users, see the section on service users in Distributing Perforce, "Perforce Replication" (http://www.perforce.com/perforce/doc.current/manuals/p4dist/chapter.replication.html#replication.serviceusers) .

  • insights.psm.baseURL.main: If you are using the optional Perforce Service Manager (PSM) component, you must also set the URL of the PSM web service, using the format http://hostname:port/api.

    For more information, see Setting up Perforce Service Manager

Jobs options

If you are using Perforce jobs, you must specify how to interpret the Perforce jobspec. Since a jobspec can vary widely in how it is structured, and there are no limitations on how the fields are used, you must configure the names of the job fields and their possible values. If you do not have experience with Perforce job views, see the Perforce System Administrator's Guide, "Customizing Perforce: Job Specifications".

  • insights.jobs.bugs.field.name: The name of the jobspec field that defines the type of job (for example, bug, defect, task, story). Defaults to 'Type'.

  • insights.jobs.bugs.field.value: The value of the Type field named above that defines a job as being a bug. Defaults to be 'BUG'.

  • insights.jobs.bugs.severity.name: The name of the jobspec field that defines the severity (or importance) of a bug. Defaults to 'Severity'.

  • insights.jobs.bugs.status.name: The name of the jobspec field that defines the status of a job (e.g. open, closed, duplicate). Defaults to 'Status'.

  • insights.jobs.bugs.status.open.value: Values for the 'Status' field that should be interpreted as open. This can contain multiple comma separated values. Defaults to 'open'.

  • insights.jobs.bugs.status.fixed.value: Values for the 'Status' field that should be interpreted as closed. This can contain multiple comma separated values. Defaults to 'fixed,closed,punted,obsolete,duplicate'.

  • insights.jobs.bugs.date.reported: The name of the jobspec field that holds the date a job was first opened. Defaults to 'ReportedDate'.

  • insights.jobs.bugs.date.modified: The name of the jobspec field that holds the date a job was last modified. Defaults to 'ModifiedDate'.

The following properties enable you to specify how job severities are displayed in the graphs. By default, there are three severities configured, and each has two settings. These are:

  • insights.jobs.bugs.values.A.weight=10

  • insights.jobs.bugs.values.A.color=red

  • insights.jobs.bugs.values.B.weight=3

  • insights.jobs.bugs.values.B.color=#ff5105

  • insights.jobs.bugs.values.C.weight=1

  • insights.jobs.bugs.values.C.color=orange

These settings give severity A bugs a weight of 10, severity B bugs a weight of 3, and a severity C bug a weight of 1. They also configure the colors used for these severities when they are displayed on the Open Bugs bar chart.

To change the weights or colors, simply edit these values.

If you do not use A, B, and C for your severities, create new settings for the severities that you do use and delete the existing ones, if necessary. For example, if your severities are CRITICAL, HIGH, MEDIUM, and LOW, then:

  • insights.jobs.bugs.values.CRITICAL.weight=12

  • insights.jobs.bugs.values.CRITICAL.color=#770000

  • insights.jobs.bugs.values.HIGH.weight=6

  • insights.jobs.bugs.values.HIGH.color=#ff0000

  • insights.jobs.bugs.values.MEDIUM.weight=3

  • insights.jobs.bugs.values.MEDIUM.color=#ff5105

  • insights.jobs.bugs.values.LOW.weight=1

  • insights.jobs.bugs.values.LOW.color=orange

If you do not use Perforce Jobs, then Insights will not be able to show you information about bugs and issues. If you use a third party defect tracking tool, such as JIRA, then it is possible to use P4DTG to import defect information into the Perforce Server, which would allow Insights to report on them. For more information on P4DTG see http://www.perforce.com/product/components/defect-tracking-gateway

Projects page: configuring projects

Initially, Insights does not know how the files in your depot are organized. In order to tell Insights how you define projects in Perforce, log in as insightsadmin and select Projects from the menu at the top right. You will see a list of all currently configured projects. Initially, the list will be empty.

Note

Any user assigned the ROLE_PROJMGR role on the User Management page can configure projects. For more information, see User Management page: assigning administrators and project managers.

Configuring projects in Perforce Insights involves giving each project a name and identifying the branches and depot paths included in the project.

Select Add Project for each project, and fill in the details.

Project Name

This is a descriptive name for the project.

Private & Perforce Group Name

Normally, information about projects can be accessed by anyone who can log in to Insights. If a project needs restricted access, check this box.

A private project can only be accessed by logged-in users who are members of a specific Perforce group. The name of this group is automatically generated from the project name and cannot be set manually. If your project is called "My Project," then the group name is insights_project_my_project.

Users who are not members of this group will not be able to select this project from the various filters. However, they will not be restricted from viewing parts of the depot within the project through other means.

Job View

This is the job query that should be used to find jobs (bugs) for this project. The nature of this query depends on how the jobspec is defined. If there is a field named project that holds the name of the project that a bug is assigned to, then the query might look like this:

project=projectname

For a complete discussion of how to write job queries, see the P4 User's Guide, "Defect Tracking".

Branches

The branches of a project define the depot paths that include files associated with the project. A project can have multiple branches, such as MAIN, DEV, or RELEASE, and each branch consists of one or more depot paths.

Add a branch, give it a name, and specify each depot path that includes files for that part of the project.

Users & Groups

Each project can have users associated with it. User filters for a project are limited to those users whom you define as members of the project. If a user is not associated with a project, then you will not be able to filter by that user, though data about them may still be displayed.

If you want to include a large number of users, it is suggested that you set up a group in the Perforce service and add that to the project.

Generating Statistics

On the main Projects page, there is a link on the bottom right-hand corner to generate statistics for projects. Once a project has been created, a process runs every night to generate statistics for that project. However, when you first create a project, you must run that process for historical data.

The Statistics Generation page enables you to select the projects for which you want to generate historical data and the time period you want to cover. The greater the time period, the longer it takes.

The time required to generate statistics from historical data depends very much on the amount of that data (number of files, submits, and jobs) in each project. We recommend that you initially generate data for six months for a single project, and then base your estimates for the rest of data generation on the results.

Regenerating data for longer time periods for a project can be done at any time.

User Management page: assigning administrators and project managers

Insights comes with two internal users:

  • insightsadmin: the default administrative user.

  • insights_template_user: a user whose dashboard preferences are used as a default to populate the Insights dashboards and widgets the first time a user logs into Insights.

    You can configure a different user as the user whose dashboards and widgets are copied as the default for first-time logins. On the Setup page, set the configuration property insights.configuration.copyDashboardsFromUser to the user whose dashboards and widgets you want to copy as the default for new users.

The Insights User Management page enables you to change the password for insightsadmin and insights_template_user and set up other users to be administrators or project managers.

Note

You cannot use this interface to change the password of users authenticated by Perforce, only the internal users insightsadmin and insights_template_user.

To access the Insights User Management page, log in as insightsadmin and select Users from the menu at the top right.

To find a user, start typing their name into the Username field. Only users who have previously logged into Insights or who were created using the User Management interface are listed.

To modify a user that has not previously logged in, use the Create option (in the toolbar). Creating users using this option does not create them in Perforce and does not allow you to set their password.

Once a user has been created, you can edit the user's role, using the Roles tab on the Create User page. A user that has the ROLE_ADMIN role has permission to configure Insights (they see the Setup option on their menu). Users with ROLE_PROJMGR have permission to modify Projects within Insights. They see the Projects option on their menu.

Setting up logging

Insights uses Apache log4j for logging. The default log location is /var/log/tomcatn/insights.log.

You can change the logging level and path in /opt/perforce/usr/share/insights/insights-config.properties.

For advanced logging configuration, you can use the sample log configuration file insights-log4j.groovy. Contact Perforce technical support for instructions about how to use this file.

The Import page displays basic DBImport logging details. For more detailed DBImport logging, change the logging level in insight-config.properties.

Setting up Perforce Service Manager

Perforce Service Manager (PSM) is an optional component of Insights that monitors the current performance of a Perforce Server instance. It reports on the current command queue length and shows the most active users and processes using Perforce.

The PSM is a standalone component that you must install on the host that runs your Perforce Server, since it monitors the log files from the Perforce Server instance in real time. PSM downloads, installation instructions, and configuration instructions are available on your Insights server here:

http://hostname:port/insights/docs/psm

Once you have set up PSM, you should ensure that the configuration option insights.psm.baseURL.main on the Setup page has been configured correctly.