Configuring Insights
This chapter discusses the following configuration tasks:
-
Configuring your PostgreSQL database (required for most installations)
-
Configuring Insights using the online administrative interface (required for all installations)
-
Setting up logging (optional)
-
Setting up Perforce Service Manager (optional)
Configuring your PostgreSQL database
By default, an installation of PostgreSQL usually requires a valid operating system account to connect to a database. Because the Insights database connection comes from a Tomcat web application, Insights cannot access the database using a system account, but must instead connect using username/password authentication. If this has not been enabled already, you must configure the database to enable it.
If you are running Ubuntu and your installation of PostgreSQL is on the same machine as Insights, you can skip this configuration. For all other configurations (Ubuntu with PostgreSQL on a remote host; Red Hat or CentOS, whether PostgreSQL is local or remote), you must ensure that the database can be accessed with a username and password. In addition, if you do not want to provide remote administrator access to the database, you must create the role and database manually.
Important
The PostgreSQL database must be configured to use UTF-8. For information about character set configuration in PostgreSQL, see http://www.postgresql.org/docs/9.0/static/multibyte.html
Enabling remote access and user/password authentication
If you are running Insights and PostgreSQL on different machines, you must enable user/password authentication for remote access to the PostgreSQL database. You must also enable user/password authentication when PostgreSQL is local if you are running Red Hat or CentOS.
For more information about setting up PostgreSQL authentication, see http://www.postgresql.org/docs/8.4/static/auth-methods.html.
To enable remote access and password authentication:
-
If you are running Insights and PostgreSQL on different machines, configure the database to allow access from remote clients.
For CentOS or Red Hat, edit the following file:
/var/lib/pgsql/data/postgresql.conf
For Ubuntu, edit the following file (Ubuntu 12.04 uses PostgreSQL 9.1):
/etc/postgresql/9.1/main/postgresql.conf
Configure
listen_addresses
to allow access from the Insights application server. The following configuration example opens up the database to clients from any host, but you may want to restrict the configuration to a narrower definition:listen_addresses = '*'
-
Enable password authentication.
For CentOS or Red Hat, edit the following file:
/var/lib/pgsql/data/pg_hba.conf
For Ubuntu, edit the following file (Ubuntu 12.04 uses PostgreSQL 9.1):
/etc/postgresql/9.1/main/pg_hba.conf
At the bottom of this file are a number of entries ending with
ident
(in PostgreSQL 9.1, these entries may end withpeer
). The default PostgreSQL configuration looks something like this:# TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all ident # IPv4 local connections: host all all 127.0.0.1/32 ident # IPv6 local connections: host all all ::1/128 ident
The simplest solution is to change each instance of
ident
(orpeer
) tomd5
.You may also want to open up the access rules for one or both of the
host
lines (depending on whether your network infrastructure uses IPv4 or IPv6). For example, to allow password access over all network types and to open up host access from all IPv4 addresses, you can use a configuration like the following:# TYPE DATABASE USER CIDR-ADDRESS METHOD # "local" is for Unix domain socket connections only local all all md5 # IPv4 local connections: host all all 0.0.0.0/0 md5 # IPv6 local connections: host all all ::1/128 md5
If you want to limit access further, consult the PostgreSQL documentation.
Note that by default, the
postgres
user does not have a password set, and you cannot log in as this user with a password until you have enabled password authentication. Until you do, you can still log in by using the postgres user account:sudo -u postgres psql
The
configure-insights.sh
script uses this technique if no database admin password is provided when you are configuring the installation.
Creating the database and role manually
If the database host is remote and you do not want to give remote administrator access to the database, then the database administrator must create the role and database manually:
-
Create a role and give it a password.
Insights expects the role name
insights
by default, but you can use any role name you like and tell Insights during configuration. -
Create a database.
Insights expects the name
insightsdb
with ownerinsights
, but you can use any name you like and tell Insights during configuration. -
Run the installed SQL files against the new database.
The
perforce-insights
Deb and RPM package installs the SQL files into/opt/perforce/usr/share/insights/db
. If you are installing manually from the .zip or .tgz file, the SQL files are in thesql
directory.-
These files create tables with owner
insights
. Rename the script owner if you created a different role name. -
Run the create_pa_tables.sql script
-
Run the create_template_user.sql script
-
When you run the configure-insights.sh
script on the Insights
application server, you will be asked if you want to create a database. Say no and
enter the role name and password you set up here.
Testing database access
To test access to the database, run the following:
psql -W -Uinsights
-hlocalhost
-dinsightsdb
-c "SELECT 1"
Replace insights
and insightsdb
with your role and database name. Replace
localhost
with the hostname of the server if the
database is remote. This command should return a single row, if access is enabled
for the user insights
against the database
insightsdb
.
Configuring Insights using the online administrative interface
Insights provides an administrative interface that you can use to import a Perforce database checkpoint, update configuration options, set up projects, and manage users.
-
Log in to
http://
.hostname
:8080/insightsThe first time you log in, you can use
insightsadmin
with a password ofP4insightsadmin
.This is the default Insights administrative account, which allows you to configure the system.
-
Go to the drop-down menu on the top right.
There are options for
, , , and . Each of these options is described below.
Import page: importing a Perforce checkpoint and enabling continuous import
The first time you install and set up Insights, you must import a Perforce database checkpoint.
-
Obtain a recent checkpoint from your Perforce Server, or create a new one.
If you do not know how to create a checkpoint, see the Perforce System Administrator's Guide, "Supporting Perforce: Backup and Recovery."
-
Upload the checkpoint to the Insights server.
Go the Insights web interface, log in as an Insights administrator (the default is
insightsadmin/P4insightsadmin
), and select from the drop-down menu on the top right.On the Import page, browse for the checkpoint file, select it, and click .
Note
Alternatively, you can copy the checkpoint file directly to the Insights server using a mechanism like ftp, scp, or network shares. You must copy the file to the directory defined in the configuration option
insights.upload.path
. The default value for this option depends on your operating system. For more information about setting configuration options, see Setup page: setting configuration options. -
On the Import page, select the checkpoint file from the Checkpoint Filename dropdown.
Click
.Verify that the checkpoint character set encoding is correct. If your Perforce service runs in Unicode-mode, Insights detects the character set and you can ignore this setting.
Click Import.
-
Enable continuous import.
Continuous import reads changes from the Perforce service as they happen in real time, keeping your Insights data up to date.
Insights needs to know where to stream changes from, so before this can happen you should configure the
insights.p4port
,insights.user
andinsights.password
configuration options on the Setup page (see below).The Import process reads any changes to the Perforce database that occur between the time you generate the checkpoint and the point at which continuous import begins, as long as there are journals available for the period.
Important
Do not use the
-z
option with p4d -jc or p4d -jj to compress checkpoints and journals. Instead use the-Z
option so that journals remain uncompressed. Insights uses the p4 export command to extract data from the Perforce database journal. In some cases the journal can be rotated before p4 export has finished extracting the data. It is important therefore that any rotated journal is left uncompressed to enable p4 export to continue.The location of the rotated journal must be known to the Perforce service. The p4 export command expects the journal to be in the P4D server root (
P4ROOT
), unless you specify a directory with the Perforce Server configurablejournalPrefix
. If your Perforce Server configuration stores journals outside of theP4ROOT
directory (which is recommended practice), then you must use thejournalPrefix
server configurable. If your journals are being moved to a different location after being rotated, or if a journal prefix is being specificed on the command line, p4 export will not be able to find them.For more information about configuring journal compression, see the Perforce System Administrator's Guide, "Supporting Perforce: Backup and Recovery".
The Import page also displays current import status, with the progress bar showing
how far through the checkpoint import the process has gone, along with the journal
number and offset reached. You can enable more detailed DBImport logging using
insight-config.properties
. For more information, see Setting up logging.
Setup page: setting configuration options
The Setup page displays a list of all of the configuration options for Insights. Not all of them are described here - most can be left unchanged.
-
To edit an option, click the option name and select Edit.
-
To delete an option, click the option name and select Delete.
-
To create a new option, click New Configuration.
Note
Initially, the values created in
pa_insights_configuration
are based on properties (for
example
/opt/perforce/usr/share/insights/insights-config.properties
).
Once a value is stored in the database on startup, modifying it in the
properties file has no effect, because the application uses the values in the
database table. Therefore, after initial startup you should use the Setup page
to add, update, or remove configuration values.
Perforce Server options
You must define the Perforce Server instance that will be used for login authentication and continuous import. Until you set the Perforce Server information, non-administrative users will be unable to log into Insights using their Perforce user name and password.
Important
You must restart the Tomcat application server after changing any of these options.
-
insights.p4port
:P4PORT
setting for the Perforce server, such asssl:myserver:1666
. -
insights.user
: The username used to authenticate against the Perforce Server. This must be asuper
user. We recommend that it be a service user. -
insights.password
: the password or ticket for the Perforce user. For security reasons, we recommend that you define a long-lived ticket for this user and enter the ticket value here rather than the password.Instead of entering the ticket value, you can place a ticket file in the home directory of the account running Tomcat. Typically for Tomcat 6 on Linux, this is
/usr/share/tomcat6
. The ticket file is called.p4tickets
and is created in your home directory when you log into the Perforce service.For more information about creating long-lived tickets for service users, see the section on service users in Distributing Perforce, "Perforce Replication" (
http://www.perforce.com/perforce/doc.current/manuals/p4dist/chapter.replication.html#replication.serviceusers
) . -
insights.psm.baseURL.main
: If you are using the optional Perforce Service Manager (PSM) component, you must also set the URL of the PSM web service, using the formathttp://hostname:port/api
.For more information, see Setting up Perforce Service Manager
Jobs options
If you are using Perforce jobs, you must specify how to interpret the Perforce jobspec. Since a jobspec can vary widely in how it is structured, and there are no limitations on how the fields are used, you must configure the names of the job fields and their possible values. If you do not have experience with Perforce job views, see the Perforce System Administrator's Guide, "Customizing Perforce: Job Specifications".
-
insights.jobs.bugs.field.name
: The name of the jobspec field that defines the type of job (for example, bug, defect, task, story). Defaults to 'Type'. -
insights.jobs.bugs.field.value
: The value of the Type field named above that defines a job as being a bug. Defaults to be 'BUG'. -
insights.jobs.bugs.severity.name
: The name of the jobspec field that defines the severity (or importance) of a bug. Defaults to 'Severity'. -
insights.jobs.bugs.status.name
: The name of the jobspec field that defines the status of a job (e.g. open, closed, duplicate). Defaults to 'Status'. -
insights.jobs.bugs.status.open.value
: Values for the 'Status' field that should be interpreted as open. This can contain multiple comma separated values. Defaults to 'open'. -
insights.jobs.bugs.status.fixed.value
: Values for the 'Status' field that should be interpreted as closed. This can contain multiple comma separated values. Defaults to 'fixed,closed,punted,obsolete,duplicate'. -
insights.jobs.bugs.date.reported
: The name of the jobspec field that holds the date a job was first opened. Defaults to 'ReportedDate'. -
insights.jobs.bugs.date.modified
: The name of the jobspec field that holds the date a job was last modified. Defaults to 'ModifiedDate'.
The following properties enable you to specify how job severities are displayed in the graphs. By default, there are three severities configured, and each has two settings. These are:
-
insights.jobs.bugs.values.A.weight
=10 -
insights.jobs.bugs.values.A.color
=red -
insights.jobs.bugs.values.B.weight
=3 -
insights.jobs.bugs.values.B.color
=#ff5105 -
insights.jobs.bugs.values.C.weight
=1 -
insights.jobs.bugs.values.C.color
=orange
These settings give severity A bugs a weight of 10, severity B bugs a weight of 3, and a severity C bug a weight of 1. They also configure the colors used for these severities when they are displayed on the Open Bugs bar chart.
To change the weights or colors, simply edit these values.
If you do not use A, B, and C for your severities, create new settings for the severities that you do use and delete the existing ones, if necessary. For example, if your severities are CRITICAL, HIGH, MEDIUM, and LOW, then:
-
insights.jobs.bugs.values.CRITICAL.weight
=12 -
insights.jobs.bugs.values.CRITICAL.color
=#770000 -
insights.jobs.bugs.values.HIGH.weight
=6 -
insights.jobs.bugs.values.HIGH.color
=#ff0000 -
insights.jobs.bugs.values.MEDIUM.weight
=3 -
insights.jobs.bugs.values.MEDIUM.color
=#ff5105 -
insights.jobs.bugs.values.LOW.weight
=1 -
insights.jobs.bugs.values.LOW.color
=orange
If you do not use Perforce Jobs, then Insights will not be able to show you information about bugs and issues. If you use a third party defect tracking tool, such as JIRA, then it is possible to use P4DTG to import defect information into the Perforce Server, which would allow Insights to report on them. For more information on P4DTG see http://www.perforce.com/product/components/defect-tracking-gateway
Projects page: configuring projects
Initially, Insights does not know how the files in your depot are organized. In
order to tell Insights how you define projects in Perforce, log in as
insightsadmin
and select from the
menu at the top right. You will see a list of all currently configured projects.
Initially, the list will be empty.
Note
Any user assigned the ROLE_PROJMGR
role on the User
Management page can configure projects. For more information, see User Management page: assigning administrators and
project managers.
Configuring projects in Perforce Insights involves giving each project a name and identifying the branches and depot paths included in the project.
Select Add Project for each project, and fill in the details.
Project Name
This is a descriptive name for the project.
Private & Perforce Group Name
Normally, information about projects can be accessed by anyone who can log in to Insights. If a project needs restricted access, check this box.
A private project can only be accessed by logged-in users who are members of a
specific Perforce group. The name of this group is automatically generated from
the project name and cannot be set manually. If your project is called "My
Project," then the group name is
insights_project_my_project
.
Users who are not members of this group will not be able to select this project from the various filters. However, they will not be restricted from viewing parts of the depot within the project through other means.
Job View
This is the job query that should be used to find jobs (bugs) for this
project. The nature of this query depends on how the jobspec is defined. If
there is a field named project
that holds the name of the
project that a bug is assigned to, then the query might look like this:
project=projectname
For a complete discussion of how to write job queries, see the P4 User's Guide, "Defect Tracking".
Branches
The branches of a project define the depot paths that include files associated with the project. A project can have multiple branches, such as MAIN, DEV, or RELEASE, and each branch consists of one or more depot paths.
Add a branch, give it a name, and specify each depot path that includes files for that part of the project.
Users & Groups
Each project can have users associated with it. User filters for a project are limited to those users whom you define as members of the project. If a user is not associated with a project, then you will not be able to filter by that user, though data about them may still be displayed.
If you want to include a large number of users, it is suggested that you set up a group in the Perforce service and add that to the project.
Generating Statistics
On the main Projects page, there is a link on the bottom right-hand corner to generate statistics for projects. Once a project has been created, a process runs every night to generate statistics for that project. However, when you first create a project, you must run that process for historical data.
The Statistics Generation page enables you to select the projects for which you want to generate historical data and the time period you want to cover. The greater the time period, the longer it takes.
The time required to generate statistics from historical data depends very much on the amount of that data (number of files, submits, and jobs) in each project. We recommend that you initially generate data for six months for a single project, and then base your estimates for the rest of data generation on the results.
Regenerating data for longer time periods for a project can be done at any time.
User Management page: assigning administrators and project managers
Insights comes with two internal users:
-
insightsadmin
: the default administrative user. -
insights_template_user
: a user whose dashboard preferences are used as a default to populate the Insights dashboards and widgets the first time a user logs into Insights.You can configure a different user as the user whose dashboards and widgets are copied as the default for first-time logins. On the Setup page, set the configuration property
insights.configuration.copyDashboardsFromUser
to the user whose dashboards and widgets you want to copy as the default for new users.
The Insights User Management page enables you to change the password for
insightsadmin
and insights_template_user
and set up other users to be administrators or project managers.
Note
You cannot use this interface to change the password of users authenticated by
Perforce, only the internal users insightsadmin
and
insights_template_user
.
To access the Insights User Management page, log in as
insightsadmin
and select from the
menu at the top right.
To find a user, start typing their name into the Username field. Only users who have previously logged into Insights or who were created using the User Management interface are listed.
To modify a user that has not previously logged in, use the Create option (in the toolbar). Creating users using this option does not create them in Perforce and does not allow you to set their password.
Once a user has been created, you can edit the user's role, using the
Roles tab on the Create User page. A
user that has the ROLE_ADMIN
role has permission to configure
Insights (they see the Setup option on their menu). Users with
ROLE_PROJMGR
have permission to modify Projects within
Insights. They see the Projects option on their menu.
Setting up logging
Insights uses Apache log4j
for logging. The default log location
is /var/log/tomcat
.
n
/insights.log
You can change the logging level and path in
/opt/perforce/usr/share/insights/insights-config.properties
.
For advanced logging configuration, you can use the sample log configuration file
insights-log4j.groovy
. Contact Perforce technical support for
instructions about how to use this file.
The Import page displays basic DBImport logging details. For more detailed DBImport
logging, change the logging level in insight-config.properties
.
Setting up Perforce Service Manager
Perforce Service Manager (PSM) is an optional component of Insights that monitors the current performance of a Perforce Server instance. It reports on the current command queue length and shows the most active users and processes using Perforce.
The PSM is a standalone component that you must install on the host that runs your Perforce Server, since it monitors the log files from the Perforce Server instance in real time. PSM downloads, installation instructions, and configuration instructions are available on your Insights server here:
http://hostname:port
/insights/docs/psm
Once you have set up PSM, you should ensure that the configuration option
insights.psm.baseURL.main
on the Setup page has been configured
correctly.