Configuration

hcpcbc uses a JSON file to read its configuration from. The loader does some basic consistency checks, but it’s required to make sure that the configuration file strictly complies to the syntax rules for JSON files. It is highly suggested not to change the structure of the file.

Warning

Even simple errors in the JSON file might trigger errors like this:

Parsing the config file "hcpcbc_config.json" failed:
    hint: Expecting property name enclosed in double quotes:
          line 66 column 13 (char 2602)

Luckily, the position given points fairly exact to where the error is - in this case, it’s just a comma too much:

58
59
60
61
62
63
64
65
66
67
        },
    "x": {
        "0 enable collection": true,
        "1 adminAllowed": false,
        "2 user": "",
        "3 password": "",
        "4 systemVisibleDescription": "",
        "9 last collected": "2015-12-30T20:36:25+0100"
    },
}

A fresh configuration file configures access to the HCP system(s) using a system-level user, only. At the very beginning of each run, hcpcbc queries the HCP systems for a list of configured Tenants. At the end of a run, the configuration file is refreshed with this gained details.

If there are Tenants where the user account used has no access to, a different (Tenant-level) user account can be configured for that specific Tenant by simply filling the provided (empty) fields with the users name and password.

Warning

As the configuration file contains user credentials, make sure that it is stored in a secure location with propper permission settings, not to disclose it to nosy eyes!

The file must be write-able by the tool itself, as it uses it to remember state between its runs.

The configuration file explained

This is the template configuration file hcpcbc is able to create for convenience, if no configuration file is available.

1
2
3
4
5
{
    "0 comment": [
        "This it the configuration file for **hcpcbc**. For a detailed",
        "description visit *https://hcpcbc.readthedocs.org*."
    ],

Target section

This section describes the HCP system(s) from which chargeback data is to be collected. Any number of HCP systems can be defined.

The user needs to be a system-level user having the Monitor role.
The folder is a sub-folder below the path(s) declared in the store sections.
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    "1 targets": [
        {
            "0 comment": [
                "An HCP system to collect from. *User* needs to be a",
                "system-level user that has the *Monitor* role enabled."
            ],
            "1 fqdn": "hcp72.archivas.com",
            "2 user": "monitor",
            "3 password": "monitor01",
            "4 folder": "hcp72",
            "9 tenants": {}
        },
        {
            "0 comment": [
                "An HCP system to collect from. *User* needs to be a",
                "system-level user that has the *Monitor* role enabled."
            ],
            "1 fqdn": "hcp73.archivas.com",
            "2 user": "monitor",
            "3 password": "monitor01",
            "4 folder": "hcp73",
            "9 tenants": {}
        }
    ],

Parameter section

The periode sets the granularity of the collected data. A list of one, two or three values (out of “hour”, “day”, “total”) may be given.
collection limit declares how many days worth of data shall be gathered in a single run. If hcpcbc fails with timeout errors, lower the value from the default of 181 days down to some ten days. Then run hcpcbc multiple times until the collection has reached the actual date.
timeout defines the TCP connection timeout in seconds.
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
    "2 parameters": {
        "0 comment": [
            "*1 periode* defines the reports granularity,",
            "*2 collection limit* defines for how many days collection",
            "takes place, starting from the last collection time."
            "*3 timeout* defines the tcp connection timeout"
        ],
        "1 periode": [
            "total",
            "day",
            "hour"
        ],
        "2 collection limit": 181,
        "3 timeout": 600
    },

Stores section

In this section, storage locations are defined for:

  • temporary storage - this will be used to store reports during processing
  • local storage - a local directory in which the collected reports will be stored
  • compliant storage - an HCP Namespace to store the reports, if required with a retention periode set
    • path - a full qualified path to the Namespace
    • user - needs to have write access to the Namespace
    • retention - see HCP manual “Using a Namespace” for valid retention patterns
  • store raw - store the raw reports (beside the split-up reports for Tenant summary and Namespaces)
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
    "3 stores": {
        "1 temporary": {
            "0 comment": [
                "The temporary directory in which the downloaded",
                "reports are stored during processing."
            ],
            "1 tempdir": "./_hcpcbc.dir"
        },
        "2 local": {
            "0 comment": [
                "The directory in which processed reports are",
                "stored locally."
            ],
            "1 enable": true,
            "2 path": "./_hcpcbc.dir",
            "9 store raw": true
        },
        "3 compliant": {
            "0 comment": [
                "This is the HCP Namespace to which the processed reports",
                "can be stored for long-term preservation"
            ],
            "1 enable": false,
            "2 path": "https://n1.m.hcp72.archivas.com/rest/chargeback",
            "3 user": "n",
            "4 password": "n01",
            "5 retention": "0",
            "9 store raw": true
        }
    },

Charts section

This section defines which charts shall be created, if at all.

  • path can be the same as for local store; report and chart folder structure are made to allow overlap without interference.
  • linear and log enable charts with linear and/or logarithmic scale.
75
76
77
78
79
80
81
82
83
84
85
86
87
    "4 charts": {
        "0 comment": [
            "Chart creation",
            "*hourly charts* will use the hour reports to create charts",
            "*daily charts* will do the same using the day reports."
        ],
        "1 enable": true,
        "2 path": "./_hcpcbc.charts",
        "3 hourly charts": true,
        "6 daily charts": true,
        "a linear": true,
        "b log": true
    },

Logging section

This section defines how logging shall take place.

88
89
90
91
92
93
94
95
96
97
98
99
    "9 logging": {
        "0 comment": [
            "This defines how hcpcbc does logging."
        ],
        "1 log to stdout": true,
        "2 log to file": false,
        "3 logfile": "./_hcpcbc.dir/_hcpcbc.log",
        "4 rotate MB": 10,
        "5 backups": 9,
        "6 debug": false
    }
}

Example configuration file after first run

After a run, the configuration file is updated with the Tenants configured per HCP system.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
    "0 comment": [
        "This it the configuration file for **hcpcbc**. For a detailed",
        "description visit *https://hcpcbc.readthedocs.org*."
    ],
    "1 targets": [
        {
            "0 comment": [
                "An HCP system to collect from. *User* needs to be a",
                "system-level user that has the *Monitor* role enabled."
            ],
            "1 fqdn": "hcp72.archivas.com",
            "2 user": "monitor",
            "3 password": "monitor01",
            "4 folder": "hcp72",
            "9 tenants": {

Tenant section

In this section, hcpcbc stores a list of all Tenants configured in HCP.

  • enable collection - if set to “false”, the Tenant will not be collected. This is usefull if you either don’t have access to the Tenant or you don’t want to collect it as it’s a never-accessed replica, for example.
  • adminAllowed - if “true”, system-level users has been granted access to Tenant management, so that no Tenant-level user is needed to access Namespace-level reports.
  • user, password - not required if adminAllowed = true; otherwise, if manually added, will allow to collect Namespace-level reports.
  • systemVisibleDescription - for convenience
  • last collected - the timestamp of the last collection from the Tenant
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
                "m": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "Der ueblicherweise als erstes erzeugte Tenant...",
                    "9 last collected": "2015-12-31T06:49:16+0100"
                },
                "s3": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:26+0100"
                },
                "s3erlei": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:48:59+0100"
                },
                "swifttest": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:48:53+0100"
                },
                "swifty": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:06+0100"
                },
                "testtenant": {
                    "0 enable collection": true,
                    "1 adminAllowed": false,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:22+0100"
                },
                "x": {
                    "0 enable collection": true,
                    "1 adminAllowed": false,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:03+0100"
                }
            }
        },
        {
            "0 comment": [
                "An HCP system to collect from. *User* needs to be a",
                "system-level user that has the *Monitor* role enabled."
            ],
            "1 fqdn": "hcp73.archivas.com",
            "2 user": "monitor",
            "3 password": "monitor01",
            "4 folder": "hcp73",
            "9 tenants": {
                "aw21": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:58+0100"
                },
                "m": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "Der ueblicherweise als erstes erzeugte Tenant...",
                    "9 last collected": "2015-12-31T06:50:24+0100"
                },
                "s3": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:54+0100"
                },
                "s3erlei": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:49:45+0100"
                },
                "swifttest": {
                    "0 enable collection": true,
                    "1 adminAllowed": true,
                    "2 user": "",
                    "3 password": "",
                    "4 systemVisibleDescription": "",
                    "9 last collected": "2015-12-31T06:50:11+0100"
                }
            }
        }
    ],
    "2 parameters": {
        "0 comment": [
            "*1 periode* defines the reports granularity,",
            "*2 collection limit* defines for how many days collection",
            "takes place, starting from the last collection time.",
            "*3 timeout* defines the tcp connection timeout"
        ],
        "1 periode": [
            "total",
            "day",
            "hour"
        ],
        "2 collection limit": 181,
        "3 timeout": 600
    },
    "3 stores": {
        "1 temporary": {
            "0 comment": [
                "The temporary directory in which the downloaded",
                "reports are stored during processing."
            ],
            "1 tempdir": "./_hcpcbc.dir"
        },
        "2 local": {
            "0 comment": [
                "The directory in which processed reports are",
                "stored locally."
            ],
            "1 enable": true,
            "2 path": "./_hcpcbc.dir",
            "9 store raw": true
        },
        "3 compliant": {
            "0 comment": [
                "This is the HCP Namespace to which the processed reports",
                "can be stored for long-term preservation"
            ],
            "1 enable": false,
            "2 path": "https://n1.m.hcp72.archivas.com/rest/chargeback",
            "3 user": "n",
            "4 password": "n01",
            "5 retention": "0",
            "9 store raw": true
        }
    },
    "4 charts": {
        "0 comment": [
            "Chart creation",
            "*hourly charts* will use the hour reports to create charts",
            "*daily charts* will do the same using the day reports."
        ],
        "1 enable": true,
        "2 path": "./_hcpcbc.charts",
        "3 hourly charts": true,
        "6 daily charts": true,
        "a linear": true,
        "b log": true
    },
    "9 logging": {
        "0 comment": [
            "This defines how hcpcbc does logging."
        ],
        "1 log to stdout": true,
        "2 log to file": false,
        "3 logfile": "./_hcpcbc.dir/_hcpcbc.log",
        "4 rotate MB": 10,
        "5 backups": 9,
        "6 debug": false
    }
}