Differences

This shows you the differences between two versions of the page.

--- public:fticks [2021/04/28 10:07] – admin
+++ public:fticks [2024/05/16 08:31] (current) – admin
@@ Line 3: / Line 3: @@
 The Jisc NRPS keep a log of successful roams between organisation connected to them. However, about 90% of roams happen within Federations so this data is not visible. To gain a complete picture of roaming Federations need to send their own roaming information to a central point.
-The mechanism for accomplishing this is Syslog, a standard (RFC 5424) approach for logging information to centralised repositories. Jisc uses ELK (Elasticsearch, Logstash and Kibana) to handle, process and display roaming data.
+The mechanism for accomplishing this is Syslog, a standard (RFC 5424) approach for logging information to centralised repositories. Jisc uses a combination of syslog-ng, loki, mysql and grafana to handle, process and display roaming data.
-For doing a similar task within eduroam GEANT devised a standard syslog message body format and called it FTICKS. Their requirements were:
+For doing a similar task within eduroam, GEANT devised a standard syslog message body format and called it FTICKS. The requirements relevant here were:
-<quote>
 . Allow receipt of statistics events in a decentralised manner (i.e. from arbitrary, but legitimate sources).
 . Support semantics of established eduroam statistics collection, in particular:
-a. Quantify number of authentications carried out, noting source country of user and country visited.
+a. Quantify number of authentications carried out
 b. Quantify number of roaming days (total number of distinct MAC addresses seen roaming on a given day).
-c. Separate actual user traffic from automatically generated probe traffic (monitoring traffic).
+. Be implementable by participants in a lightweight manner; ideally completely stateless for the participant.
-. Be implementable by participants (FLR or SP/IdP servers) in a lightweight manner; ideally completely
-stateless for the participant.
 . Contain a reliable duplicate detection.
@@ Line 30: / Line 26: @@
 . Be independent of the server software used by participants.
-</quote>
+The format would be:
+<code>
+F-TICKS/govroam/1.0#REALM=%R#VISINST=%{Operator-Name}#VISCOUNTRY=GB#CSI=%{Calling-Station-Id}#RESULT=OK#
+</code>
+# as a field separate makes sense because it doesn't appear in realms or the Calling-Station-Id.
-The eduroam model is very flat with just NRPS->ORPS->IdP, but the govroam model goes a bit further with, potentially, multiple levels of Region Proxies e.g. NRPS->RRPS->RRPS->...->ORPS->IdP.
+This format can easily be machine parsed by the aforementioned tools.
-Each site keeps logs internally which is enough for identifying local issues and passing on details for audit purposes. JISC's NRPS can see all the roaming interactions between sites and aren't interested in local users using eduroam on their own site.
+The REALM field contains the realm extracted from the username in the form 'holby.nhs.uk' and the CSI contains the Calling Station ID of the device making the authentication request (the user's device). More on the CSI later.
-However, the govroam model means that there could (and are) users roaming between sites which have their own RRP and thus the traffic isn't seen by the JISC NRPS. People are more likely to roam to a nearby site as opposed to a more remote one so most of the roaming traffic will be seen only by the RRPS and much less by the NRPS. Thus JISC is missing out on a considerable amount of meta-data describing the roaming activity of govroam users.
+VISINT is the identity of the organisation sending the authetication request. Ideally this should be the Operator-Name of the site from which the Visitor is making their request. e.g. '1localgp.holby.nhs.uk' but the RFO should insert the originating site's identity if the originating site is unable to do so themselves. The least desirable default option is for the RFO to insert their own Operator-Name as a last resort e.g. '1holby.nhs.uk'.
-There's a requirement to see if and how logging information could be passed from the RRPS to a central JISC logging site. The obvious answer would be to have the RRPS configured to send FTICKS (the standard logging format used by eduroam and govroam) to the same logging server receiving the NRPS logs.
+RESULT should always be 'OK' because only successful authentication should be reported. Furthermore, only successful authentication attempts between two members of the same Federation should be reported. Specifically, authentication attempts within the same realm are NOT roams, by definition, and authentication attempts with realms outside of the Federation will already by logged by the Jisc NRPS.
-The problems start with the RADIUS software. Whilst it's easy to configure software such as RADIATOR (used by eduroam NRPS) and FreeRADIUS (used by the govroam NRPS) to generate logs in arbitary formats and send them to arbitary destinations, other software, such as Microsoft NPS appears to have no ability to do this. However, there might be tools which take the logs and convert them to the right format to be sent out. A quick look suggests that it's possible but there might be costs associated with the software. There might also be licence issues with buying software to then give to someone else to use. An alternative would be writing something in-house which would avoid both of these problems.
+The CSI, or Calling-Station-Id, is useful for de-duplication of messages but it's not necessary for the CSI to be the actual MAC address of the client device. For security purposes obfuscating this address is recommended. As long as the pseudo MAC sent is always the same for the same MAC (i.e. 1-to-1 mapping) then the requirement for deduplication is met and the actual device can not be identified. The suggestion is to leave the first half of the MAC untouched because this OUI can be used for high-level device type analysis.
-ELK (ElasticSearch, Logstash, Kibana) is a monitoring tool currently being used to process the NRPS logs and is likely to be the ultimate destination of any logs generated by the RRPS/ORPS so integration with this is vital. Logstash can take logs in a variety of formats, including Windows Event logs (from a local machine), syslog and any number of log formats as long as data can be pattern matched. Logstash feeds directly into ElasticSearch though, making it a fairly heavy duty solution. Along with this triumverate is 'filebeat', a light-weight piece of software that can gather logs and send them onto Logstash. It's less capable of processing data but is easier to install and configure. The responsibility then falls on Logstash to process the data and store it in ElasticSearch.
+=====Syslog Configuration=====
-Filebeat is available for Linux and Windows so should cover most installs. It's capable of reading the data from file logs generated by NPS at least.
+The most basic form of syslog is to use UDP on port 514 to send through simple messages. This is an acceptable approach. TCP is also an option, as is TLS. Please discuss with Jisc which approach you'd like to take for sending syslog.
-Discussions would be needed with various sites running various different type of RADIUS software to determine what's possible and acceptable. Security could be an issue at a policy level. The ELK software is all capable of communicating securely but sites may have security policies which wouldn't allow traffic to be sent directly from internal RADIUS servers offsite.
+====Unix Syslog====
-===Log Attributes===
+Syslog tools are available on all versions of Unix.
-If extracting the data from the RADIUS server and sending it on to a log processing system has issues that can be overcome then the next problem, less surmountable, is the actual data generated by the RADIUS servers. Most information is available to the IdP, which has visibility of the inner tunnel, but there is no expectation, or even desire, for JISC to be presented with this data. Conversely, a RADIUS proxy can only see the data in the outer tunnel but can add extra information, such as source IP, to the log data. This is what which JISC needs to see.
+[[siteadmin:syslog_f-ticks_logging|Configuration fragments for syslog-ng]] are available but other syslog software, such as rsyslog, are just as capable.
-One of the key data is the Operator-Name. This is an attribute set in the RADIUS server configuration and identifies the site sending the request. Manchester University would have the Operator-Name set to '1manchester.ac.uk', for example. If the sending site doesn't set this then the proxy can log information which identifies the source. This is less desirable as it only identifies the previous hop. With eduroam this wouldn't be a problem due to the flat nature but is more significant with govroam due to the number of layers of the hierarchy.
-Insisting that all sites set the Operator-Name is problematic because some RADIUS servers don't have the capability of adding attributes (Microsoft NPS for example), but should still proxy any existing such attribute.
+====FTICKS configuration====
-In a perfect world the first proxy to see an authentication request should set its Operator-Name to a globally unique (based on DNS for example) name and then pass the request up the chain. If that happened then whichever proxy is at the 'top' of the chain will have a log of exactly which site generated the initial request. Since the request also contains the realm of the visitor there is then a nice mapping of a roam. If all sites were capable of generating, collecting and forwarding this data then JISC would have a complete set of data describing roaming activity between Govroam enabled sites.
+There are configuration fragments for [[siteadmin:basic_orps_radsecproxy_configuration|RadSecProxy]], [[siteadmin:basic_freeradius_orps_configuration|FreeRADIUS]] and [[siteadmin:fticks_logging_for_radiator|RADIATOR]].
-We may have to accept that, until NPS is fixed/improved, that there are going to be big holes in the collected data. As for other systems, documentation will be needed covering the install and configuration of whatever software used to generate and pass on F-TICKS or equivalent.