🔎Searching Logs
Various ways to search logs
Grafana Logs Browser
Grafana Logs Browser is the defacto UI for searching logs in Episilia. A Grafana installation is required. Episilia is to be configured as a Loki datasource, the server:port is of episilia-gateway node of the cluster.
The Epislia query syntax is compliant with Grafana/Loki query syntax, documented at https://grafana.com/docs/loki/latest/logql/
User can run the query as-is they would run for a grafana/loki system. User can type the query in the query window or use the drop-down menu available.
Query Syntax
Structure of query is:
{LabelSelector1, LabelSelector2, LabelSelector3,.. } SeacrhFilter1 SeacrhFilter2 SeacrhFilter3
The query comprises of two parts:
Label Selector Part
Search Filter Part
Label Selector Part:
There can be one or more label selectors.
{LabelSelector1, LabelSelector2, LabelSelector3,.. }
Each label selector would choose one available label (as per the label name in the input log) and select any of the four operators below:
( EQUAL ) Label = “Val”
(NOT EQ) Label != “Val”
(REGEX ) Label =~ “Val”
(NOTREG) Label !~ “Val”
Search Filter Part:
There can be multiple search filters applied on the label selectors.
*** SeacrhFilter1 SeacrhFilter2 SeacrhFilter3 …
Search filter can have any of the following operators as below:
(Exactly contain) |= “Filter”
(Doesn’t contain) != “Filter”
(Regex match) |~ “Filter”
(Regex not match) !~ “Filter”
For detailed regexp syntax refer to: https://www.pcre.org/original/doc/html/pcrepattern.html
Note on App ID and Tenant ID:
Episilia uses/creates a unique app id & tenant id label (as part of the deployment). These two unique labels are available to be used for query as:
__app__
__tenant__
The App ID label (__app__) is provided by log indexer configuration. It can be a fixed value or a combination of multiple labels from the input log.
Similarly, the Tenant ID label (__tenant__) is provided by log indexer configuration. It can be a fixed value or a combination of multiple labels from the input log.
References
Grafana loki query syntax: https://grafana.com/docs/loki/latest/logql/
Regexp filter patterns: https://www.pcre.org/original/doc/html/pcrepattern.html
Query Examples
Query all the logs for a given app-id:
{__app__=”Fuduntu12.04”}
Query all the logs for a given label:
{release=”14.04LTS”}
Query by selecting multiple labels:
{distr=”Fuduntu”,release=”12.04”}
Query for a given app-id with simple match filter:
{__app__=”Fuduntu12.04”} |= “signal”
Query for a given app-id with multiple match filter:
{__app__=”Fuduntu12.04”} |= “signal” |= “SIGTERM”
Query all the logs based on regexp match of labels:
{release=~”.*LTS”} |= “host”
Query for a given app-id with regexp match filter:
{__app__=”openSUSE12.04”} |~ “TID \d3\d\d”
Running Multiple Queries:
Multiple queries can be selected using “Add Query” in Grafana, which will enable a summing (OR) of log outputs. This is same as running more than one query serially and concatenating the results.
Running for multiple values of a label:
Multiple values of a label can be passed using regexp as below:
{release=~”(20.10|16.04LTS)”} |= “Error”
To search for double-quoted strings in the input log, escape the double quotes as below:
{__app__=”Fuduntu12.04”} |= “\“pnmf4.py\””
Retrieving surrounding context lines along with log results:
To retrieve surrounding context lines for each log line, add the following predefined label selector
__ctx__=<number> # number of lines to include above and below the log
To get contextual logs of 2 lines above and below each log line
{__app__=”fedora14.04LTS”,__ctx__=2} |= “shutdown”
Downloading log results to S3 and laptop
Grafana log browser works for result sizes upto 5000 lines. To work large results of 100K or 1M log lines, they can exported to S3 directly and downloaded. The label selector is
__s3__=1 # 1 indicates true
Example:
{__app__=”fedora14.04LTS”,__s3__=1}
The above query will provide the s3 url in the grafana browser, and the url will be available in a few seconds to a couple of minutes depending on the query.
The total number of logs in the resulting s3 file can be specified with this label selector
__s3limit__=500000 # limit to 500K lines
Example:
{__app__=”fedora14.04LTS”,__s3limit__=5000}
Regex Examples
Phone
{__app__="Ubuntu20.10"} |= "phone" |~ "[6-9][0-9]{9}"
SSN
{__app__="Ubuntu20.10"} |~ "\([2-9][0-9]{2}\)\s[0-9]{3}-[0-9]{4}"
Aadhar
{__app__="Ubuntu20.10"} |="aadhar" |~ "[2-9]{1}[0-9]{3}-[0-9]{4}-[0-9]{4}"
{__app__="Ubuntu20.10"} |~ "[A-Z0-9a-z.%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,6}"
AWS SecretKey
{__app__="Ubuntu20.10"} |~ "(?i)secret.*[A-Z0-9]{20}"
Size >= 2000000 and < 4000000
{__app__="Ubuntu20.10"} |~ "size [2-4]\d\d\d\d\d\d.*"
IP Address
{__app__="Ubuntu20.10"} |~ "\d+\.\d+\.\d+\.\d+"
Not sync Ips
{__app__="Ubuntu20.10"} |~ "\d+\.\d+\.\d+\.\d+" != "synchronized"
Any word
{__app__="Ubuntu20.10"} |~ "(block|cancel)"
Last updated