False Positive Filtering through Globalyzer Rule Sets

From Lingoport Wiki
Revision as of 18:15, 31 August 2016 by Masnes (talk | contribs) (Globalyzer Workbench)
Jump to: navigation, search

Reducing false positives by customizing rule sets.

The default Globalyzer Rule Sets come pre-configured to be generally applicable to all applications written in the relevant programming language. This places some limits on their accuracy, as applications can vary wildly in form and coding style. Rule sets are designed to detect issues regardless of the application coding style. However, as a consequence of this, they may also detect a number of false positives. These false positives can be greatly reduced via tuning a rule set to match an applications coding style.

Rule Set Tuning Basics

Rule sets contain 4 categories of detection types:

  • Embedded Strings
    • Any hard coded string in the application that will need to be translated.
  • Locale Sensitive Methods
    • For example, Date/Time, Encoding, or String Concatenation methods.
  • General Patterns
    • For example, hard coded fonts, encodings, or date formats: 'ASCII', 'ARIAL', 'mm/dd/yy'.
  • Static File References
    • Application references to static files, some of which may need to be localized.

For each category, there are regex based rules used to detect and filter issues. Rule sets start with pre-configured rules, but are fully modifiable after creation.

Rule Types

For each category, there are detection and filtering rules. You can mostly leave the detection rules alone - they are already well configured. This article will focus on creating, modifying, and testing filtering rules.

Filtering Rules

For each rule, the given regex will be matched against a part of the issues context. The context could be the line the issue is on, the text of the issue itself, a method call, or a variable. If the regex pattern matches some or all of the context (depending), then the issue will be filtered.

Filtering rules come in the following categories:

Literal Filters (Embedded String Only)

String literal filters match against the string contents. The string will beis filtered if part of its content matches.

Example:

String example = "This is a string.";

Matching filter examples:

  • a string
  • This
Line Filters (All types)

Line filters match against the code line containing the issue. If the match succeeds for the code line, then the issue will be filtered.

Example:

String example = "This is a string with the general pattern 'mm/dd/yy'";

Matching filter examples:

  • example
  • String example
  • This is a string
  • mm/dd/yy
Method Filters (Embedded Strings only)

String method filters match against a method in use which may contain one or more embedded strings. All strings passed to the method are filtered.

Example:

someObject.doSomething("Input one", "Input two");

Matching filter examples:

  • doSomething
  • someObject\.doSomething
Operand Filters (Embedded Strings only)

String operand filters match against a variable that is compared to or assigned to the string via an operand.

Examples:

String example = "This is a string.";
if (example != "This is not a string") { ...

Matching filter example:

  • example

Note: =, == and != are nearly universal operators, but many other operators may also be matched against, depending on the language.

Filtering Tactics

Pre-Setup

Rule sets include an inheritance feature. You may wish to have an overarching rule set where you place filters for common application wide issues. Then multiple sub rule sets for specific sub projects. Rules are distinguished as either from a parent rule set, or part of the current rule set. This make it easier to see the changes that you have made to a given rule set.

Tools

Globalyzer Workbench

The Globalyzer Workbench is the ideal platform for rule set tuning. It allows you to test your filters against your code as you create them. Before you begin, you should be familiar with the workbench. Knowledge of the following is essential:

  1. Workbench usage basics (create a project, manage scans, scan a project)
  2. Searching through scan results.
    • Be aware that the search can only look through issues currently displayed in the scan results. This is a maximum of 5,000 issues at a time.
  1. The 'Add filters/detections' dialog (From the menu: Scan -> Add Rule Set Filters/Detections...).
  2. The scan views
    • View "all active" issues while searching for issues to filter.
    • Search through issues in the view "filtered" to check that your rule filtered the correct issues. And to make sure the rule did not filter out real problems.
Globalyzer Server

You should mostly be testing new rules from the Globalyzer Workbench interface. However, you will need to visit the Globalyzer Server website in order to modify already existing rules.


Regex Testing Websites

There are multiple websites that allow you to test regex patterns. These provide a fast way to test rule patterns before trying them in the Workbench. A couple of sites to consider are regexpal.com and regex101.com. Multiple regex variants are supported on each site. Use the 'pcre' variant - this is closest to the form that the Workbench's Java rule set engine uses. Using another variant, such as javascript, will not always give accurate results.

General Tactics

If scanning a large project in the workbench, create two scans using the same rule set. Set one scan to apply to the entire project, but have the other only scan a small portion of the code. The partial scan will be quick - use it as you iterate over your rules. Once you are happy with a given set of rules, try them with the full scan. Then submit them to the server and restart the process.

When filtering a fresh project, it's best to filter widely used patterns first. There will be some application specific patterns that are present in a large number of false positives. Look for these patterns. You will need to do less work overall if you take the time to first filter the most widely present patterns.

The most obvious patterns include similar method and variable names. You can create a single filter that will match many similar method/variable names. More obscure patterns in the results will be discussed below.

Regex Patterns

A great deal of creativity is possible when creating regex based rules. This section provides a tutorial to help you make the most of this power.


To

 * Advanced Filters
   * The Embedded String category has all filter types. Will use it for
     example.
   * Method
     * log
     * log\.debug
     * log[\w\._]*
     * (CharConversion|Unimplemented|I?IO)Exception[\w\._]*
     * 'myMethod(Long)?'
     * '(possible)?Example\w+'
   * Literal
     * \A[A-Z\s]+\Z
     * \A(ON|OFF)\Z
     * \A(word1|word2|word3)\Z
     * html: \A(\s*\b(a|b|body|button|colgroup|dd|div|dt|em|file|font|footer|form|geo_name|h1|h2|h3|h4|h5|head|header|html|i|iframe|li|ol|p|pane|pre|seq|span|tab|tbody|td|text|textarea|th|tr|tt|type|ul|wrapper)\b[,\s]*)+\Z
     * \A[A-Z\W]*\b(GET|POST)\b[A-Z\W]*\Z  # [A-Z\W] = UPPERCASE or non-word
   * Operand
     * (error)?[Mm]essage
     * [^\s]+Mode                # [^\s] at start won't work for method filters
     * [A-Z]+_[A-Z]+(_|[A-Z]+)*
   * Line
     * String stringName
     * \A[A-Z]+_[A-Z](_[A-Z]|[A-Z]+)+\Z   # Line constant, e.g. CONSTANT_NAME

* Notes:

   * Avoid initial .* (computationally expensive, also may not work)
   * Monster Literal: 
     ((\\[bwst.]|\\[AZSP]|\.[*+]|\^\{0,6}\]).*?){3,}
     * Filters out strings containing regex expressions.
       Looks for 3 or more of the following:
       "\b \w \s \t \. .* .+ [q-y,$etc] ^ $ \A \Z \S \P"
       Components:
       \\[bwst.]      # \b \w \s \t \.
       \\[AZSP.]      # \A \Z \S \P \.
       \.[*+]         # .* .+
       \^\{0,6}\] # [a-z,etc]
       ((a).*?){3,}
       apples          # no match
       apples fall     # no match
       apples fall far # match
      Same principle:
      ((\\[bwst.]|\\[AZSP]|\.[*+]|\^\{0,6}\]).*?){3,}
   * Less obvious patterns:
   * Similar string content
* Sort the issues by different categories. Then scroll through, looking for
  patterns
 * Code Line and Issue are the most useful categories to sort for. Priority
   and File can also be useful.