Simpson’s Paradox is a kind of statistical brain teaser that provides lessons on text analytics and choosing the best tools to work with enterprise content.
The “paradox” is that sometimes trends that seem apparent when data are analyzed as separate groups become reversed or disappear when the groups are combined. An example of Simpson’s Paradox is shown in the following graph:
When both blue and red groups are combined and plotted there is a downward trend line as represented by the dashed black line. However, if blue or red dots are plotted as separate groups they each show upward trends as shown in the solid blue and red lines.
The table below presents an example of the real-life consequences that can result from occurrences of Simpson’s Paradox. It shows the success rates of two different treatments for kidney stones within two groups, one with small stones, another with large:
When both small-stone and large-stone groups are combined in the Total row, Treatment B appears to yield the best prognosis – 83%. However, within the subgroups having either small kidney stones or large ones, Treatment A yields the best results. Patients facing treatment for kidney stones of any size would be better off NOT relying on the overall success rate when deciding which treatment to have performed!
People who examine organizational data (e.g., corporate auditors trying to discern underlying facts or lawyers examining electronic discovery during investigations or litigation) will want to examine all versions of particular data, both aggregated and stratified, to avoid results like these.
There are at least two lessons about text analytics that can be drawn from this discussion:
Specific data elements matter. The type of analysis performed to disclose Simpson’s Paradox requires the analysis of specific data elements. Text analytics may assign broad classifications to files but by itself does not extract specific data elements from them. It treats text strings as occurring in a one-dimensional space – words are either before or after other words without the concept of being above or below other words. Text analytics may also ignore numeric-only strings and analyze only words that occur in sentences. This makes it uniquely unsuited to working with data tables.
Data triangulation is critical. Decision makers who rely on data presented in tables need to be able to drill down to examine included subgroups to confirm whether trend lines for subgroups are different from the trend lines for aggregated data. Finding related tables not only helps identify occurrences of Simpson’s Paradox but can also tell if one set of figures was presented to one group, e.g., the IRS, while other sets were presented to other groups, e.g., investors or managers.
Tables as Identifiable & Extractable Objects
Visual classification has nine levels of objects it identifies in files, ranging from individual graphical elements to documents. One of the intermediate-level objects is the table. Visual classification works in two dimensions and can track when values (regardless of whether alpha, numeric, or both) appear in side-by-side columns, thereby enabling it to automatically identify table objects. Users then have a number of options on how to work with table content, including:
- Searching by absolute or relational page coordinates, e.g., the word “entertainment” in the left column with an amount greater than $2,000 in the same row in a column to the right.
- Having table information output in CSV format with table titles, column headings, row headings, and data values identified. The CSV data can then be imported into database, spreadsheet, or other structured analysis tools.
Users have options to find other tables that have the same row or column headings, and can find other document types that contain the same types of tables.
Visual classification’s ability to identify tables, extract table values for use in other analytics platforms, and identify where similar tables occur provides auditors, lawyers, and business people significant advantages over simple text analytics when working with unstructured content.
For more information on working with unstructured content, request your copy of the forthcoming, Guide to Managing Unstructured Content, Practical Advice on Gaining Control of Unstructured Content (including information on BR’s Fully Automated Attribute Tables), at https://beyondrecognition.net/guide-to-managing-unstructured-content/