Debugging CAS with the Endeca RecordStore Inspector

Introduction

When it comes to debugging Endeca Content Acquisition System (CAS) related issues, there are few tools that Endeca developers have at their disposal to aid them in their troubleshooting. Most know to review the CAS service logs, but occasionally an issue arises where a peek inside a record store can be very revealing. If you’re a savvy Endeca developer you already know that you can export record store content using the “recordstore-cmd” command. Combined with a good text editor this command can be very useful, but CLI’s can be a bit tedious to work with at times, so when I recently ran into such an issue I decided to write my own visual tool for inspecting Endeca Record Stores, which I aptly named the Endeca RecordStore Inspector. In this article, I introduce the Endeca RecordStore Inspector utility and show how it can be used to debug Endeca CAS related issues.

Main Article

I was recently assisting with a CAS issue where the CAS Console was reporting failed records for an incremental crawl of a Record Store Merger where one of the data sources contained deleted records. The origin record store was reporting the deleted records correctly, but when the last-mile-crawl merger was run it would report the deleted records as failed records.

For each failed record, I could see the following messages in cas-service.log:

WARN [cas] [cas-B2BDemo-last-mile-crawl-worker-1] com.endeca.itl.executor.ErrorChannelImpl.[B2BDemo-last-mile-crawl]: Record failed in MDEX Output with message: missing record spec record.id

DEBUG [cas] [cas-B2BDemo-last-mile-crawl-worker-1] com.endeca.itl.executor.ErrorChannelImpl.[B2BDemo-last-mile-crawl]: Record failed in MDEX Output (MdexOutputSink-826115787) with Record Content: [Endeca.Action=DELETE]

The messages were not intuitive at first, but I could tell that a different record spec identifier was being used, so to see what was going into the record stores I decided to create a tool for visualizing the contents of a record store in a familiar tabular format. Using this tool, I could see that the records contained both an “Endeca.Id” property as well as the “record.id” property:

However, when one of the source files was removed and the acquisition re-run, the new generation contained the delete records with only the “Endeca.Id” property:

So when the last mile record store merger was run, it didn’t know how to merge the delete records because the record spec identifier (as well as the Record ID property on the data sources) had been changed to “record.id”, thereby producing the above warning message (“missing record spec record.id“) for the DELETE action entries.

Of course the same diagnosis could have been made using recordstore-cmd and a text editor, but some things are easier done in a GUI. For example, sorting records by a specific column type. The Record Store Inspector allows you to sort on any column, as well as filter which columns are visible using regular expression syntax. You can open two different generations of a record store and compare them side-by-side. You can even export the contents of a record store (with or without filters applied) to a comma-separated value (CSV) text file, Microsoft Excel file, or Endeca Record XML file. These sorts of operations are more difficult to do when using just recordstore-cmd and a text editor, and my goal in creating this tool was to make the Endeca community more productive in their ability to diagnose CAS related issues on their own.

About the Endeca RecordStore Inspector

The Endeca RecordStore Inspector utility was written using JavaFX 8 and Endeca 11.1, and runs on Windows, Linux, or any environment that supports the Java 8 runtime. Below I’ve provided download links for two versions; a portable version optimized for Windows, and a self-contained Java jar file for all other environments. To run the Windows version, simply extract the contents of the attached archive and double-click on the rs_inspector.exe file. This version includes a Java 8 runtime environment, so there is no need to install Java or Endeca to run it. To run the self-contained jar file you will need a Java 8 Runtime Environment installed. If one is already present, just copy the attached file below and run the command: “java -jar rs_inspector-1.0-all.jar” to launch the application.

When the application has started you can press CTRL-R to select the record store and generation that you want to view. If your CAS Server runs on a host/port other than localhost:8500, then you can use ALT-S to change the default settings.

Once the record store loads, you can further refine the view using Java regular expression syntax in the column and value fields. This will restrict the columns and rows visible to only records matching the regular expression syntax specified. For example, to view only the columns for “product.id” and “product.short_desc” you can specify a Column Text filter of “product\.id|product\.short_desc” and click the Apply Filter button. To further refine the view to show only products with an id value greater than 3000000, you can use a Value Text filter of “[3-9][0-9]{6}[0-9]*|[1-9][0-9]{7}[0-9]*”.

It is important to point out that this tool currently loads all rows in the selected record store into the table view, so depending on how large the record store is, if your JVM doesn’t have sufficient memory to store all the data, you will receive an OutOfMemoryError. If your environment has sufficient memory to store the entire record store in memory, then you can increase the JVM memory settings (-Xmx) to support your record store. If your record store is larger than 2GB, you should run the RecordStore Inspector in a 64-bit JVM. If the memory in your environment is limited, then you may not be able to load your record store using this tool. Perhaps later versions will offer the ability to incrementally load data into the view. If this is important to you, then please let me know.

Summary

Endeca content acquisition can be somewhat of a black box. To provide some transparency to this process I’ve created the Endeca RecordStore Inspector. The Endeca RecordStore Inspector is a visual tool intended to aid in the debugging of issues pertaining to Endeca CAS data ingestion. In this article we’ve seen one example of how this tool was used to make sense of a seemingly enigmatic error message, but the applications of this tool are much broader in scope, not only in its use as debugging aid, but as a medium for understanding Endeca CAS in general.

Below are links to download the Endeca RecordStore Inspector. Please note that this tool is provided “as-is”, without guarantee or warranty of any kind. It is not part of the Endeca or Oracle Commerce product suite, and therefore not supported by Oracle. However, a link to the complete source code is provided below, and you are free to fix any issues or enhance the tool in any way you like.

Comments

  1. Prabakaran S K says:

    I couldn’t use this tool because of the need for Java 8 Runtime. In a machine without admin rights and pre-installed with JDK 7, all we can do is to stick with it. It would’ve been great, if the tool is made compatible to vastly used JDK 7 also.

    • I originally started with JDK 7, but JavaFX support is better in Java 8. With the exception of some lambda expressions, I don’t believe I do anything else that *requires* Java 8. The source is provided, so feel free to change it. Shouldn’t be much of an effort. It’s a small project.

  2. Hello Mohamed,

    One of the best parts of sharing a project like this is community members, like yourself, who are eager to contribute and help improve it. Yes, you are correct, active transactions are retained until they are either committed or rolled back (though it’s only the txId that’s maintained). I’ve already modified the code and updated Git, as well as the links above.

    Regarding the OutOfMemory issue, there are a couple of things you can try: (1) in the app directory there is a file named “rs_inspector.cfg”. In this file you can increase the JVM heap settings to allocate more memory for larger stores. If you’re limited on memory, another option is (2) when loading a large record store, hit cancel, then update the filter criteria to return a smaller set of data that you’re interested in. At this time, there’s not much more that can be done, but if there’s enough demand, I may enhance the tool to allow loading of records in batch sets.

    Hope that helps,

    -jim

    • Mohamed Nazeemudeen says:

      Great thanks for making the changes!

      1. Have noticed a method in the record store API, However not sure whether it can be used to read in batches from the baseline read…

      — can pass this – Integer recordsPerTransfer,
      createBaselineReader(RecordStore recordStore, TransactionId transactionId, GenerationId generationId, Integer recordsPerTransfer)

      • Ultimately, that just ends up calling readRecords on the RecordStore, which is set to 500 by default. That behavior is already there. What I meant by batching results is how many records I fetch into the client UI at a time. Currently, all records are fetched into the tableview, which for large recordstores could require a lot of memory. One possible enhancement is to fetch a fixed number of records into the client UI at a time. This means sorting and filtering would apply only to the smaller data sets, but at least you wouldn’t need as much memory to run.

  3. Mohamed Nazeemudeen says:

    Hi Jim,

    It’s a great tool, Thanks for sharing this! We wrote a similar one our team.. One thing I’ve noticed is, for example, if we run this as an executable jar, and if the record set is large (eg: 100k) then the tool throws OOM error/hangs. of course we need to have appropriate heap while running this. However the problem is, in the “RecordReaderService.java” the program is not rolling back or committing the current transaction in case of exceptions. Apparently, it’s not catching any other exceptions other than “IOException” or “RecordStoreException”. This will leave the “Active” READ transaction in the CAS. Since the active transactions are not cleaned up automatically, the clean job will fail. As you know this will lead to piling up of stale generations and tax the system….

    Guess its a good to have this? Any thoughts?

    catch (Exception exception) {
    exception.printStackTrace();
    if (transactionId != null) {
    try {
    recordStore.rollbackTransaction(transactionId);
    } catch (RecordStoreException anotherException) {
    System.out.println(“Failed to roll back transaction.”);
    anotherException.printStackTrace();
    }
    }
    } finally {
    if (writer != null) {
    writer.close();
    }
    writer = null;
    }

Add Your Comment