lowrance_usr – Lowrance USR File Parser

The Lowrance USR file is a binary file, and parsing it is a fairly complex exercise.

See https://www.gpsbabel.org/htmldoc-1.7.0/fmt_lowranceusr.html

The documentation for GPSBabel has a number of errors. It does, however, provide some advice on the overall structure of the USR file and how to decode it one field at a time.

See https://github.com/GPSBabel/gpsbabel/blob/master/lowranceusr.cc for the code, which appears to work.

Data Encodings

Latitude and Logitude encoding:

Latitude and longitude for USR coords are in the lowrance mercator meter format in WGS84. The below code converts them to degrees.

  • lon = x / (DEGREESTORADIANS * SEMIMINOR)

  • lat = (2.0 * atan(exp(x / SEMIMINOR)) - M_PI / 2.0) / DEGREESTORADIANS

Where

  • static constexpr double SEMIMINOR = 6356752.3142;

  • static constexpr double DEGREESTORADIANS = M_PI/180.0;

(See https://en.wikipedia.org/wiki/World_Geodetic_System#1984_version This says \(s_b = 6356752.314245\), but that may not be the constant that Lowrance actually uses.)

Generally, the \(\phi\) values are N-S latitude, and \(\lambda\) values are E-W longitude.

The above formula combine two things.

\[\lambda = \frac{x}{s_b} \times \frac{180}{\pi}\]

The \(\frac{x}{s_b}\) term converts millimeters to radians. These are then converted to degrees.

\[\phi = \Big( 2 \arctan( e^{\frac{x}{s_b}} ) - \frac{\pi}{2} \Big) \times \frac{180}{\pi}\]

As with the longitude, we convert mm to radians and then radians to degrees.

Dates are Julian Day Numbers. Use fromordinal(JD-1721425) to convert to a datetime.date

Times are milliseconds past midnight. Either use seconds=time/1000 or microseconds=time*1000. to create a datetime.timedelta that can be added to the date to create a usable datetime.datetime object.

See the lowranceusr4_parse_waypt() function for the decoding of a waypoint.

Field Extraction

The general approach is to leverage struct to handle decoding little-endian values.

Many structures are a sequence of fields, often a repeating sequence of fields where the repeat factor is a field.

For example:

AtomicField("count", "<i"),
AtomicField("data", "<{count}f"),

The first field has the four-byte count. The second field depends on this count to define an array of floats.

Implementation

Here’s the UML overview of this module.

@startuml 'navtools.lowrance_usr' allow_mixing component lowrance_usr { abstract class Field { name: str extract(UnpackContext) } class AtomicField { encoding: str } class FieldList { field_list: List[Field] } class FieldRepeat { name: str field_list: Field count: str } Field <|-- AtomicField Field <|-- FieldList Field <|-- FieldRepeat FieldList *-- "*" Field FieldRepeat *-- Field class UnpackContext { source: BinaryIO fields: dict[str, Any] extract(Field) } class Lowrance_USR { {static} load(BinaryIO): Lowrance_USR } AtomicField ..> UnpackContext Lowrance_USR *-- "*" Field Lowrance_USR -- "1" UnpackContext } @enduml

Parse a Lowrance .USR file.

We define a recursive data structure that defines fields. It must be traversed in order to correctly locate dependencies among field definitions. This approach will capture repeat factors and lengths.

The Lowrance_USR class contians the field definitions. Once loaded, it behaves like a dictionary.

See https://github.com/GPSBabel/gpsbabel/blob/master/lowranceusr.h and https://github.com/GPSBabel/gpsbabel/blob/master/lowranceusr.cc

The Format 6 file layout can be described as follows:

name

format

size

usr

usr - format

<I

4

usr - data_stream_version

<I

4

usr - file_title_length

<i

4

usr - file_title

varies

usr - file_creation_date_length

<i

4

usr - file_creation_date_text

varies

usr - file_creation_date

<I

4

usr - file_creation_time

<I

4

usr - unknown

<b

1

usr - unit_serial_number

<I

4

usr - file_description_length

<i

4

usr - file_description

varies

usr - number_waypoints

<i

4

usr - waypoints

depends on number_waypoints

usr - waypoints - waypoint

usr - waypoints - waypoint - uuid

<16s

16

usr - waypoints - waypoint - UID_unit_number

<I

4

usr - waypoints - waypoint - UID_sequence_number

<Q

8

usr - waypoints - waypoint - waypt_stream_version

<h

2

usr - waypoints - waypoint - waypt_name_length

<i

4

usr - waypoints - waypoint - waypt_name

varies

usr - waypoints - waypoint - UID_unit_number_2

<I

4

usr - waypoints - waypoint - longitude

<i

4

usr - waypoints - waypoint - latitude

<i

4

usr - waypoints - waypoint - flags

<I

4

usr - waypoints - waypoint - icon_id

<h

2

usr - waypoints - waypoint - color_id

<h

2

usr - waypoints - waypoint - waypt_description_length

<i

4

usr - waypoints - waypoint - waypt_description

varies

usr - waypoints - waypoint - alarm_radius

<f

4

usr - waypoints - waypoint - waypt_creation_date

<I

4

usr - waypoints - waypoint - waypt_creation_time

<I

4

usr - waypoints - waypoint - unknown_2

<b

1

usr - waypoints - waypoint - depth

<f

4

usr - waypoints - waypoint - LORAN_GRI

<i

4

usr - waypoints - waypoint - LORAN_Tda

<i

4

usr - waypoints - waypoint - LORAN_Tdb

<i

4

usr - number_routes

<i

4

usr - routes

depends on number_routes

usr - routes - route

usr - routes - route - uuid

<16s

16

usr - routes - route - UID_unit_number

<I

4

usr - routes - route - UID_sequence_number

<Q

8

usr - routes - route - route_stream_version

<h

2

usr - routes - route - route_name_length

<i

4

usr - routes - route - route_name

varies

usr - routes - route - UID_unit_number_3

<I

4

usr - routes - route - number_legs

<i

4

usr - routes - route - leg_uuids

depends on number_legs

usr - routes - route - leg_uuids - leg_uuid

<16s

16

usr - routes - route - route_unknown

<10s

10

usr - number_event_markers

<i

4

usr - number_trails

<i

4
navtools.lowrance_usr.dump_next(source: BinaryIO, count: int) → None

This peeks at a bunch of bytes in a file so we can diagnose problems with decoding them.

navtools.lowrance_usr.icon_parse(icon_code: str) → Iterator[tuple]
class navtools.lowrance_usr.Field(name: str)

A generic Field.

The report output is CSV name,format,size.

abstract extract(context: navtools.lowrance_usr.UnpackContext) → Any
report(context: str = '') → Iterable[dict]
_abc_impl = <_abc._abc_data object>
class navtools.lowrance_usr.AtomicField(name: str, encoding: str, conversion: Optional[Callable[[Tuple[Any, ...]], Any]] = None)

An isolated, atomic field or sequence of fields that we are not examining more deeply.

The name must be unique, otherwise previous values will be overwritten.

The encoding is a struct format. It is extended to permit including {name} to refer to a previously loaded field’s value. For example:

AtomicField("count", "<i"),
AtomicField("data", "<{count}f"),

This defines a count field followed by a data field. The data field will be a number of float values, defined by the value of the preceeding field.

The conversion is additional conversion beyond what struct does. For example:

AtomicField("name_len", "<i"),
AtomicField("name", "<{name_len}s", lambda x: x[0].decode("ASCII"))
extract(context: navtools.lowrance_usr.UnpackContext) → Any
report(context: str = '') → Iterable[dict]
_abc_impl = <_abc._abc_data object>
class navtools.lowrance_usr.FieldList(name: str, field_list: list)

A sequence of Field instances. This is a “block” of data.

extract(context: navtools.lowrance_usr.UnpackContext) → dict
report(context: str = '') → Iterable[dict]
_abc_impl = <_abc._abc_data object>
class navtools.lowrance_usr.FieldRepeat(name: str, field_list: navtools.lowrance_usr.Field, count: str)

A repeating AtomicField or FieldList where the repeat count comes from another field.

extract(context: navtools.lowrance_usr.UnpackContext) → list
report(context: str = '') → Iterable[dict]
_abc_impl = <_abc._abc_data object>
class navtools.lowrance_usr.UnpackContext(source: BinaryIO)

Used to unpack a binary file. This is used to manage the input buffer and extract fields using AtomicField, FieldList, FieldRepeat definitions.

THe fields includes the currently named fields being processed. This makes them visible for resolving dependencies in repeating fields and formats that depend on the values of other fields.

extract(field_list: navtools.lowrance_usr.Field) → Union[Any, dict, list]

Extracts the next fields present in the file of bytes.

peek(field: navtools.lowrance_usr.AtomicField) → Any

Peeks ahead in the file of bytes to see what follows.

eof() → bool

At EOF? Try to read another byte. Might be better to get the file size and compare the tell location.

navtools.lowrance_usr.lon_deg(mm: int) → float

Convert millimeters to degrees of longitude

navtools.lowrance_usr.lat_deg(mm: int) → float

Convert millimeters to degrees of latitude

navtools.lowrance_usr.b2i_le(value: bytes) → int

Converts a sequence of bytes (in little-endian order) to a single integer.

class navtools.lowrance_usr.Lowrance_USR

Read a Lowrance USR file, creating a complex dict[str, Any] structure that reflects the header fields, waypoints, routes, event markers, and trails.

classmethod format_6() → navtools.lowrance_usr.Field
classmethod load(source: BinaryIO) → navtools.lowrance_usr.Lowrance_USR
navtools.lowrance_usr.t3() → None
navtools.lowrance_usr.layout(usr_fields: navtools.lowrance_usr.Field = <navtools.lowrance_usr.FieldList object>) → None

Atomic Field

An isolated, atomic field or sequence of fields that we are not examining more deeply.

The name must be unique, otherwise previous values will be overwritten.

The encoding is a struct format. It is extended to permit including {name} to refer to a previously loaded field’s value. For example:

AtomicField("count", "<i"),
AtomicField("data", "<{count}f"),

This defines a count field followed by a data field. The data field will be a number of float values, defined by the value of the preceeding field.

The conversion is additional conversion beyond what struct does. For example:

AtomicField("name_len", "<i"),
AtomicField("name", "<{name_len}s", lambda x: x[0].decode("ASCII"))

Collection of Fields

A sequence of Field instances. This is a “block” of data.

Repeated of Field

A repeating AtomicField or FieldList where the repeat count comes from another field.

The Stateful Context

This is where we maintain state, reading the binary file. This allows fields to be defined independently, only sharing this context object.

Used to unpack a binary file. This is used to manage the input buffer and extract fields using AtomicField, FieldList, FieldRepeat definitions.

THe fields includes the currently named fields being processed. This makes them visible for resolving dependencies in repeating fields and formats that depend on the values of other fields.

Extracts the next fields present in the file of bytes.

Peeks ahead in the file of bytes to see what follows.

At EOF? Try to read another byte. Might be better to get the file size and compare the tell location.

Conversions

Convert millimeters to degrees of longitude

Convert millimeters to degrees of latitude

Converts a sequence of bytes (in little-endian order) to a single integer.

Unpack The File

Read a Lowrance USR file, creating a complex dict[str, Any] structure that reflects the header fields, waypoints, routes, event markers, and trails.