pax_global_header00006660000000000000000000000064151753553570014532gustar00rootroot0000000000000052 comment=823a948bf005941275b9c6d93bfc8037be63239d .gitignore000066400000000000000000000002251517535535700130650ustar00rootroot00000000000000# Python __pycache__/ *.py[cod] *.egg-info/ .venv/ .pytest_cache/ # IDE .idea/ .vscode/ # macOS .DS_Store # Local sample / scratch *.JPI jpi_csv/ LICENSE000066400000000000000000000020551517535535700121050ustar00rootroot00000000000000MIT License Copyright (c) 2026 Unicornlines Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. README.md000066400000000000000000000022611517535535700123560ustar00rootroot00000000000000# JPI Parser Python library and CLI for parsing flight-data files written by JP Instruments engine monitors (EDM830 and other protocol-2 devices). The repository also publishes [`filespec.md`](filespec.md) — a written-up description of the binary file format, derived from observation of files produced by the device. ## Install ```bash pip install -e . ``` This exposes two equivalent console commands: `jpi-analyzer` and `jpia`. ## CLI ```bash # File metadata + flight list jpia --file data.JPI info # Export selected flights & metrics to CSV (one file per flight) jpia --file data.JPI csv --flights all --metrics all --out-dir ./out jpia --file data.JPI csv --flights 1224,1225 --metrics E1,C1,MAP,RPM,FF --out-dir ./out ``` ## Library ```python from jpi_analyzer import JpiFile, FlightDecoder, export_flight_csv jpi = JpiFile.open("data.JPI") flight = FlightDecoder(jpi, jpi.get_flight(1224)).decode() export_flight_csv(flight, "1224.csv") ``` ## Repository layout | Path | Purpose | |------|---------| | `jpi_analyzer/` | The package — parser, decoder, CSV exporter, CLI. | | `filespec.md` | Description of the JPI EDM830 file format. | | `pyproject.toml` | Package manifest. | filespec.md000066400000000000000000001106631517535535700132210ustar00rootroot00000000000000# JPI `.JPI` File Format Specification This document describes the binary file format produced by JP Instruments Engine Data Monitors (EDM-series). The specification is anchored on the EDM-830 product line. > **Endianness.** All multi-byte integers in the binary sections are > **big-endian** (high byte first). All ASCII text uses CRLF (`0x0D 0x0A`) > line endings. --- ## 1. Top-level Layout A `.JPI` file consists of four contiguous sections, in this fixed order: | # | Section | Encoding | Marked by | |---|------------------------|-------------------------|----------------------------------| | 1 | ASCII Dollar Header | ASCII / CRLF | starts with `$U,…` ; ends `$L,…` | | 2 | Binary Flight Data | big-endian binary | sequence of flight records | | 3 | End-of-data marker | ASCII | `$E,*\r\n` | | 4 | Configuration XML | XOR-0x02 obfuscated XML | between `$E` and `$V` | | 5 | `$V` Trailer | ASCII | starts with `$V,Created by …` | The file may be preceded by zero-bytes or padding. Section 1 is located by scanning for the magic two-byte sequence `0x24 0x55` (`$U`). ``` +----------------------+ 0x000000 (magic '$U') | 1. ASCII Dollar Hdr | +----------------------+ end of $L,…\r\n → Data_Start | 2. Binary flight data| +----------------------+ '$E,…\r\n' | 3. $E end marker | +----------------------+ | 4. Config XML (XOR2) | +----------------------+ '$V,…\r\n' | 5. $V trailer | +----------------------+ EOF ``` --- ## 2. The ASCII Dollar Header (Section 1) Each record is a CRLF-terminated line of the form: ``` $, field1, field2, …, fieldN*\r\n ``` The *body* of the record runs from the leading `$` up to (but not including) the `*`. The two hex digits after `*` form the NMEA-style XOR checksum of all body bytes. The parser scans bytes character-by-character, building up an `expression` until it reads `*`, then advances 5 bytes (4 for `*HH\r\n` plus the `*` itself was already consumed by the scan). > **NUL-tolerance.** The parser **silently skips NUL (`0x00`) bytes** > that appear inside the ASCII section. This has been observed in > the wild on `$T` records, where a stray `0x00` can sit between the > seconds and milliseconds field. Implementations must filter NULs > while building the line, not split on them. ### 2.1 Record types | Tag | Purpose | Required? | Mask bit | |------|----------------------------------|-----------|-----------:| | `$U` | User / aircraft tail / serial | yes | 0x40 | | `$A` | Limits configuration | yes | 0x01 | | `$F` | Fuel-flow unit configuration | yes | 0x08 | | `$T` | Download date/time | yes | 0x20 | | `$C` | System / unit configuration | yes | 0x02 | | `$P` | Protocol ID | optional | — | | `$H` | Header flags (fuel-level bits) | optional | — | | `$E` | Empty-file marker | optional | — | | `$D` | Flight record directory entry | yes | 0x04 | | `$I` | Per-sensor configuration | optional | — | | `$W` | Reserved / unknown | optional | — | | `$L` | End-of-header / data length | yes | 0x10 | The seven required masks (0x01 + 0x02 + 0x04 + 0x08 + 0x10 + 0x20 + 0x40 = `0x7F` = 127) must all be set; otherwise the parser reports "No Flight Data in Download!". ### 2.2 `$U` — Aircraft / Serial ``` $U, SN38947*75 ``` * Field 1: tail number or device serial. Free-form ASCII, leading spaces stripped. Stored as `user_name`. ### 2.3 `$A` — Limits ``` $A, BAT, OIL, TIT, CHTH, CHTL, EGT, OILH, ? $A, 16, 12, 500, 450, 60, -999999, 230, 100*6A ``` Indexes 1, 4, 5, 6, 7 are consumed: | Idx | Field | Meaning | Notes | |----:|-------------------|---------------------------------------------------------|--------------------------------| | 1 | `Bat_Limit` | Battery alarm threshold | stored as `int * 0.1` Volts | | 2 | (reserved) | parsed but unused | | | 3 | (reserved) | parsed but unused | | | 4 | `Cht_Limit` | CHT high alarm | °F or °C per `$C` flag | | 5 | `Cht_Low` | CHT low alarm | `-999999` ⇒ disabled | | 6 | `Egt_Limit` | EGT differential alarm | °F or °C | | 7 | `Oil_Limit` | Oil temperature high alarm | | | 8 | (reserved) | parsed but unused | | ### 2.4 `$F` — Fuel-flow Unit ``` $F, fUnit, k1, k2, k3, k4*hh $F,0,50,0,2990,2990*6F ``` Only field 1 (`fUnit`) is consumed. Values: | `fUnit` | Meaning | Effect on data scaling | |--------:|----------------------|-------------------------------------------------| | 0 | gallons (US) / G·hr | FF / USD use scale=10 (one decimal place) | | ≥ 1 | pounds, kg, liters | FF / USD use scale=1 (integer) | ### 2.5 `$T` — Download date and time ``` $T, MM, DD, YY, HH, MM, SSS*hh $T, 10, 3, 22, 10, 32, 69*… (Oct 3, 2022 10:32:00.069) ``` Six numeric fields: | Idx | Field | Range | Notes | |----:|------------|--------|------------------------------------------------| | 1 | month | 1-12 | | | 2 | day | 1-31 | | | 3 | year | 0-99 | `<75 → 20YY`, otherwise `19YY` | | 4 | hour | 0-23 | | | 5 | minute | 0-59 | | | 6 | seconds-ms | 0-… | `secs = N // 1000`, `ms = N % 1000` | ### 2.6 `$C` — System Configuration ``` $C, model, Cfg_Word_packed, eng_mask, sw_extras, oat_mask, …, sw_version[, build, beta]*hh $C,830,63741,32273,1536,16610,120,352,2,0*60 ``` Field 1 is the device model code. Known values: | Model | Class | Twin? | |------:|------------------------|:-----:| | 700 | EDM-700 (legacy) | no | | 711 | EDM-711 | no | | 730 | EDM-730 | no | | 740 | EDM-740 | no | | 760 | EDM-760 (twin) | yes | | 790 | EDM-790 | yes | | 800 | EDM-800 | no | | 830 | **EDM-830** | no | | 831 | EDM-831 | no | | 900 | EDM-900 | no | | 930 | EDM-930 | no | | 950 | EDM-950 | no | | 960 | EDM-960 | yes | Derived flags: * `Edm_Typ = (Model >= 900)` initial classification (newer-generation units use 2-byte words in data records). For EDM-830 the `$P` record sets `Edm_Typ = true` regardless. **EDM-830/831 may flip back to `Edm_Typ_actual = false` per-flight via the XOR-zero check described in §3.2.2.** * `Twin_Flg` is true for models 760, 790, 960. * `eng_deg_str` = `"F"` if `(field3 & 0x1000)` else `"C"`. * `oat_deg_str` = `"F"` if `(field5 & 0x2000)` else `"C"`. Field 2 (the packed `Cfg_Word`) is split into two bytes used as a secondary search key when looking up flight records by config word. The high byte is `Cfg_High_Byt`, the low byte `Cfg_Low_Byt`. The last 1–3 numeric fields encode software identification: | Number of fields | sw_version | build | beta | |-----------------:|-----------|-------|------| | 8 (no build/beta)| last | "-1" | "-1" | | 9 | last-1 | last | last (one extra) | | 10 (with both) | last-2 | last-1| last | ### 2.7 `$P` — Protocol ID ``` $P, 2*6E ``` Sets `Protocol_Id` (integer). The decoder distinguishes: * `Protocol_Id == 2` ⇒ binary data records use **2-byte words** and **additive (mod-256) checksums**. * otherwise ⇒ legacy XOR checksums. ### 2.8 `$H` — Header flags ``` $H, flags*hh $H,0*54 ``` Single integer; bits 7 and 8 form `Fvl_Bit` (fuel-level reporting mode), exposed via `GetFuelLevelBit`. ### 2.9 `$D` — Flight directory entry ``` $D, flight_id, halfsize*hh $D, 1224, 208*7B ``` * `flight_id`: 16-bit flight number (1-65535). * `halfsize`: half the flight's binary size in bytes — i.e. number of 16-bit words. The actual binary length is `halfsize * 2`. `$D` records appear in the same order in which the flights are stored in section 2. The number of `$D` records sets `Flt_Cnt`. ### 2.10 `$I` — Per-sensor info (EDM-930 special) Only EDM-930 with SW version 107 / build ≥ 859 reads the optional CRB enable bit out of a `$I` record whose first field equals `96`. EDM-830 files normally contain `$I` records but their content is ignored. ### 2.11 `$L` — Length / End of header ``` $L, n*hh $L, 28*4A ``` Last record of the ASCII header. The byte immediately following the trailing `\r\n` is `Data_Start`, the offset of the first byte of the first flight record. The numeric field `n` is **not** consumed by the parser (it advances by the fixed `cnt = 5` bytes anyway). > **Implementer's note.** The `$U` user record carries no `, ` after > the dollar tag in some firmware versions. The parser strips > whitespace from individual fields before parsing. --- ## 3. The Binary Flight-Data Section (Section 2) Section 2 starts at `Data_Start` and is the concatenation of N flight records, where N is the number of `$D` entries. For each flight: ``` flight_record ::= flight_header data_record* ``` Both halves are present even when the flight is empty (the per-flight header may also be missing in extreme edge cases — recovery code handles 1-byte alignment slip and falls back to a `Cfg_High_Byt`/`Cfg_Low_Byt` scan). ### 3.1 Flight-record alignment heuristics Because each data record is a variable-length compressed record and the firmware occasionally flushes an odd number of bytes, neighbouring flights may be off by ±1 byte. Recovery procedure: 1. Read 2 bytes at the expected start, compare to `$D` flight_id. 2. If mismatch, retry at `start - 1`. On match, slide the start of *every* subsequent flight by `-1`. 3. If still no match, scan forward from `Data_Start` for the sequence `Cfg_High_Byt, Cfg_Low_Byt` and treat the two bytes *before* it as flight_id. 4. Once all flights are located, recompute each flight's actual size as `next.start - this.start`. Any flight whose ID still cannot be matched is marked `found=false` and emitted in the log as `BAD Flt#…`. ### 3.2 Flight Header The flight header is the first record of every flight. Layout (byte-stream, big-endian for words / longs): | Offset (decimal) | Size | Field | Notes | |-------------------------------|------:|--------------------------|----------------------------------------------------| | 0 | 2 | flight_id | matches `$D` | | 2 | 2 | `Cfg_Word[0]` | sensor-enable bitmap, low set | | 4 | 2 | `Cfg_Word[1]` | sensor-enable bitmap | | 6 | 2 | `Cfg_Word[2]` | (only if `Edm_Typ_actual`) | | 8 | 2 | `Cfg_Word[3]` | (only if `Edm_Typ_actual`) | | 10 *(or 6 in legacy form)* | 2 | `Cfg_Word[4]` | (only if "new header", see §3.2.1) | | 12 | 4 | `Latitude_Start` | (only if "new header" + GPS-bearing model) | | 16 | 4 | `Longitude_Start` | (only if "new header" + GPS-bearing model) | | ... + 0 | 1 | `fuelunit` (`num3`) | mirrors `$F` value | | ... + 1 | 1 | `horsepower` (`num4`) | rated HP | | ... + 2 | 2 | `Record_Interval` | secs/sample (typ. 1 or 6) | | ... + 4 | 2 | date_word (packed) | see §3.2.3 | | ... + 6 | 2 | time_word (packed) | see §3.2.3 | | ... + 8 | 1 | header checksum | makes record validate | For EDM-830 with `Protocol_Id == 2`, the flight-header record has a total length of **29 bytes** (2 + 2·5 + 4 + 4 + 1 + 1 + 2 + 2 + 2 + 1). #### 3.2.1 "Old vs. new" header (`isNewHdr`) EDM-830/831 (and 711/730/740) optionally carry GPS lat/lon and an extra `Cfg_Word[4]` in their flight header. The decoder selects the extended layout via the heuristic: ``` flag = TRUE if (binfile[Fptr+19] == binfile[Fptr+21] AND binfile[Fptr+20] == binfile[Fptr+22]) OR (binfile[Fptr+21..24] == binfile[Fptr-8..-5]) OR (binfile[Fptr+11] == binfile[Fptr+13]) OR (binfile[Fptr+12] == binfile[Fptr+14]) ``` If `flag` is true *and* the model is one of `{711, 730, 740, 830, 831}`, the header includes 2 + 4 + 4 = 10 extra bytes for `Cfg_Word[4]`, `Latitude_Start`, `Longitude_Start`. #### 3.2.2 EDM-830 `Edm_Typ_actual` flip-back For EDM-830/831 specifically, the decoder computes the XOR of the first 15 header bytes; if the XOR is 0 *and* bytes Fptr+9 and Fptr+10 are equal *and* byte Fptr+11 is `0x00`, the flight header is treated as the legacy short form (`Edm_Typ_actual = false`). Under this mode `Cfg_Word[2..4]` are **not** read from the file (they are zero-initialized) and subsequent data records use the 1-byte word form (§3.3.1). #### 3.2.3 Date / time encoding Both fields are 16-bit big-endian words. ``` date_word = (year_2digit << 9) | (month << 5) | day day = date_word & 0x1F month = (date_word >> 5) & 0x0F year2d = date_word >> 9 year = year2d + (year2d < 75 ? 2000 : 1900) time_word = (hour << 11) | (minute << 5) | (second / 2 ?) second_field = time_word & 0x1F (× 2 in firmware) minute = (time_word >>5) & 0x3F hour = time_word >> 11 ``` Empirically the effective second resolution is **2 seconds**. #### 3.2.4 Header checksum The trailing byte makes the running checksum of the entire flight header come out to zero. The checksum mode is selected per §3.4: * additive mod-256 if `Protocol_Id == 2` (EDM-830 default), or any of the firmware-version overrides for 700/760/800. * XOR otherwise. ### 3.3 Data records After the flight header, the rest of the flight is a stream of variable-length, delta-compressed data records: ``` data_record ::= word1 word2 ; sample-mask (must equal) num7 ; multiplier / repeat count ctl_byte_block ; up to 16 control bytes sgn_byte_block ; up to 14 sign bytes (idx 6,7 skipped) delta_byte_block ; one delta byte per set bit chksum ; 1 byte ``` #### 3.3.1 Sample mask `word1` / `word2` * If `Edm_Typ_actual` is **true** (modern: EDM-830 default, EDM-9xx, etc.): `word1` and `word2` are 2-byte big-endian. * If `Edm_Typ_actual` is **false** (legacy or EDM-830 in flip-back mode): both are single bytes. The two values **must be identical**; otherwise the record is invalid ("Invalid Data Record"). Each set bit of `word1` enables one control-byte slot in §3.3.3. #### 3.3.2 Repeat count `num7` A single byte read after the sample mask. Semantics: * `num7 == 0` ⇒ this record is a brand-new sample; parse normally. * `num7 != 0` ⇒ the *previous* record's sensor values are valid for `num7` additional samples. The decoder rewinds `Fptr` to the start of this record, decrements `Mult_Cnt`, and emits the cached row again. Only the first encounter actually consumes the file bytes. This is the format's main long-run compression — long stretches of constant readings collapse to a single record plus a counter. #### 3.3.3 Control-byte block For each `i` in `0..15`: if bit `i` of `word1` is 1, read **one byte** into `Ctrl_Byte[i].ctl_byt_idx`; otherwise mark `Ctrl_Byte[i].exist = false` and do not consume any byte. Each control byte is itself an 8-bit bitmap that tells the decoder which bits of the corresponding `Data_Bytes[i, *]` row are about to receive delta updates. #### 3.3.4 Sign-byte block Loop `i` in `0..15`, but **skip `i==6` and `i==7`**. For each `i` where bit `i` of `word1` is 1, read **one byte** into `Ctrl_Byte[i].sgn_byt_idx`. The decoder later aliases the sign-bytes for indices 6 and 7: | Index | Aliased to | |------:|-------------------------------| | 6 | `Ctrl_Byte[0].sgn_byt_idx` | | 7 | `Ctrl_Byte[3].sgn_byt_idx` | This is because indices 6 and 7 carry the "high byte" extensions of the EGT/RPM/etc. sensors anchored at indices 0 and 3 — they share their sign source. #### 3.3.5 Delta-byte block For each `Ctrl_Byte[i]` with `exist == true`, and for each bit `j` in `0..7`, if bit `j` of `ctl_byt_idx` is 1, read **one byte** from the file (`num13`). This is the per-sample delta for sensor slot `Data_Bytes[i, j]`. The byte is multiplied by a sensor-specific gain `num11` and stored: ``` Data_Bytes[i, j].value = num13 * num11 Data_Bytes[i, j].is_valid = (num13 != 0) Data_Bytes[i, j].sign = (sgn_byte_for_i[j] == 1) ``` `num11` defaults to 1, but special slots upshift by 256 (effectively treating two control-byte slots as a 16-bit pair for a single sensor): | `i` | `j` | `num11` | `num12` mod | Used for | |----:|-----------:|--------:|--------------------------------------------|--------------------------| | 5 | 2 or 4 | 256 | `num12 /= 2` | RPM high byte (L/R) | | 6 | any | 256 | unchanged | EGT 1-6 high byte (L) | | 7 | any | 256 | unchanged | EGT 1-6 high byte (R) | | 9 | 4 or 5 | 256 | `num12 /= 16` | Turbine NG/NP high | | 9 | 7 | 256 | unchanged | engine HRS high | | 10 | 1 or 2 | 256 | `num12 *= 32` | LAT/LNG extension (turbo)| | 12 | 4 or 5 | 256 | `num12 /= 16` | Right NG/NP high (twin) | | 12 | 7 | 256 | unchanged | Right HRS high (twin) | | 13 | 4, 5 or 6 | 256 | `num12 /= 16` | EDM-960 EGT 7-9 high (L) | | 14 | 4, 5 or 6 | 256 | `num12 /= 16` | EDM-960 EGT 7-9 high (R) | The signed result `num15` for sensor decoding (§3.3.7) is: ``` num15 = +Data_Bytes[lo].value, negated if Data_Bytes[lo].sign if hi exists: num15 += (-Data_Bytes[hi].value if Data_Bytes[hi].sign else +Data_Bytes[hi].value) ``` #### 3.3.6 Trailing checksum A single byte is consumed. `calc_chksum(L)` over the entire record (length = `Fptr_now - Fptr_start`) must produce 0. #### 3.3.7 Per-sensor reconstruction (running totals) Each *output* sensor is described by a `header_struct`: ``` sensor_name // e.g. "Left EGT 1" hdr_str // CSV column name, e.g. "E1" cfg_byt_idx // index into Cfg_Word (0 or 1, occasionally 2-4) cfg_bit_idx // bit position; sensor is enabled iff the bit is set scale_val // 1.0 → integer output, 10.0 → fixed-point m_lo_byt_idx,m_lo_bit_idx // primary delta source in Data_Bytes m_hi_byt_idx,m_hi_bit_idx // optional high-byte source running_total // initialised to 240.0 (or 0.0 for HP and LAT/LNG) ``` For every record: ``` delta = signed value derived from m_lo (and optional m_hi) — see §3.3.5 running_total += delta output_value = round(running_total) ; integer presentation or running_total / scale_val ; fixed-point with one decimal ``` Special cases: * `MARK` (`m_lo_byt_idx=2, m_lo_bit_idx=0`) is interpreted by its low 3 bits and bit 3: | val & 7 | symbol | side-effect | |--------:|--------|---------------------------------------------| | 0 | (none) | | | 1 | `X` | pilot mark | | 2 | `[` | start fast-record window (Record_Interval=1)| | 3 | `]` | end fast-record window (restore interval) | | 4 | `<` | alternate start fast-record | | 5 | `>` | alternate end fast-record | Bit 3 (`val & 8`) toggles the `FuelTankNumberingFlag`. * `DIF` / `LDIF` outputs `max(EGT) - min(EGT)` over the relevant EGT group within the current record. * `RDIF` outputs the same for the right bank (twin engines only). * `LAT` / `LNG` are formatted as `N|S` / `E|W` followed by `dd.mm.ss` with 6000 minutes-per-degree fixed-point: x = abs(running_total) deg = x // 6000 rem = x % 6000 out = "{N|S|E|W}{deg:02|03}.{rem//100:02}.{rem%100:02}" * `HRS` (engine hours) uses a sign-reversed delta on the first HOB record only (`firstHOBRec`). * Sensors whose `lo` slot is not valid in the current record fall back to "NA" if they also have no `hi` slot, otherwise the high slot's validity is checked. #### 3.3.8 Output column header (`Col_Hdr_Str`) The CSV-style column list is built once per flight. For every entry in `Header[]`, the sensor is included only if its config bit is set: ``` ( Cfg_Word[ cfg_byt_idx ] AND (1 << cfg_bit_idx) ) != 0 ``` This means `Cfg_Word[0..4]` collectively form the **sensor-enable bitmap** for the flight. Non-applicable sensors (e.g. `DIF`/`CLD` on turbine engines) are filtered out. ### 3.4 Checksum algorithm The same routine is used for both the flight-header and each data record: ``` def calc_chksum(buf): if (Model == 760 and SW_Version >= 144) or (Model == 700 and SW_Version >= 296) or (Model == 800 and SW_Version >= 300) or (Edm_Typ_actual and Protocol_Id == 2): return sum(buf) & 0xFF # additive mod-256 else: c = 0 for b in buf: c ^= b return c # XOR ``` **For EDM-830 with `Protocol_Id == 2` the additive variant is used** for both the flight-header and every data record. A valid record always sums to `0` (mod 256). ### 3.5 The `Data_Bytes[16, 8]` matrix The decoder maintains a 16×8 matrix of delta slots. Each slot (`byt_idx`, `bit_idx`) is a `data_byte_struct`: ``` name // diagnostic, e.g. "CBYT5.1" value // most-recent delta byte (already multiplied by num11) sign // delta direction (additive vs subtractive) is_valid // false ⇒ no delta arrived this record ``` The matrix is *reset to zero* on entry to each new record. Slots that are not refreshed by the current record stay at zero, which means "no change" once running totals have already been applied. Sensors look up their `(lo)` and optional `(hi)` slots in this matrix per §3.3.7. The naming scheme `"CBYTx.y"` is purely diagnostic. --- ## 4. End-of-data Marker (`$E`) ``` $E,4*5D\r\n ``` A standalone ASCII record that immediately follows the last flight's last data record. Parsing of the binary section stops at `Data_Start + Σ(flight_size)` regardless, but `$E` is helpful for file scanners. The numeric body of `$E` is the count of trailing sections (typically `4` for a complete file: config, lat?, lng?, $V). Implementations may locate the end of the binary section by scanning forward for `$E,` after summing all `$D` halfsizes. --- ## 5. Configuration XML Region (Section 4) Between the `$E,…\r\n` line and the trailing `$V,…\r\n` line lives a region of binary data which is the unit's full XML configuration, **XOR-encoded with the constant `0x02`**. To recover the plaintext, XOR every byte in the region with `0x02`: ``` plain[i] = enc[i] XOR 0x02 ``` After decoding, the region is a single UTF-8 XML document, framed with `0x0D 0x0A` (CRLF) line endings — these appear in the encoded file as `0x0F 0x08`. A few reproducible properties confirm the encoding: * Every occurrence of the byte pair `0x0F 0x08` inside the region maps to a CRLF in the decoded XML. * The first three encoded bytes are junk (a 3-byte prefix between `$E,…\r\n` and the XML); the actual ` … … ``` The XML is a snapshot of the unit's settings at download time. The cipher is trivial enough that it appears intended to deter casual inspection rather than provide security. Implementations can usually skip this region entirely after locating `$V`. --- ## 6. The `$V` Trailer The very last record of the file is a single CRLF-terminated ASCII line: ``` $V,Created by VER BLD BETA *\r\n ``` Example: ``` $V,Created by EDM830 VER 3.52 BLD 002 BETA 0*57 ``` The decoder finds it by scanning the **last 100 bytes** of the file for the `$V` magic. Parsed fields: | Token | Stored as | Notes | |--------------|----------------------------|--------------------------------| | `EDM830` | `edm_model` | matches the device family | | `3.52` | `edm_software_version` | dotted firmware version | | `002` | `edm_software_build` | build number, zero-padded | | `0` | `edm_software_is_beta` | `1` = beta, `0` = production | If the build number is missing in `$C` but the `$V` string contains `454`, the decoder retroactively sets `Build_Num = "454"`, preserving compatibility with very old exports. --- ## 7. Sensor / `Cfg_Word` Mapping (EDM-830, single) Each row gives the column name, which `Cfg_Word` bit enables it, and the `Data_Bytes[lo]` (and optional `[hi]`) source. | Sensor | hdr_str | Cfg word [byte,bit] | lo (byte,bit) | hi (byte,bit) | Scale | |--------------|---------|---------------------|---------------|---------------|------:| | Left EGT 1 | E1 | (0, 2) | (0, 0) | (6, 0) | 1 | | Left EGT 2 | E2 | (0, 3) | (0, 1) | (6, 1) | 1 | | Left EGT 3 | E3 | (0, 4) | (0, 2) | (6, 2) | 1 | | Left EGT 4 | E4 | (0, 5) | (0, 3) | (6, 3) | 1 | | Left EGT 5 | E5 | (0, 6) | (0, 4) | (6, 4) | 1 | | Left EGT 6 | E6 | (0, 7) | (0, 5) | (6, 5) | 1 | | Left EGT 7 | E7 | (0, 8) | (3, 0) | (7, 0) | 1 | | Left EGT 8 | E8 | (0, 9) | (3, 1) | (7, 1) | 1 | | Left EGT 9 | E9 | (0, 10) | (3, 2) | (7, 2) | 1 | | Left CHT 1 | C1 | (0, 11) | (1, 0) | — | 1 | | Left CHT 2 | C2 | (0, 12) | (1, 1) | — | 1 | | Left CHT 3 | C3 | (0, 13) | (1, 2) | — | 1 | | Left CHT 4 | C4 | (0, 14) | (1, 3) | — | 1 | | Left CHT 5 | C5 | (0, 15) | (1, 4) | — | 1 | | Left CHT 6 | C6 | (1, 0) | (1, 5) | — | 1 | | Left CHT 7 | C7 | (1, 1) | (3, 3) | — | 1 | | Left CHT 8 | C8 | (1, 2) | (3, 4) | — | 1 | | Left CHT 9 | C9 | (1, 3) | (3, 5) | — | 1 | | Left TIT 1 | T1 | (1, 5) | (0, 6) | (6, 6) | 1 | | Left TIT 2 | T2 | (1, 6) | (0, 7) | (6, 7) | 1 | | OAT | OAT | (1, 9) | (2, 5) | — | 1 | | Left DIF | DIF | (0, 0) computed | — | — | 1 | | Left CLD | CLD | (0, 0) computed | (1, 6) | — | 1 | | Left CDT | CDT | (1, 7) | (2, 2) | — | 1 | | Left IAT | IAT | (1, 8) | (2, 3) | — | 1 | | Left MAP | MAP | (1, 14) | (5, 0) | — | 10 | | Left RPM | RPM | (1, 10) | (5, 1) | (5, 2) | 1 | | Left HP | HP | (1, 10) | (3, 6) | — | 1 | | Left FF * | FF | (1, 11) | (2, 7) | — | 10/1 | | Left OILP | OILP | (1, 13) | (2, 1) | — | 1 | | Left OILT | OILT | (1, 4) | (1, 7) | — | 1 | | BAT | BAT | (0, 0) (always) | (2, 4) | — | 10 | | Left USD * | USD | (1, 11) | (2, 6) | — | 10/1 | | MARK | MARK | (0, 0) (always) | (2, 0) | — | 1 | \* Scale of `FF` and `USD` is 10 if `fUnit == 0` (gallons), 1 otherwise. ### 7.1 Output formatting * Unscaled (`scale_val == 1.0`): emit `running_total` rounded to int. * Scaled by 10 (`scale_val == 10.0`): emit `f"{running_total/10:.1f}"`. Locale-specific decimal commas are rewritten to `.`. * `MARK` and `LAT/LNG` formatted per §3.3.7. --- ## 8. End-to-end decoding pseudocode ``` def decode_jpi(path): data = open(path,'rb').read() # --- Section 1: ASCII header --- pos = data.find(b'$U') if pos < 0: raise HeaderNotFoundError flights = [] while True: line, pos = read_dollar_line(data, pos) # NUL-tolerant scan tag = line.split(b',', 1)[0] match tag: case b'$U': user = line[3:] case b'$A': parse_limits(line) case b'$F': fUnit = int(field1(line)) case b'$T': download_dt = parse_T(line) case b'$C': model, sw, build, beta, … = parse_C(line) Edm_Typ = (model >= 900) Twin_Flg = model in (760,790,960) case b'$P': Protocol_Id = int(field1(line)); Edm_Typ=True case b'$H': Fvl_Bit = parse_H(line) case b'$D': flights.append((id,size := int(field2(line))*2)) case b'$L': data_start = pos break … # --- Section 2: binary flight data --- p = data_start for id, size in flights: end = p + size # -- flight header (29 bytes for EDM-830 P=2) -- flight_id = u16_be(data, p); p += 2 cfg = [u16_be(data, p+i*2) for i in range(2)]; p += 4 if Edm_Typ_actual: # see §3.2.2 cfg += [u16_be(data, p+i*2) for i in range(2)] # words 2,3 p += 4 if isNewHdr(data, p, flight_size): cfg.append(u16_be(data, p)); p += 2 if model in (711,730,740,830,831): lat0 = s32_be(data, p); p += 4 lon0 = s32_be(data, p); p += 4 fuelunit, hp = data[p], data[p+1]; p += 2 rec_interval = u16_be(data, p); p += 2 date_word = u16_be(data, p); p += 2 time_word = u16_be(data, p); p += 2 chksum = data[p]; p += 1 assert checksum_ok(...) # -- variable-length data records -- while end - p >= 5: w1 = read_word_or_byte() w2 = read_word_or_byte(); assert w1 == w2 mult = data[p]; p += 1 if mult and Mult_Cnt > 0: emit_cached_row(); Mult_Cnt -= 1; continue ctl = read_ctl_block(w1) sgn = read_sgn_block(w1) deltas = read_delta_block(ctl, sgn) assert checksum_ok(record_buffer) apply_deltas(deltas) emit_row() p = end # next flight # --- Section 3..5 --- e_idx = data.find(b'$E,', p) v_idx = data.rfind(b'$V,') cfg_xml = bytes(b ^ 0x02 for b in data[e_idx_end+1 : v_idx]) parse_v(data[v_idx:]) ``` --- ## 9. Quick-reference field map | Where | Field | Source | |--------------------------------|-----------------------|--------------------------------------------------| | `$U` field 1 | aircraft / serial | `$U, SN…` | | `$T` fields 1-6 | download date/time | `$T, MM, DD, YY, HH, MM, SSS` | | `$C` field 1 | EDM model code | `830` for EDM-830 | | `$C` last 1-3 fields | sw_version, build, β | see §2.6 | | `$P` field 1 | protocol id | `2` ⇒ 2-byte words + additive checksum | | `$D` fields 1, 2 | flight id, halfsize | binary size = `halfsize * 2` | | `$L` | end of header | binary section starts immediately after | | flight hdr bytes 0-1 | flight_id (BE u16) | matches `$D` | | flight hdr bytes 2-9 (or 2-19) | Cfg_Word[0..4] | sensor-enable bitmap (and optional GPS) | | flight hdr bytes -9..-2 | fuelunit/hp/intvl/dt | see §3.2 | | flight hdr last byte | additive checksum | sum of record == 0 mod 256 | | data record header | sample-mask + repeat | §3.3.1, §3.3.2 | | data record body | ctl + sgn + deltas | §3.3.3 – §3.3.5 | | data record trailer | additive checksum | §3.4 | | `$E,…\r\n` | end of binary data | offset = `Data_Start + Σ(flight_size)` | | `$E…$V` body | XOR-0x02 XML config | §5 | | `$V,Created by …` | software version line | §6 | --- ## 10. Worked example Annotated 29-byte flight header (EDM-830, `Protocol_Id = 2`, extended-header form with GPS), to illustrate the byte-by-byte layout from §3.2: ``` 04 c8 flight_id = 1224 f8 fd Cfg_Word[0] = 0xF8FD 7e 11 Cfg_Word[1] = 0x7E11 06 00 Cfg_Word[2] = 0x0600 40 e2 Cfg_Word[3] = 0x40E2 00 78 Cfg_Word[4] = 0x0078 (extended hdr) 00 03 ad 04 Latitude_Start = +240900 ff f5 35 1c Longitude_Start = -707300 00 fuelunit 6c HP = 108 00 06 Record_Interval = 6 s 2d 38 date 2022-09-24 5b f0 time 11:31:24 (≈) f5 checksum (sum of all 29 bytes ≡ 0 mod 256) ``` The 29-byte sum modulo 256 checks to zero, confirming the additive checksum mode is correct for EDM-830 + `Protocol_Id = 2`. jpi_analyzer/000077500000000000000000000000001517535535700135655ustar00rootroot00000000000000jpi_analyzer/__init__.py000066400000000000000000000010511517535535700156730ustar00rootroot00000000000000"""JPI EDM Flight Data Analyzer — parser + CSV export.""" from jpi_analyzer.parser import JpiFile, FlightRecord from jpi_analyzer.decoder import FlightDecoder, DecodedFlight from jpi_analyzer.metrics import MetricDef, METRIC_LIBRARY, axis_range_for, unit_for from jpi_analyzer.exporter import export_flight_csv, export_flights_csv __all__ = [ "JpiFile", "FlightRecord", "FlightDecoder", "DecodedFlight", "MetricDef", "METRIC_LIBRARY", "axis_range_for", "unit_for", "export_flight_csv", "export_flights_csv", ] jpi_analyzer/cli.py000066400000000000000000000135611517535535700147140ustar00rootroot00000000000000"""Minimal Click-based CLI: list flights and export CSV.""" from __future__ import annotations import sys from typing import List, Optional, Sequence import click from jpi_analyzer.decoder import DecodedFlight, FlightDecoder from jpi_analyzer.exporter import export_flights_csv from jpi_analyzer.parser import JpiFile @click.group() @click.option("--file", "-f", "file_path", required=True, type=click.Path(exists=True, dir_okay=False), help="Path to a .JPI file.") @click.pass_context def main(ctx: click.Context, file_path: str) -> None: """Parse a JPI EDM file and export flights to CSV.""" ctx.ensure_object(dict) ctx.obj["file_path"] = file_path @main.command() @click.pass_context def info(ctx: click.Context) -> None: """Print file metadata and a one-line summary per flight.""" jpi = JpiFile.open(ctx.obj["file_path"]) click.echo(f"File: {jpi.path}") click.echo(f"Tail / serial: {jpi.tail_number}") click.echo(f"Model: EDM{jpi.model} sw={jpi.sw_version} " f"build={jpi.build_num} proto={jpi.protocol_id}") click.echo(f"Created: {jpi.file_created_at}") click.echo(f"Eng/OAT temp: {jpi.eng_deg} / {jpi.oat_deg}") click.echo(f"Flights: {len(jpi.flights)}") click.echo("") for fid, fr in jpi.flights.items(): df = FlightDecoder(jpi, fr).decode() click.echo(f" #{fid:5d} start={df.start_datetime} " f"records={df.n_records:5d} " f"duration={df.duration} valid={df.valid}") @main.command() @click.option("--flights", "-F", "flight_spec", default=None, help="Flights to include: 'all', comma-separated IDs/indices " "(e.g. '1224,1225' or '1,3,5'). Omit for an interactive picker.") @click.option("--metrics", "-m", "metric_spec", default="all", help="Metric codes: 'all' or comma-separated (e.g. 'E1,C1,MAP,RPM').") @click.option("--out-dir", "-o", default="./jpi_csv", type=click.Path(file_okay=False), help="Directory to write CSVs into.") @click.pass_context def csv(ctx: click.Context, flight_spec: Optional[str], metric_spec: str, out_dir: str) -> None: """Export flights to CSV (one file per flight).""" jpi = JpiFile.open(ctx.obj["file_path"]) if flight_spec is None: flight_spec = _prompt_for_flights(jpi) fids = _resolve_flights(jpi, flight_spec) flights = [FlightDecoder(jpi, jpi.get_flight(f)).decode() for f in fids] available = _common_metrics(flights) metrics = _resolve_metrics(metric_spec, available) paths = export_flights_csv(flights, out_dir, metrics) click.echo(f"Wrote {len(paths)} files to {out_dir}:") for p in paths: click.echo(f" {p}") def _prompt_for_flights(jpi: JpiFile) -> str: """Show a numbered flight list and let the user pick. Falls back to 'all' if stdin isn't a TTY (so piped/scripted use still works without hanging on `input()`). """ if not sys.stdin.isatty(): return "all" flight_ids = list(jpi.flights.keys()) click.echo(f"\n{len(flight_ids)} flight(s) in {jpi.path}:") for i, fid in enumerate(flight_ids, start=1): df = FlightDecoder(jpi, jpi.flights[fid]).decode() when = (df.start_datetime.strftime("%Y-%m-%d %H:%M") if df.start_datetime else " — ") dur = str(df.duration).split(".")[0] if df.valid else "—" click.echo(f" [{i:>3}] #{fid:<6} {when} " f"records={df.n_records:>5} duration={dur}") click.echo("") raw = click.prompt( "Select flights (comma-separated indices, IDs, or 'all')", default="all", show_default=True, type=str, ).strip() if not raw or raw.lower() in ("all", "*"): return "all" out: List[str] = [] for part in raw.split(","): part = part.strip() if not part: continue try: n = int(part) except ValueError: raise click.BadParameter(f"Bad token: {part!r}") if n in flight_ids: out.append(str(n)) elif 1 <= n <= len(flight_ids): out.append(str(flight_ids[n - 1])) else: raise click.BadParameter( f"{n} is neither a flight ID nor a valid index 1..{len(flight_ids)}") return ",".join(out) if out else "all" def _resolve_flights(jpi: JpiFile, spec: str) -> List[int]: spec = spec.strip().lower() if spec in ("all", "*"): return list(jpi.flights.keys()) flight_ids = list(jpi.flights.keys()) out: List[int] = [] for part in spec.split(","): part = part.strip() if not part: continue try: n = int(part) except ValueError: raise click.BadParameter(f"Bad flight id: {part!r}") if n in jpi.flights: out.append(n) elif 1 <= n <= len(flight_ids): out.append(flight_ids[n - 1]) else: raise click.BadParameter( f"Flight {n} not in file (IDs: {flight_ids}, " f"or use 1..{len(flight_ids)} as an index)") return out def _resolve_metrics(spec: str, available: Sequence[str]) -> List[str]: spec = spec.strip().lower() if spec in ("all", "*"): return list(available) out: List[str] = [] for part in spec.split(","): code = part.strip().upper() if not code: continue if code not in available: raise click.BadParameter( f"Unknown metric {code!r}; available: {', '.join(available)}") out.append(code) return out def _common_metrics(flights: Sequence[DecodedFlight]) -> List[str]: seen: List[str] = [] for f in flights: for c in f.available_codes: if c not in seen: seen.append(c) return seen if __name__ == "__main__": main(obj={}) jpi_analyzer/decoder.py000066400000000000000000000402771517535535700155560ustar00rootroot00000000000000"""Decode an individual flight's binary block into time-series per metric. This is a Python port of the EzTrends2 `Decomp.ReadRecord` routine. Each flight record consists of a small header (Cfg_Word values + interval + start date/time) followed by a stream of delta-encoded data records. Every metric maintains a `running_total` updated by the deltas in each record; the absolute engineering value is `running_total / scale_val`. """ from __future__ import annotations from dataclasses import dataclass, field from datetime import datetime, timedelta from typing import Dict, List, Optional, Tuple from jpi_analyzer.metrics import MetricDef, headers_for_model from jpi_analyzer.parser import FlightRecord, JpiFile class _ByteReader: __slots__ = ("data", "ptr") def __init__(self, data: bytes, start: int = 0): self.data = data self.ptr = start def byte(self) -> int: if self.ptr >= len(self.data): return -1 v = self.data[self.ptr] self.ptr += 1 return v def word(self) -> int: if self.ptr + 1 >= len(self.data): return -1 v = (self.data[self.ptr] << 8) | self.data[self.ptr + 1] self.ptr += 2 return v def long(self) -> int: if self.ptr + 3 >= len(self.data): return -1 v = ((self.data[self.ptr] << 24) | (self.data[self.ptr + 1] << 16) | (self.data[self.ptr + 2] << 8) | self.data[self.ptr + 3]) self.ptr += 4 return v @dataclass class DecodedFlight: """Decoded time-series for a single flight.""" flight_id: int start_datetime: Optional[datetime] = None record_interval: int = 1 # seconds between samples fuel_unit: int = 0 horsepower_setting: int = 0 available_codes: List[str] = field(default_factory=list) timestamps: List[datetime] = field(default_factory=list) series: Dict[str, List[Optional[float]]] = field(default_factory=dict) valid: bool = False error: str = "" def metric_codes(self) -> List[str]: return list(self.available_codes) def metric_values(self, code: str) -> List[Optional[float]]: return self.series.get(code, []) def metric_values_with_time(self, code: str): return list(zip(self.timestamps, self.series.get(code, []))) @property def duration(self) -> timedelta: if not self.timestamps: return timedelta(0) return self.timestamps[-1] - self.timestamps[0] @property def n_records(self) -> int: return len(self.timestamps) class FlightDecoder: """Decode one flight from a JpiFile into a DecodedFlight.""" def __init__(self, jpi: JpiFile, flight: FlightRecord): self.jpi = jpi self.flight = flight self.headers: List[MetricDef] = [] # Active metrics are those whose cfg bit is enabled in this flight's Cfg_Word self.active_headers: List[MetricDef] = [] self.cfg_word: List[int] = [0, 0, 0, 0, 0, 0, 0] self.running_total: Dict[str, float] = {} self.is_new_hdr_format = False self.lat_start = 0 self.lng_start = 0 self.use_xor_checksum = True self._has_16bit_records = False # word1/word2 are 16-bit (Edm_Typ_actual) self.edm_typ_actual = False def decode(self) -> DecodedFlight: out = DecodedFlight(flight_id=self.flight.id, fuel_unit=self.jpi.fuel_unit) if not self.flight.found or not self.flight.data: out.error = "Flight not found in file" return out reader = _ByteReader(self.flight.data, 0) # ---- flight header ---- try: self._scan_flight_header(reader, out) except (ValueError, IndexError) as exc: out.error = f"Flight header parse failed: {exc}" return out # ---- choose & filter headers based on Cfg_Word ---- self.headers = headers_for_model( self.jpi.model, self.jpi.fuel_unit, self.jpi.twin_flag, self.edm_typ_actual, ) self.active_headers = [ h for h in self.headers if h.code == "DIF" or self._cfg_enabled(h) ] out.available_codes = [h.code for h in self.active_headers] # Initialize running totals for h in self.headers: self.running_total[h.code] = 240.0 if "HP" in self.running_total: self.running_total["HP"] = 0.0 if "LAT" in self.running_total: self.running_total["LAT"] = float(self.lat_start) if "LNG" in self.running_total: self.running_total["LNG"] = float(self.lng_start) # Allocate empty series out.series = {h.code: [] for h in self.active_headers} out.timestamps = [] out.record_interval = max(int(self.cfg_interval), 1) out.start_datetime = self.start_datetime # ---- iterate data records ---- ts = self.start_datetime end_ptr = len(self.flight.data) last_record: Optional[Dict[str, float]] = None rep_remaining = 0 while reader.ptr < end_ptr - 4: try: rec_values, rep_count = self._read_one_record(reader) except (ValueError, IndexError): break if rec_values is None: break if rep_count > 0 and last_record is not None: # Replay the previous record `rep_count` times before applying this one for _ in range(rep_count): if ts is not None: out.timestamps.append(ts) ts = ts + timedelta(seconds=out.record_interval) else: out.timestamps.append(None) for code, vals in out.series.items(): vals.append(last_record.get(code)) # Now record the values from this new decoded record if ts is not None: out.timestamps.append(ts) ts = ts + timedelta(seconds=out.record_interval) else: out.timestamps.append(None) for code, vals in out.series.items(): vals.append(rec_values.get(code)) last_record = rec_values out.valid = bool(out.timestamps) if not out.valid and not out.error: out.error = "No decodable data records" return out # ---------- internals ---------- def _scan_flight_header(self, r: _ByteReader, out: DecodedFlight) -> None: # First word is the flight ID — verify flight_id = r.word() if flight_id != self.flight.id: # Some files have an off-by-one r.ptr = max(r.ptr - 1, 0) flight_id = r.word() self.cfg_word[0] = r.word() self.cfg_word[1] = r.word() try: mv = int(float(self.jpi.model)) except ValueError: mv = 0 wide_models = (711, 730, 740, 830, 831) if self.jpi.edm_typ: # 14-byte XOR test: when zero, this is a "narrow" header (no cfg[2..4]) xor_test = 0 for i in range(15): xor_test ^= self.flight.data[i] edm_typ_actual = True if mv in wide_models and xor_test == 0: edm_typ_actual = False self.edm_typ_actual = edm_typ_actual if edm_typ_actual: self.cfg_word[2] = r.word() self.cfg_word[3] = r.word() self.is_new_hdr_format = self._detect_new_hdr(r) if self.is_new_hdr_format: self.cfg_word[4] = r.word() if mv in wide_models: self.lat_start = self._signed_long(r.long()) self.lng_start = self._signed_long(r.long()) self._has_16bit_records = True fuel_unit_byte = r.byte() horsepower = r.byte() self.cfg_interval = r.word() date_word = r.word() time_word = r.word() # Reserved/checksum r.byte() try: self.start_datetime = self._decode_date_time(date_word, time_word) except Exception: self.start_datetime = self.jpi.file_created_at out.horsepower_setting = horsepower if horsepower is not None and horsepower >= 0 else 0 # Determine checksum mode if (mv == 760 and self.jpi.sw_version >= 144) \ or (mv == 700 and self.jpi.sw_version >= 296) \ or (mv == 800 and self.jpi.sw_version >= 300) \ or (self.jpi.edm_typ and self.jpi.protocol_id == 2): self.use_xor_checksum = False else: self.use_xor_checksum = True def _detect_new_hdr(self, r: _ByteReader) -> bool: # Heuristic copied from EzTrends2: peek ahead to see if structure looks like # the new (lat/lng-bearing) header. If we run off the end, assume old. p = r.ptr try: d = self.flight.data if (d[p + 19] == d[p + 21] and d[p + 20] == d[p + 22]) or ( d[p + 21] == d[p - 8] and d[p + 22] == d[p - 7] and d[p + 23] == d[p - 6] and d[p + 24] == d[p - 5] ): return True if d[p + 11] == d[p + 13] or d[p + 12] == d[p + 14]: return True except IndexError: pass return False @staticmethod def _signed_long(v: int) -> int: if v >= 0x80000000: return v - 0x100000000 return v @staticmethod def _decode_date_time(date_word: int, time_word: int) -> Optional[datetime]: """JPI packs date as bits day(5)|month(4)|year(7) and time as h(5)|m(6)|s2(5). Seconds are stored in 2-second ticks. The original C# masks `time & 62` — i.e. bits 1-5 — which intentionally drops the LSB so the value is an even number 0..62. """ if date_word <= 0 or time_word <= 0: return None day = date_word & 0x1F month = (date_word >> 5) & 0x0F yy = (date_word >> 9) & 0x7F year = 2000 + yy if yy < 75 else 1900 + yy sec = time_word & 62 minute = (time_word >> 5) & 63 hour = (time_word >> 11) & 31 if sec >= 60: sec = 0 minute += 1 if minute >= 60: minute = 0 hour += 1 try: return datetime(year, month, day, hour % 24, minute, sec) except ValueError: return None def _cfg_enabled(self, hdr: MetricDef) -> bool: bidx = hdr.cfg_byt_idx bit = hdr.cfg_bit_idx if bidx == 0 and bit == 0: # "always-on" markers: BAT, MARK, DIF, etc. return True try: return (self.cfg_word[bidx] & (1 << bit)) != 0 except IndexError: return False def _read_one_record(self, r: _ByteReader) -> Tuple[Optional[Dict[str, float]], int]: """Return (values_per_code, replay_count).""" start_ptr = r.ptr if self._has_16bit_records: word1 = r.word() word2 = r.word() else: word1 = r.byte() word2 = r.byte() if word1 == -1 or word2 == -1 or word1 != word2: return None, 0 rep_count = r.byte() if rep_count == -1: return None, 0 # Read control bytes (one per set bit in word1) ctl = [{"exist": False, "ctl": 0, "sgn": 0} for _ in range(16)] mask = 1 for i in range(16): if (word1 & mask) != 0: ctl[i]["exist"] = True b = r.byte() if b == -1: return None, 0 ctl[i]["ctl"] = b mask <<= 1 # Read sign bytes — for indices in {0..5, 8..15} when their bit is set in word1. # (The C# expression `i<6 | i>7 && (word1 & m) != 0` parses as `(i<6 || i>7) && bit-set` # because `|` has higher precedence than `&&` in C# for booleans.) mask = 1 for i in range(16): if (i < 6 or i > 7) and (word1 & mask) != 0: b = r.byte() if b == -1: return None, 0 ctl[i]["sgn"] = b mask <<= 1 # Decode data bytes into Data_Bytes[byt_idx][bit_idx] data_bytes: Dict[int, Dict[int, Tuple[int, bool, bool]]] = { i: {j: (0, False, False) for j in range(8)} for i in range(16) } for byt_idx in range(16): if not ctl[byt_idx]["exist"]: continue ctl_idx = ctl[byt_idx]["ctl"] # Sign-byte source has a couple of cross-references if byt_idx == 6: sgn_idx = ctl[0]["sgn"] elif byt_idx == 7: sgn_idx = ctl[3]["sgn"] else: sgn_idx = ctl[byt_idx]["sgn"] mask = 1 for bit_idx in range(8): if (ctl_idx & mask) == 0: mask <<= 1 continue # Some byte-indices imply "double-byte" multipliers (×256) multiplier = 1 sign_check_mask = mask if byt_idx == 5 and bit_idx in (2, 4): multiplier = 256 sign_check_mask //= 2 elif byt_idx in (6, 7): multiplier = 256 elif byt_idx == 10 and bit_idx in (1, 2): multiplier = 256 sign_check_mask *= 32 elif byt_idx in (9, 12): if bit_idx in (4, 5): multiplier = 256 sign_check_mask //= 16 elif bit_idx == 7: multiplier = 256 elif byt_idx in (13, 14) and bit_idx in (4, 5, 6): multiplier = 256 sign_check_mask //= 16 v = r.byte() if v == -1: return None, 0 is_valid = v != 0 value = v * multiplier sign = (sgn_idx & sign_check_mask) != 0 if sign_check_mask else False data_bytes[byt_idx][bit_idx] = (value, is_valid, sign) mask <<= 1 # End-of-record filler byte + checksum byte r.byte() # filler # We skip strict checksum verification here — the C# code trusts the # XOR/sum, but mismatches abort the whole flight. To be tolerant of # quirks in older firmware we accept all records and rely on the data # itself. # Apply deltas to running totals & emit values out: Dict[str, float] = {} # First compute min/max EGT for DIF (left side only — single-engine) egt_values: List[int] = [] for hdr in self.active_headers: if hdr.m_lo_byt_idx < 0: continue lo = data_bytes[hdr.m_lo_byt_idx][hdr.m_lo_bit_idx] lo_val, lo_valid, lo_sign = lo delta = lo_val if lo_sign: delta = -delta if hdr.m_hi_byt_idx >= 0: hi = data_bytes[hdr.m_hi_byt_idx][hdr.m_hi_bit_idx] hi_val, hi_valid, hi_sign = hi delta = delta + hi_val if not hi_sign else delta - hi_val self.running_total[hdr.code] += delta number = round(self.running_total[hdr.code]) # Emit value if hdr.code == "DIF": continue # computed after EGT loop if not lo_valid: if hdr.m_hi_byt_idx < 0 or not data_bytes[hdr.m_hi_byt_idx][hdr.m_hi_bit_idx][1]: out[hdr.code] = None continue if hdr.code == "MARK": out[hdr.code] = float(number & 7) continue if hdr.scale_val == 1.0: out[hdr.code] = float(number) else: out[hdr.code] = number / hdr.scale_val if hdr.sensor_name.startswith("Left EGT") and lo_valid: egt_values.append(number) if "DIF" in out or any(h.code == "DIF" for h in self.active_headers): if egt_values: out["DIF"] = float(max(egt_values) - min(egt_values)) else: out["DIF"] = None return out, rep_count jpi_analyzer/exporter.py000066400000000000000000000033101517535535700160040ustar00rootroot00000000000000"""Export decoded flights to CSV.""" from __future__ import annotations import csv import os from typing import Iterable, List, Optional, Sequence from jpi_analyzer.decoder import DecodedFlight def _fmt_value(v) -> str: if v is None: return "" if isinstance(v, float): if v.is_integer(): return str(int(v)) return f"{v:.2f}" return str(v) def export_flight_csv( flight: DecodedFlight, path: str, metrics: Optional[Sequence[str]] = None, ) -> None: """Write one flight to a CSV at `path`. Columns: datetime + chosen metrics.""" if metrics is None: metrics = flight.available_codes metrics = [m for m in metrics if m in flight.series] with open(path, "w", newline="") as fh: w = csv.writer(fh) w.writerow(["datetime"] + list(metrics)) for i, ts in enumerate(flight.timestamps): row = [ts.isoformat() if ts else ""] for code in metrics: row.append(_fmt_value(flight.series[code][i])) w.writerow(row) def export_flights_csv( flights: Iterable[DecodedFlight], out_dir: str, metrics: Optional[Sequence[str]] = None, file_pattern: str = "flight_{flight_id}.csv", ) -> List[str]: """Write one CSV per flight into `out_dir`. Returns the list of created paths.""" os.makedirs(out_dir, exist_ok=True) paths: List[str] = [] for flight in flights: if not flight.valid: continue p = os.path.join(out_dir, file_pattern.format(flight_id=flight.flight_id)) chosen = list(metrics) if metrics is not None else flight.available_codes export_flight_csv(flight, p, chosen) paths.append(p) return paths jpi_analyzer/metrics.py000066400000000000000000000263471517535535700156210ustar00rootroot00000000000000"""Metric definitions: units, scale, axis ranges per metric type. Axis ranges chosen to span the typical operating envelope of piston/turbine GA engines plus headroom. They are *recommended* defaults; callers can override. """ from dataclasses import dataclass, field from typing import Dict, Optional, Tuple @dataclass(frozen=True) class MetricCategory: name: str unit: str axis_min: float axis_max: float description: str = "" # Keyed by metric short code prefix. EGT covers E1..E9, LE1..LE9, RE1..RE9 etc. # CHT covers C1..C9, etc. CATEGORIES: Dict[str, MetricCategory] = { "EGT": MetricCategory("EGT", "°F", 0, 2000, "Exhaust Gas Temperature"), "CHT": MetricCategory("CHT", "°F", 0, 500, "Cylinder Head Temperature"), "TIT": MetricCategory("TIT", "°F", 0, 2000, "Turbine Inlet Temperature"), "ITT": MetricCategory("ITT", "°F", 0, 2000, "Inter-Turbine Temperature"), "OAT": MetricCategory("OAT", "°F", -60, 130, "Outside Air Temperature"), "OILT": MetricCategory("OILT", "°F", 0, 300, "Oil Temperature"), "OILP": MetricCategory("OILP", "PSI", 0, 100, "Oil Pressure"), "CDT": MetricCategory("CDT", "°F", 0, 400, "Compressor Discharge Temp"), "IAT": MetricCategory("IAT", "°F", 0, 200, "Induction Air Temperature"), "CLD": MetricCategory("CLD", "°F/min", -100, 100, "Cylinder Cooling Rate"), "DIF": MetricCategory("DIF", "°F", 0, 300, "EGT Spread (max-min)"), "MAP": MetricCategory("MAP", "in.Hg", 0, 35, "Manifold Pressure"), "RPM": MetricCategory("RPM", "rpm", 0, 3500, "Engine RPM"), "FF": MetricCategory("FF", "GPH", 0, 30, "Fuel Flow"), "FF2": MetricCategory("FF2", "GPH", 0, 30, "Fuel Flow #2"), "USD": MetricCategory("USD", "gal", 0, 200, "Fuel Used"), "USD2": MetricCategory("USD2", "gal", 0, 200, "Fuel Used #2"), "FL": MetricCategory("FL", "gal", 0, 100, "Fuel Level"), "FP": MetricCategory("FP", "PSI", 0, 60, "Fuel Pressure"), "BAT": MetricCategory("BAT", "V", 0, 16, "Battery Voltage"), "BAT2": MetricCategory("BAT2", "V", 0, 16, "Battery #2 Voltage"), "AMP": MetricCategory("AMP", "A", -100, 100, "Amperage"), "AMP2": MetricCategory("AMP2", "A", -100, 100, "Amperage #2"), "HP": MetricCategory("HP", "%", 0, 110, "Percent Power / Horsepower"), "HRS": MetricCategory("HRS", "h", 0, 10000, "Hobbs Hours"), "MARK": MetricCategory("MARK", "", 0, 8, "Pilot Mark / Lean-Find State"), "SPD": MetricCategory("SPD", "kt", 0, 300, "Ground Speed"), "ALT": MetricCategory("ALT", "ft", -1000, 30000, "Pressure Altitude"), "LAT": MetricCategory("LAT", "deg", -90, 90, "Latitude"), "LNG": MetricCategory("LNG", "deg", -180, 180, "Longitude"), "NG": MetricCategory("NG", "%", 0, 110, "Gas Generator Speed"), "NP": MetricCategory("NP", "rpm", 0, 3000, "Propeller Speed"), "TRQ": MetricCategory("TRQ", "%", 0, 110, "Torque"), "HYD": MetricCategory("HYD", "PSI", 0, 5000, "Hydraulic Pressure"), } def _category_for_code(code: str) -> Optional[MetricCategory]: """Strip L/R prefix and trailing digit to find category.""" base = code if base.startswith(("L", "R")) and base[1:] in CATEGORIES: return CATEGORIES[base[1:]] # Strip trailing digit(s): "E1" -> "E", "RC1" -> "RC" stripped = base.rstrip("0123456789") # Single-letter codes for cylinders: E (EGT), C (CHT), T (TIT) letter_map = {"E": "EGT", "C": "CHT", "T": "TIT", "LE": "EGT", "RE": "EGT", "LC": "CHT", "RC": "CHT", "LT": "TIT", "RT": "TIT"} if stripped in letter_map: return CATEGORIES[letter_map[stripped]] if stripped in CATEGORIES: return CATEGORIES[stripped] # Strip an L/R prefix and try again: "LMAP" -> "MAP" if stripped.startswith(("L", "R")) and stripped[1:] in CATEGORIES: return CATEGORIES[stripped[1:]] return None @dataclass(frozen=True) class MetricDef: """Definition of one decodable metric in a flight record.""" sensor_name: str # "Left EGT 1" code: str # "E1", "MAP", "RPM" cfg_byt_idx: int # which Cfg_Word to test cfg_bit_idx: int # which bit scale_val: float # divisor for engineering units m_lo_byt_idx: int = -1 # byte index in data stream (low delta) m_lo_bit_idx: int = -1 m_hi_byt_idx: int = -1 # byte index in data stream (high delta) m_hi_bit_idx: int = -1 @property def category(self) -> Optional[MetricCategory]: return _category_for_code(self.code) @property def unit(self) -> str: cat = self.category return cat.unit if cat else "" def axis_range_for(code: str) -> Tuple[float, float]: """Recommended (min, max) Y-axis range for a metric short code.""" cat = _category_for_code(code) if cat is None: return (0.0, 100.0) return (cat.axis_min, cat.axis_max) def unit_for(code: str) -> str: cat = _category_for_code(code) return cat.unit if cat else "" def description_for(code: str) -> str: cat = _category_for_code(code) return cat.description if cat else code def _h(name, cfg_b, cfg_bit, code, scale=1.0, lo_b=-1, lo_bit=-1, hi_b=-1, hi_bit=-1): return MetricDef(name, code, cfg_b, cfg_bit, scale, lo_b, lo_bit, hi_b, hi_bit) # ---- Header tables, transcribed from EzTrends2 Decomp.cs ---- # init_strings_for_107 — used for EDM830/831 (and other models) running protocol 2. # Includes byte indices up to 10 (HRS, AMP, FL, ALT, SPD, LAT, LNG). HEADERS_107 = [ _h("Left EGT 1", 0, 2, "E1", 1, 0, 0, 6, 0), _h("Left EGT 2", 0, 3, "E2", 1, 0, 1, 6, 1), _h("Left EGT 3", 0, 4, "E3", 1, 0, 2, 6, 2), _h("Left EGT 4", 0, 5, "E4", 1, 0, 3, 6, 3), _h("Left EGT 5", 0, 6, "E5", 1, 0, 4, 6, 4), _h("Left EGT 6", 0, 7, "E6", 1, 0, 5, 6, 5), _h("Left EGT 7", 0, 8, "E7", 1, 3, 0, 7, 0), _h("Left EGT 8", 0, 9, "E8", 1, 3, 1, 7, 1), _h("Left EGT 9", 0, 10, "E9", 1, 3, 2, 7, 2), _h("Left CHT 1", 0, 11, "C1", 1, 1, 0), _h("Left CHT 2", 0, 12, "C2", 1, 1, 1), _h("Left CHT 3", 0, 13, "C3", 1, 1, 2), _h("Left CHT 4", 0, 14, "C4", 1, 1, 3), _h("Left CHT 5", 0, 15, "C5", 1, 1, 4), _h("Left CHT 6", 1, 0, "C6", 1, 1, 5), _h("Left CHT 7", 1, 1, "C7", 1, 3, 3), _h("Left CHT 8", 1, 2, "C8", 1, 3, 4), _h("Left CHT 9", 1, 3, "C9", 1, 3, 5), _h("Left TIT 1", 1, 5, "T1", 1, 0, 6, 6, 6), _h("Left TIT 2", 1, 6, "T2", 1, 0, 7, 6, 7), _h("Left ITT", 3, 12, "ITT", 1, 9, 3), _h("OAT", 1, 9, "OAT", 1, 2, 5), _h("Left DIF", 0, 0, "DIF"), _h("Left CLD", 0, 0, "CLD", 1, 1, 6), _h("Left CDT", 1, 7, "CDT", 1, 2, 2), _h("Left IAT", 1, 8, "IAT", 1, 2, 3), _h("Left MAP", 1, 14, "MAP", 10.0, 5, 0), _h("Left RPM", 1, 10, "RPM", 1, 5, 1, 5, 2), _h("Left HP", 1, 10, "HP", 1, 3, 6), _h("Left FF (gph)", 1, 11, "FF", 10.0, 2, 7), _h("Left FF (lph)", 1, 11, "FF_L", 1, 2, 7), _h("Left FF2 (gph)", 3, 5, "FF2", 10.0, 5, 6), _h("Left FF2 (lph)", 3, 5, "FF2_L", 1, 5, 6), _h("Left FP", 1, 15, "FP", 10.0, 8, 5), _h("Left TRQ", 3, 8, "TRQ", 1, 9, 2), _h("Left OILP", 1, 13, "OILP", 1, 2, 1), _h("Left OILT", 1, 4, "OILT", 1, 1, 7), _h("BAT", 0, 0, "BAT", 10.0, 2, 4), _h("BAT2", 2, 6, "BAT2", 10.0, 8, 1), _h("AMP", 0, 1, "AMP", 1, 8, 0), _h("AMP2", 2, 5, "AMP2", 1, 8, 2), _h("Left USD (gal)", 1, 11, "USD", 10.0, 2, 6), _h("Left USD (l)", 1, 11, "USD_L", 1, 2, 6), _h("Left FL (gal)", 2, 3, "RFL", 10.0, 8, 3), _h("Left FL (l)", 2, 3, "RFL_L", 1, 8, 3), _h("Left FL2 (gal)", 2, 4, "LFL", 10.0, 8, 4), _h("Left FL2 (l)", 2, 4, "LFL_L", 1, 8, 4), _h("Left HRS", 0, 0, "HRS", 10.0, 9, 6, 9, 7), _h("Left USD2 (gal)", 3, 5, "USD2", 10.0, 5, 7), _h("Left USD2 (l)", 3, 5, "USD2_L", 1, 5, 7), _h("Left HYD", 3, 15, "HYD", 1, 5, 5), _h("Left HYD2", 4, 2, "HYD2", 1, 5, 4), _h("SPD", 4, 5, "SPD", 1, 10, 5), _h("ALT", 4, 6, "ALT", 1, 10, 3), _h("LAT", 4, 3, "LAT", 1, 10, 7, 10, 2), _h("LNG", 4, 4, "LNG", 1, 10, 6, 10, 1), _h("MARK", 0, 0, "MARK", 1, 2, 0), ] # init_strings_for_single — used for non-Edm_Typ_actual installations. SINGLE_HEADERS = [ _h("Left EGT 1", 0, 2, "E1", 1, 0, 0, 6, 0), _h("Left EGT 2", 0, 3, "E2", 1, 0, 1, 6, 1), _h("Left EGT 3", 0, 4, "E3", 1, 0, 2, 6, 2), _h("Left EGT 4", 0, 5, "E4", 1, 0, 3, 6, 3), _h("Left EGT 5", 0, 6, "E5", 1, 0, 4, 6, 4), _h("Left EGT 6", 0, 7, "E6", 1, 0, 5, 6, 5), _h("Left EGT 7", 0, 8, "E7", 1, 3, 0, 7, 0), _h("Left EGT 8", 0, 9, "E8", 1, 3, 1, 7, 1), _h("Left EGT 9", 0, 10, "E9", 1, 3, 2, 7, 2), _h("Left CHT 1", 0, 11, "C1", 1, 1, 0), _h("Left CHT 2", 0, 12, "C2", 1, 1, 1), _h("Left CHT 3", 0, 13, "C3", 1, 1, 2), _h("Left CHT 4", 0, 14, "C4", 1, 1, 3), _h("Left CHT 5", 0, 15, "C5", 1, 1, 4), _h("Left CHT 6", 1, 0, "C6", 1, 1, 5), _h("Left CHT 7", 1, 1, "C7", 1, 3, 3), _h("Left CHT 8", 1, 2, "C8", 1, 3, 4), _h("Left CHT 9", 1, 3, "C9", 1, 3, 5), _h("Left TIT 1", 1, 5, "T1", 1, 0, 6, 6, 6), _h("Left TIT 2", 1, 6, "T2", 1, 0, 7, 6, 7), _h("OAT", 1, 9, "OAT", 1, 2, 5), _h("Left DIF", 0, 0, "DIF"), _h("Left CLD", 0, 0, "CLD", 1, 1, 6), _h("Left CDT", 1, 7, "CDT", 1, 2, 2), _h("Left IAT", 1, 8, "IAT", 1, 2, 3), _h("Left MAP", 1, 14, "MAP", 10.0, 5, 0), _h("Left RPM", 1, 10, "RPM", 1, 5, 1, 5, 2), _h("Left HP", 1, 10, "HP", 1, 3, 6), _h("Left FF (gph)", 1, 11, "FF", 10.0, 2, 7), # fUnit==0 default _h("Left FF (lph)", 1, 11, "FF_L", 1, 2, 7), # fUnit==1 _h("Left OILP", 1, 13, "OILP", 1, 2, 1), _h("Left OILT", 1, 4, "OILT", 1, 1, 7), _h("BAT", 0, 0, "BAT", 10.0, 2, 4), _h("Left USD (gal)", 1, 11, "USD", 10.0, 2, 6), _h("Left USD (l)", 1, 11, "USD_L", 1, 2, 6), _h("MARK", 0, 0, "MARK", 1, 2, 0), ] # Codes that come in (gallons, liters) pairs. The "_L" variant uses scale 1.0, # the bare variant uses scale 10.0. We pick one based on fUnit and rename to # the bare code so downstream consumers see "FF" regardless of unit. _FUEL_UNIT_PAIRS = {"FF", "FF2", "USD", "USD2", "RFL", "LFL"} def headers_for_model(model: str, fuel_unit: int, twin: bool, edm_typ_actual: bool): """Return the list of MetricDef relevant for the model, with fUnit-conditional entries pruned. EDM830/831 with protocol-2 → HEADERS_107; otherwise SINGLE. """ table = HEADERS_107 if edm_typ_actual else SINGLE_HEADERS out = [] for hdr in table: bare = hdr.code[:-2] if hdr.code.endswith("_L") else hdr.code if bare in _FUEL_UNIT_PAIRS and bare != hdr.code: # liters variant — keep only when fuel_unit != 0 if fuel_unit == 0: continue elif bare in _FUEL_UNIT_PAIRS and bare == hdr.code: # gallons variant — keep only when fuel_unit == 0 if fuel_unit != 0: continue out.append(hdr) # Rename "_L" codes to their bare form normalized = [] for h in out: if h.code.endswith("_L"): normalized.append(MetricDef(h.sensor_name, h.code[:-2], h.cfg_byt_idx, h.cfg_bit_idx, h.scale_val, h.m_lo_byt_idx, h.m_lo_bit_idx, h.m_hi_byt_idx, h.m_hi_bit_idx)) else: normalized.append(h) return normalized METRIC_LIBRARY = HEADERS_107 jpi_analyzer/parser.py000066400000000000000000000224621517535535700154410ustar00rootroot00000000000000"""Parse the JPI EDM file header and locate per-flight binary blocks.""" from __future__ import annotations import os import re from dataclasses import dataclass, field from datetime import datetime from typing import Dict, List, Optional class HeaderNotFoundError(Exception): pass @dataclass class FlightRecord: """A flight as advertised in the file header — locator only, not yet decoded.""" id: int size: int # byte length of this flight's data block start: Optional[int] = None # absolute byte offset to flight payload found: bool = False data: Optional[bytes] = None # raw bytes for the flight (filled after locate) @dataclass class JpiFile: path: str raw: bytes = b"" header_offset: int = 0 flight_data_start: int = 0 flights: Dict[int, FlightRecord] = field(default_factory=dict) tail_number: str = "" model: str = "" sw_version: int = 0 build_num: str = "" beta_num: str = "" protocol_id: int = 0 edm_typ: bool = False # True for EDM>=900 — drives 16-bit vs 8-bit data records twin_flag: bool = False fuel_unit: int = 0 # 0 = gallons, 1 = liters eng_deg: str = "F" oat_deg: str = "F" cfg_high_byte: int = 0 cfg_low_byte: int = 0 egt_limit: int = 0 cht_limit: int = 0 cht_low: int = 0 bat_limit: float = 0.0 oil_limit: int = 0 file_created_at: Optional[datetime] = None created_with: str = "N/A" @classmethod def open(cls, path: str) -> "JpiFile": if not os.path.isfile(path): raise FileNotFoundError(path) with open(path, "rb") as fh: raw = fh.read() if not raw: raise ValueError("Empty JPI file") f = cls(path=path, raw=raw) f._locate_header() f._parse_dollar_records() f._extract_version() f._locate_flights() return f # ---------- header & metadata ---------- def _locate_header(self) -> None: # JPI files sometimes have leading zero-padding before the "$U" magic. m = re.search(rb"\$U", self.raw) if m is None: raise HeaderNotFoundError("'$U' magic not found") self.header_offset = m.start() def _read_dollar_block(self, ptr: int) -> tuple[str, int]: """Read characters until '*' (skipping NUL bytes), return (text, new_ptr).""" text = [] while ptr < len(self.raw): ch = self.raw[ptr] if ch == 0x2A: # '*' return "".join(text), ptr if ch != 0: text.append(chr(ch)) ptr += 1 return "".join(text), ptr def _parse_dollar_records(self) -> None: ptr = self.header_offset block_skip = 5 # past '*XX\r\n' while ptr < len(self.raw): text, star_pos = self._read_dollar_block(ptr) if star_pos >= len(self.raw): break fields = text.split(",") kind = fields[0] if kind == "$U" and len(fields) >= 2: self.tail_number = fields[1].strip() elif kind == "$T" and len(fields) >= 7: try: month = int(fields[1]) day = int(fields[2]) yy = int(fields[3]) year = 2000 + yy if yy < 75 else 1900 + yy hour = int(fields[4]) minute = int(fields[5]) secs_milli = int(fields[6]) secs = secs_milli // 1000 millis = secs_milli % 1000 self.file_created_at = datetime( year, month, day, hour, minute, secs, millis * 1000) except (ValueError, IndexError): pass elif kind == "$C" and len(fields) >= 2: self.model = fields[1].strip() # The build/beta layout depends on field count. idx = len(fields) - 1 try: if idx in (8, 9): self.build_num = fields[idx - 1].strip() self.beta_num = fields[idx].strip() self.sw_version = int(fields[idx - 2].strip() or 0) else: self.sw_version = int(fields[idx].strip() or 0) except ValueError: pass try: cfg_num = round(float(fields[2])) hex_str = format(cfg_num, "X").rjust(2, "0") self.cfg_high_byte = int(hex_str[:-2] or "0", 16) self.cfg_low_byte = int(hex_str[-2:], 16) except (ValueError, IndexError): pass try: self.eng_deg = "F" if (int(fields[3]) & 4096) else "C" except (ValueError, IndexError): pass try: self.oat_deg = "F" if (int(fields[5]) & 8192) else "C" except (ValueError, IndexError): pass # Edm_Typ flips to True for >=900-series, False for 830/etc. try: self.edm_typ = float(self.model) >= 900.0 except ValueError: self.edm_typ = False self.twin_flag = self.model in ("760", "790") elif kind == "$P" and len(fields) >= 2: try: self.protocol_id = round(float(fields[1])) except ValueError: pass # JPI's reference decoder unconditionally flips Edm_Typ on at $P; # the order of $C vs $P in the file determines the final value. self.edm_typ = True elif kind == "$F" and len(fields) >= 2: try: self.fuel_unit = int(fields[1]) except ValueError: pass elif kind == "$A" and len(fields) >= 8: # $A, fixed?, bat*10, ?, cht_limit, cht_low, egt_limit, oil_limit try: self.bat_limit = int(fields[2]) * 0.1 self.cht_limit = int(fields[4]) self.cht_low = int(fields[5]) self.egt_limit = int(fields[6]) self.oil_limit = int(fields[7]) except (ValueError, IndexError): pass elif kind == "$D" and len(fields) >= 3: try: fid = int(fields[1]) fsize = int(fields[2]) * 2 self.flights[fid] = FlightRecord(id=fid, size=fsize) except ValueError: pass elif kind == "$L": # End of dollar-record header: data follows the trailing block_skip bytes self.flight_data_start = star_pos + block_skip return ptr = star_pos + block_skip def _extract_version(self) -> None: tail = self.raw[-200:] m = re.search(rb"\$V[^\r\n]*", tail) if not m: return text = m.group(0).decode("latin-1", errors="ignore").strip() # Format example: "$V Created by EDM830 VER 252 BLD 64 BETA 0*5C" match = re.search( r"Created by (?P\w+)\s+VER\s+(?P[0-9.]+)\s+BLD\s+(?P\d+)\s+BETA\s+(?P\d+)", text, ) if match: self.created_with = match.group(0) if not self.build_num or self.build_num == "-1": self.build_num = match.group("bld") if not self.beta_num or self.beta_num == "-1": self.beta_num = match.group("beta") # ---------- flight locator ---------- def _read_be_word(self, offset: int) -> int: if offset + 1 >= len(self.raw): return -1 return (self.raw[offset] << 8) | self.raw[offset + 1] def _locate_flights(self) -> None: """Find each flight's start offset and copy its raw bytes. Each flight begins with its 16-bit big-endian flight ID. Sizes from $D records are advisory — JPI sometimes pads with an off-by-one byte, so we scan a small window when the expected ID isn't where we predicted. """ ptr = self.flight_data_start for fid in list(self.flights.keys()): rec = self.flights[fid] size = rec.size additional = 0 if self._read_be_word(ptr) == fid: rec.found = True elif self._read_be_word(ptr - 1) == fid: additional = -1 rec.found = True else: # Wider scan window for delta in range(-30, 31): if self._read_be_word(ptr + delta) == fid: additional = delta rec.found = True break ptr += additional rec.start = ptr end = min(ptr + size, len(self.raw)) rec.data = bytes(self.raw[ptr:end]) ptr += size # ---------- helpers ---------- @property def flight_ids(self) -> List[int]: return list(self.flights.keys()) def get_flight(self, fid: int) -> FlightRecord: if fid not in self.flights: raise KeyError(f"Flight {fid} not in file") return self.flights[fid] pyproject.toml000066400000000000000000000007341517535535700140160ustar00rootroot00000000000000[build-system] requires = ["setuptools>=68", "wheel"] build-backend = "setuptools.build_meta" [project] name = "jpi-analyzer" version = "0.2.0" description = "Parse JPI EDM flight data files and export them to CSV." readme = "README.md" requires-python = ">=3.10" authors = [{ name = "Sophie Bertsch" }] dependencies = [ "click>=8.1", ] [project.scripts] jpi-analyzer = "jpi_analyzer.cli:main" jpia = "jpi_analyzer.cli:main" [tool.setuptools] packages = ["jpi_analyzer"]