Compare commits
2 Commits
release_1.
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
ec95ea26c3 | ||
|
|
a6ed05a38a |
1
.gitignore
vendored
1
.gitignore
vendored
@ -61,3 +61,4 @@ docker-compose.override.yml
|
|||||||
*.backup
|
*.backup
|
||||||
/tmp/
|
/tmp/
|
||||||
/~archive/
|
/~archive/
|
||||||
|
firefly-import-preprocessor.code-workspace
|
||||||
|
|||||||
15
AGENTS.md
15
AGENTS.md
@ -14,15 +14,17 @@ composer psalm # Psalm static analysis
|
|||||||
|
|
||||||
### Test Suite Overview
|
### Test Suite Overview
|
||||||
|
|
||||||
85 tests across 5 test classes:
|
129 tests across 7 test classes:
|
||||||
|
|
||||||
| File | Tests | Scope |
|
| File | Tests | Scope |
|
||||||
|------|-------|-------|
|
| ------ | -------: | ------- |
|
||||||
| `tests/ColumnTransformerTest.php` | 37 | All 13 transformation types, edge cases |
|
| `tests/ColumnTransformerTest.php` | 51 | All 14 transformation types, edge cases |
|
||||||
| `tests/ConfigurationLoaderTest.php` | 18 | JSON loading, dot-notation access, validation |
|
| `tests/ConfigurationLoaderTest.php` | 18 | JSON loading, dot-notation access, validation |
|
||||||
| `tests/CsvReaderTest.php` | 15 | CSV parsing, BOM handling, delimiter, encoding |
|
| `tests/CsvReaderTest.php` | 15 | CSV parsing, BOM handling, delimiter, encoding |
|
||||||
| `tests/MetadataExtractorTest.php` | 14 | Pre-header regex extraction, edge cases |
|
| `tests/MetadataExtractorTest.php` | 14 | Pre-header regex extraction, edge cases |
|
||||||
| `tests/ConfigIntegrationTest.php` | 1× per fixture | Golden-file integration tests (see below) |
|
| `tests/ConfigIntegrationTest.php` | 1× per fixture | Golden-file integration tests (see below) |
|
||||||
|
| `tests/RowFilterTest.php` | 19 | skipIf conditions, all operators, nested AND/OR groups |
|
||||||
|
| `tests/FireflyImporterChunkStateTest.php` | 11 | Chunk state persistence, resume, reset |
|
||||||
|
|
||||||
### Integration Tests (Golden-File Pattern)
|
### Integration Tests (Golden-File Pattern)
|
||||||
|
|
||||||
@ -85,7 +87,10 @@ bin/transformer.php → TransformerEngine
|
|||||||
|
|
||||||
- **PSR-12** enforced via phpcs using `phpcs.xml` (auto-discovered at root). Line length: soft 120, hard 150 chars.
|
- **PSR-12** enforced via phpcs using `phpcs.xml` (auto-discovered at root). Line length: soft 120, hard 150 chars.
|
||||||
- **PHPStan level 8** with `checkMissingCallableSignature: true`. `phpstan-baseline.neon` is empty — do not add suppressions without good reason.
|
- **PHPStan level 8** with `checkMissingCallableSignature: true`. `phpstan-baseline.neon` is empty — do not add suppressions without good reason.
|
||||||
- **All source comments and docblocks are written in German.**
|
- **All source comments and docblocks are written in English.**
|
||||||
|
- **Documentation language:** `README.md` is the primary documentation in **English**. `README.de.md` is the German translation. Both cross-link to each other at the top.
|
||||||
|
- **`showHelp()` in `bin/transformer.php`** is locale-aware: English is the default; German is shown when `isGermanLocale()` returns `true` (checks `LANG`, `LC_ALL`, `LC_MESSAGES`, `LANGUAGE` env vars for a `de` prefix).
|
||||||
|
- **License:** GPL-3.0.
|
||||||
- Namespace `UbsCsvTransformer\` (PSR-4 → `src/`); tests use `UbsCsvTransformer\Tests\` (→ `tests/`).
|
- Namespace `UbsCsvTransformer\` (PSR-4 → `src/`); tests use `UbsCsvTransformer\Tests\` (→ `tests/`).
|
||||||
- No runtime package dependencies — only `ext-json` and `ext-mbstring`.
|
- No runtime package dependencies — only `ext-json` and `ext-mbstring`.
|
||||||
|
|
||||||
@ -103,4 +108,4 @@ See [config/config.example.json](config/config.example.json) for a full referenc
|
|||||||
- `"outputAction": "create"` vs `"overwrite"` — controls whether the result is a new column or replaces an existing one
|
- `"outputAction": "create"` vs `"overwrite"` — controls whether the result is a new column or replaces an existing one
|
||||||
- `MetadataExtractor` uses 1-based `lineNumber` in config; it converts to 0-based array index internally
|
- `MetadataExtractor` uses 1-based `lineNumber` in config; it converts to 0-based array index internally
|
||||||
|
|
||||||
Supported transformation types: `map`, `replace`, `regex`, `regexextract`, `dateformat`, `split`, `trim`, `uppercase`, `lowercase`, `ucwordsfirst`, `truncate`, `constantvalue`, `pipeline`
|
Supported transformation types: `map`, `replace`, `regex`, `regexextract`, `dateformat`, `split`, `trim`, `uppercase`, `lowercase`, `ucwordsfirst`, `truncate`, `constantvalue`, `pipeline`, `timeperiod`
|
||||||
|
|||||||
767
README.de.md
Normal file
767
README.de.md
Normal file
@ -0,0 +1,767 @@
|
|||||||
|
# Firefly Import Preprocessor — Dokumentation
|
||||||
|
|
||||||
|
**Version:** 1.0.0
|
||||||
|
**Datum:** 03. Mai 2026
|
||||||
|
**Status:** Production Ready
|
||||||
|
|
||||||
|
🌐 [English](README.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📋 Inhaltsverzeichnis
|
||||||
|
|
||||||
|
1. [Überblick](#überblick)
|
||||||
|
2. [Installation & Setup](#installation--setup)
|
||||||
|
3. [Schnellstart](#schnellstart)
|
||||||
|
4. [Konfiguration](#konfiguration)
|
||||||
|
5. [Transformationstypen](#transformationstypen)
|
||||||
|
6. [CLI-Referenz](#cli-referenz)
|
||||||
|
7. [Debug-Modus](#debug-modus)
|
||||||
|
8. [Firefly III Integration](#firefly-iii-integration)
|
||||||
|
9. [Architektur](#architektur)
|
||||||
|
10. [Fehlerbehandlung](#fehlerbehandlung)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Überblick
|
||||||
|
|
||||||
|
Der **Firefly Import Preprocessor** ist ein produktionsreifer PHP-Preprocessor für Banken-CSV-Exportdateien. Er transformiert Bankdaten in ein standardisiertes Format und kann sie optional in Firefly III importieren.
|
||||||
|
|
||||||
|
### Kernfeatures
|
||||||
|
|
||||||
|
✅ **Vollständige CSV-Transformation** mit komplexen Pipelines
|
||||||
|
✅ **Metadaten-Extraktion** mit Regex (IBAN, Währung, Kontoname)
|
||||||
|
✅ **14 Transformationstypen** für flexible Datenverarbeitung
|
||||||
|
✅ **Firefly III Integration** — CLI, Docker und HTTP-Upload
|
||||||
|
✅ **Debug-Modus** für Transparenz bei Verarbeitung
|
||||||
|
✅ **Production Ready** mit vollständiger Fehlerbehandlung
|
||||||
|
✅ **Zero Dependencies** für Core-Funktionalität
|
||||||
|
|
||||||
|
### Workflow
|
||||||
|
|
||||||
|
```text
|
||||||
|
Input CSV
|
||||||
|
↓
|
||||||
|
Metadaten extrahieren (Regex)
|
||||||
|
↓
|
||||||
|
Datenzeilen transformieren (Pipeline)
|
||||||
|
↓
|
||||||
|
Output CSV schreiben
|
||||||
|
↓
|
||||||
|
[Optional] In Firefly III importieren
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Installation & Setup
|
||||||
|
|
||||||
|
### Voraussetzungen
|
||||||
|
|
||||||
|
- PHP 8.1+
|
||||||
|
- Composer (empfohlen)
|
||||||
|
- [Optional] Docker für Firefly III Integration
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Repository clonen/kopieren
|
||||||
|
cd ff-imp-preprocessor
|
||||||
|
|
||||||
|
# 2. Abhängigkeiten installieren (optional)
|
||||||
|
composer install
|
||||||
|
|
||||||
|
# 3. Konfiguration erstellen
|
||||||
|
cp config/config.example.json config/config.json
|
||||||
|
# Bearbeite config/config.json mit deinen Einstellungen
|
||||||
|
|
||||||
|
# 4. Directories erstellen
|
||||||
|
mkdir -p config/import/{source,output,archive,error}
|
||||||
|
chmod 755 config/import/{source,output,archive,error}
|
||||||
|
|
||||||
|
# 5. Test durchführen
|
||||||
|
php bin/transformer.php validate config/config.json input.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Schnellstart
|
||||||
|
|
||||||
|
### 1. Konfiguration anpassen
|
||||||
|
|
||||||
|
Bearbeite `config/config.json` und stelle sicher, dass die Extraction-Rules zu deinem CSV-Format passen:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"extractionRules": [
|
||||||
|
{
|
||||||
|
"name": "account_iban",
|
||||||
|
"lineNumber": 2,
|
||||||
|
"regex": "IBAN:\\s*([A-Z0-9 ]+)",
|
||||||
|
"captureGroup": 1
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"csvStructure": {
|
||||||
|
"headerLine": 5,
|
||||||
|
"delimiter": ";",
|
||||||
|
"encoding": "UTF-8"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. CSV validieren
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php bin/transformer.php validate config/config.json input.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Transformation durchführen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php bin/transformer.php transform input.csv config/config.json
|
||||||
|
|
||||||
|
# Mit Debug-Modus für Fehlersuche
|
||||||
|
php bin/transformer.php transform input.csv config/config.json --debug
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Output prüfen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php bin/transformer.php test input.csv config/config.json --debug
|
||||||
|
# Zeigt max. 10 transformierte Zeilen und Debug-Logs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Konfiguration
|
||||||
|
|
||||||
|
### config.json Struktur
|
||||||
|
|
||||||
|
#### `metadata` - Metadaten-Extraktion
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"extractionRules": [
|
||||||
|
{
|
||||||
|
"name": "account_iban",
|
||||||
|
"lineNumber": 2,
|
||||||
|
"regex": "IBAN:\\s*([A-Z0-9 ]+)",
|
||||||
|
"captureGroup": 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "currency_code",
|
||||||
|
"lineNumber": 3,
|
||||||
|
"regex": "Währung:\\s*([A-Z]{3})",
|
||||||
|
"captureGroup": 1
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Feld | Typ | Beschreibung |
|
||||||
|
| ------ | ----- | ------------- |
|
||||||
|
| `name` | string | Name der Metadaten-Variable (verwendet in constantvalue) |
|
||||||
|
| `lineNumber` | int | Zeilennummer in CSV (1-basiert, menschenlesbar) |
|
||||||
|
| `regex` | string | Regex-Pattern zur Extraktion (ohne Delimiter) |
|
||||||
|
| `captureGroup` | int | Nummer der Klammer-Gruppe (0=komplett, 1=erste Klammer, etc.) |
|
||||||
|
|
||||||
|
**Beispiel Regex:**
|
||||||
|
|
||||||
|
- Pattern: `IBAN:\s*([A-Z0-9 ]+)`
|
||||||
|
- Input: `IBAN: CH93 0077 2020 6262 5252 7`
|
||||||
|
- Capture Group 1: `CH93 0077 2020 6262 5252 7`
|
||||||
|
|
||||||
|
#### `csvStructure` - CSV-Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"csvStructure": {
|
||||||
|
"headerLine": 5,
|
||||||
|
"delimiter": ";",
|
||||||
|
"encoding": "UTF-8",
|
||||||
|
"hasBom": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Feld | Typ | Default | Beschreibung |
|
||||||
|
| ------ | ----- | --------- | ------------- |
|
||||||
|
| `headerLine` | int | 5 | Zeilennummer der Header (1-basiert) |
|
||||||
|
| `delimiter` | string | `;` | CSV-Delimiter |
|
||||||
|
| `encoding` | string | `UTF-8` | Zeichenkodierung (UTF-8, ISO-8859-1, CP1252) |
|
||||||
|
| `hasBom` | bool | false | Hat die Datei BOM (Byte Order Mark)? |
|
||||||
|
|
||||||
|
#### `columnTransformations` - Spalten-Transformationen
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"columnTransformations": [
|
||||||
|
{
|
||||||
|
"sourceColumn": "Buchungsdatum",
|
||||||
|
"transformations": [
|
||||||
|
{
|
||||||
|
"type": "dateformat",
|
||||||
|
"fromFormat": "d.m.Y",
|
||||||
|
"toFormat": "Y-m-d"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"outputColumn": "date",
|
||||||
|
"outputAction": "overwrite"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**outputAction:**
|
||||||
|
|
||||||
|
| Wert | Verhalten |
|
||||||
|
|---|---|
|
||||||
|
| `overwrite` | Ziel-Spalte mit dem Transformations-Ergebnis überschreiben (Standard) |
|
||||||
|
| `create` | Ergebnis in eine neue Ausgabe-Spalte schreiben |
|
||||||
|
| `append` | Ergebnis ans Ende des bestehenden Spalten-Werts anhängen. Mit `"appendDelimiter": " "` (beliebige Zeichenkette) wird ein Trennzeichen zwischen bestehendem und neuem Wert eingefügt — der Trennzeichen entfällt, wenn die Ziel-Spalte noch leer ist |
|
||||||
|
| `append-if-not-empty` | Wie `append` (inkl. `appendDelimiter`), aber überspringt die Operation vollständig, wenn das Transformations-Ergebnis leer ist — geeignet für optionale Werte wie Tags oder Notiz-Zeilen |
|
||||||
|
| `append-line` | Wie `append`, aber als Trennzeichen wird immer ein Zeilenumbruch `\n` verwendet; kein führender Zeilenumbruch wenn die Ziel-Spalte leer ist |
|
||||||
|
| `overwrite-if-empty` | Ergebnis nur schreiben, wenn die Ziel-Spalte aktuell leer ist |
|
||||||
|
| `overwrite-if-not-empty` | Ergebnis nur schreiben, wenn das Transformations-Ergebnis nicht leer ist |
|
||||||
|
|
||||||
|
#### `directories` - Dateisystem
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"directories": {
|
||||||
|
"source": "/opt/ff-imp-preprocessor/import/source",
|
||||||
|
"output": "/opt/ff-imp-preprocessor/import/output",
|
||||||
|
"archive": "/opt/ff-imp-preprocessor/import/archive",
|
||||||
|
"error": "/opt/ff-imp-preprocessor/import/error"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Feld | Beschreibung |
|
||||||
|
| ------ | ------------- |
|
||||||
|
| `source` | Eingabe-Verzeichnis |
|
||||||
|
| `output` | Ausgabe-Verzeichnis |
|
||||||
|
| `archive` | Archiv für verarbeitete Dateien |
|
||||||
|
| `error` | Error-Verzeichnis für ungültige Dateien |
|
||||||
|
|
||||||
|
#### `fireflyImport` - Firefly III Integration
|
||||||
|
|
||||||
|
Optional. Mit dem Flag `--do-import` beim `transform`-Kommando (oder via `auto-import`) wird der Firefly III Data Importer nach dem Schreiben der Output-CSV aufgerufen.
|
||||||
|
|
||||||
|
Details und vollständige Beispiele: [Firefly III Integration](#firefly-iii-integration).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Transformationstypen
|
||||||
|
|
||||||
|
Es gibt **14 unterstützte Transformationstypen**, die als Pipeline kombiniert werden können:
|
||||||
|
|
||||||
|
### 1. **trim** - Leerzeichen entfernen
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "trim" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: ` Coop Pronto ` → Output: `Coop Pronto`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. **lowercase** - Zu Kleinbuchstaben
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "lowercase" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: `COOP PRONTO CHUR` → Output: `coop pronto chur`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. **uppercase** - Zu Grossbuchstaben
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "uppercase" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: `Coop Pronto Chur` → Output: `COOP PRONTO CHUR`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. **ucwordsfirst** - Grossschreibung nach Trennzeichen
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "ucwordsfirst" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- `COOP PRONTO CHUR` → `Coop Pronto Chur`
|
||||||
|
- `migros-rail city` → `Migros-Rail City`
|
||||||
|
- `O'NEILL STORE` → `O'Neill Store`
|
||||||
|
|
||||||
|
Trennzeichen: Leerzeichen, Bindestrich, Apostroph, Slash, Punkt, Komma, Semikolon, Doppelpunkt, Klammern.
|
||||||
|
|
||||||
|
> **Guard:** Wenn der Eingabe-String bereits sowohl Groß- als auch Kleinbuchstaben enthält (gemischte Groß-/Kleinschreibung), wird er unverändert zurückgegeben. So werden bereits korrekt formatierte Strings wie `"Coop pronto chur"` nicht verändert. Vollständig groß- oder kleingeschriebene Strings werden weiterhin verarbeitet.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. **replace** - String-Replacement
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "replace", "search": " ", "replace": " " }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: `Coop Pronto` → Output: `Coop Pronto`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. **split** - Spalte teilen
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "split", "delimiter": ";", "part": 0 }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: `Coop Pronto Chur;7007 Chur` → Output: `Coop Pronto Chur`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. **regex** - Regex-Ersetzung
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "regex", "pattern": "^(.*?);.*$", "replace": "$1" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Kein Match → Originalwert bleibt **unverändert** (pipeline-sicher)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 8. **regexextract** - Regex-Extraktion
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "regexextract", "pattern": "(\\d{4,} [^;]+)" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Kein Match → leerer String (**nicht** pipeline-sicher)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 9. **dateformat** - Datum-Umformat
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "dateformat", "fromFormat": "d.m.Y", "toFormat": "Y-m-d" }
|
||||||
|
```
|
||||||
|
|
||||||
|
- Input: `10.12.2025` → Output: `2025-12-10`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 10. **truncate** - String kürzen
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "truncate", "maxLength": 100 }
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 11. **constantvalue** - Konstanten-Wert aus Metadaten
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sourceColumn": "_constant_",
|
||||||
|
"transformations": [{ "type": "constantvalue", "metadataKey": "account_iban" }],
|
||||||
|
"outputColumn": "account_iban",
|
||||||
|
"outputAction": "create"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 12. **map** - Spalte kopieren
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "type": "map" }
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 13. **pipeline** - Verschachtelte Pipeline
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "pipeline",
|
||||||
|
"steps": [
|
||||||
|
{ "type": "trim" },
|
||||||
|
{ "type": "lowercase" },
|
||||||
|
{ "type": "ucwordsfirst" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 14. **timeperiod** - Zeit einer Tagesperiode zuordnen
|
||||||
|
|
||||||
|
Parst eine Zeitangabe und gibt das Label des passenden Perioden-Bereichs zurück.
|
||||||
|
Unterstützt mitternachtübergreifende Bereiche (z. B. 22:00–03:59).
|
||||||
|
Gibt `default` (standardmäßig leer) zurück, wenn keine Periode passt oder die Eingabe ungültig ist.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "timeperiod",
|
||||||
|
"timeFormat": "H:i:s",
|
||||||
|
"periods": [
|
||||||
|
{ "from": "04:00:00", "to": "08:59:59", "label": "Morgen" },
|
||||||
|
{ "from": "09:00:00", "to": "10:59:59", "label": "Vormittag" },
|
||||||
|
{ "from": "11:00:00", "to": "13:59:59", "label": "Mittag" },
|
||||||
|
{ "from": "14:00:00", "to": "17:59:59", "label": "Nachmittag" },
|
||||||
|
{ "from": "18:00:00", "to": "21:59:59", "label": "Abend" },
|
||||||
|
{ "from": "22:00:00", "to": "03:59:59", "label": "Nacht" }
|
||||||
|
],
|
||||||
|
"default": ""
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `"09:30:00"` → `"Vormittag"`
|
||||||
|
- `"23:00:00"` → `"Nacht"` (mitternachtübergreifender Bereich)
|
||||||
|
- `"02:00:00"` → `"Nacht"` (mitternachtübergreifender Bereich)
|
||||||
|
- `""` oder nicht parsbare Eingabe → `""`
|
||||||
|
|
||||||
|
`timeFormat` folgt der PHP-Syntax `DateTime::createFromFormat` (Standard: `H:i:s`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Zeilen-Filterung — `skipIf`
|
||||||
|
|
||||||
|
Zeilen können durch einen Top-Level-Schlüssel `skipIf` in der Konfiguration ausgeschlossen werden.
|
||||||
|
Der Wert ist ein Filter-Knoten — entweder eine einzelne Bedingung oder eine verschachtelte `and`/`or`-Gruppe.
|
||||||
|
|
||||||
|
**Einzelne Bedingung:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
"skipIf": { "column": "Buchungstext", "operator": "equals", "value": "Saldovortrag" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**AND-Gruppe:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
"skipIf": {
|
||||||
|
"and": [
|
||||||
|
{ "column": "Beschreibung1", "operator": "empty" },
|
||||||
|
{ "column": "Beschreibung2", "operator": "empty" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verschachtelte AND/OR-Gruppen:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
"skipIf": {
|
||||||
|
"or": [
|
||||||
|
{ "column": "Amount", "operator": "gt", "value": "10000" },
|
||||||
|
{
|
||||||
|
"and": [
|
||||||
|
{ "column": "Type", "operator": "equals", "value": "Saldo" },
|
||||||
|
{ "column": "Notes", "operator": "empty" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Unterstützte Operatoren:**
|
||||||
|
|
||||||
|
| Operator | Passt wenn… |
|
||||||
|
|---|---|
|
||||||
|
| `empty` | Spaltenwert ist leer |
|
||||||
|
| `not-empty` | Spaltenwert ist nicht leer |
|
||||||
|
| `equals` | Spaltenwert gleich `"value"` |
|
||||||
|
| `not-equals` | Spaltenwert ungleich `"value"` |
|
||||||
|
| `contains` | Spaltenwert enthält `"value"` |
|
||||||
|
| `not-contains` | Spaltenwert enthält `"value"` nicht |
|
||||||
|
| `matches` | Spaltenwert entspricht Regex `"pattern"` |
|
||||||
|
| `not-matches` | Spaltenwert entspricht Regex `"pattern"` nicht |
|
||||||
|
| `gt` | `(float) Spalte > (float) value` |
|
||||||
|
| `gte` | `(float) Spalte >= (float) value` |
|
||||||
|
| `lt` | `(float) Spalte < (float) value` |
|
||||||
|
| `lte` | `(float) Spalte <= (float) value` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Pipeline-Beispiel
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sourceColumn": "Buchungstext",
|
||||||
|
"transformations": [
|
||||||
|
{ "type": "trim" },
|
||||||
|
{ "type": "replace", "search": " ", "replace": " " },
|
||||||
|
{ "type": "lowercase" },
|
||||||
|
{ "type": "ucwordsfirst" }
|
||||||
|
],
|
||||||
|
"outputColumn": "description",
|
||||||
|
"outputAction": "overwrite"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verarbeitung:**
|
||||||
|
|
||||||
|
1. `" COOP PRONTO "` → trim → `"COOP PRONTO"`
|
||||||
|
2. `"COOP PRONTO"` → replace → `"COOP PRONTO"`
|
||||||
|
3. `"COOP PRONTO"` → lowercase → `"coop pronto"`
|
||||||
|
4. `"coop pronto"` → ucwordsfirst → `"Coop Pronto"`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-Referenz
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php bin/transformer.php <command> [input] [config] [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Kommandos
|
||||||
|
|
||||||
|
| Kommando | Beschreibung |
|
||||||
|
| -------- | ------------- |
|
||||||
|
| `test` | Test-Run (max. 10 Zeilen) |
|
||||||
|
| `transform` | Vollständige Transformation |
|
||||||
|
| `validate` | Konfiguration validieren |
|
||||||
|
| `auto-import` | Verzeichnis-Überwachung |
|
||||||
|
| `help` | Hilfe anzeigen |
|
||||||
|
|
||||||
|
### Optionen
|
||||||
|
|
||||||
|
| Option | Beschreibung |
|
||||||
|
| -------- | ------------- |
|
||||||
|
| `--debug`, `-d` | Debug-Modus aktivieren |
|
||||||
|
| `--rows=N` | Max. N Zeilen (test-Kommando) |
|
||||||
|
| `--output=FILE`, `-o` | Output-Pfad |
|
||||||
|
| `--do-import` | Nach der Transformation in Firefly III importieren (`transform`) |
|
||||||
|
| `--strict` | Strikte Validierung |
|
||||||
|
| `--watch` | Kontinuierliche Überwachung |
|
||||||
|
| `--interval=SEC` | Prüfintervall in Sekunden |
|
||||||
|
| `--dry-run` | Simulationsmodus |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Debug-Modus
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php bin/transformer.php test input.csv config/config.json --debug
|
||||||
|
```
|
||||||
|
|
||||||
|
Der Debug-Modus protokolliert Ereignisse in folgenden Kategorien:
|
||||||
|
|
||||||
|
| Kategorie | Wann |
|
||||||
|
| ----------- | ------ |
|
||||||
|
| `transformer` | Anfang/Ende Transformation |
|
||||||
|
| `csv_reader` | Beim CSV lesen |
|
||||||
|
| `metadata` | Bei Metadaten-Extraktion |
|
||||||
|
| `metadata_warning` | Bei Problemen |
|
||||||
|
| `transformation` | Bei jeder Transformation |
|
||||||
|
| `csv_writer` | Beim CSV schreiben |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Firefly III Integration
|
||||||
|
|
||||||
|
Drei Betriebsmodi decken alle typischen Deployment-Szenarien ab.
|
||||||
|
|
||||||
|
**`chunkSize`** (optional, Standard: 0 = deaktiviert): Die Output-CSV wird vor dem Import in Blöcke von maximal N Datenzeilen aufgeteilt. Jeder Block wird als separate Anfrage gesendet. Das verhindert serverseitige Timeouts bei grossen Dateien (Faustregel: ~3–4 s/Transaktion im HTTP-Modus). Der `timeout`-Wert gilt pro Block, nicht für den gesamten Lauf.
|
||||||
|
|
||||||
|
**`chunkRetries`** (optional, Standard: 0 = kein Retry): Anzahl zusätzlicher Importversuche pro Block nach dem ersten. Bei einem Fehler wiederholt der Importer den Upload bis zu dieser Anzahl, bevor er abbricht. Nur wirksam wenn `chunkSize > 0`.
|
||||||
|
|
||||||
|
**`chunkRetryDelay`** (optional, Standard: 0 = keine Pause): Pause in Sekunden vor jedem Block-Request ab dem zweiten Block sowie zwischen Wiederholungsversuchen desselben fehlgeschlagenen Blocks. Ein einziger Wert für Cooldown und Retry-Back-off. Nur wirksam wenn `chunkSize > 0`.
|
||||||
|
|
||||||
|
**`connectionTimeout`** (optional, Standard: 10): Maximale Wartezeit in Sekunden für den Aufbau der TCP-Verbindung zum Importer-Server. Unabhängig von `timeout` (der die gesamte Übertragungsdauer begrenzt). Nur im Modus `http`.
|
||||||
|
|
||||||
|
### Modus `cli`
|
||||||
|
|
||||||
|
Transformer und Importer auf demselben Server.
|
||||||
|
|
||||||
|
```json
|
||||||
|
"fireflyImport": {
|
||||||
|
"mode": "cli",
|
||||||
|
"jsonConfig": "/opt/firefly-data-importer/storage/configurations/ubs-import.json",
|
||||||
|
"importerCommand": "php /opt/firefly-data-importer/artisan importer:import",
|
||||||
|
"chunkSize": 50,
|
||||||
|
"chunkRetries": 3,
|
||||||
|
"chunkRetryDelay": 10,
|
||||||
|
"timeout": 300,
|
||||||
|
"environment": {
|
||||||
|
"FIREFLY_III_URL": "https://localhost",
|
||||||
|
"FIREFLY_III_ACCESS_TOKEN": "your-token-here"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Modus `docker`
|
||||||
|
|
||||||
|
Transformer lokal, Importer in Docker. Das Ausgabeverzeichnis muss als Volume eingebunden sein. `jsonConfig` ist der Pfad **innerhalb des Containers**.
|
||||||
|
|
||||||
|
```json
|
||||||
|
"fireflyImport": {
|
||||||
|
"mode": "docker",
|
||||||
|
"jsonConfig": "/import/configs/ubs-import.json",
|
||||||
|
"importerCommand": "docker exec firefly-importer php artisan importer:import",
|
||||||
|
"chunkSize": 50,
|
||||||
|
"chunkRetries": 3,
|
||||||
|
"chunkRetryDelay": 10,
|
||||||
|
"timeout": 300
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Modus `http`
|
||||||
|
|
||||||
|
Transformer lokal, Importer über HTTP(S) erreichbar. Benötigt `ext-curl`.
|
||||||
|
|
||||||
|
**Voraussetzungen auf dem Importer-Server:**
|
||||||
|
|
||||||
|
```text
|
||||||
|
CAN_POST_FILES=true
|
||||||
|
AUTO_IMPORT_SECRET=<secret> # mindestens 16 Zeichen
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
"fireflyImport": {
|
||||||
|
"mode": "http",
|
||||||
|
"importerUrl": "https://importer.your-server.com",
|
||||||
|
"personalSecret": "your-auto-import-secret-min-16-chars",
|
||||||
|
"accessToken": "your-firefly-iii-personal-access-token",
|
||||||
|
"jsonConfig": "config/ubs-import.json",
|
||||||
|
"chunkSize": 50,
|
||||||
|
"chunkRetries": 3,
|
||||||
|
"chunkRetryDelay": 10,
|
||||||
|
"connectionTimeout": 10,
|
||||||
|
"timeout": 300
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Die Anfrage geht an `POST {importerUrl}/autoupload?secret={personalSecret}` mit CSV und JSON-Config als Multipart-Felder. `accessToken` wird als `Authorization: Bearer` gesendet. Falls `FIREFLY_III_ACCESS_TOKEN` bereits in der Importer-Umgebung gesetzt ist, kann `accessToken` weggelassen werden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Serverseitige Konfiguration
|
||||||
|
|
||||||
|
Bei grossen Importen liegt der Engpass meist auf dem Firefly III Data Importer-Server, nicht im Transformer. Die folgenden Einstellungen gehören in die Umgebung des Importers (`.env` oder `docker-compose.yml`):
|
||||||
|
|
||||||
|
| Einstellung | Empfohlener Wert | Hinweis |
|
||||||
|
|---|---|---|
|
||||||
|
| `PHP_MEMORY_LIMIT` | `512M` – `2048M` | Docker-Umgebungsvariable. Erhöhen, wenn PHP mit „Allowed memory size exhausted" abbricht. |
|
||||||
|
| `CONNECTION_TIMEOUT` | `60` | Sekunden für den TCP-Verbindungsaufbau zu Firefly III. Standard ~31 s (π × 10). |
|
||||||
|
| `IGNORE_DUPLICATE_ERRORS` | `true` | Doppelte Transaktionswarnungen bei Wiederholungsimporten unterdrücken. |
|
||||||
|
|
||||||
|
**nginx Reverse Proxy** (falls vorhanden):
|
||||||
|
```nginx
|
||||||
|
proxy_read_timeout 600s; # muss länger sein als der längste Einzelblock-Import
|
||||||
|
client_max_body_size 64M; # muss die grösste Chunk-CSV abdecken
|
||||||
|
```
|
||||||
|
|
||||||
|
**Docker Compose** Beispiel:
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
firefly-importer:
|
||||||
|
environment:
|
||||||
|
- PHP_MEMORY_LIMIT=1024M
|
||||||
|
- CONNECTION_TIMEOUT=60
|
||||||
|
- IGNORE_DUPLICATE_ERRORS=true
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Verwendung
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Nur transformieren (kein Import)
|
||||||
|
php bin/transformer.php transform input.csv config/config.json
|
||||||
|
|
||||||
|
# Transformieren und in Firefly III importieren
|
||||||
|
php bin/transformer.php transform input.csv config/config.json --do-import
|
||||||
|
|
||||||
|
# Watch-Modus: automatisch transformieren und importieren bei neuer CSV
|
||||||
|
php bin/transformer.php auto-import config/config.json --watch
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architektur
|
||||||
|
|
||||||
|
```text
|
||||||
|
bin/transformer.php (CLI Entry Point)
|
||||||
|
↓
|
||||||
|
TransformerEngine (Orchestrierung)
|
||||||
|
├─ ConfigurationLoader (Config laden/validieren)
|
||||||
|
├─ CsvReader (CSV einlesen)
|
||||||
|
├─ MetadataExtractor (Metadaten mit Regex)
|
||||||
|
├─ ColumnTransformer (Transformationen anwenden)
|
||||||
|
├─ CsvWriter (CSV schreiben)
|
||||||
|
├─ FireflyImporter (Firefly III Integration)
|
||||||
|
└─ DebugLogger (Debug-Protokolle)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Klasse | Verantwortung |
|
||||||
|
| -------- | --------------- |
|
||||||
|
| `TransformerEngine` | Orchestriert gesamten Workflow |
|
||||||
|
| `ConfigurationLoader` | Lädt und validiert JSON-Konfiguration |
|
||||||
|
| `CsvReader` | Liest CSV mit Metadaten |
|
||||||
|
| `MetadataExtractor` | Extrahiert Metadaten mit Regex |
|
||||||
|
| `ColumnTransformer` | Transformiert Spalten (Pipeline) |
|
||||||
|
| `CsvWriter` | Schreibt CSV |
|
||||||
|
| `FireflyImporter` | Importiert in Firefly III |
|
||||||
|
| `DebugLogger` | Statischer Logger für Debug |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fehlerbehandlung
|
||||||
|
|
||||||
|
### Häufige Fehler
|
||||||
|
|
||||||
|
#### "Input file not found"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Prüfe Dateipfad
|
||||||
|
ls -la input.csv
|
||||||
|
|
||||||
|
# Nutze absoluten Pfad wenn relativ nicht funktioniert
|
||||||
|
php bin/transformer.php transform /absolute/path/input.csv config.json
|
||||||
|
```
|
||||||
|
|
||||||
|
#### "Missing metadata: account_iban"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Prüfe erste Zeilen des CSV
|
||||||
|
head -5 input.csv
|
||||||
|
|
||||||
|
# Überprüfe lineNumber und regex in config.json
|
||||||
|
php bin/transformer.php validate config.json input.csv --debug
|
||||||
|
```
|
||||||
|
|
||||||
|
#### "Invalid JSON"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php -r "json_decode(file_get_contents('config/config.json'), true) or die('JSON invalid');"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### "Configuration: 'csvStructure.headerLine' required"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
diff config/config.json config/config.example.json
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Version & Änderungen
|
||||||
|
|
||||||
|
**v1.0.0 (03. Mai 2026)**
|
||||||
|
|
||||||
|
- ✅ Initial Release
|
||||||
|
- ✅ 14 Transformationstypen
|
||||||
|
- ✅ Metadaten-Extraktion mit Regex
|
||||||
|
- ✅ Debug-Modus
|
||||||
|
- ✅ Firefly III Integration (cli / docker / http)
|
||||||
|
- ✅ Vollständige Dokumentation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Lizenz:** GPL-3.0
|
||||||
|
**Author:** PHP CSV Transformer Project
|
||||||
|
**Repository:** [git.andare.ch/david.reindl/ff-imp-preprocessor](https://git.andare.ch/david.reindl/ff-imp-preprocessor)
|
||||||
@ -16,9 +16,10 @@ require_once __DIR__ . '/../vendor/autoload.php';
|
|||||||
use UbsCsvTransformer\TransformerEngine;
|
use UbsCsvTransformer\TransformerEngine;
|
||||||
use UbsCsvTransformer\ConfigurationLoader;
|
use UbsCsvTransformer\ConfigurationLoader;
|
||||||
use UbsCsvTransformer\FireflyImporter;
|
use UbsCsvTransformer\FireflyImporter;
|
||||||
|
use UbsCsvTransformer\DebugLogger;
|
||||||
|
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// CLI-Argument-Verarbeitung
|
// CLI argument processing
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
|
|
||||||
$argc = $_SERVER['argc'] ?? 0;
|
$argc = $_SERVER['argc'] ?? 0;
|
||||||
@ -29,10 +30,10 @@ if ($argc < 2) {
|
|||||||
exit(0);
|
exit(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Debug-Modus aktivierbar
|
// Debug mode can be enabled
|
||||||
$debug = in_array('--debug', $argv) || in_array('-d', $argv);
|
$debug = in_array('--debug', $argv) || in_array('-d', $argv);
|
||||||
|
|
||||||
// Extrahiere Kommando
|
// Extract command
|
||||||
$command = $argv[1];
|
$command = $argv[1];
|
||||||
|
|
||||||
try {
|
try {
|
||||||
@ -54,11 +55,26 @@ try {
|
|||||||
// ============================================================================
|
// ============================================================================
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Zeige Hilfe und Verwendungsanleitung
|
* Returns true when the active shell locale is German (de_*)
|
||||||
|
*/
|
||||||
|
function isGermanLocale(): bool
|
||||||
|
{
|
||||||
|
foreach (['LANG', 'LC_ALL', 'LC_MESSAGES', 'LANGUAGE'] as $var) {
|
||||||
|
$val = getenv($var);
|
||||||
|
if ($val !== false && $val !== '') {
|
||||||
|
return str_starts_with(strtolower($val), 'de');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Show help and usage instructions
|
||||||
*/
|
*/
|
||||||
function showHelp(): void
|
function showHelp(): void
|
||||||
{
|
{
|
||||||
echo <<<'HELP'
|
if (isGermanLocale()) {
|
||||||
|
echo <<<'HELP_DE'
|
||||||
╔════════════════════════════════════════════════════════════════════════════╗
|
╔════════════════════════════════════════════════════════════════════════════╗
|
||||||
║ Firefly Import Preprocessor - Kommandozeilen-Tool ║
|
║ Firefly Import Preprocessor - Kommandozeilen-Tool ║
|
||||||
║ ║
|
║ ║
|
||||||
@ -84,10 +100,10 @@ KOMMANDOS:
|
|||||||
Transformiert eine komplette CSV-Datei
|
Transformiert eine komplette CSV-Datei
|
||||||
Optionen:
|
Optionen:
|
||||||
--output=FILE, -o Output-Pfad (Standard: input-transformed.csv)
|
--output=FILE, -o Output-Pfad (Standard: input-transformed.csv)
|
||||||
--no-import Nicht automatisch in Firefly III importieren
|
--do-import Nach der Transformation in Firefly III importieren
|
||||||
Beispiel:
|
Beispiel:
|
||||||
transformer transform ubs-export.csv config.json
|
transformer transform ubs-export.csv config.json
|
||||||
transformer transform ubs-export.csv config.json -o import.csv
|
transformer transform ubs-export.csv config.json --do-import
|
||||||
|
|
||||||
validate [config] [options]
|
validate [config] [options]
|
||||||
Validiert die Konfigurationsdatei
|
Validiert die Konfigurationsdatei
|
||||||
@ -170,26 +186,150 @@ KONFIGURATION:
|
|||||||
|
|
||||||
DOKUMENTATION:
|
DOKUMENTATION:
|
||||||
|
|
||||||
Siehe README.md und UBS_Transformer_Guide.md für vollständige Dokumentation
|
Siehe README.md für vollständige Dokumentation
|
||||||
|
|
||||||
LIZENZ:
|
LIZENZ:
|
||||||
|
|
||||||
MIT License
|
GPL 3
|
||||||
|
|
||||||
HELP;
|
HELP_DE;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
echo <<<'HELP_EN'
|
||||||
|
╔════════════════════════════════════════════════════════════════════════════╗
|
||||||
|
║ Firefly Import Preprocessor - Command Line Tool ║
|
||||||
|
║ ║
|
||||||
|
║ A lightweight PHP 8 tool for transforming UBS E-Banking exports ║
|
||||||
|
║ into a Firefly III compatible format. ║
|
||||||
|
╚════════════════════════════════════════════════════════════════════════════╝
|
||||||
|
|
||||||
|
USAGE:
|
||||||
|
transformer [command] [options]
|
||||||
|
|
||||||
|
COMMANDS:
|
||||||
|
|
||||||
|
test [input] [config] [options]
|
||||||
|
Tests the transformation with a limited number of rows
|
||||||
|
Options:
|
||||||
|
--rows=N Process only N rows (default: 10)
|
||||||
|
--output=FILE, -o Also write result to file
|
||||||
|
Example:
|
||||||
|
transformer test ubs-export.csv config.json --rows=5
|
||||||
|
transformer test ubs-export.csv config.json -o test-output.csv
|
||||||
|
|
||||||
|
transform [input] [config] [options]
|
||||||
|
Transforms a complete CSV file
|
||||||
|
Options:
|
||||||
|
--output=FILE, -o Output path (default: input-transformed.csv)
|
||||||
|
--do-import Import into Firefly III after transformation
|
||||||
|
Example:
|
||||||
|
transformer transform ubs-export.csv config.json
|
||||||
|
transformer transform ubs-export.csv config.json --do-import
|
||||||
|
|
||||||
|
validate [config] [options]
|
||||||
|
Validates the configuration file
|
||||||
|
Options:
|
||||||
|
--strict Strict validation (recommended)
|
||||||
|
Example:
|
||||||
|
transformer validate config.json
|
||||||
|
transformer validate config.json --strict
|
||||||
|
|
||||||
|
auto-import [config] [options]
|
||||||
|
Monitors source directory and processes new files
|
||||||
|
Options:
|
||||||
|
--watch Continuous monitoring (daemon mode)
|
||||||
|
--interval=SEC Check interval in seconds (default: 60)
|
||||||
|
--dry-run Show what would be done (no actual processing)
|
||||||
|
Example:
|
||||||
|
transformer auto-import config.json
|
||||||
|
transformer auto-import config.json --watch --interval=30
|
||||||
|
|
||||||
|
help, -h, --help
|
||||||
|
Show this help
|
||||||
|
|
||||||
|
GLOBAL OPTIONS:
|
||||||
|
--debug, -d Enable debug mode (detailed output)
|
||||||
|
|
||||||
|
INSTALLATION:
|
||||||
|
|
||||||
|
1. PHP 8.1+ must be installed
|
||||||
|
php --version
|
||||||
|
|
||||||
|
2. Autoloader setup (choose one):
|
||||||
|
Option A: With Composer (recommended)
|
||||||
|
composer install
|
||||||
|
Option B: Manual - files in directory structure:
|
||||||
|
ff-imp-preprocessor/
|
||||||
|
├── bin/transformer.php
|
||||||
|
├── src/*.php
|
||||||
|
└── config/config.json
|
||||||
|
|
||||||
|
3. Make executable:
|
||||||
|
chmod +x bin/transformer.php
|
||||||
|
|
||||||
|
4. Adjust configuration:
|
||||||
|
cp config/config.example.json config/config.json
|
||||||
|
nano config/config.json
|
||||||
|
|
||||||
|
EXAMPLES:
|
||||||
|
|
||||||
|
# Test transformation (first 5 rows)
|
||||||
|
./bin/transformer test data/ubs-export.csv config/config.json --rows=5
|
||||||
|
|
||||||
|
# Full transformation
|
||||||
|
./bin/transformer transform data/ubs-export.csv config/config.json \
|
||||||
|
--output=output/firefly-import.csv
|
||||||
|
|
||||||
|
# Validate configuration
|
||||||
|
./bin/transformer validate config/config.json --strict
|
||||||
|
|
||||||
|
# Start auto-import with monitoring
|
||||||
|
./bin/transformer auto-import config/config.json --watch
|
||||||
|
|
||||||
|
# Process only next file
|
||||||
|
./bin/transformer auto-import config/config.json
|
||||||
|
|
||||||
|
CONFIGURATION:
|
||||||
|
|
||||||
|
The config.json must have the following structure:
|
||||||
|
{
|
||||||
|
"metadata": { "extractionRules": {...} },
|
||||||
|
"csvStructure": { "delimiter": ";", ... },
|
||||||
|
"columnTransformations": { ... },
|
||||||
|
"fireflyImport": { "apiUrl": "...", "apiKey": "..." },
|
||||||
|
"directories": {
|
||||||
|
"source": "./import/source",
|
||||||
|
"output": "./import/output",
|
||||||
|
"archive": "./import/archive",
|
||||||
|
"error": "./import/error"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
DOCUMENTATION:
|
||||||
|
|
||||||
|
See README.md for full documentation
|
||||||
|
|
||||||
|
LICENSE:
|
||||||
|
|
||||||
|
GPL 3
|
||||||
|
|
||||||
|
HELP_EN;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Expandiert ~ zu absolutem Home-Verzeichnis und löst relative Pfade auf
|
* Expands ~ to absolute home directory and resolves relative paths
|
||||||
*/
|
*/
|
||||||
function expandPath(string $path): string
|
function expandPath(string $path): string
|
||||||
{
|
{
|
||||||
if (str_starts_with($path, '~/') || $path === '~') {
|
if (str_starts_with($path, '~/') || $path === '~') {
|
||||||
$home = getenv('HOME') ?: posix_getpwuid(posix_getuid())['dir'];
|
$homeEnv = getenv('HOME');
|
||||||
|
$pwInfo = posix_getpwuid(posix_getuid());
|
||||||
|
$home = $homeEnv !== false && $homeEnv !== '' ? $homeEnv : ($pwInfo !== false ? $pwInfo['dir'] : '~');
|
||||||
$path = $home . substr($path, 1);
|
$path = $home . substr($path, 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Relative Pfade gegen cwd auflösen (ohne realpath, damit nicht-existierende Dirs erlaubt sind)
|
// Resolve relative paths against cwd (without realpath, so non-existent dirs are allowed)
|
||||||
if (!str_starts_with($path, '/')) {
|
if (!str_starts_with($path, '/')) {
|
||||||
$path = getcwd() . '/' . $path;
|
$path = getcwd() . '/' . $path;
|
||||||
}
|
}
|
||||||
@ -198,7 +338,7 @@ function expandPath(string $path): string
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Parse CLI-Optionen in assoziatives Array
|
* Parses CLI options into an associative array
|
||||||
*/
|
*/
|
||||||
function parseOptions(array $argv, int $startIndex = 0): array
|
function parseOptions(array $argv, int $startIndex = 0): array
|
||||||
{
|
{
|
||||||
@ -217,9 +357,9 @@ function parseOptions(array $argv, int $startIndex = 0): array
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Teste Transformation mit begrenzter Zeilenzahl
|
* Tests transformation with a limited number of rows
|
||||||
*/
|
*/
|
||||||
function handleTest($argc, $argv): void
|
function handleTest(int $argc, array $argv): void
|
||||||
{
|
{
|
||||||
if ($argc < 4) {
|
if ($argc < 4) {
|
||||||
throw new Exception("Usage: transformer test [input-file] [config-file] [options]");
|
throw new Exception("Usage: transformer test [input-file] [config-file] [options]");
|
||||||
@ -234,10 +374,10 @@ function handleTest($argc, $argv): void
|
|||||||
$outputFile = $options['output'] ?? $options['o'] ?? null;
|
$outputFile = $options['output'] ?? $options['o'] ?? null;
|
||||||
|
|
||||||
if (!file_exists($inputFile)) {
|
if (!file_exists($inputFile)) {
|
||||||
throw new Exception("Input-Datei nicht gefunden: $inputFile");
|
throw new Exception("Input file not found: $inputFile");
|
||||||
}
|
}
|
||||||
if (!file_exists($configFile)) {
|
if (!file_exists($configFile)) {
|
||||||
throw new Exception("Konfigurationsdatei nicht gefunden: $configFile");
|
throw new Exception("Configuration file not found: $configFile");
|
||||||
}
|
}
|
||||||
|
|
||||||
echo "\n📊 TEST-MODUS: Verarbeite max. $maxRows Zeilen\n";
|
echo "\n📊 TEST-MODUS: Verarbeite max. $maxRows Zeilen\n";
|
||||||
@ -288,13 +428,17 @@ function handleTest($argc, $argv): void
|
|||||||
echo "\n💾 Output-Datei: $outputFile\n";
|
echo "\n💾 Output-Datei: $outputFile\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if ($debug) {
|
||||||
|
echo DebugLogger::format(true);
|
||||||
|
}
|
||||||
|
|
||||||
echo "\n✅ Test erfolgreich!\n\n";
|
echo "\n✅ Test erfolgreich!\n\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transformiere komplette CSV-Datei
|
* Transforms a complete CSV file
|
||||||
*/
|
*/
|
||||||
function handleTransform($argc, $argv): void
|
function handleTransform(int $argc, array $argv): void
|
||||||
{
|
{
|
||||||
if ($argc < 4) {
|
if ($argc < 4) {
|
||||||
throw new Exception("Usage: transformer transform [input-file] [config-file] [options]");
|
throw new Exception("Usage: transformer transform [input-file] [config-file] [options]");
|
||||||
@ -306,21 +450,23 @@ function handleTransform($argc, $argv): void
|
|||||||
$debug = isset($options['debug']) || isset($options['d']);
|
$debug = isset($options['debug']) || isset($options['d']);
|
||||||
|
|
||||||
$outputFile = $options['output'] ?? $options['o'] ?? null;
|
$outputFile = $options['output'] ?? $options['o'] ?? null;
|
||||||
|
$doImport = isset($options['do-import']);
|
||||||
|
$resetImport = isset($options['reset-import']);
|
||||||
|
|
||||||
if (!file_exists($inputFile)) {
|
if (!file_exists($inputFile)) {
|
||||||
throw new Exception("Input-Datei nicht gefunden: $inputFile");
|
throw new Exception("Input file not found: $inputFile");
|
||||||
}
|
}
|
||||||
if (!file_exists($configFile)) {
|
if (!file_exists($configFile)) {
|
||||||
throw new Exception("Konfigurationsdatei nicht gefunden: $configFile");
|
throw new Exception("Configuration file not found: $configFile");
|
||||||
}
|
}
|
||||||
|
|
||||||
echo "\n🚀 TRANSFORMATION STARTEN\n";
|
echo "\n🚀 TRANSFORMATION\n";
|
||||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n";
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n";
|
||||||
|
|
||||||
$configLoader = new ConfigurationLoader($configFile);
|
$configLoader = new ConfigurationLoader($configFile);
|
||||||
$configLoader->load();
|
$config = $configLoader->load();
|
||||||
|
|
||||||
// --output überschreibt Zielverzeichnis und Dateiname aus der Konfiguration
|
// --output overrides target directory and filename from configuration
|
||||||
if ($outputFile !== null) {
|
if ($outputFile !== null) {
|
||||||
$outputFile = expandPath($outputFile);
|
$outputFile = expandPath($outputFile);
|
||||||
$configLoader->set('directories.output', dirname($outputFile));
|
$configLoader->set('directories.output', dirname($outputFile));
|
||||||
@ -330,17 +476,164 @@ function handleTransform($argc, $argv): void
|
|||||||
$engine = new TransformerEngine($configLoader, $debug);
|
$engine = new TransformerEngine($configLoader, $debug);
|
||||||
$result = $engine->transform($inputFile);
|
$result = $engine->transform($inputFile);
|
||||||
|
|
||||||
echo "✅ Transformation erfolgreich!\n";
|
echo "✅ Transformation complete!\n";
|
||||||
echo " Output-Datei: " . ($result['outputFile'] ?? 'N/A') . "\n";
|
echo " Output file: " . ($result['outputFile'] ?? 'N/A') . "\n";
|
||||||
echo " Zeilen transformiert: " . ($result['rowsProcessed'] ?? 0) . "\n";
|
echo " Rows transformed: " . ($result['rowsProcessed'] ?? 0) . "\n";
|
||||||
|
|
||||||
echo "\n✅ Fertig!\n\n";
|
if ($doImport) {
|
||||||
|
if (!empty($config['fireflyImport'])) {
|
||||||
|
echo "\n🚀 FIREFLY III IMPORT\n";
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n";
|
||||||
|
|
||||||
|
$fireflyConfig = $config['fireflyImport'];
|
||||||
|
$importer = new FireflyImporter($fireflyConfig);
|
||||||
|
|
||||||
|
$outputCsv = $result['outputFile'] ?? '';
|
||||||
|
|
||||||
|
if ($resetImport) {
|
||||||
|
$importer->resetImportState($outputCsv);
|
||||||
|
echo " ℹ️ Import state cleared — starting fresh.\n";
|
||||||
|
} elseif ($importer->hasResumeState($outputCsv)) {
|
||||||
|
$stateRaw = @file_get_contents($outputCsv . '.ffi-state.json');
|
||||||
|
$stateData = is_string($stateRaw) ? json_decode($stateRaw, true) : null;
|
||||||
|
if (is_array($stateData)) {
|
||||||
|
$doneSoFar = count((array) ($stateData['completed_chunks'] ?? []));
|
||||||
|
$totalSoFar = (int) ($stateData['total_chunks'] ?? 0);
|
||||||
|
echo " ℹ️ Resuming previous import: {$doneSoFar}/{$totalSoFar} chunks already completed.\n";
|
||||||
|
echo " Add --reset-import to start over from scratch.\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$inChunkedMode = false;
|
||||||
|
|
||||||
|
// Detect the system timezone: PHP CLI often defaults to UTC even when the OS
|
||||||
|
// is configured otherwise. Read /etc/localtime symlink to get the real zone.
|
||||||
|
$localTzName = date_default_timezone_get();
|
||||||
|
if (is_link('/etc/localtime')) {
|
||||||
|
$link = (string) readlink('/etc/localtime');
|
||||||
|
if (preg_match('#zoneinfo/(.+)$#', $link, $tzMatch) === 1) {
|
||||||
|
$localTzName = $tzMatch[1];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$localTz = new \DateTimeZone($localTzName);
|
||||||
|
|
||||||
|
$importer->setProgressCallback(
|
||||||
|
function (string $event, array $data) use (&$inChunkedMode, $localTz): void {
|
||||||
|
static $chunkHadRetry = false;
|
||||||
|
$ts = '[' . (new \DateTimeImmutable('now', $localTz))->format('H:i:s') . ']';
|
||||||
|
if ($event === 'chunk_start') {
|
||||||
|
$inChunkedMode = true;
|
||||||
|
$chunkHadRetry = false;
|
||||||
|
echo " ⏳ {$ts} Chunk {$data['chunk']}/{$data['total']} ({$data['rows']} rows)...";
|
||||||
|
flush();
|
||||||
|
} elseif ($event === 'chunk_done') {
|
||||||
|
$d = round((float) ($data['result']['duration'] ?? 0), 1);
|
||||||
|
$status = $data['result']['success'] ? 'done' : 'failed';
|
||||||
|
if ($chunkHadRetry) {
|
||||||
|
// After retries the line is already terminated — print a full new line
|
||||||
|
echo " ✅ {$ts} Chunk {$data['chunk']}/{$data['total']}: {$status} ({$d}s)\n";
|
||||||
|
} else {
|
||||||
|
echo " {$status} ({$d}s)\n";
|
||||||
|
}
|
||||||
|
flush();
|
||||||
|
} elseif ($event === 'chunk_retry') {
|
||||||
|
$chunkHadRetry = true;
|
||||||
|
$err = (string) ($data['error'] ?? '');
|
||||||
|
$msg = $err !== '' ? " — {$err}" : '';
|
||||||
|
echo "\n 🔄 {$ts} Chunk {$data['chunk']}/{$data['total']}: attempt {$data['attempt']}/{$data['max_attempts']} failed{$msg}\n";
|
||||||
|
flush();
|
||||||
|
} elseif ($event === 'chunk_delay') {
|
||||||
|
$ctx = ($data['context'] ?? '') === 'retry' ? 'retry' : 'next chunk';
|
||||||
|
echo " ⏸ {$ts} Waiting {$data['seconds']}s before {$ctx}...\n";
|
||||||
|
flush();
|
||||||
|
} elseif ($event === 'chunk_skip') {
|
||||||
|
echo " ⏭ {$ts} Chunk {$data['chunk']}/{$data['total']} already completed — skipping\n";
|
||||||
|
flush();
|
||||||
|
} elseif ($event === 'request_start' && !$inChunkedMode) {
|
||||||
|
echo " ⏳ {$ts} Sending to importer...\n";
|
||||||
|
flush();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
$outputDelimiter = (string) ($config['csvStructure']['outputDelimiter'] ?? ',');
|
||||||
|
$importResult = $importer->importChunked($outputCsv, $outputDelimiter);
|
||||||
|
|
||||||
|
$duration = $importResult['duration'] ?? null;
|
||||||
|
$chunks = $importResult['chunks'] ?? null;
|
||||||
|
$summary = $importResult['summary'] ?? null;
|
||||||
|
|
||||||
|
if ($importResult['success']) {
|
||||||
|
if (is_array($summary)) {
|
||||||
|
$created = $summary['created'] ?? 0;
|
||||||
|
$byType = $summary['by_type'] ?? [];
|
||||||
|
$completed = $summary['completed'] ?? false;
|
||||||
|
$duplicates = $summary['duplicates'] ?? 0;
|
||||||
|
$errors = $summary['errors'] ?? [];
|
||||||
|
|
||||||
|
$status = $completed ? '✅ Import complete!' : '⚠️ Import finished (no "Done!" marker received)';
|
||||||
|
echo $status . ($duration !== null ? " ({$duration}s)" : '') . "\n";
|
||||||
|
echo " Transactions created: {$created}\n";
|
||||||
|
|
||||||
|
$typeLabels = ['deposit' => 'Deposits', 'withdrawal' => 'Withdrawals', 'transfer' => 'Transfers'];
|
||||||
|
foreach ($byType as $type => $count) {
|
||||||
|
$label = $typeLabels[$type] ?? ucfirst($type);
|
||||||
|
echo " {$label}: {$count}\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($duplicates > 0) {
|
||||||
|
echo " ⚠️ Duplicates skipped: {$duplicates}\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!empty($errors)) {
|
||||||
|
$errorCount = count($errors);
|
||||||
|
echo " ❌ Errors ({$errorCount}):\n";
|
||||||
|
foreach ($errors as $err) {
|
||||||
|
echo " - {$err}\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
echo "✅ Import complete!" . ($duration !== null ? " ({$duration}s)" : '') . "\n";
|
||||||
|
if (!empty($importResult['output']['stdout'])) {
|
||||||
|
echo $importResult['output']['stdout'] . "\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
$errorMsg = $importResult['error']
|
||||||
|
?? ('HTTP ' . ($importResult['exit_code'] ?? '?'));
|
||||||
|
$chunksData = $importResult['chunks'] ?? null;
|
||||||
|
if (is_array($chunksData) && $chunksData['total'] > 1) {
|
||||||
|
$failedChunk = $chunksData['done'] + 1;
|
||||||
|
echo "❌ Import failed at chunk {$failedChunk}/{$chunksData['total']}: {$errorMsg}\n";
|
||||||
|
echo " Run the same command again to resume from where it stopped.\n";
|
||||||
|
echo " Add --reset-import to start over from scratch.\n";
|
||||||
|
} else {
|
||||||
|
echo "❌ Import failed: {$errorMsg}\n";
|
||||||
|
}
|
||||||
|
// Only dump the raw response body in debug mode
|
||||||
|
if ($debug && !empty($importResult['output']['stdout'])) {
|
||||||
|
echo $importResult['output']['stdout'] . "\n";
|
||||||
|
}
|
||||||
|
if (!empty($importResult['output']['stderr'])) {
|
||||||
|
echo $importResult['output']['stderr'] . "\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
echo "\n⚠️ --do-import specified but no fireflyImport section found in config.\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($debug) {
|
||||||
|
echo DebugLogger::format(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "\n✅ Done!\n\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Validiere Konfigurationsdatei
|
* Validates the configuration file
|
||||||
*/
|
*/
|
||||||
function handleValidate($argc, $argv): void
|
function handleValidate(int $argc, array $argv): void
|
||||||
{
|
{
|
||||||
if ($argc < 3) {
|
if ($argc < 3) {
|
||||||
throw new Exception("Usage: transformer validate [config-file] [options]");
|
throw new Exception("Usage: transformer validate [config-file] [options]");
|
||||||
@ -351,7 +644,7 @@ function handleValidate($argc, $argv): void
|
|||||||
$strict = isset($options['strict']);
|
$strict = isset($options['strict']);
|
||||||
|
|
||||||
if (!file_exists($configFile)) {
|
if (!file_exists($configFile)) {
|
||||||
throw new Exception("Konfigurationsdatei nicht gefunden: $configFile");
|
throw new Exception("Configuration file not found: $configFile");
|
||||||
}
|
}
|
||||||
|
|
||||||
echo "\n✔️ KONFIGURATION VALIDIEREN\n";
|
echo "\n✔️ KONFIGURATION VALIDIEREN\n";
|
||||||
@ -362,7 +655,7 @@ function handleValidate($argc, $argv): void
|
|||||||
try {
|
try {
|
||||||
$config = $configLoader->load();
|
$config = $configLoader->load();
|
||||||
|
|
||||||
// Basis-Validierung
|
// Basic validation
|
||||||
echo "✅ JSON-Format valide\n";
|
echo "✅ JSON-Format valide\n";
|
||||||
|
|
||||||
$required = ['metadata', 'csvStructure', 'columnTransformations'];
|
$required = ['metadata', 'csvStructure', 'columnTransformations'];
|
||||||
@ -377,7 +670,7 @@ function handleValidate($argc, $argv): void
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Firefly-Validierung
|
// Firefly validation
|
||||||
if (isset($config['fireflyImport'])) {
|
if (isset($config['fireflyImport'])) {
|
||||||
echo "✅ Firefly III Konfiguration vorhanden\n";
|
echo "✅ Firefly III Konfiguration vorhanden\n";
|
||||||
if (empty($config['fireflyImport']['apiUrl'])) {
|
if (empty($config['fireflyImport']['apiUrl'])) {
|
||||||
@ -396,7 +689,7 @@ function handleValidate($argc, $argv): void
|
|||||||
echo "⚠️ Firefly III Konfiguration nicht vorhanden (optional)\n";
|
echo "⚠️ Firefly III Konfiguration nicht vorhanden (optional)\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
// Verzeichnisse-Validierung
|
// Directory validation
|
||||||
if (isset($config['directories'])) {
|
if (isset($config['directories'])) {
|
||||||
echo "✅ Verzeichnisse konfiguriert\n";
|
echo "✅ Verzeichnisse konfiguriert\n";
|
||||||
$dirs = ['source', 'output', 'archive', 'error'];
|
$dirs = ['source', 'output', 'archive', 'error'];
|
||||||
@ -416,14 +709,14 @@ function handleValidate($argc, $argv): void
|
|||||||
echo "\n⚠️ Konfiguration hat Warnungen aber ist funktional\n\n";
|
echo "\n⚠️ Konfiguration hat Warnungen aber ist funktional\n\n";
|
||||||
}
|
}
|
||||||
} catch (Exception $e) {
|
} catch (Exception $e) {
|
||||||
throw new Exception("Validierungsfehler: " . $e->getMessage());
|
throw new Exception("Validation error: " . $e->getMessage());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Auto-Import mit Verzeichnis-Überwachung
|
* Auto-import with directory monitoring
|
||||||
*/
|
*/
|
||||||
function handleAutoImport($argc, $argv): void
|
function handleAutoImport(int $argc, array $argv): void
|
||||||
{
|
{
|
||||||
if ($argc < 3) {
|
if ($argc < 3) {
|
||||||
throw new Exception("Usage: transformer auto-import [config-file] [options]");
|
throw new Exception("Usage: transformer auto-import [config-file] [options]");
|
||||||
@ -434,7 +727,7 @@ function handleAutoImport($argc, $argv): void
|
|||||||
$debug = isset($options['debug']) || isset($options['d']);
|
$debug = isset($options['debug']) || isset($options['d']);
|
||||||
|
|
||||||
if (!file_exists($configFile)) {
|
if (!file_exists($configFile)) {
|
||||||
throw new Exception("Konfigurationsdatei nicht gefunden: $configFile");
|
throw new Exception("Configuration file not found: $configFile");
|
||||||
}
|
}
|
||||||
|
|
||||||
$configLoader = new ConfigurationLoader($configFile);
|
$configLoader = new ConfigurationLoader($configFile);
|
||||||
@ -448,7 +741,7 @@ function handleAutoImport($argc, $argv): void
|
|||||||
$watch = isset($options['watch']);
|
$watch = isset($options['watch']);
|
||||||
$interval = isset($options['interval']) ? (int)$options['interval'] : 60;
|
$interval = isset($options['interval']) ? (int)$options['interval'] : 60;
|
||||||
|
|
||||||
// Verzeichnisse erstellen
|
// Create directories
|
||||||
foreach ([$sourceDir, $outputDir, $archiveDir, $errorDir] as $dir) {
|
foreach ([$sourceDir, $outputDir, $archiveDir, $errorDir] as $dir) {
|
||||||
if (!is_dir($dir)) {
|
if (!is_dir($dir)) {
|
||||||
mkdir($dir, 0755, true);
|
mkdir($dir, 0755, true);
|
||||||
@ -476,7 +769,9 @@ function handleAutoImport($argc, $argv): void
|
|||||||
|
|
||||||
if ($watch) {
|
if ($watch) {
|
||||||
echo "⏳ Drücke Ctrl+C zum Beenden.\n\n";
|
echo "⏳ Drücke Ctrl+C zum Beenden.\n\n";
|
||||||
while (true) {
|
$running = true;
|
||||||
|
/** @phpstan-ignore while.alwaysTrue (intentional infinite loop — terminated only via Ctrl+C / SIGINT) */
|
||||||
|
while ($running) {
|
||||||
processImportDirectory($sourceDir, $outputDir, $archiveDir, $errorDir, $config, $configFile, $dryRun, $debug);
|
processImportDirectory($sourceDir, $outputDir, $archiveDir, $errorDir, $config, $configFile, $dryRun, $debug);
|
||||||
sleep($interval);
|
sleep($interval);
|
||||||
}
|
}
|
||||||
@ -486,9 +781,9 @@ function handleAutoImport($argc, $argv): void
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Verarbeite Verzeichnis mit CSV-Dateien
|
* Processes directory containing CSV files
|
||||||
*/
|
*/
|
||||||
function processImportDirectory($sourceDir, $outputDir, $archiveDir, $errorDir, $config, $configFile, $dryRun = false, $debug = false): void
|
function processImportDirectory(string $sourceDir, string $outputDir, string $archiveDir, string $errorDir, array $config, string $configFile, bool $dryRun = false, bool $debug = false): void
|
||||||
{
|
{
|
||||||
if (!is_dir($sourceDir)) {
|
if (!is_dir($sourceDir)) {
|
||||||
return;
|
return;
|
||||||
@ -516,10 +811,10 @@ function processImportDirectory($sourceDir, $outputDir, $archiveDir, $errorDir,
|
|||||||
$result = $engine->transform($file);
|
$result = $engine->transform($file);
|
||||||
$outputFile = $result['outputFile'] ?? $outputFile;
|
$outputFile = $result['outputFile'] ?? $outputFile;
|
||||||
|
|
||||||
// Archiviere Original-Datei
|
// Archive original file
|
||||||
$archiveFile = $archiveDir . '/' . $basename;
|
$archiveFile = $archiveDir . '/' . $basename;
|
||||||
if (!rename($file, $archiveFile)) {
|
if (!rename($file, $archiveFile)) {
|
||||||
throw new Exception("Konnte nicht archivieren");
|
throw new Exception("Could not archive file");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Firefly Import
|
// Firefly Import
|
||||||
@ -534,7 +829,7 @@ function processImportDirectory($sourceDir, $outputDir, $archiveDir, $errorDir,
|
|||||||
echo "❌ " . $e->getMessage() . "\n";
|
echo "❌ " . $e->getMessage() . "\n";
|
||||||
|
|
||||||
if (!$dryRun) {
|
if (!$dryRun) {
|
||||||
// Verschiebe zu Error-Verzeichnis
|
// Move to error directory
|
||||||
$errorFile = $errorDir . '/' . $basename;
|
$errorFile = $errorDir . '/' . $basename;
|
||||||
@rename($file, $errorFile);
|
@rename($file, $errorFile);
|
||||||
}
|
}
|
||||||
|
|||||||
@ -40,6 +40,7 @@
|
|||||||
"vimeo/psalm": "^5.0"
|
"vimeo/psalm": "^5.0"
|
||||||
},
|
},
|
||||||
"suggest": {
|
"suggest": {
|
||||||
|
"ext-curl": "Benötigt für Modus fireflyImport.mode=http (HTTP-Upload an den Data Importer)",
|
||||||
"monolog/monolog": "For advanced logging capabilities (optional)",
|
"monolog/monolog": "For advanced logging capabilities (optional)",
|
||||||
"guzzlehttp/guzzle": "For Firefly III HTTP client integration (optional)"
|
"guzzlehttp/guzzle": "For Firefly III HTTP client integration (optional)"
|
||||||
},
|
},
|
||||||
|
|||||||
@ -199,11 +199,20 @@
|
|||||||
],
|
],
|
||||||
|
|
||||||
"fireflyImport": {
|
"fireflyImport": {
|
||||||
"jsonConfig": "/opt/firefly/import-config.json",
|
"mode": "docker",
|
||||||
"importerCommand": "docker exec -it firefly-importer php artisan importer:import",
|
|
||||||
|
"jsonConfig": "/import/configs/ubs-import.json",
|
||||||
|
|
||||||
|
"importerCommand": "docker exec firefly-importer php artisan importer:import",
|
||||||
|
|
||||||
"autoImport": false,
|
"autoImport": false,
|
||||||
"deleteAfterImport": false,
|
"deleteAfterImport": false,
|
||||||
|
"chunkSize": 0,
|
||||||
|
"chunkRetries": 0,
|
||||||
|
"chunkRetryDelay": 0,
|
||||||
|
"connectionTimeout": 10,
|
||||||
"timeout": 300,
|
"timeout": 300,
|
||||||
|
|
||||||
"environment": {
|
"environment": {
|
||||||
"FIREFLY_III_URL": "https://your-firefly.com",
|
"FIREFLY_III_URL": "https://your-firefly.com",
|
||||||
"FIREFLY_III_ACCESS_TOKEN": "your-token-here"
|
"FIREFLY_III_ACCESS_TOKEN": "your-token-here"
|
||||||
|
|||||||
79
config/ff-import-ubs-account.json
Normal file
79
config/ff-import-ubs-account.json
Normal file
@ -0,0 +1,79 @@
|
|||||||
|
{
|
||||||
|
"version": 3,
|
||||||
|
"source": "ff3-importer-2.1.1",
|
||||||
|
"created_at": "2026-05-04T22:22:39+02:00",
|
||||||
|
"date": "Y-m-d",
|
||||||
|
"default_account": 1,
|
||||||
|
"delimiter": "comma",
|
||||||
|
"headers": true,
|
||||||
|
"rules": true,
|
||||||
|
"webhooks": true,
|
||||||
|
"skip_form": false,
|
||||||
|
"add_import_tag": true,
|
||||||
|
"roles": [
|
||||||
|
"amount_debit",
|
||||||
|
"amount_credit",
|
||||||
|
"date_transaction",
|
||||||
|
"date_process",
|
||||||
|
"opposing-name",
|
||||||
|
"tags-comma",
|
||||||
|
"description",
|
||||||
|
"opposing-iban",
|
||||||
|
"opposing-number",
|
||||||
|
"note",
|
||||||
|
"account-iban",
|
||||||
|
"currency-code"
|
||||||
|
],
|
||||||
|
"do_mapping": [
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false,
|
||||||
|
false
|
||||||
|
],
|
||||||
|
"mapping": {},
|
||||||
|
"duplicate_detection_method": "classic",
|
||||||
|
"ignore_duplicate_lines": false,
|
||||||
|
"unique_column_index": 0,
|
||||||
|
"unique_column_type": "note",
|
||||||
|
"pseudo_identifier": [],
|
||||||
|
"flow": "file",
|
||||||
|
"content_type": "csv",
|
||||||
|
"camt_type": "",
|
||||||
|
"custom_tag": "test001",
|
||||||
|
"identifier": "0",
|
||||||
|
"connection": "0",
|
||||||
|
"ignore_spectre_categories": false,
|
||||||
|
"grouped_transaction_handling": "",
|
||||||
|
"use_entire_opposing_address": false,
|
||||||
|
"map_all_data": true,
|
||||||
|
"pending_transactions": false,
|
||||||
|
"access_token": "",
|
||||||
|
"accounts": {},
|
||||||
|
"new_accounts": [],
|
||||||
|
"date_range": "",
|
||||||
|
"date_range_number": 30,
|
||||||
|
"date_range_unit": "d",
|
||||||
|
"date_range_not_after_unit": "",
|
||||||
|
"date_range_not_after_number": 0,
|
||||||
|
"date_not_before": "",
|
||||||
|
"date_not_after": "",
|
||||||
|
"nordigen_country": "",
|
||||||
|
"nordigen_bank": "",
|
||||||
|
"nordigen_requisitions": {},
|
||||||
|
"nordigen_max_days": "90",
|
||||||
|
"lunch_flow_api_key": "",
|
||||||
|
"enable_banking_country": "",
|
||||||
|
"enable_banking_bank": "",
|
||||||
|
"enable_banking_auth_id": "",
|
||||||
|
"enable_banking_sessions": [],
|
||||||
|
"conversion": false,
|
||||||
|
"ignore_duplicate_transactions": true
|
||||||
|
}
|
||||||
53
config/firefly-import-config.example.json
Normal file
53
config/firefly-import-config.example.json
Normal file
@ -0,0 +1,53 @@
|
|||||||
|
{
|
||||||
|
"_comment_1": "Firefly III Data Importer – configuration file (format version 3)",
|
||||||
|
"_comment_2": "Created for the output of config-ubs-account.json (11 columns, comma-delimited)",
|
||||||
|
"_comment_3": "Adjust: set 'default_account' to your Firefly III asset account ID (number, not name)",
|
||||||
|
"_comment_4": "Docs: https://docs.firefly-iii.org/references/data-importer/json/",
|
||||||
|
|
||||||
|
"version": 3,
|
||||||
|
"flow": "csv",
|
||||||
|
|
||||||
|
"date": "Y-m-d",
|
||||||
|
"delimiter": "comma",
|
||||||
|
"headers": true,
|
||||||
|
"conversion": false,
|
||||||
|
|
||||||
|
"default_account": 1,
|
||||||
|
|
||||||
|
"rules": true,
|
||||||
|
"skip_form": true,
|
||||||
|
"add_import_tag": true,
|
||||||
|
"duplicate_detection_method": "classic",
|
||||||
|
"ignore_duplicate_lines": true,
|
||||||
|
"ignore_duplicate_transactions": true,
|
||||||
|
|
||||||
|
"roles": [
|
||||||
|
"amount_debit",
|
||||||
|
"amount_credit",
|
||||||
|
"date_transaction",
|
||||||
|
"date_process",
|
||||||
|
"opposing-name",
|
||||||
|
"tags-comma",
|
||||||
|
"description",
|
||||||
|
"opposing-iban",
|
||||||
|
"note",
|
||||||
|
"account-iban",
|
||||||
|
"currency-code"
|
||||||
|
],
|
||||||
|
|
||||||
|
"do_mapping": {
|
||||||
|
"0": false,
|
||||||
|
"1": false,
|
||||||
|
"2": false,
|
||||||
|
"3": false,
|
||||||
|
"4": false,
|
||||||
|
"5": false,
|
||||||
|
"6": false,
|
||||||
|
"7": false,
|
||||||
|
"8": false,
|
||||||
|
"9": false,
|
||||||
|
"10": false
|
||||||
|
},
|
||||||
|
|
||||||
|
"mapping": {}
|
||||||
|
}
|
||||||
@ -3,30 +3,36 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transformiert Spalten gemäß Konfiguration
|
* Transforms columns according to configuration
|
||||||
*
|
*
|
||||||
* Unterstützte Transformationstypen (canonical names):
|
* Supported transformation types (canonical names):
|
||||||
* - map: Spalte kopieren/umbenennen (Standard)
|
* - map: Copy/rename column (default)
|
||||||
* - replace: String-Replacement (str_replace)
|
* - replace: String replacement (str_replace)
|
||||||
* - regex: Regex-Replace mit preg_replace (Backreferenzen: $1, $2 …)
|
* - regex: Regex replace via preg_replace (backreferences: $1, $2 …)
|
||||||
* - dateformat: Datum-Formatierung
|
* - dateformat: Date formatting (toFormat: 'l' yields English weekday name)
|
||||||
* - split: Spalte bei Delimiter teilen
|
* - split: Split column at delimiter
|
||||||
* - regexextract: Mit Regex extrahieren
|
* - regexextract: Extract using regex
|
||||||
* - trim: Whitespace entfernen
|
* - trim: Remove whitespace
|
||||||
* - uppercase: In Grossbuchstaben umwandeln
|
* - uppercase: Convert to uppercase
|
||||||
* - lowercase: In Kleinbuchstaben umwandeln
|
* - lowercase: Convert to lowercase
|
||||||
* - ucwordsfirst: Ersten Buchstaben nach Worttrennern gross
|
* - ucwordsfirst: Capitalise first letter after word boundaries (only when input
|
||||||
* - truncate: String auf maximale Länge kürzen
|
* has no lowercase letters; strings already mixed-case are returned
|
||||||
* - constantvalue: Konstanten-Wert aus Metadaten
|
* unchanged)
|
||||||
* - pipeline: Mehrere Transformationen hintereinander (via steps[])
|
* - truncate: Truncate string to maximum length
|
||||||
* - custom: Custom PHP-Callback
|
* - constantvalue: Constant value from metadata
|
||||||
|
* - pipeline: Chain multiple transformations (via steps[])
|
||||||
|
* - custom: Custom PHP callback
|
||||||
|
* - timeperiod: Map a time string to a period label (morning, evening, …)
|
||||||
*
|
*
|
||||||
* Unterstützte outputAction-Werte:
|
* Supported outputAction values:
|
||||||
* - create / overwrite: Ziel-Spalte setzen (Standard)
|
* - create / overwrite: Set target column (default)
|
||||||
* - append: Wert anhängen
|
* - append: Append value directly; optional "appendDelimiter" inserts a separator
|
||||||
* - append-line: Wert auf neuer Zeile anhängen (kein Leerzeichen wenn Ziel leer)
|
* between existing and new value (skipped when target is still empty)
|
||||||
* - overwrite-if-empty: Nur setzen wenn Ziel-Spalte leer
|
* - append-if-not-empty: Like append, but skips entirely when the transformation result is
|
||||||
* - overwrite-if-not-empty: Nur setzen wenn Ergebnis nicht leer
|
* empty (safe for optional values like tags and notes lines)
|
||||||
|
* - append-line: Append value on new line (no leading newline if target is empty)
|
||||||
|
* - overwrite-if-empty: Only set if target column is empty
|
||||||
|
* - overwrite-if-not-empty: Only set if transformation result is not empty
|
||||||
*/
|
*/
|
||||||
class ColumnTransformer
|
class ColumnTransformer
|
||||||
{
|
{
|
||||||
@ -36,11 +42,11 @@ class ColumnTransformer
|
|||||||
private array $globalExceptions;
|
private array $globalExceptions;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialisiert ColumnTransformer mit Transformationsregeln
|
* Initialises ColumnTransformer with transformation rules
|
||||||
*
|
*
|
||||||
* @param array $transformations Transformationskonfiguration aus config.json
|
* @param array $transformations Transformation configuration from config.json
|
||||||
* @param array $metadata Extrahierte Metadaten aus CSV-Header
|
* @param array $metadata Extracted metadata from CSV header
|
||||||
* @param array $globalExceptions Globale Ausnahmeliste für ucwordsfirst
|
* @param array $globalExceptions Global exceptions list for ucwordsfirst
|
||||||
*/
|
*/
|
||||||
public function __construct(array $transformations, array $metadata = [], array $globalExceptions = [])
|
public function __construct(array $transformations, array $metadata = [], array $globalExceptions = [])
|
||||||
{
|
{
|
||||||
@ -51,36 +57,36 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transformiert eine einzelne Datenzeile
|
* Transforms a single data row
|
||||||
*
|
*
|
||||||
* Wendet alle definierten Transformationen auf die Zeile an.
|
* Applies all defined transformations to the row.
|
||||||
* Kann neue Spalten generieren (z.B. bei regex_extract).
|
* Can generate new columns (e.g. for regex_extract).
|
||||||
*
|
*
|
||||||
* @param array $row Datenzeile mit Header-Keys als Array-Keys
|
* @param array $row Data row with header keys as array keys
|
||||||
*
|
*
|
||||||
* @return array Transformierte Datenzeile
|
* @return array Transformed data row
|
||||||
*/
|
*/
|
||||||
public function transformRow(array $row): array
|
public function transformRow(array $row): array
|
||||||
{
|
{
|
||||||
$transformedRow = $row;
|
$transformedRow = $row;
|
||||||
|
|
||||||
foreach ($this->transformations as $config) {
|
foreach ($this->transformations as $config) {
|
||||||
// Multi-Output Detection (für split)
|
// Multi-output detection (for split)
|
||||||
if (isset($config['outputs']) && is_array($config['outputs'])) {
|
if (isset($config['outputs']) && is_array($config['outputs'])) {
|
||||||
// Multi-Output Transformation (z.B. split in mehrere Spalten)
|
// Multi-output transformation (e.g. split into multiple columns)
|
||||||
$multiOutputResult = $this->handleMultiOutputTransformation($transformedRow, $config);
|
$multiOutputResult = $this->handleMultiOutputTransformation($transformedRow, $config);
|
||||||
|
|
||||||
// Merge Ergebnisse in transformedRow
|
// Merge results into transformedRow
|
||||||
foreach ($multiOutputResult as $columnName => $value) {
|
foreach ($multiOutputResult as $columnName => $value) {
|
||||||
$transformedRow[$columnName] = $value;
|
$transformedRow[$columnName] = $value;
|
||||||
|
|
||||||
// Registriere neue Spalten
|
// Register new columns
|
||||||
if (!in_array($columnName, $this->outputColumns)) {
|
if (!in_array($columnName, $this->outputColumns)) {
|
||||||
$this->outputColumns[] = $columnName;
|
$this->outputColumns[] = $columnName;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fahre mit nächster Transformation fort
|
// Continue with next transformation
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -90,7 +96,7 @@ class ColumnTransformer
|
|||||||
|
|
||||||
if (empty($targetColumn)) {
|
if (empty($targetColumn)) {
|
||||||
throw new \RuntimeException(
|
throw new \RuntimeException(
|
||||||
"Transformation fehlt 'outputColumn' Feld: " . json_encode($config)
|
"Transformation missing 'outputColumn' field: " . json_encode($config)
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -124,23 +130,38 @@ class ColumnTransformer
|
|||||||
// Apply output action
|
// Apply output action
|
||||||
switch ($outputAction) {
|
switch ($outputAction) {
|
||||||
case 'append':
|
case 'append':
|
||||||
$transformedRow[$targetColumn] = ($transformedRow[$targetColumn] ?? '') . $resultValue;
|
$existing = $transformedRow[$targetColumn] ?? '';
|
||||||
|
if (isset($config['appendDelimiter']) && $existing !== '') {
|
||||||
|
$transformedRow[$targetColumn] = $existing . (string) $config['appendDelimiter'] . $resultValue;
|
||||||
|
} else {
|
||||||
|
$transformedRow[$targetColumn] = $existing . $resultValue;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case 'append-if-not-empty':
|
||||||
|
if ($resultValue !== '') {
|
||||||
|
$existing = $transformedRow[$targetColumn] ?? '';
|
||||||
|
if (isset($config['appendDelimiter']) && $existing !== '') {
|
||||||
|
$transformedRow[$targetColumn] = $existing . (string) $config['appendDelimiter'] . $resultValue;
|
||||||
|
} else {
|
||||||
|
$transformedRow[$targetColumn] = $existing . $resultValue;
|
||||||
|
}
|
||||||
|
}
|
||||||
break;
|
break;
|
||||||
case 'append-line':
|
case 'append-line':
|
||||||
// Wert auf neuer Zeile anhängen; kein führender Zeilenumbruch wenn Ziel leer
|
// Append value on new line; no leading newline if target is empty
|
||||||
if ($resultValue !== '') {
|
if ($resultValue !== '') {
|
||||||
$existing = $transformedRow[$targetColumn] ?? '';
|
$existing = $transformedRow[$targetColumn] ?? '';
|
||||||
$transformedRow[$targetColumn] = $existing !== '' ? $existing . "\n" . $resultValue : $resultValue;
|
$transformedRow[$targetColumn] = $existing !== '' ? $existing . "\n" . $resultValue : $resultValue;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case 'overwrite-if-empty':
|
case 'overwrite-if-empty':
|
||||||
// Nur überschreiben wenn Ziel-Spalte leer ist
|
// Only overwrite if target column is empty
|
||||||
if (($transformedRow[$targetColumn] ?? '') === '') {
|
if (($transformedRow[$targetColumn] ?? '') === '') {
|
||||||
$transformedRow[$targetColumn] = $resultValue;
|
$transformedRow[$targetColumn] = $resultValue;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case 'overwrite-if-not-empty':
|
case 'overwrite-if-not-empty':
|
||||||
// Nur überschreiben wenn das Transformations-Ergebnis nicht leer ist
|
// Only overwrite if the transformation result is not empty
|
||||||
if ($resultValue !== '') {
|
if ($resultValue !== '') {
|
||||||
$transformedRow[$targetColumn] = $resultValue;
|
$transformedRow[$targetColumn] = $resultValue;
|
||||||
}
|
}
|
||||||
@ -157,14 +178,14 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Wendet eine einzelne Transformation auf einen Stringwert an
|
* Applies a single transformation to a string value
|
||||||
*
|
*
|
||||||
* Normalisiert den Typ-Namen (snake_case, PascalCase, no-separator alle akzeptiert)
|
* Normalises the type name (snake_case, PascalCase, no-separator all accepted)
|
||||||
* und delegiert an die jeweilige transformXxx()-Methode.
|
* and delegates to the respective transformXxx() method.
|
||||||
*
|
*
|
||||||
* @param string $value Eingabewert
|
* @param string $value Input value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function applySingleTransformation(string $value, array $config): string
|
private function applySingleTransformation(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -205,6 +226,9 @@ class ColumnTransformer
|
|||||||
case 'pipeline':
|
case 'pipeline':
|
||||||
return $this->transformPipeline($value, $config);
|
return $this->transformPipeline($value, $config);
|
||||||
|
|
||||||
|
case 'timeperiod':
|
||||||
|
return $this->transformTimePeriod($value, $config);
|
||||||
|
|
||||||
case 'truncate':
|
case 'truncate':
|
||||||
$maxLength = (int)($config['maxLength'] ?? 255);
|
$maxLength = (int)($config['maxLength'] ?? 255);
|
||||||
return mb_substr($value, 0, $maxLength, 'UTF-8');
|
return mb_substr($value, 0, $maxLength, 'UTF-8');
|
||||||
@ -219,8 +243,8 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Normalisiert Transformationstyp-Namen: lowercase, Trennzeichen entfernt.
|
* Normalises transformation type names: lowercase, separators removed.
|
||||||
* Erlaubt z.B. dass 'dateformat' und 'dateFormat' beide funktionieren.
|
* Allows e.g. 'dateformat' and 'dateFormat' to both work.
|
||||||
*/
|
*/
|
||||||
private function normalizeTransformType(string $type): string
|
private function normalizeTransformType(string $type): string
|
||||||
{
|
{
|
||||||
@ -228,19 +252,19 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* String-Replacement Transformation
|
* String replacement transformation
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "replace",
|
* "type": "replace",
|
||||||
* "search": "Alt",
|
* "search": "old",
|
||||||
* "replace": "Neu"
|
* "replace": "new"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformReplace(string $value, array $config): string
|
private function transformReplace(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -255,22 +279,22 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Regex-Replace Transformation
|
* Regex replace transformation
|
||||||
*
|
*
|
||||||
* Wendet einen regulären Ausdruck auf den Wert an und ersetzt den Treffer.
|
* Applies a regular expression to the value and replaces the match.
|
||||||
* Backreferenz-Syntax: $1, $2 usw. im replace-String.
|
* Backreference syntax: $1, $2 etc. in the replace string.
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "regex",
|
* "type": "regex",
|
||||||
* "pattern": "SumUp \\*+(.*)",
|
* "pattern": "SumUp \\*+(.*)",
|
||||||
* "replace": "[$1]"
|
* "replace": "[$1]"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformRegex(string $value, array $config): string
|
private function transformRegex(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -288,19 +312,19 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Datum-Format Transformation
|
* Date format transformation
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "date_format",
|
* "type": "date_format",
|
||||||
* "fromFormat": "d.m.Y",
|
* "fromFormat": "d.m.Y",
|
||||||
* "toFormat": "Y-m-d"
|
* "toFormat": "Y-m-d"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformDate(string $value, array $config): string
|
private function transformDate(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -323,26 +347,26 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Split Transformation
|
* Split transformation
|
||||||
*
|
*
|
||||||
* Teilt einen Wert bei einem Delimiter und behaelt einen definierten Teil
|
* Splits a value at a delimiter and keeps a defined part
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: "Coop Pronto Chur;7007 Chur"
|
* Input: "Coop Pronto Chur;7007 Chur"
|
||||||
* Config: delimiter=";", part=0
|
* Config: delimiter=";", part=0
|
||||||
* Output: "Coop Pronto Chur"
|
* Output: "Coop Pronto Chur"
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "split",
|
* "type": "split",
|
||||||
* "delimiter": ";",
|
* "delimiter": ";",
|
||||||
* "part": 0
|
* "part": 0
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformSplit(string $value, array $config): string
|
private function transformSplit(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -370,16 +394,16 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Regex Extract Transformation
|
* Regex extract transformation
|
||||||
*
|
*
|
||||||
* Extrahiert einen Teil mit Regex und erstellt neue Spalte
|
* Extracts a portion using regex and creates a new column
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: "Coop Pronto Chur;7007 Chur"
|
* Input: "Coop Pronto Chur;7007 Chur"
|
||||||
* Config: pattern="(\d{4,} .*)"
|
* Config: pattern="(\d{4,} .*)"
|
||||||
* Output: "7007 Chur" (in neuer Spalte "Location")
|
* Output: "7007 Chur" (in new column "Location")
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "Location": {
|
* "Location": {
|
||||||
* "type": "regex_extract",
|
* "type": "regex_extract",
|
||||||
@ -388,10 +412,10 @@ class ColumnTransformer
|
|||||||
* }
|
* }
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return string|null Extrahierter Wert oder null
|
* @return string|null Extracted value or null
|
||||||
*/
|
*/
|
||||||
private function transformRegexExtract(string $value, array $config): ?string
|
private function transformRegexExtract(string $value, array $config): ?string
|
||||||
{
|
{
|
||||||
@ -421,22 +445,22 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Trim Transformation
|
* Trim transformation
|
||||||
*
|
*
|
||||||
* Entfernt Leerzeichen am Anfang und Ende eines Strings
|
* Removes whitespace from the beginning and end of a string
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "trim"
|
* "type": "trim"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: " Coop Pronto "
|
* Input: " Coop Pronto "
|
||||||
* Output: "Coop Pronto"
|
* Output: "Coop Pronto"
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformTrim(string $value): string
|
private function transformTrim(string $value): string
|
||||||
{
|
{
|
||||||
@ -444,22 +468,22 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lowercase Transformation
|
* Lowercase transformation
|
||||||
*
|
*
|
||||||
* Wandelt einen String in Kleinbuchstaben um (UTF-8 safe)
|
* Converts a string to lowercase (UTF-8 safe)
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "lowercase"
|
* "type": "lowercase"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: "COOP PRONTO CHUR"
|
* Input: "COOP PRONTO CHUR"
|
||||||
* Output: "coop pronto chur"
|
* Output: "coop pronto chur"
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformLowercase(string $value): string
|
private function transformLowercase(string $value): string
|
||||||
{
|
{
|
||||||
@ -467,22 +491,22 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Uppercase Transformation
|
* Uppercase transformation
|
||||||
*
|
*
|
||||||
* Wandelt einen String in Grossbuchstaben um (UTF-8 safe)
|
* Converts a string to uppercase (UTF-8 safe)
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "uppercase"
|
* "type": "uppercase"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: "Coop Pronto Chur"
|
* Input: "Coop Pronto Chur"
|
||||||
* Output: "COOP PRONTO CHUR"
|
* Output: "COOP PRONTO CHUR"
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformUppercase(string $value): string
|
private function transformUppercase(string $value): string
|
||||||
{
|
{
|
||||||
@ -490,65 +514,73 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Ucwords First Transformation
|
* Ucwords first transformation
|
||||||
*
|
*
|
||||||
* Grossschreibung nur des ersten Buchstabens nach Worttrennern.
|
* Capitalises only the first letter after word boundaries.
|
||||||
* Alle anderen Buchstaben werden zu Kleinbuchstaben.
|
* All other letters are converted to lowercase.
|
||||||
* Funktioniert auch, wenn Input komplett in Grossbuchstaben vorliegt.
|
* Works even when input is entirely in uppercase.
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "ucwords_first"
|
* "type": "ucwords_first"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Mit Ausnahmeliste (Wörter, die exakt erhalten bleiben):
|
* With exceptions list (words that are preserved exactly):
|
||||||
* ```
|
* ```
|
||||||
* "type": "ucwords_first",
|
* "type": "ucwords_first",
|
||||||
* "exceptions": ["SBB", "UBS", "AG", "GmbH"]
|
* "exceptions": ["SBB", "UBS", "AG", "GmbH"]
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Beispiele:
|
* Examples:
|
||||||
* "COOP PRONTO CHUR" → "Coop Pronto Chur"
|
* "COOP PRONTO CHUR" → "Coop Pronto Chur"
|
||||||
* "migros-rail city zuerich" → "Migros-Rail City Zuerich"
|
* "migros-rail city zuerich" → "Migros-Rail City Zuerich"
|
||||||
* "O'NEILL STORE" → "O'Neill Store"
|
* "O'NEILL STORE" → "O'Neill Store"
|
||||||
* "SAINT-JEAN-DE-MAURIENNE" → "Saint-Jean-De-Maurienne"
|
* "SAINT-JEAN-DE-MAURIENNE" → "Saint-Jean-De-Maurienne"
|
||||||
*
|
*
|
||||||
* Wortgrenzen definiert durch: Leerzeichen, Bindestrich, Apostroph,
|
* Word boundaries defined by: space, hyphen, apostrophe,
|
||||||
* Slash, Punkt, Komma, Semikolon, Doppelpunkt, Klammern, Anführungszeichen
|
* slash, period, comma, semicolon, colon, brackets, quotation marks
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert
|
* @return string Transformed value
|
||||||
*/
|
*/
|
||||||
private function transformUcwordsFirst(string $value, array $config = []): string
|
private function transformUcwordsFirst(string $value, array $config = []): string
|
||||||
{
|
{
|
||||||
// Schritt 1: Alles zu Kleinbuchstaben
|
// Guard: if the string already contains both uppercase and lowercase letters
|
||||||
|
// (i.e. mixed-case), it has already been intentionally formatted — leave it alone.
|
||||||
|
// Fully-uppercase or fully-lowercase strings are still processed so that patterns
|
||||||
|
// like "lowercase → ucwordsfirst" continue to work as expected.
|
||||||
|
if (preg_match('/\p{Lu}/u', $value) && preg_match('/\p{Ll}/u', $value)) {
|
||||||
|
return $value;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 1: Convert everything to lowercase
|
||||||
$value = mb_strtolower($value, 'UTF-8');
|
$value = mb_strtolower($value, 'UTF-8');
|
||||||
|
|
||||||
// Schritt 2: Definiere Wortgrenzen (Trennzeichen)
|
// Step 2: Define word boundaries (delimiters)
|
||||||
// Diese Zeichen markieren Grenzen, nach denen grossgeschrieben wird
|
// These characters mark boundaries after which capitalisation is applied
|
||||||
$delimiters = [
|
$delimiters = [
|
||||||
' ', // Leerzeichen
|
' ', // space
|
||||||
'-', // Bindestrich
|
'-', // hyphen
|
||||||
'\'', // Apostroph
|
'\'', // apostrophe
|
||||||
'/', // Slash
|
'/', // slash
|
||||||
'.', // Punkt
|
'.', // period
|
||||||
',', // Komma
|
',', // comma
|
||||||
';', // Semikolon
|
';', // semicolon
|
||||||
':', // Doppelpunkt
|
':', // colon
|
||||||
'(', // Oeffnende Klammer
|
'(', // opening parenthesis
|
||||||
')', // Schliessende Klammer
|
')', // closing parenthesis
|
||||||
'[', // Oeffnende eckige Klammer
|
'[', // opening square bracket
|
||||||
']', // Schliessende eckige Klammer
|
']', // closing square bracket
|
||||||
'{', // Oeffnende geschweifte Klammer
|
'{', // opening curly bracket
|
||||||
'}', // Schliessende geschweifte Klammer
|
'}', // closing curly bracket
|
||||||
'"', // Anführungszeichen
|
'"', // quotation mark
|
||||||
'&', // Ampersand
|
'&', // ampersand
|
||||||
'+' // Plus
|
'+' // plus
|
||||||
];
|
];
|
||||||
|
|
||||||
// Schritt 3: Regex-Pattern fuer "Stringanfang ODER Delimiter, gefolgt von Buchstabe"
|
// Step 3: Regex pattern for "start of string OR delimiter, followed by letter"
|
||||||
// Die u-Flag ermoeglicht Unicode-Unterstaetzung (\p{L})
|
// The u-flag enables Unicode support (\p{L})
|
||||||
$escapedDelimiters = array_map(function ($char) {
|
$escapedDelimiters = array_map(function ($char) {
|
||||||
return preg_quote($char, '/');
|
return preg_quote($char, '/');
|
||||||
}, $delimiters);
|
}, $delimiters);
|
||||||
@ -556,18 +588,18 @@ class ColumnTransformer
|
|||||||
|
|
||||||
$pattern = '/(^|[' . $delimiterPattern . '])(\p{L})/u';
|
$pattern = '/(^|[' . $delimiterPattern . '])(\p{L})/u';
|
||||||
|
|
||||||
// Schritt 4: Callback fuer preg_replace_callback
|
// Step 4: Callback for preg_replace_callback
|
||||||
// Grossschreibe den gefangenen Buchstaben (Capture Group 2)
|
// Capitalise the captured letter (capture group 2)
|
||||||
$callback = function (array $matches): string {
|
$callback = function (array $matches): string {
|
||||||
// $matches[1] = Stringanfang oder Trennzeichen
|
// $matches[1] = start of string or delimiter
|
||||||
// $matches[2] = Buchstabe, der grossgeschrieben werden soll
|
// $matches[2] = letter to be capitalised
|
||||||
return $matches[1] . mb_strtoupper($matches[2], 'UTF-8');
|
return $matches[1] . mb_strtoupper($matches[2], 'UTF-8');
|
||||||
};
|
};
|
||||||
|
|
||||||
// Schritt 5: Anwende Transformation
|
// Step 5: Apply transformation
|
||||||
$result = preg_replace_callback($pattern, $callback, $value) ?? $value;
|
$result = preg_replace_callback($pattern, $callback, $value) ?? $value;
|
||||||
|
|
||||||
// Schritt 6: Ausnahmeliste anwenden (Wörter die exakt erhalten bleiben sollen, z.B. SBB, UBS, GmbH)
|
// Step 6: Apply exceptions list (words to be preserved exactly, e.g. SBB, UBS, GmbH)
|
||||||
$exceptions = $config['exceptions'] ?? $this->globalExceptions;
|
$exceptions = $config['exceptions'] ?? $this->globalExceptions;
|
||||||
foreach ($exceptions as $exception) {
|
foreach ($exceptions as $exception) {
|
||||||
if (!is_string($exception) || $exception === '') {
|
if (!is_string($exception) || $exception === '') {
|
||||||
@ -581,12 +613,12 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Pipeline Transformation
|
* Pipeline transformation
|
||||||
*
|
*
|
||||||
* Wendet mehrere Transformationen nacheinander auf einen Wert an.
|
* Applies multiple transformations sequentially to a value.
|
||||||
* Jeder Schritt benutzt das Ergebnis des vorherigen Schrittes.
|
* Each step uses the result of the previous step.
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "Merchant": {
|
* "Merchant": {
|
||||||
* "type": "pipeline",
|
* "type": "pipeline",
|
||||||
@ -599,17 +631,17 @@ class ColumnTransformer
|
|||||||
* }
|
* }
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* Input: " COOP PRONTO CHUR "
|
* Input: " COOP PRONTO CHUR "
|
||||||
* Step 1 (trim): "COOP PRONTO CHUR"
|
* Step 1 (trim): "COOP PRONTO CHUR"
|
||||||
* Step 2 (lowercase): "coop pronto chur"
|
* Step 2 (lowercase): "coop pronto chur"
|
||||||
* Step 3 (ucwords_first): "Coop Pronto Chur"
|
* Step 3 (ucwords_first): "Coop Pronto Chur"
|
||||||
* Output: "Coop Pronto Chur"
|
* Output: "Coop Pronto Chur"
|
||||||
*
|
*
|
||||||
* @param string $value Ursprungswert
|
* @param string $value Source value
|
||||||
* @param array $config Transformationskonfiguration mit 'steps' Array
|
* @param array $config Transformation configuration with 'steps' array
|
||||||
*
|
*
|
||||||
* @return string Transformierter Wert nach allen Schritten
|
* @return string Transformed value after all steps
|
||||||
*/
|
*/
|
||||||
private function transformPipeline(string $value, array $config): string
|
private function transformPipeline(string $value, array $config): string
|
||||||
{
|
{
|
||||||
@ -619,7 +651,7 @@ class ColumnTransformer
|
|||||||
return $value;
|
return $value;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Wende jeden Schritt nacheinander an
|
// Apply each step sequentially
|
||||||
foreach ($steps as $step) {
|
foreach ($steps as $step) {
|
||||||
if (!empty($step['type'] ?? $step['transform'] ?? null)) {
|
if (!empty($step['type'] ?? $step['transform'] ?? null)) {
|
||||||
$value = $this->applySingleTransformation($value, $step);
|
$value = $this->applySingleTransformation($value, $step);
|
||||||
@ -630,23 +662,23 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Custom Callback Transformation
|
* Custom callback transformation
|
||||||
*
|
*
|
||||||
* Ruft eine Custom-Funktion auf, die komplexe Logik implementiert
|
* Calls a custom function implementing complex logic
|
||||||
*
|
*
|
||||||
* Konfiguration:
|
* Configuration:
|
||||||
* ```
|
* ```
|
||||||
* "type": "custom",
|
* "type": "custom",
|
||||||
* "callback": "myCustomFunction"
|
* "callback": "myCustomFunction"
|
||||||
* ```
|
* ```
|
||||||
*
|
*
|
||||||
* Die Callback-Funktion erhaelt die gesamte Zeile und gibt die
|
* The callback function receives the entire row and returns the
|
||||||
* modifizierte Zeile zurueck.
|
* modified row.
|
||||||
*
|
*
|
||||||
* @param array $row Gesamte Datenzeile
|
* @param array $row Complete data row
|
||||||
* @param array $config Transformationskonfiguration
|
* @param array $config Transformation configuration
|
||||||
*
|
*
|
||||||
* @return array Transformierte Datenzeile
|
* @return array Transformed data row
|
||||||
*/
|
*/
|
||||||
private function transformCustom(array $row, array $config): array
|
private function transformCustom(array $row, array $config): array
|
||||||
{
|
{
|
||||||
@ -664,10 +696,10 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Behandelt Multi-Output Transformationen
|
* Handles multi-output transformations
|
||||||
* Aktuell nur für 'split' implementiert.
|
* Currently only implemented for 'split'.
|
||||||
*
|
*
|
||||||
* Config-Beispiel:
|
* Config example:
|
||||||
* {
|
* {
|
||||||
* "outputs": ["FirstName", "LastName"],
|
* "outputs": ["FirstName", "LastName"],
|
||||||
* "sourceColumn": "FullName",
|
* "sourceColumn": "FullName",
|
||||||
@ -675,10 +707,10 @@ class ColumnTransformer
|
|||||||
* "delimiter": " "
|
* "delimiter": " "
|
||||||
* }
|
* }
|
||||||
*
|
*
|
||||||
* @param array $row Input-Zeile
|
* @param array $row Input row
|
||||||
* @param array $config Transformations-Konfiguration
|
* @param array $config Transformation configuration
|
||||||
* @return array Assoziatives Array: columnName => value
|
* @return array Associative array: columnName => value
|
||||||
* @throws \RuntimeException wenn Transformation-Type nicht unterstützt
|
* @throws \RuntimeException if transformation type is not supported
|
||||||
*/
|
*/
|
||||||
private function handleMultiOutputTransformation(array $row, array $config): array
|
private function handleMultiOutputTransformation(array $row, array $config): array
|
||||||
{
|
{
|
||||||
@ -687,39 +719,39 @@ class ColumnTransformer
|
|||||||
$transformType = $this->normalizeTransformType($config['type'] ?? '');
|
$transformType = $this->normalizeTransformType($config['type'] ?? '');
|
||||||
|
|
||||||
if (empty($outputs) || empty($sourceColumn) || empty($transformType)) {
|
if (empty($outputs) || empty($sourceColumn) || empty($transformType)) {
|
||||||
throw new \RuntimeException("Multi-Output Transformation benötigt 'outputs', 'sourceColumn' und 'type'");
|
throw new \RuntimeException("Multi-output transformation requires 'outputs', 'sourceColumn' and 'type'");
|
||||||
}
|
}
|
||||||
|
|
||||||
$sourceValue = $row[$sourceColumn] ?? '';
|
$sourceValue = $row[$sourceColumn] ?? '';
|
||||||
|
|
||||||
if ($transformType !== 'split') {
|
if ($transformType !== 'split') {
|
||||||
throw new \RuntimeException("Multi-Output nur für 'split' unterstützt, gegeben: {$transformType}");
|
throw new \RuntimeException("Multi-output only supported for 'split', given: {$transformType}");
|
||||||
}
|
}
|
||||||
|
|
||||||
return $this->handleMultiOutputSplit($sourceValue, $outputs, $config);
|
return $this->handleMultiOutputSplit($sourceValue, $outputs, $config);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Split-Transformation mit Multi-Output
|
* Split transformation with multi-output
|
||||||
* Teilt einen String und verteilt die Teile auf mehrere Spalten
|
* Splits a string and distributes the parts across multiple columns
|
||||||
*
|
*
|
||||||
* @param string $value Zu teilender String
|
* @param string $value String to split
|
||||||
* @param array $outputs Liste der Ziel-Spaltennamen
|
* @param array $outputs List of target column names
|
||||||
* @param array $config Transformation-Config
|
* @param array $config Transformation configuration
|
||||||
* @return array Assoziatives Array: columnName => value
|
* @return array Associative array: columnName => value
|
||||||
*/
|
*/
|
||||||
|
|
||||||
private function handleMultiOutputSplit(string $value, array $outputs, array $config): array
|
private function handleMultiOutputSplit(string $value, array $outputs, array $config): array
|
||||||
{
|
{
|
||||||
$delimiter = $config['delimiter'] ?? ';';
|
$delimiter = $config['delimiter'] ?? ';';
|
||||||
|
|
||||||
// Führe Split durch
|
// Perform split
|
||||||
$parts = explode($delimiter, $value);
|
$parts = explode($delimiter, $value);
|
||||||
|
|
||||||
// Mappe Parts zu Output-Spalten
|
// Map parts to output columns
|
||||||
$result = [];
|
$result = [];
|
||||||
foreach ($outputs as $index => $columnName) {
|
foreach ($outputs as $index => $columnName) {
|
||||||
// Wenn Teil existiert: verwenden (getrimmt) // Wenn nicht: leerer String
|
// If part exists: use it (trimmed) // If not: empty string
|
||||||
$result[$columnName] = isset($parts[$index]) ? trim($parts[$index]) : '';
|
$result[$columnName] = isset($parts[$index]) ? trim($parts[$index]) : '';
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -730,14 +762,89 @@ class ColumnTransformer
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Anzahl der Output-Spalten zurueck
|
* Returns the number of output columns
|
||||||
*
|
*
|
||||||
* Zaehlt Original-Spalten plus neu generierte Spalten (z.B. bei regex_extract)
|
* Counts original columns plus newly generated columns (e.g. from regex_extract)
|
||||||
*
|
*
|
||||||
* @return int Anzahl Output-Spalten
|
* @return int Number of output columns
|
||||||
*/
|
*/
|
||||||
public function getOutputColumns(): int
|
public function getOutputColumns(): int
|
||||||
{
|
{
|
||||||
return count(array_unique($this->outputColumns));
|
return count(array_unique($this->outputColumns));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Time-period transformer
|
||||||
|
*
|
||||||
|
* Maps a time string to a period label via a configurable list of ranges.
|
||||||
|
* Supports midnight-spanning ranges (e.g. "22:00:00" to "03:59:59").
|
||||||
|
* Returns the configured default (empty string by default) when no range matches
|
||||||
|
* or the input cannot be parsed.
|
||||||
|
*
|
||||||
|
* Configuration:
|
||||||
|
* ```json
|
||||||
|
* {
|
||||||
|
* "type": "timeperiod",
|
||||||
|
* "timeFormat": "H:i:s",
|
||||||
|
* "periods": [
|
||||||
|
* {"from": "04:00:00", "to": "08:59:59", "label": "Morgen"},
|
||||||
|
* {"from": "22:00:00", "to": "03:59:59", "label": "Nacht"}
|
||||||
|
* ],
|
||||||
|
* "default": ""
|
||||||
|
* }
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @param string $value Time string to evaluate
|
||||||
|
* @param array<string, mixed> $config Transformation configuration
|
||||||
|
* @return string Period label or default
|
||||||
|
*/
|
||||||
|
private function transformTimePeriod(string $value, array $config): string
|
||||||
|
{
|
||||||
|
$default = (string) ($config['default'] ?? '');
|
||||||
|
$timeFormat = (string) ($config['timeFormat'] ?? 'H:i:s');
|
||||||
|
/** @var array<int, array<string, string>> $periods */
|
||||||
|
$periods = $config['periods'] ?? [];
|
||||||
|
|
||||||
|
if ($value === '' || empty($periods)) {
|
||||||
|
return $default;
|
||||||
|
}
|
||||||
|
|
||||||
|
$parsed = \DateTime::createFromFormat($timeFormat, $value);
|
||||||
|
if ($parsed === false) {
|
||||||
|
return $default;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Represent time as total minutes from midnight for easy comparison
|
||||||
|
$minutes = (int) $parsed->format('H') * 60 + (int) $parsed->format('i');
|
||||||
|
|
||||||
|
foreach ($periods as $period) {
|
||||||
|
$fromStr = (string) ($period['from'] ?? '');
|
||||||
|
$toStr = (string) ($period['to'] ?? '');
|
||||||
|
$label = (string) ($period['label'] ?? '');
|
||||||
|
|
||||||
|
$fromParsed = \DateTime::createFromFormat($timeFormat, $fromStr);
|
||||||
|
$toParsed = \DateTime::createFromFormat($timeFormat, $toStr);
|
||||||
|
|
||||||
|
if ($fromParsed === false || $toParsed === false) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
$fromMin = (int) $fromParsed->format('H') * 60 + (int) $fromParsed->format('i');
|
||||||
|
$toMin = (int) $toParsed->format('H') * 60 + (int) $toParsed->format('i');
|
||||||
|
|
||||||
|
if ($fromMin <= $toMin) {
|
||||||
|
// Normal range (e.g. 04:00 – 08:59)
|
||||||
|
if ($minutes >= $fromMin && $minutes <= $toMin) {
|
||||||
|
return $label;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Midnight-spanning range (e.g. 22:00 – 03:59)
|
||||||
|
if ($minutes >= $fromMin || $minutes <= $toMin) {
|
||||||
|
return $label;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return $default;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -3,7 +3,7 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lädt und validiert JSON-Konfigurationsdateien
|
* Loads and validates JSON configuration files
|
||||||
*/
|
*/
|
||||||
class ConfigurationLoader
|
class ConfigurationLoader
|
||||||
{
|
{
|
||||||
@ -16,19 +16,19 @@ class ConfigurationLoader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lädt die Konfigurationsdatei
|
* Loads the configuration file
|
||||||
*
|
*
|
||||||
* @return array Die geladene und validierte Konfiguration
|
* @return array The loaded and validated configuration
|
||||||
* @throws \RuntimeException wenn Datei nicht gefunden oder ungültig
|
* @throws \RuntimeException if file not found or invalid
|
||||||
*/
|
*/
|
||||||
public function load(): array
|
public function load(): array
|
||||||
{
|
{
|
||||||
if (!file_exists($this->configFile)) {
|
if (!file_exists($this->configFile)) {
|
||||||
throw new \RuntimeException("Konfigurationsdatei nicht gefunden: {$this->configFile}");
|
throw new \RuntimeException("Configuration file not found: {$this->configFile}");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pathinfo($this->configFile, PATHINFO_EXTENSION) !== 'json') {
|
if (pathinfo($this->configFile, PATHINFO_EXTENSION) !== 'json') {
|
||||||
throw new \RuntimeException("Konfigurationsdatei muss eine JSON-Datei sein: {$this->configFile}");
|
throw new \RuntimeException("Configuration file must be a JSON file: {$this->configFile}");
|
||||||
}
|
}
|
||||||
|
|
||||||
$this->config = $this->loadJson($this->configFile);
|
$this->config = $this->loadJson($this->configFile);
|
||||||
@ -38,96 +38,96 @@ class ConfigurationLoader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lädt eine JSON-Datei
|
* Loads a JSON file
|
||||||
*
|
*
|
||||||
* @param string $file Pfad zur JSON-Datei
|
* @param string $file Path to JSON file
|
||||||
* @return array Geparste Konfiguration
|
* @return array Parsed configuration
|
||||||
*/
|
*/
|
||||||
private function loadJson(string $file): array
|
private function loadJson(string $file): array
|
||||||
{
|
{
|
||||||
$json = file_get_contents($file);
|
$json = file_get_contents($file);
|
||||||
if ($json === false) {
|
if ($json === false) {
|
||||||
throw new \RuntimeException("Konnte JSON-Datei nicht lesen: {$file}");
|
throw new \RuntimeException("Could not read JSON file: {$file}");
|
||||||
}
|
}
|
||||||
|
|
||||||
$config = json_decode($json, true);
|
$config = json_decode($json, true);
|
||||||
|
|
||||||
if ($config === null && json_last_error() !== JSON_ERROR_NONE) {
|
if ($config === null && json_last_error() !== JSON_ERROR_NONE) {
|
||||||
throw new \RuntimeException("Ungültiges JSON: " . json_last_error_msg());
|
throw new \RuntimeException("Invalid JSON: " . json_last_error_msg());
|
||||||
}
|
}
|
||||||
|
|
||||||
return $config;
|
return $config;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Validiert die geladene Konfiguration auf erforderliche Felder
|
* Validates the loaded configuration for required fields
|
||||||
*
|
*
|
||||||
* @throws \RuntimeException wenn erforderliche Felder fehlen
|
* @throws \RuntimeException if required fields are missing
|
||||||
*/
|
*/
|
||||||
private function validate(): void
|
private function validate(): void
|
||||||
{
|
{
|
||||||
// Metadata erforderlich
|
// Metadata required
|
||||||
if (empty($this->config['metadata'])) {
|
if (empty($this->config['metadata'])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'metadata' Section erforderlich");
|
throw new \RuntimeException("Configuration: 'metadata' section required");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!isset($this->config['metadata']['extractionRules']) || !is_array($this->config['metadata']['extractionRules'])) {
|
if (!isset($this->config['metadata']['extractionRules']) || !is_array($this->config['metadata']['extractionRules'])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'metadata.extractionRules' erforderlich (kann leer sein: [])");
|
throw new \RuntimeException("Configuration: 'metadata.extractionRules' required (may be empty: [])");
|
||||||
}
|
}
|
||||||
|
|
||||||
// CSV-Struktur erforderlich
|
// CSV structure required
|
||||||
if (empty($this->config['csvStructure'])) {
|
if (empty($this->config['csvStructure'])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'csvStructure' Section erforderlich");
|
throw new \RuntimeException("Configuration: 'csvStructure' section required");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!isset($this->config['csvStructure']['headerLine'])) {
|
if (!isset($this->config['csvStructure']['headerLine'])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'csvStructure.headerLine' erforderlich");
|
throw new \RuntimeException("Configuration: 'csvStructure.headerLine' required");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Column Transformations erforderlich
|
// Column transformations required
|
||||||
if (empty($this->config['columnTransformations'])) {
|
if (empty($this->config['columnTransformations'])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'columnTransformations' erforderlich");
|
throw new \RuntimeException("Configuration: 'columnTransformations' required");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Directories validieren (wenn auto-import genutzt wird)
|
// Validate directories (if auto-import is used)
|
||||||
if (!empty($this->config['directories'])) {
|
if (!empty($this->config['directories'])) {
|
||||||
foreach (['source', 'output', 'archive', 'error'] as $dir) {
|
foreach (['source', 'output', 'archive', 'error'] as $dir) {
|
||||||
if (empty($this->config['directories'][$dir])) {
|
if (empty($this->config['directories'][$dir])) {
|
||||||
throw new \RuntimeException("Konfiguration: 'directories.{$dir}' erforderlich für Auto-Import");
|
throw new \RuntimeException("Configuration: 'directories.{$dir}' required for auto-import");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Validiere CSV-Struktur Werte
|
// Validate CSV structure values
|
||||||
$headerLine = $this->config['csvStructure']['headerLine'] ?? 1;
|
$headerLine = $this->config['csvStructure']['headerLine'] ?? 1;
|
||||||
if (!is_int($headerLine) || $headerLine < 1) {
|
if (!is_int($headerLine) || $headerLine < 1) {
|
||||||
throw new \Exception(
|
throw new \Exception(
|
||||||
'Konfiguration csvStructure.headerLine muss eine positive Ganzzahl sein'
|
'Configuration: csvStructure.headerLine must be a positive integer'
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
$delimiter = $this->config['csvStructure']['inputDelimiter'] ?? '';
|
$delimiter = $this->config['csvStructure']['inputDelimiter'] ?? '';
|
||||||
if (strlen($delimiter) === 0) {
|
if (strlen($delimiter) === 0) {
|
||||||
throw new \Exception(
|
throw new \Exception(
|
||||||
'Konfiguration csvStructure.inputDelimiter darf nicht leer sein'
|
'Configuration: csvStructure.inputDelimiter must not be empty'
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Validiere Encoding
|
// Validate encoding
|
||||||
$encoding = $this->config['csvStructure']['encoding'] ?? 'UTF-8';
|
$encoding = $this->config['csvStructure']['encoding'] ?? 'UTF-8';
|
||||||
if (!in_array($encoding, ['UTF-8', 'ISO-8859-1', 'CP1252'])) {
|
if (!in_array($encoding, ['UTF-8', 'ISO-8859-1', 'CP1252'])) {
|
||||||
throw new \Exception(
|
throw new \Exception(
|
||||||
'Konfiguration csvStructure.encoding: ' . $encoding . ' nicht unterstützt'
|
'Configuration: csvStructure.encoding: ' . $encoding . ' not supported'
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt eine einzelne Konfigurationsoption zurück
|
* Returns a single configuration option
|
||||||
*
|
*
|
||||||
* @param string $key Dot-Notation Key (z.B. 'metadata.extractionRules')
|
* @param string $key Dot-notation key (e.g. 'metadata.extractionRules')
|
||||||
* @param mixed $default Standardwert wenn Key nicht existiert
|
* @param mixed $default Default value if key does not exist
|
||||||
* @return mixed Der Konfigurationswert
|
* @return mixed The configuration value
|
||||||
*/
|
*/
|
||||||
public function get(string $key, mixed $default = null): mixed
|
public function get(string $key, mixed $default = null): mixed
|
||||||
{
|
{
|
||||||
@ -145,9 +145,9 @@ class ConfigurationLoader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die vollständige Konfiguration zurück
|
* Returns the complete configuration
|
||||||
*
|
*
|
||||||
* @return array Die komplette Konfiguration
|
* @return array The full configuration
|
||||||
*/
|
*/
|
||||||
public function getAll(): array
|
public function getAll(): array
|
||||||
{
|
{
|
||||||
@ -155,10 +155,10 @@ class ConfigurationLoader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Setzt einen Konfigurationswert (überschreibt bestehenden Wert)
|
* Sets a configuration value (overwrites existing value)
|
||||||
*
|
*
|
||||||
* @param string $key Dot-Notation Key (z.B. 'directories.output')
|
* @param string $key Dot-notation key (e.g. 'directories.output')
|
||||||
* @param mixed $value Neuer Wert
|
* @param mixed $value New value
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
public function set(string $key, mixed $value): void
|
public function set(string $key, mixed $value): void
|
||||||
@ -179,9 +179,9 @@ class ConfigurationLoader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Prüft ob ein Konfigurationsschlüssel existiert
|
* Checks whether a configuration key exists
|
||||||
*
|
*
|
||||||
* @param string $key Dot-Notation Key
|
* @param string $key Dot-notation key
|
||||||
* @return bool
|
* @return bool
|
||||||
*/
|
*/
|
||||||
public function has(string $key): bool
|
public function has(string $key): bool
|
||||||
|
|||||||
@ -3,10 +3,10 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Liest und parst CSV-Dateien
|
* Reads and parses CSV files
|
||||||
*
|
*
|
||||||
* Diese Klasse liest CSV-Dateien mit konfigurierbarem Delimiter
|
* Reads CSV files with a configurable delimiter and separates
|
||||||
* und separiert Metadaten-Zeilen von den eigentlichen Datenzeilen.
|
* metadata lines from the actual data rows.
|
||||||
*/
|
*/
|
||||||
class CsvReader
|
class CsvReader
|
||||||
{
|
{
|
||||||
@ -16,8 +16,8 @@ class CsvReader
|
|||||||
private bool $hasBom;
|
private bool $hasBom;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param string $filePath Pfad zur CSV-Datei
|
* @param string $filePath Path to the CSV file
|
||||||
* @param array $csvStructure CSV-Struktur aus Konfiguration
|
* @param array $csvStructure CSV structure from configuration
|
||||||
*/
|
*/
|
||||||
public function __construct(string $filePath, array $csvStructure)
|
public function __construct(string $filePath, array $csvStructure)
|
||||||
{
|
{
|
||||||
@ -28,25 +28,25 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Liest alle Zeilen aus der Datei
|
* Reads all lines from the file
|
||||||
*
|
*
|
||||||
* @param int $maxLines Maximale Anzahl Zeilen (0 = alle)
|
* @param int $maxLines Maximum number of lines (0 = all)
|
||||||
* @return array Array mit Zeilen (ohne Newlines)
|
* @return array Array of lines (without newlines)
|
||||||
* @throws \RuntimeException wenn Datei nicht gelesen werden kann
|
* @throws \RuntimeException if file cannot be read
|
||||||
*/
|
*/
|
||||||
public function readLines(int $maxLines = 0): array
|
public function readLines(int $maxLines = 0): array
|
||||||
{
|
{
|
||||||
if (!file_exists($this->filePath) || !is_readable($this->filePath)) {
|
if (!file_exists($this->filePath) || !is_readable($this->filePath)) {
|
||||||
throw new \RuntimeException("Konnte Datei nicht lesen: {$this->filePath}");
|
throw new \RuntimeException("Could not read file: {$this->filePath}");
|
||||||
}
|
}
|
||||||
|
|
||||||
$lines = file($this->filePath, FILE_IGNORE_NEW_LINES);
|
$lines = file($this->filePath, FILE_IGNORE_NEW_LINES);
|
||||||
|
|
||||||
if ($lines === false) {
|
if ($lines === false) {
|
||||||
throw new \RuntimeException("Konnte Datei nicht lesen: {$this->filePath}");
|
throw new \RuntimeException("Could not read file: {$this->filePath}");
|
||||||
}
|
}
|
||||||
|
|
||||||
// BOM entfernen falls vorhanden
|
// Remove BOM if present
|
||||||
if ($this->hasBom && !empty($lines)) {
|
if ($this->hasBom && !empty($lines)) {
|
||||||
$lines[0] = $this->removeBom($lines[0]);
|
$lines[0] = $this->removeBom($lines[0]);
|
||||||
}
|
}
|
||||||
@ -59,9 +59,9 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Liest die Metadaten-Zeilen (vor der Header-Zeile)
|
* Reads the metadata lines (before the header line)
|
||||||
*
|
*
|
||||||
* @return array Array mit Metadaten-Zeilen
|
* @return array Array of metadata lines
|
||||||
*/
|
*/
|
||||||
public function readMetadataLines(): array
|
public function readMetadataLines(): array
|
||||||
{
|
{
|
||||||
@ -75,28 +75,28 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Liest die CSV-Daten mit Headers
|
* Reads CSV data with headers
|
||||||
*
|
*
|
||||||
* @param int $maxDataRows Maximale Anzahl Datenzeilen (0 = alle)
|
* @param int $maxDataRows Maximum number of data rows (0 = all)
|
||||||
* @return array Array von assoziativen Arrays (mit Spalten-Namen als Keys)
|
* @return array Array of associative arrays (with column names as keys)
|
||||||
* @throws \RuntimeException wenn Header-Zeile nicht gefunden
|
* @throws \RuntimeException if header line is not found
|
||||||
*/
|
*/
|
||||||
public function readCsvData(int $maxDataRows = 0): array
|
public function readCsvData(int $maxDataRows = 0): array
|
||||||
{
|
{
|
||||||
$lines = $this->readLines();
|
$lines = $this->readLines();
|
||||||
|
|
||||||
if ($this->headerLine > count($lines)) {
|
if ($this->headerLine > count($lines)) {
|
||||||
throw new \RuntimeException("Header-Zeile {$this->headerLine} nicht gefunden in Datei mit " . count($lines) . " Zeilen");
|
throw new \RuntimeException("Header line {$this->headerLine} not found in file with " . count($lines) . " lines");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Header parsen
|
// Parse header
|
||||||
$headerLineContent = $lines[$this->headerLine - 1];
|
$headerLineContent = $lines[$this->headerLine - 1];
|
||||||
$headers = str_getcsv($headerLineContent, $this->delimiter, '"', '\\');
|
$headers = str_getcsv($headerLineContent, $this->delimiter, '"', '\\');
|
||||||
$headers = array_map(static fn(?string $v): string => trim($v ?? ''), $headers);
|
$headers = array_map(static fn(?string $v): string => trim($v ?? ''), $headers);
|
||||||
|
|
||||||
// Datenzeilen parsen
|
// Parse data rows
|
||||||
$data = [];
|
$data = [];
|
||||||
$dataStartLine = $this->headerLine; // 0-basiert
|
$dataStartLine = $this->headerLine; // 0-based
|
||||||
$lineCount = 0;
|
$lineCount = 0;
|
||||||
|
|
||||||
for ($i = $dataStartLine; $i < count($lines); $i++) {
|
for ($i = $dataStartLine; $i < count($lines); $i++) {
|
||||||
@ -106,7 +106,7 @@ class CsvReader
|
|||||||
|
|
||||||
$lineContent = $lines[$i];
|
$lineContent = $lines[$i];
|
||||||
|
|
||||||
// Leere Zeilen überspringen
|
// Skip empty lines
|
||||||
if (trim($lineContent) === '') {
|
if (trim($lineContent) === '') {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@ -114,7 +114,7 @@ class CsvReader
|
|||||||
$row = str_getcsv($lineContent, $this->delimiter, '"', '\\');
|
$row = str_getcsv($lineContent, $this->delimiter, '"', '\\');
|
||||||
$row = array_map(static fn(?string $v): string => trim($v ?? ''), $row);
|
$row = array_map(static fn(?string $v): string => trim($v ?? ''), $row);
|
||||||
|
|
||||||
// Zeile mit Header-Keys kombinieren
|
// Combine row with header keys
|
||||||
$rowData = [];
|
$rowData = [];
|
||||||
foreach ($headers as $index => $header) {
|
foreach ($headers as $index => $header) {
|
||||||
$rowData[$header] = $row[$index] ?? '';
|
$rowData[$header] = $row[$index] ?? '';
|
||||||
@ -128,17 +128,17 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Spalten-Header zurück
|
* Returns the column headers
|
||||||
*
|
*
|
||||||
* @return array Array mit Spalten-Namen
|
* @return array Array of column names
|
||||||
* @throws \RuntimeException wenn Header-Zeile nicht gefunden
|
* @throws \RuntimeException if header line is not found
|
||||||
*/
|
*/
|
||||||
public function getHeaders(): array
|
public function getHeaders(): array
|
||||||
{
|
{
|
||||||
$lines = $this->readLines();
|
$lines = $this->readLines();
|
||||||
|
|
||||||
if ($this->headerLine > count($lines)) {
|
if ($this->headerLine > count($lines)) {
|
||||||
throw new \RuntimeException("Header-Zeile {$this->headerLine} nicht gefunden");
|
throw new \RuntimeException("Header line {$this->headerLine} not found");
|
||||||
}
|
}
|
||||||
|
|
||||||
$headerLineContent = $lines[$this->headerLine - 1];
|
$headerLineContent = $lines[$this->headerLine - 1];
|
||||||
@ -148,10 +148,10 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Entfernt UTF-8 BOM (Byte Order Mark) von String
|
* Removes UTF-8 BOM (Byte Order Mark) from string
|
||||||
*
|
*
|
||||||
* @param string $text String mit potenziellem BOM
|
* @param string $text String with potential BOM
|
||||||
* @return string String ohne BOM
|
* @return string String without BOM
|
||||||
*/
|
*/
|
||||||
private function removeBom(string $text): string
|
private function removeBom(string $text): string
|
||||||
{
|
{
|
||||||
@ -162,9 +162,9 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Gesamtzahl der Zeilen in der Datei zurück
|
* Returns the total number of lines in the file
|
||||||
*
|
*
|
||||||
* @return int Anzahl Zeilen
|
* @return int Number of lines
|
||||||
*/
|
*/
|
||||||
public function countLines(): int
|
public function countLines(): int
|
||||||
{
|
{
|
||||||
@ -172,9 +172,9 @@ class CsvReader
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Anzahl der Datenzeilen zurück (ohne Header und Metadaten)
|
* Returns the number of data rows (excluding header and metadata)
|
||||||
*
|
*
|
||||||
* @return int Anzahl Datenzeilen
|
* @return int Number of data rows
|
||||||
*/
|
*/
|
||||||
public function countDataRows(): int
|
public function countDataRows(): int
|
||||||
{
|
{
|
||||||
|
|||||||
@ -3,10 +3,9 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Schreibt transformierte Daten in CSV-Datei
|
* Writes transformed data to a CSV file
|
||||||
*
|
*
|
||||||
* Diese Klasse schreibt die transformierten Daten in eine
|
* Writes transformed data into a Firefly III-compatible CSV file.
|
||||||
* Firefly III-kompatible CSV-Datei.
|
|
||||||
*/
|
*/
|
||||||
class CsvWriter
|
class CsvWriter
|
||||||
{
|
{
|
||||||
@ -14,8 +13,8 @@ class CsvWriter
|
|||||||
private string $delimiter;
|
private string $delimiter;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param string $outputFile Pfad zur Output-Datei
|
* @param string $outputFile Path to the output file
|
||||||
* @param array $csvStructure CSV-Struktur aus Konfiguration
|
* @param array $csvStructure CSV structure from configuration
|
||||||
*/
|
*/
|
||||||
public function __construct(string $outputFile, array $csvStructure = [])
|
public function __construct(string $outputFile, array $csvStructure = [])
|
||||||
{
|
{
|
||||||
@ -24,39 +23,39 @@ class CsvWriter
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Schreibt Daten in CSV-Datei
|
* Writes data to a CSV file
|
||||||
*
|
*
|
||||||
* @param array $data Array von assoziativen Arrays (Zeilen)
|
* @param array $data Array of associative arrays (rows)
|
||||||
* @throws \RuntimeException wenn Datei nicht geschrieben werden kann
|
* @throws \RuntimeException if file cannot be written
|
||||||
*/
|
*/
|
||||||
public function write(array $data): void
|
public function write(array $data): void
|
||||||
{
|
{
|
||||||
if (empty($data)) {
|
if (empty($data)) {
|
||||||
throw new \RuntimeException("Keine Daten zum Schreiben");
|
throw new \RuntimeException("No data to write");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Output-Verzeichnis erstellen falls nicht vorhanden
|
// Create output directory if it does not exist
|
||||||
$dir = dirname($this->outputFile);
|
$dir = dirname($this->outputFile);
|
||||||
if (!is_dir($dir)) {
|
if (!is_dir($dir)) {
|
||||||
if (!mkdir($dir, 0755, true)) {
|
if (!mkdir($dir, 0755, true)) {
|
||||||
throw new \RuntimeException("Konnte Output-Verzeichnis nicht erstellen: {$dir}");
|
throw new \RuntimeException("Could not create output directory: {$dir}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
$fp = fopen($this->outputFile, 'w');
|
$fp = fopen($this->outputFile, 'w');
|
||||||
|
|
||||||
if ($fp === false) {
|
if ($fp === false) {
|
||||||
throw new \RuntimeException("Konnte Output-Datei nicht erstellen: {$this->outputFile}");
|
throw new \RuntimeException("Could not create output file: {$this->outputFile}");
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Headers schreiben (Spalten-Namen aus erster Zeile)
|
// Write headers (column names from first row)
|
||||||
$headers = array_keys($data[0]);
|
$headers = array_keys($data[0]);
|
||||||
$this->writeCsvLine($fp, $headers);
|
$this->writeCsvLine($fp, $headers);
|
||||||
|
|
||||||
// Datenzeilen schreiben
|
// Write data rows
|
||||||
foreach ($data as $row) {
|
foreach ($data as $row) {
|
||||||
// Sicherstellen dass alle Spalten vorhanden sind
|
// Ensure all columns are present
|
||||||
$values = [];
|
$values = [];
|
||||||
foreach ($headers as $header) {
|
foreach ($headers as $header) {
|
||||||
$values[] = $row[$header] ?? '';
|
$values[] = $row[$header] ?? '';
|
||||||
@ -70,25 +69,25 @@ class CsvWriter
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Schreibt eine CSV-Zeile mit fputcsv
|
* Writes a CSV line using fputcsv
|
||||||
*
|
*
|
||||||
* @param resource $fp File-Handle
|
* @param resource $fp File handle
|
||||||
* @param array $values Array mit Werten
|
* @param array $values Array of values
|
||||||
* @throws \RuntimeException wenn Schreiben fehlschlägt
|
* @throws \RuntimeException if writing fails
|
||||||
*/
|
*/
|
||||||
private function writeCsvLine($fp, array $values): void
|
private function writeCsvLine($fp, array $values): void
|
||||||
{
|
{
|
||||||
$result = fputcsv($fp, $values, $this->delimiter, '"', '\\');
|
$result = fputcsv($fp, $values, $this->delimiter, '"', '\\');
|
||||||
|
|
||||||
if ($result === false) {
|
if ($result === false) {
|
||||||
throw new \RuntimeException("Fehler beim Schreiben der CSV-Zeile");
|
throw new \RuntimeException("Error writing CSV row");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt den Pfad zur Output-Datei zurück
|
* Returns the path to the output file
|
||||||
*
|
*
|
||||||
* @return string Output-Dateipfad
|
* @return string Output file path
|
||||||
*/
|
*/
|
||||||
public function getOutputFile(): string
|
public function getOutputFile(): string
|
||||||
{
|
{
|
||||||
@ -96,9 +95,9 @@ class CsvWriter
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Prüft ob Output-Datei erstellt wurde
|
* Checks whether the output file was created
|
||||||
*
|
*
|
||||||
* @return bool True wenn Datei existiert
|
* @return bool True if file exists
|
||||||
*/
|
*/
|
||||||
public function fileExists(): bool
|
public function fileExists(): bool
|
||||||
{
|
{
|
||||||
@ -106,9 +105,9 @@ class CsvWriter
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Größe der Output-Datei zurück
|
* Returns the size of the output file
|
||||||
*
|
*
|
||||||
* @return int|false Dateigröße in Bytes oder false bei Fehler
|
* @return int|false File size in bytes or false on error
|
||||||
*/
|
*/
|
||||||
public function getFileSize(): int|false
|
public function getFileSize(): int|false
|
||||||
{
|
{
|
||||||
|
|||||||
@ -3,42 +3,42 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Zentraler Debug-Logger für Transparenz
|
* Central debug logger for transparency
|
||||||
*
|
*
|
||||||
* Sammelt Debug-Informationen aus allen Komponenten und macht die
|
* Collects debug information from all components and makes the
|
||||||
* Verarbeitung nachvollziehbar. Ermöglicht Transparenz über alle
|
* processing traceable. Provides transparency over all
|
||||||
* Verarbeitungsschritte: Metadaten-Extraktion, Transformationen,
|
* processing steps: metadata extraction, transformations,
|
||||||
* CSV-Lesevorgänge etc.
|
* CSV reads etc.
|
||||||
*
|
*
|
||||||
* Verwendung:
|
* Usage:
|
||||||
* - DebugLogger::enable() → Debug-Modus aktivieren
|
* - DebugLogger::enable() → activate debug mode
|
||||||
* - DebugLogger::log('category', 'message', $data) → Nachricht loggen
|
* - DebugLogger::log('category', 'message', $data) → log message
|
||||||
* - DebugLogger::getLogs() → Alle Logs abrufen
|
* - DebugLogger::getLogs() → retrieve all logs
|
||||||
* - DebugLogger::reset() → Logs zurücksetzen
|
* - DebugLogger::reset() → reset logs
|
||||||
*
|
*
|
||||||
* Beispiel:
|
* Example:
|
||||||
* ```php
|
* ```php
|
||||||
* DebugLogger::enable();
|
* DebugLogger::enable();
|
||||||
* DebugLogger::log('metadata', 'IBAN extrahiert', ['iban' => 'CH9300762011623852957']);
|
* DebugLogger::log('metadata', 'IBAN extracted', ['iban' => 'CH9300762011623852957']);
|
||||||
* $logs = DebugLogger::getLogs();
|
* $logs = DebugLogger::getLogs();
|
||||||
* ```
|
* ```
|
||||||
*/
|
*/
|
||||||
class DebugLogger
|
class DebugLogger
|
||||||
{
|
{
|
||||||
/**
|
/**
|
||||||
* @var bool Ist Debug-Modus aktiviert?
|
* @var bool Whether debug mode is enabled
|
||||||
*/
|
*/
|
||||||
private static bool $enabled = false;
|
private static bool $enabled = false;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @var array Gesammelte Logs mit Timestamp, Kategorie, Nachricht und Daten
|
* @var array Collected logs with timestamp, category, message and data
|
||||||
*/
|
*/
|
||||||
private static array $logs = [];
|
private static array $logs = [];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Aktiviert den Debug-Modus
|
* Enables debug mode
|
||||||
*
|
*
|
||||||
* Nach Aktivierung werden alle DebugLogger::log() Aufrufe protokolliert.
|
* Once enabled, all DebugLogger::log() calls are recorded.
|
||||||
*
|
*
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
@ -48,9 +48,9 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Deaktiviert den Debug-Modus
|
* Disables debug mode
|
||||||
*
|
*
|
||||||
* Nach Deaktivierung werden DebugLogger::log() Aufrufe ignoriert.
|
* Once disabled, DebugLogger::log() calls are ignored.
|
||||||
*
|
*
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
@ -60,16 +60,16 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Protokolliert eine Debug-Nachricht
|
* Records a debug message
|
||||||
*
|
*
|
||||||
* Sammelt Informationen über jeden Verarbeitungsschritt mit Timestamp,
|
* Collects information about each processing step with timestamp,
|
||||||
* Kategorie, Nachricht und optionalen Daten. Die Logs werden nur
|
* category, message and optional data. Logs are only collected
|
||||||
* gesammelt, wenn der Debug-Modus aktiviert ist.
|
* when debug mode is enabled.
|
||||||
*
|
*
|
||||||
* @param string $category Kategorie der Log-Nachricht
|
* @param string $category Log message category
|
||||||
* z.B. 'metadata', 'transformation', 'csv_reader', 'config'
|
* e.g. 'metadata', 'transformation', 'csv_reader', 'config'
|
||||||
* @param string $message Beschreibung der Aktion oder des Ereignisses
|
* @param string $message Description of the action or event
|
||||||
* @param mixed $data Zusätzliche Kontextdaten (Array oder beliebiger Wert)
|
* @param mixed $data Additional context data (array or any value)
|
||||||
*
|
*
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
@ -88,16 +88,16 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt alle gesammelten Logs zurück
|
* Returns all collected logs
|
||||||
*
|
*
|
||||||
* Liefert ein Array aller protokollierten Ereignisse mit vollständigen
|
* Delivers an array of all recorded events with complete
|
||||||
* Informationen für Analyse und Debugging.
|
* information for analysis and debugging.
|
||||||
*
|
*
|
||||||
* @return array Array von Log-Einträgen, jeder mit:
|
* @return array Array of log entries, each with:
|
||||||
* - timestamp: Mikrosekunden-Zeitstempel
|
* - timestamp: microsecond timestamp
|
||||||
* - category: Log-Kategorie
|
* - category: log category
|
||||||
* - message: Beschreibung
|
* - message: description
|
||||||
* - data: Zusätzliche Daten
|
* - data: additional data
|
||||||
*/
|
*/
|
||||||
public static function getLogs(): array
|
public static function getLogs(): array
|
||||||
{
|
{
|
||||||
@ -105,10 +105,10 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Setzt alle Logs zurück
|
* Resets all logs
|
||||||
*
|
*
|
||||||
* Löscht den gesamten Log-Buffer. Nützlich um zwischen mehreren
|
* Clears the entire log buffer. Useful for maintaining a clean
|
||||||
* Transformationen einen sauberen State zu haben.
|
* state between multiple transformations.
|
||||||
*
|
*
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
@ -118,9 +118,9 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Anzahl der gesammelten Log-Einträge zurück
|
* Returns the number of collected log entries
|
||||||
*
|
*
|
||||||
* @return int Anzahl protokollierter Ereignisse
|
* @return int Number of recorded events
|
||||||
*/
|
*/
|
||||||
public static function count(): int
|
public static function count(): int
|
||||||
{
|
{
|
||||||
@ -128,9 +128,9 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Prüft ob Debug-Modus aktiviert ist
|
* Checks whether debug mode is enabled
|
||||||
*
|
*
|
||||||
* @return bool true wenn aktiviert, false sonst
|
* @return bool true if enabled, false otherwise
|
||||||
*/
|
*/
|
||||||
public static function isEnabled(): bool
|
public static function isEnabled(): bool
|
||||||
{
|
{
|
||||||
@ -138,18 +138,18 @@ class DebugLogger
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt einen formattierten String aller Logs zurück
|
* Returns a formatted string of all logs
|
||||||
*
|
*
|
||||||
* Konvertiert den Log-Buffer in ein lesbares Format für Konsolen-Ausgabe.
|
* Converts the log buffer into a readable format for console output.
|
||||||
*
|
*
|
||||||
* @param bool $includeData true = auch Daten ausgeben, false = nur Messages
|
* @param bool $includeData true = also output data, false = messages only
|
||||||
*
|
*
|
||||||
* @return string Formatierte Log-Ausgabe
|
* @return string Formatted log output
|
||||||
*/
|
*/
|
||||||
public static function format(bool $includeData = true): string
|
public static function format(bool $includeData = true): string
|
||||||
{
|
{
|
||||||
if (empty(self::$logs)) {
|
if (empty(self::$logs)) {
|
||||||
return "Keine Debug-Logs vorhanden.\n";
|
return "No debug logs available.\n";
|
||||||
}
|
}
|
||||||
|
|
||||||
$output = "\n=== DEBUG LOGS ===\n";
|
$output = "\n=== DEBUG LOGS ===\n";
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@ -3,10 +3,10 @@
|
|||||||
namespace UbsCsvTransformer;
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extrahiert Metadaten aus Header-Zeilen mit Regex
|
* Extracts metadata from header lines using regex
|
||||||
*
|
*
|
||||||
* Diese Klasse extrahiert konstante Werte aus den Metadatenzeilen
|
* Extracts constant values from metadata lines
|
||||||
* (Header-Zeilen vor der eigentlichen CSV-Tabelle) mittels Regex-Regeln.
|
* (header lines before the actual CSV table) using regex rules.
|
||||||
*/
|
*/
|
||||||
class MetadataExtractor
|
class MetadataExtractor
|
||||||
{
|
{
|
||||||
@ -18,17 +18,17 @@ class MetadataExtractor
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extrahiert Metadaten aus den übergebenen Zeilen
|
* Extracts metadata from the provided lines
|
||||||
*
|
*
|
||||||
* @param array $lines Array von Zeilen aus dem CSV-Header
|
* @param array $lines Array of lines from the CSV header
|
||||||
* @return array Extrahierte Metadaten
|
* @return array Extracted metadata
|
||||||
*/
|
*/
|
||||||
public function extract(array $lines): array
|
public function extract(array $lines): array
|
||||||
{
|
{
|
||||||
$metadata = [];
|
$metadata = [];
|
||||||
|
|
||||||
foreach ($this->rules as $rule) {
|
foreach ($this->rules as $rule) {
|
||||||
// Validiere erforderliche Felder
|
// Validate required fields
|
||||||
if (empty($rule['name']) || empty($rule['regex'])) {
|
if (empty($rule['name']) || empty($rule['regex'])) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@ -37,15 +37,15 @@ class MetadataExtractor
|
|||||||
$lineNumber = $rule['lineNumber'] ?? 1;
|
$lineNumber = $rule['lineNumber'] ?? 1;
|
||||||
$regex = $rule['regex'];
|
$regex = $rule['regex'];
|
||||||
|
|
||||||
// ✅ KORRIGIERT: Off-by-One Fix
|
// Off-by-one fix
|
||||||
// config.json: "lineNumber": 1, 2, 3 (1-basiert, für Menschen lesbar)
|
// config.json: "lineNumber": 1, 2, 3 (1-based, human-readable)
|
||||||
// PHP Arrays: $lines[0], $lines[1], $lines[2] (0-basiert)
|
// PHP arrays: $lines[0], $lines[1], $lines[2] (0-based)
|
||||||
// Konvertierung: arrayIndex = lineNumber - 1
|
// Conversion: arrayIndex = lineNumber - 1
|
||||||
$arrayIndex = $lineNumber - 1;
|
$arrayIndex = $lineNumber - 1;
|
||||||
|
|
||||||
// Prüfe ob Zeile existiert
|
// Check if line exists
|
||||||
if (!isset($lines[$arrayIndex])) {
|
if (!isset($lines[$arrayIndex])) {
|
||||||
// Zeile existiert nicht - Debug-Info für Support
|
// Line does not exist - debug info for support
|
||||||
DebugLogger::log('metadata_warning', "Extraction rule not found", [
|
DebugLogger::log('metadata_warning', "Extraction rule not found", [
|
||||||
'rule_name' => $ruleName,
|
'rule_name' => $ruleName,
|
||||||
'expected_lineNumber' => $lineNumber,
|
'expected_lineNumber' => $lineNumber,
|
||||||
@ -57,7 +57,7 @@ class MetadataExtractor
|
|||||||
|
|
||||||
$line = $lines[$arrayIndex];
|
$line = $lines[$arrayIndex];
|
||||||
|
|
||||||
// Regex mit '#' als Delimiter (erlaubt '/' in User-Patterns); '#' im Pattern escapen
|
// Regex with '#' as delimiter (allows '/' in user patterns); escape '#' in pattern
|
||||||
$pattern = '#' . str_replace('#', '\#', $regex) . '#u';
|
$pattern = '#' . str_replace('#', '\#', $regex) . '#u';
|
||||||
$matchResult = @preg_match_all($pattern, $line, $matches);
|
$matchResult = @preg_match_all($pattern, $line, $matches);
|
||||||
if ($matchResult === false) {
|
if ($matchResult === false) {
|
||||||
@ -68,7 +68,7 @@ class MetadataExtractor
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if ($matchResult === 0) {
|
if ($matchResult === 0) {
|
||||||
// Regex matched nicht auf dieser Zeile
|
// Regex did not match on this line
|
||||||
DebugLogger::log('metadata_warning', "Regex did not match", [
|
DebugLogger::log('metadata_warning', "Regex did not match", [
|
||||||
'rule_name' => $ruleName,
|
'rule_name' => $ruleName,
|
||||||
'lineNumber' => $lineNumber,
|
'lineNumber' => $lineNumber,
|
||||||
@ -78,20 +78,20 @@ class MetadataExtractor
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ✅ KORRIGIERT: captureGroup benutzen
|
// Use captureGroup to select the extraction group
|
||||||
// captureGroup definiert welche Klammer-Gruppe extrahiert wird
|
// captureGroup defines which capture group is extracted
|
||||||
// 0 = komplette Match
|
// 0 = complete match
|
||||||
// 1 = erste Klammer-Gruppe (...)
|
// 1 = first capture group (...)
|
||||||
// 2 = zweite Klammer-Gruppe, etc.
|
// 2 = second capture group, etc.
|
||||||
$captureGroup = isset($rule['captureGroup']) ? intval($rule['captureGroup']) : 1;
|
$captureGroup = isset($rule['captureGroup']) ? intval($rule['captureGroup']) : 1;
|
||||||
|
|
||||||
// Sicherstellen dass die Capture Group existiert
|
// Ensure the capture group exists
|
||||||
if (!isset($matches[$captureGroup]) || empty($matches[$captureGroup])) {
|
if (!isset($matches[$captureGroup]) || empty($matches[$captureGroup])) {
|
||||||
// Fallback: Nutze komplette Match wenn Gruppe nicht existiert
|
// Fallback: use complete match if group does not exist
|
||||||
$metadata[$ruleName] = $matches[0][0] ?? '';
|
$metadata[$ruleName] = $matches[0][0] ?? '';
|
||||||
// echo "DEBUG: extraction_rule '{$ruleName}' - captureGroup {$captureGroup} not found, falling back to complete match\n";
|
// echo "DEBUG: extraction_rule '{$ruleName}' - captureGroup {$captureGroup} not found, falling back to complete match\n";
|
||||||
} else {
|
} else {
|
||||||
// Nutze die spezifische Capture Group
|
// Use the specific capture group
|
||||||
$metadata[$ruleName] = $matches[$captureGroup][0] ?? '';
|
$metadata[$ruleName] = $matches[$captureGroup][0] ?? '';
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -105,9 +105,9 @@ class MetadataExtractor
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Anzahl der definierten Extraction-Rules zurück
|
* Returns the number of defined extraction rules
|
||||||
*
|
*
|
||||||
* @return int Anzahl Rules
|
* @return int Number of rules
|
||||||
*/
|
*/
|
||||||
public function getRuleCount(): int
|
public function getRuleCount(): int
|
||||||
{
|
{
|
||||||
@ -115,9 +115,9 @@ class MetadataExtractor
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt alle definierten Extraction-Rules zurück
|
* Returns all defined extraction rules
|
||||||
*
|
*
|
||||||
* @return array Die Rules
|
* @return array The rules
|
||||||
*/
|
*/
|
||||||
public function getRules(): array
|
public function getRules(): array
|
||||||
{
|
{
|
||||||
|
|||||||
161
src/RowFilter.php
Normal file
161
src/RowFilter.php
Normal file
@ -0,0 +1,161 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
namespace UbsCsvTransformer;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Evaluates row-filter conditions defined in the "skipIf" config key.
|
||||||
|
*
|
||||||
|
* A node is either:
|
||||||
|
*
|
||||||
|
* - A bare condition:
|
||||||
|
* { "column": "A", "operator": "empty" }
|
||||||
|
*
|
||||||
|
* - An AND group:
|
||||||
|
* { "and": [ <node>, <node>, ... ] }
|
||||||
|
*
|
||||||
|
* - An OR group:
|
||||||
|
* { "or": [ <node>, <node>, ... ] }
|
||||||
|
*
|
||||||
|
* Groups may be nested arbitrarily.
|
||||||
|
*
|
||||||
|
* Supported operators for conditions:
|
||||||
|
*
|
||||||
|
* | Operator | Matches when … |
|
||||||
|
* |----------------|------------------------------------------------------|
|
||||||
|
* | empty | column value is empty string |
|
||||||
|
* | not-empty | column value is not empty |
|
||||||
|
* | equals | value === "value" (string compare) |
|
||||||
|
* | not-equals | value !== "value" |
|
||||||
|
* | contains | strpos(value, "value") !== false |
|
||||||
|
* | not-contains | strpos(value, "value") === false |
|
||||||
|
* | matches | preg_match("pattern", value) === 1 |
|
||||||
|
* | not-matches | preg_match("pattern", value) === 0 |
|
||||||
|
* | gt | (float) value > (float) "value" |
|
||||||
|
* | gte | (float) value >= (float) "value" |
|
||||||
|
* | lt | (float) value < (float) "value" |
|
||||||
|
* | lte | (float) value <= (float) "value" |
|
||||||
|
*
|
||||||
|
* Usage in config:
|
||||||
|
* ```json
|
||||||
|
* "skipIf": {
|
||||||
|
* "and": [
|
||||||
|
* { "column": "Beschreibung1", "operator": "empty" },
|
||||||
|
* { "column": "Beschreibung2", "operator": "empty" }
|
||||||
|
* ]
|
||||||
|
* }
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* ```json
|
||||||
|
* "skipIf": {
|
||||||
|
* "or": [
|
||||||
|
* { "column": "Amount", "operator": "gt", "value": "10000" },
|
||||||
|
* { "and": [
|
||||||
|
* { "column": "Type", "operator": "equals", "value": "Saldo" },
|
||||||
|
* { "column": "Notes", "operator": "empty" }
|
||||||
|
* ]}
|
||||||
|
* ]
|
||||||
|
* }
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
class RowFilter
|
||||||
|
{
|
||||||
|
/**
|
||||||
|
* Evaluates a filter node against a data row.
|
||||||
|
*
|
||||||
|
* Returns true when the row should be skipped.
|
||||||
|
*
|
||||||
|
* @param array<string, mixed> $node Filter node (condition or group)
|
||||||
|
* @param array<string, string> $row Data row with column values
|
||||||
|
*
|
||||||
|
* @throws \InvalidArgumentException on unknown operator
|
||||||
|
*/
|
||||||
|
public static function evaluate(array $node, array $row): bool
|
||||||
|
{
|
||||||
|
// AND group
|
||||||
|
if (isset($node['and'])) {
|
||||||
|
/** @var array<int, array<string, mixed>> $children */
|
||||||
|
$children = $node['and'];
|
||||||
|
foreach ($children as $child) {
|
||||||
|
if (!self::evaluate($child, $row)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// OR group
|
||||||
|
if (isset($node['or'])) {
|
||||||
|
/** @var array<int, array<string, mixed>> $children */
|
||||||
|
$children = $node['or'];
|
||||||
|
foreach ($children as $child) {
|
||||||
|
if (self::evaluate($child, $row)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bare condition
|
||||||
|
return self::evaluateCondition($node, $row);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Evaluates a single leaf condition.
|
||||||
|
*
|
||||||
|
* @param array<string, mixed> $condition
|
||||||
|
* @param array<string, string> $row
|
||||||
|
*
|
||||||
|
* @throws \InvalidArgumentException on unknown operator
|
||||||
|
*/
|
||||||
|
private static function evaluateCondition(array $condition, array $row): bool
|
||||||
|
{
|
||||||
|
$column = (string) ($condition['column'] ?? '');
|
||||||
|
$operator = strtolower((string) ($condition['operator'] ?? ''));
|
||||||
|
$colValue = (string) ($row[$column] ?? '');
|
||||||
|
$cmpValue = (string) ($condition['value'] ?? '');
|
||||||
|
$pattern = (string) ($condition['pattern'] ?? '');
|
||||||
|
|
||||||
|
switch ($operator) {
|
||||||
|
case 'empty':
|
||||||
|
return $colValue === '';
|
||||||
|
|
||||||
|
case 'not-empty':
|
||||||
|
return $colValue !== '';
|
||||||
|
|
||||||
|
case 'equals':
|
||||||
|
return $colValue === $cmpValue;
|
||||||
|
|
||||||
|
case 'not-equals':
|
||||||
|
return $colValue !== $cmpValue;
|
||||||
|
|
||||||
|
case 'contains':
|
||||||
|
return str_contains($colValue, $cmpValue);
|
||||||
|
|
||||||
|
case 'not-contains':
|
||||||
|
return !str_contains($colValue, $cmpValue);
|
||||||
|
|
||||||
|
case 'matches':
|
||||||
|
$delimited = '#' . str_replace('#', '\#', $pattern) . '#u';
|
||||||
|
return preg_match($delimited, $colValue) === 1;
|
||||||
|
|
||||||
|
case 'not-matches':
|
||||||
|
$delimited = '#' . str_replace('#', '\#', $pattern) . '#u';
|
||||||
|
return preg_match($delimited, $colValue) !== 1;
|
||||||
|
|
||||||
|
case 'gt':
|
||||||
|
return (float) $colValue > (float) $cmpValue;
|
||||||
|
|
||||||
|
case 'gte':
|
||||||
|
return (float) $colValue >= (float) $cmpValue;
|
||||||
|
|
||||||
|
case 'lt':
|
||||||
|
return (float) $colValue < (float) $cmpValue;
|
||||||
|
|
||||||
|
case 'lte':
|
||||||
|
return (float) $colValue <= (float) $cmpValue;
|
||||||
|
|
||||||
|
default:
|
||||||
|
throw new \InvalidArgumentException("Unknown RowFilter operator: '{$operator}'");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -8,18 +8,19 @@ use UbsCsvTransformer\ConfigurationLoader;
|
|||||||
use UbsCsvTransformer\MetadataExtractor;
|
use UbsCsvTransformer\MetadataExtractor;
|
||||||
use UbsCsvTransformer\ColumnTransformer;
|
use UbsCsvTransformer\ColumnTransformer;
|
||||||
use UbsCsvTransformer\FireflyImporter;
|
use UbsCsvTransformer\FireflyImporter;
|
||||||
|
use UbsCsvTransformer\RowFilter;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Orchestriert die gesamte CSV-Transformations-Pipeline
|
* Orchestrates the complete CSV transformation pipeline
|
||||||
*
|
*
|
||||||
* Koordiniert alle Schritte von CSV-Einlesen über Metadaten-Extraktion
|
* Coordinates all steps from reading the CSV through metadata extraction
|
||||||
* und Spalten-Transformation bis zur Ausgabe und optional zum Import in Firefly III.
|
* and column transformation to output and optional import into Firefly III.
|
||||||
*
|
*
|
||||||
* @property ConfigurationLoader $configLoader Verwaltet Konfiguration
|
* @property ConfigurationLoader $configLoader Manages configuration
|
||||||
* @property CsvWriter $csvWriter Schreibt Output-CSV
|
* @property CsvWriter $csvWriter Writes output CSV
|
||||||
* @property MetadataExtractor $metadataExtractor Extrahiert Metadaten aus Header
|
* @property MetadataExtractor $metadataExtractor Extracts metadata from header
|
||||||
* @property ColumnTransformer $columnTransformer Transformiert Spalten
|
* @property ColumnTransformer $columnTransformer Transforms columns
|
||||||
* @property array $csvStructure CSV-Struktur-Konfiguration
|
* @property array $csvStructure CSV structure configuration
|
||||||
*/
|
*/
|
||||||
class TransformerEngine
|
class TransformerEngine
|
||||||
{
|
{
|
||||||
@ -33,21 +34,24 @@ class TransformerEngine
|
|||||||
private bool $debugMode = false;
|
private bool $debugMode = false;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialisiert TransformerEngine mit Konfiguration
|
* Initialises TransformerEngine with configuration
|
||||||
*
|
*
|
||||||
* Lädt alle erforderlichen Konfigurationen und initialisiert
|
* Loads all required configurations and initialises
|
||||||
* die Komponenten (MetadataExtractor, ColumnTransformer, CsvWriter).
|
* the components (MetadataExtractor, ColumnTransformer, CsvWriter).
|
||||||
* CsvReader wird später in transform() und validate() initialisiert mit dem Dateipfad.
|
* CsvReader is instantiated later in transform() and validate() with the file path.
|
||||||
*
|
*
|
||||||
* @param ConfigurationLoader $configLoader Lädt Konfigurationsdateien
|
* @param ConfigurationLoader $configLoader Loads configuration files
|
||||||
* @param bool $debugMode true = Debug-Modus aktivieren
|
* @param bool $debugMode true = enable debug mode
|
||||||
*
|
*
|
||||||
* @throws \RuntimeException wenn erforderliche Konfigurationen fehlen
|
* @throws \RuntimeException if required configurations are missing
|
||||||
*/
|
*/
|
||||||
public function __construct(ConfigurationLoader $configLoader, bool $debugMode = false)
|
public function __construct(ConfigurationLoader $configLoader, bool $debugMode = false)
|
||||||
{
|
{
|
||||||
$this->configLoader = $configLoader;
|
$this->configLoader = $configLoader;
|
||||||
$this->debugMode = $debugMode;
|
$this->debugMode = $debugMode;
|
||||||
|
if ($debugMode) {
|
||||||
|
DebugLogger::enable();
|
||||||
|
}
|
||||||
|
|
||||||
$config = $configLoader->getAll();
|
$config = $configLoader->getAll();
|
||||||
|
|
||||||
@ -63,7 +67,7 @@ class TransformerEngine
|
|||||||
$config['capitalizationExceptions'] ?? []
|
$config['capitalizationExceptions'] ?? []
|
||||||
);
|
);
|
||||||
|
|
||||||
// Bestimme Output-Dateiname aus Konfiguration
|
// Determine output file name from configuration
|
||||||
$outputDir = $config['directories']['output'] ?? './output';
|
$outputDir = $config['directories']['output'] ?? './output';
|
||||||
$outputFileName = $config['csvStructure']['outputFilename'] ?? 'transformed.csv';
|
$outputFileName = $config['csvStructure']['outputFilename'] ?? 'transformed.csv';
|
||||||
$outputFile = rtrim($outputDir, '/') . '/' . $outputFileName;
|
$outputFile = rtrim($outputDir, '/') . '/' . $outputFileName;
|
||||||
@ -75,9 +79,9 @@ class TransformerEngine
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Aktiviert oder deaktiviert den Debug-Modus
|
* Enables or disables debug mode
|
||||||
*
|
*
|
||||||
* @param bool $enabled true = Debug-Modus aktiviert
|
* @param bool $enabled true = debug mode enabled
|
||||||
* @return void
|
* @return void
|
||||||
*/
|
*/
|
||||||
public function setDebugMode(bool $enabled): void
|
public function setDebugMode(bool $enabled): void
|
||||||
@ -91,30 +95,30 @@ class TransformerEngine
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transformiert eine CSV-Datei
|
* Transforms a CSV file
|
||||||
*
|
*
|
||||||
* Führt folgende Schritte durch:
|
* Performs the following steps:
|
||||||
* 1. CSV-Datei einlesen mit CsvReader
|
* 1. Read CSV file with CsvReader
|
||||||
* 2. Metadaten aus Header extrahieren
|
* 2. Extract metadata from header
|
||||||
* 3. Spalten gemäß Konfiguration transformieren
|
* 3. Transform columns according to configuration
|
||||||
* 4. Daten in Output-CSV schreiben
|
* 4. Write data to output CSV
|
||||||
* 5. Beispiel-Daten sammeln (maximal 3 Zeilen oder maxRows)
|
* 5. Collect sample data (maximum 3 rows or maxRows)
|
||||||
*
|
*
|
||||||
* Der Output-Dateipfad wird aus der Konfiguration bestimmt und kann nicht überschrieben werden.
|
* The output file path is determined from the configuration and cannot be overridden.
|
||||||
*
|
*
|
||||||
* @param string $inputFile Pfad zur Input-CSV-Datei
|
* @param string $inputFile Path to the input CSV file
|
||||||
* @param int $maxRows Maximale Anzahl Datenzeilen zu transformieren (0 = alle).
|
* @param int $maxRows Maximum number of data rows to transform (0 = all).
|
||||||
* Beispiel-Daten werden begrenzt auf min(3, maxRows)
|
* Sample data is limited to min(3, maxRows)
|
||||||
*
|
*
|
||||||
* @return array Transformations-Ergebnis mit:
|
* @return array Transformation result with:
|
||||||
* - success: bool (true = erfolgreich, false = Fehler)
|
* - success: bool (true = successful, false = error)
|
||||||
* - inputFile: string (Input-Dateipfad, nur bei Erfolg)
|
* - inputFile: string (input file path, on success only)
|
||||||
* - outputFile: string (Output-Dateipfad, nur bei Erfolg)
|
* - outputFile: string (output file path, on success only)
|
||||||
* - rowsProcessed: int (tatsächlich verarbeitete Datenzeilen)
|
* - rowsProcessed: int (actually processed data rows)
|
||||||
* - sampleData: array (Erste Beispiel-Zeilen, max 3 oder maxRows)
|
* - sampleData: array (first sample rows, max 3 or maxRows)
|
||||||
* - metadata: array (Extrahierte Metadaten, nur bei Erfolg)
|
* - metadata: array (extracted metadata, on success only)
|
||||||
* - outputColumns: int (Anzahl Output-Spalten)
|
* - outputColumns: int (number of output columns)
|
||||||
* - error: string (Fehlermeldung, nur bei Fehler)
|
* - error: string (error message, on failure only)
|
||||||
*/
|
*/
|
||||||
public function transform(string $inputFile, int $maxRows = 0): array
|
public function transform(string $inputFile, int $maxRows = 0): array
|
||||||
{
|
{
|
||||||
@ -130,50 +134,59 @@ class TransformerEngine
|
|||||||
]);
|
]);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Validiere Input-Datei
|
// Validate input file
|
||||||
if (!file_exists($inputFile)) {
|
if (!file_exists($inputFile)) {
|
||||||
throw new \RuntimeException("Input-Datei nicht gefunden: {$inputFile}");
|
throw new \RuntimeException("Input file not found: {$inputFile}");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Initialisiere CsvReader mit Dateipfad und Konfiguration
|
// Initialise CsvReader with file path and configuration
|
||||||
$csvReader = new CsvReader($inputFile, $this->csvStructure);
|
$csvReader = new CsvReader($inputFile, $this->csvStructure);
|
||||||
|
|
||||||
// Lese Metadaten-Zeilen (vor der Header-Zeile)
|
// Read metadata lines (before the header line)
|
||||||
$metadataLines = $csvReader->readMetadataLines();
|
$metadataLines = $csvReader->readMetadataLines();
|
||||||
|
|
||||||
// Extrahiere Metadaten aus den Metadaten-Zeilen
|
// Extract metadata from the metadata lines
|
||||||
$metadata = $this->metadataExtractor->extract($metadataLines);
|
$metadata = $this->metadataExtractor->extract($metadataLines);
|
||||||
|
|
||||||
// Initialisiere ColumnTransformer mit extrahierten Metadaten
|
// Initialise ColumnTransformer with extracted metadata
|
||||||
$this->columnTransformer = new ColumnTransformer(
|
$this->columnTransformer = new ColumnTransformer(
|
||||||
$this->configLoader->get('columnTransformations', []),
|
$this->configLoader->get('columnTransformations', []),
|
||||||
$metadata,
|
$metadata,
|
||||||
$this->configLoader->get('capitalizationExceptions', [])
|
$this->configLoader->get('capitalizationExceptions', [])
|
||||||
);
|
);
|
||||||
|
|
||||||
// Lese CSV-Daten mit Header-Keys als Array-Keys
|
// Read CSV data with header keys as array keys
|
||||||
$dataRows = $csvReader->readCsvData($maxRows);
|
$dataRows = $csvReader->readCsvData($maxRows);
|
||||||
if (empty($dataRows)) {
|
if (empty($dataRows)) {
|
||||||
throw new \RuntimeException("Keine Datenzeilen in CSV-Datei");
|
throw new \RuntimeException("No data rows in CSV file");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Berechne Limit für Beispiel-Daten
|
// Calculate limit for sample data
|
||||||
$sampleLimit = $maxRows == 0 ? 3 : $maxRows;
|
$sampleLimit = $maxRows == 0 ? 3 : $maxRows;
|
||||||
|
|
||||||
// Transformiere Zeilen und sammle sie
|
// Transform rows and collect them
|
||||||
$transformedData = [];
|
$transformedData = [];
|
||||||
|
|
||||||
|
/** @var array<string, mixed>|null $skipIfNode */
|
||||||
|
$skipIfNode = $this->configLoader->get('skipIf', null);
|
||||||
|
|
||||||
foreach ($dataRows as $row) {
|
foreach ($dataRows as $row) {
|
||||||
// Prüfe ob maxRows erreicht
|
// Check if maxRows reached
|
||||||
if ($maxRows > 0 && $this->rowsProcessed >= $maxRows) {
|
if ($maxRows > 0 && $this->rowsProcessed >= $maxRows) {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Transformiere Zeile
|
// Skip row if filter condition matches
|
||||||
|
if ($skipIfNode !== null && RowFilter::evaluate($skipIfNode, $row)) {
|
||||||
|
DebugLogger::log('transformer', 'Row skipped by skipIf filter', ['row' => $row]);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Transform row
|
||||||
$transformedRow = $this->columnTransformer->transformRow($row);
|
$transformedRow = $this->columnTransformer->transformRow($row);
|
||||||
$transformedData[] = $transformedRow;
|
$transformedData[] = $transformedRow;
|
||||||
|
|
||||||
// Speichere Beispiel-Daten
|
// Save sample data
|
||||||
if (count($this->sampleData) < $sampleLimit) {
|
if (count($this->sampleData) < $sampleLimit) {
|
||||||
$this->sampleData[] = $transformedRow;
|
$this->sampleData[] = $transformedRow;
|
||||||
}
|
}
|
||||||
@ -181,7 +194,7 @@ class TransformerEngine
|
|||||||
$this->rowsProcessed++;
|
$this->rowsProcessed++;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Entferne Spalten die aus dem Output ausgeschlossen werden sollen
|
// Remove columns to be excluded from the output
|
||||||
$excludeColumns = $this->csvStructure['excludeOutputColumns'] ?? [];
|
$excludeColumns = $this->csvStructure['excludeOutputColumns'] ?? [];
|
||||||
if (!empty($excludeColumns)) {
|
if (!empty($excludeColumns)) {
|
||||||
$excludeMap = array_flip($excludeColumns);
|
$excludeMap = array_flip($excludeColumns);
|
||||||
@ -195,7 +208,7 @@ class TransformerEngine
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Schreibe alle transformierten Daten in Output-CSV
|
// Write all transformed data to output CSV
|
||||||
$this->csvWriter->write($transformedData);
|
$this->csvWriter->write($transformedData);
|
||||||
|
|
||||||
$result = [
|
$result = [
|
||||||
@ -225,43 +238,43 @@ class TransformerEngine
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transformiert und importiert CSV in Firefly III
|
* Transforms and imports CSV into Firefly III
|
||||||
*
|
*
|
||||||
* Führt Transformation durch und importiert die Ausgabe-Datei
|
* Performs transformation and imports the output file
|
||||||
* in Firefly III wenn in der Konfiguration aktiviert.
|
* into Firefly III if enabled in the configuration.
|
||||||
*
|
*
|
||||||
* Rückwärts-kompatibel mit legacy Signatur.
|
* Backwards-compatible with legacy signature.
|
||||||
*
|
*
|
||||||
* @param string $inputFile Pfad zur Input-CSV-Datei
|
* @param string $inputFile Path to the input CSV file
|
||||||
* @param int $maxRows Maximale Anzahl Datenzeilen zu verarbeiten (0 = alle)
|
* @param int $maxRows Maximum number of data rows to process (0 = all)
|
||||||
*
|
*
|
||||||
* @return array Transformations- und Import-Ergebnis mit:
|
* @return array Transformation and import result with:
|
||||||
* - success: bool (true = transformation erfolgreich)
|
* - success: bool (true = transformation successful)
|
||||||
* - inputFile: string
|
* - inputFile: string
|
||||||
* - outputFile: string
|
* - outputFile: string
|
||||||
* - rowsProcessed: int
|
* - rowsProcessed: int
|
||||||
* - sampleData: array
|
* - sampleData: array
|
||||||
* - metadata: array
|
* - metadata: array
|
||||||
* - outputColumns: int
|
* - outputColumns: int
|
||||||
* - import: array (Firefly Import-Ergebnis, wenn autoImport aktiv)
|
* - import: array (Firefly import result, if autoImport active)
|
||||||
* - error: string (falls Fehler)
|
* - error: string (if error)
|
||||||
*/
|
*/
|
||||||
public function transformAndImport(string $inputFile, int $maxRows = 0): array
|
public function transformAndImport(string $inputFile, int $maxRows = 0): array
|
||||||
{
|
{
|
||||||
// Zuerst transformieren
|
// Transform first
|
||||||
$transformResult = $this->transform($inputFile, $maxRows);
|
$transformResult = $this->transform($inputFile, $maxRows);
|
||||||
|
|
||||||
if (!$transformResult['success']) {
|
if (!$transformResult['success']) {
|
||||||
return $transformResult;
|
return $transformResult;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Prüfe ob Auto-Import in Konfiguration aktiviert ist
|
// Check whether auto-import is enabled in configuration
|
||||||
$fireflyConfig = $this->configLoader->get('fireflyImport', []);
|
$fireflyConfig = $this->configLoader->get('fireflyImport', []);
|
||||||
if (empty($fireflyConfig['autoImport'])) {
|
if (empty($fireflyConfig['autoImport'])) {
|
||||||
return $transformResult;
|
return $transformResult;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Führe Firefly-Import durch
|
// Perform Firefly import
|
||||||
try {
|
try {
|
||||||
$importer = new FireflyImporter($fireflyConfig);
|
$importer = new FireflyImporter($fireflyConfig);
|
||||||
$importResult = $importer->import($transformResult['outputFile']);
|
$importResult = $importer->import($transformResult['outputFile']);
|
||||||
@ -278,19 +291,19 @@ class TransformerEngine
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Validiert eine CSV-Datei gegen die Konfiguration
|
* Validates a CSV file against the configuration
|
||||||
*
|
*
|
||||||
* Prüft ob erforderliche Metadaten vorhanden sind
|
* Checks whether required metadata is present
|
||||||
* und ob die CSV-Struktur der Konfiguration entspricht.
|
* and whether the CSV structure matches the configuration.
|
||||||
*
|
*
|
||||||
* @param string $inputFile Pfad zur zu validierenden CSV-Datei
|
* @param string $inputFile Path to the CSV file to validate
|
||||||
*
|
*
|
||||||
* @return array Validierungs-Ergebnis mit:
|
* @return array Validation result with:
|
||||||
* - valid: bool (true = Validierung erfolgreich)
|
* - valid: bool (true = validation successful)
|
||||||
* - metadata: array (Extrahierte Metadaten, wenn valid)
|
* - metadata: array (extracted metadata, when valid)
|
||||||
* - line_count: int (Gesamtzahl Zeilen, wenn valid)
|
* - line_count: int (total number of lines, when valid)
|
||||||
* - error: string (Fehlermeldung, wenn nicht valid)
|
* - error: string (error message, when not valid)
|
||||||
* - metadata_found: array (Gefundene Metadaten trotz Fehler)
|
* - metadata_found: array (found metadata despite error)
|
||||||
*/
|
*/
|
||||||
public function validate(string $inputFile): array
|
public function validate(string $inputFile): array
|
||||||
{
|
{
|
||||||
@ -298,18 +311,18 @@ class TransformerEngine
|
|||||||
if (!file_exists($inputFile)) {
|
if (!file_exists($inputFile)) {
|
||||||
return [
|
return [
|
||||||
'valid' => false,
|
'valid' => false,
|
||||||
'error' => "Datei nicht gefunden: {$inputFile}",
|
'error' => "File not found: {$inputFile}",
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
||||||
// Initialisiere CsvReader mit Dateipfad
|
// Initialise CsvReader with file path
|
||||||
$csvReader = new CsvReader($inputFile, $this->csvStructure);
|
$csvReader = new CsvReader($inputFile, $this->csvStructure);
|
||||||
|
|
||||||
// Extrahiere Metadaten-Zeilen (vor der Header-Zeile)
|
// Extract metadata lines (before the header line)
|
||||||
$metadataLines = $csvReader->readMetadataLines();
|
$metadataLines = $csvReader->readMetadataLines();
|
||||||
$metadata = $this->metadataExtractor->extract($metadataLines);
|
$metadata = $this->metadataExtractor->extract($metadataLines);
|
||||||
|
|
||||||
// Prüfe auf erforderliche Metadaten
|
// Check for required metadata
|
||||||
$requiredMetadata = [
|
$requiredMetadata = [
|
||||||
'account_iban',
|
'account_iban',
|
||||||
'currency_code',
|
'currency_code',
|
||||||
@ -325,12 +338,12 @@ class TransformerEngine
|
|||||||
if (!empty($missingMetadata)) {
|
if (!empty($missingMetadata)) {
|
||||||
return [
|
return [
|
||||||
'valid' => false,
|
'valid' => false,
|
||||||
'error' => 'Fehlende Metadaten: ' . implode(', ', $missingMetadata),
|
'error' => 'Missing metadata: ' . implode(', ', $missingMetadata),
|
||||||
'metadata_found' => $metadata,
|
'metadata_found' => $metadata,
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
||||||
// Zähle Gesamtzahl Zeilen
|
// Count total number of lines
|
||||||
$lineCount = $csvReader->countLines();
|
$lineCount = $csvReader->countLines();
|
||||||
|
|
||||||
return [
|
return [
|
||||||
@ -341,15 +354,15 @@ class TransformerEngine
|
|||||||
} catch (\Exception $e) {
|
} catch (\Exception $e) {
|
||||||
return [
|
return [
|
||||||
'valid' => false,
|
'valid' => false,
|
||||||
'error' => 'Validierungs-Fehler: ' . $e->getMessage(),
|
'error' => 'Validation error: ' . $e->getMessage(),
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die gesammelten Beispiel-Daten zurück
|
* Returns the collected sample data
|
||||||
*
|
*
|
||||||
* @return array Beispiel-Daten (maximal 3 oder maxRows Zeilen)
|
* @return array Sample data (maximum 3 or maxRows rows)
|
||||||
*/
|
*/
|
||||||
public function getSampleData(): array
|
public function getSampleData(): array
|
||||||
{
|
{
|
||||||
@ -357,9 +370,9 @@ class TransformerEngine
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Gibt die Anzahl verarbeiteter Datenzeilen zurück
|
* Returns the number of processed data rows
|
||||||
*
|
*
|
||||||
* @return int Anzahl transformierter Zeilen
|
* @return int Number of transformed rows
|
||||||
*/
|
*/
|
||||||
public function getRowsProcessed(): int
|
public function getRowsProcessed(): int
|
||||||
{
|
{
|
||||||
|
|||||||
@ -435,6 +435,90 @@ class ColumnTransformerTest extends TestCase
|
|||||||
$this->assertSame('Hello World', $result['B']);
|
$this->assertSame('Hello World', $result['B']);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public function testOutputActionAppendWithDelimiter(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append',
|
||||||
|
'appendDelimiter' => ', ',
|
||||||
|
], ['A' => 'World', 'B' => 'Hello']);
|
||||||
|
$this->assertSame('Hello, World', $result['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOutputActionAppendWithDelimiterSkippedWhenTargetEmpty(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append',
|
||||||
|
'appendDelimiter' => ', ',
|
||||||
|
], ['A' => 'Hello', 'B' => '']);
|
||||||
|
$this->assertSame('Hello', $result['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOutputActionAppendLine(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append-line',
|
||||||
|
], ['A' => 'Line2', 'B' => 'Line1']);
|
||||||
|
$this->assertSame("Line1\nLine2", $result['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOutputActionAppendLineNoLeadingNewlineWhenEmpty(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append-line',
|
||||||
|
], ['A' => 'Line1', 'B' => '']);
|
||||||
|
$this->assertSame('Line1', $result['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOutputActionOverwriteIfEmpty(): void
|
||||||
|
{
|
||||||
|
$resultEmpty = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'overwrite-if-empty',
|
||||||
|
], ['A' => 'new', 'B' => '']);
|
||||||
|
$this->assertSame('new', $resultEmpty['B']);
|
||||||
|
|
||||||
|
$resultFilled = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'overwrite-if-empty',
|
||||||
|
], ['A' => 'new', 'B' => 'existing']);
|
||||||
|
$this->assertSame('existing', $resultFilled['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOutputActionOverwriteIfNotEmpty(): void
|
||||||
|
{
|
||||||
|
$resultNotEmpty = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'overwrite-if-not-empty',
|
||||||
|
], ['A' => 'new', 'B' => 'old']);
|
||||||
|
$this->assertSame('new', $resultNotEmpty['B']);
|
||||||
|
|
||||||
|
$resultEmpty = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'overwrite-if-not-empty',
|
||||||
|
], ['A' => '', 'B' => 'old']);
|
||||||
|
$this->assertSame('old', $resultEmpty['B']);
|
||||||
|
}
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
// multi-output split
|
// multi-output split
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
@ -504,4 +588,139 @@ class ColumnTransformerTest extends TestCase
|
|||||||
$transformer->transformRow(['A' => '1', 'B' => '2', 'C' => '3']);
|
$transformer->transformRow(['A' => '1', 'B' => '2', 'C' => '3']);
|
||||||
$this->assertSame(2, $transformer->getOutputColumns());
|
$this->assertSame(2, $transformer->getOutputColumns());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// timeperiod
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** @var array<int, array<string, string>> */
|
||||||
|
private array $testPeriods = [
|
||||||
|
['from' => '04:00:00', 'to' => '08:59:59', 'label' => 'Morgen'],
|
||||||
|
['from' => '09:00:00', 'to' => '10:59:59', 'label' => 'Vormittag'],
|
||||||
|
['from' => '11:00:00', 'to' => '13:59:59', 'label' => 'Mittag'],
|
||||||
|
['from' => '14:00:00', 'to' => '17:59:59', 'label' => 'Nachmittag'],
|
||||||
|
['from' => '18:00:00', 'to' => '21:59:59', 'label' => 'Abend'],
|
||||||
|
['from' => '22:00:00', 'to' => '03:59:59', 'label' => 'Nacht'],
|
||||||
|
];
|
||||||
|
|
||||||
|
public function testTimePeriodBasicMapping(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'Time',
|
||||||
|
'outputColumn' => 'Period',
|
||||||
|
'type' => 'timeperiod',
|
||||||
|
'timeFormat' => 'H:i:s',
|
||||||
|
'periods' => $this->testPeriods,
|
||||||
|
'default' => '',
|
||||||
|
], ['Time' => '09:30:00', 'Period' => '']);
|
||||||
|
$this->assertSame('Vormittag', $result['Period']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTimePeriodMidnightSpanning(): void
|
||||||
|
{
|
||||||
|
$result1 = $this->applyOne([
|
||||||
|
'sourceColumn' => 'Time',
|
||||||
|
'outputColumn' => 'Period',
|
||||||
|
'type' => 'timeperiod',
|
||||||
|
'timeFormat' => 'H:i:s',
|
||||||
|
'periods' => $this->testPeriods,
|
||||||
|
'default' => '',
|
||||||
|
], ['Time' => '23:00:00', 'Period' => '']);
|
||||||
|
$this->assertSame('Nacht', $result1['Period']);
|
||||||
|
|
||||||
|
$result2 = $this->applyOne([
|
||||||
|
'sourceColumn' => 'Time',
|
||||||
|
'outputColumn' => 'Period',
|
||||||
|
'type' => 'timeperiod',
|
||||||
|
'timeFormat' => 'H:i:s',
|
||||||
|
'periods' => $this->testPeriods,
|
||||||
|
'default' => '',
|
||||||
|
], ['Time' => '02:00:00', 'Period' => '']);
|
||||||
|
$this->assertSame('Nacht', $result2['Period']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTimePeriodNoMatch(): void
|
||||||
|
{
|
||||||
|
// 03:45 falls outside all labelled ranges except Nacht (00:00-03:59)
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'Time',
|
||||||
|
'outputColumn' => 'Period',
|
||||||
|
'type' => 'timeperiod',
|
||||||
|
'timeFormat' => 'H:i:s',
|
||||||
|
'periods' => [
|
||||||
|
['from' => '09:00:00', 'to' => '17:59:59', 'label' => 'Day'],
|
||||||
|
],
|
||||||
|
'default' => 'Unknown',
|
||||||
|
], ['Time' => '03:45:00', 'Period' => '']);
|
||||||
|
$this->assertSame('Unknown', $result['Period']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testTimePeriodInvalidInput(): void
|
||||||
|
{
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'Time',
|
||||||
|
'outputColumn' => 'Period',
|
||||||
|
'type' => 'timeperiod',
|
||||||
|
'timeFormat' => 'H:i:s',
|
||||||
|
'periods' => $this->testPeriods,
|
||||||
|
'default' => 'N/A',
|
||||||
|
], ['Time' => '', 'Period' => '']);
|
||||||
|
$this->assertSame('N/A', $result['Period']);
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// ucwordsfirst guard
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testUcwordsFirstSkipsLowercase(): void
|
||||||
|
{
|
||||||
|
// Input already contains lowercase letters → must be returned unchanged
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'A',
|
||||||
|
'type' => 'ucwordsfirst',
|
||||||
|
], ['A' => 'Coop pronto chur']);
|
||||||
|
$this->assertSame('Coop pronto chur', $result['A']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testUcwordsFirstAppliesAllCaps(): void
|
||||||
|
{
|
||||||
|
// Fully uppercase input → capitalise first letter of each word
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'A',
|
||||||
|
'type' => 'ucwordsfirst',
|
||||||
|
], ['A' => 'COOP PRONTO']);
|
||||||
|
$this->assertSame('Coop Pronto', $result['A']);
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// append-if-not-empty
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testAppendIfNotEmptySkipsEmpty(): void
|
||||||
|
{
|
||||||
|
// Result is empty → target column must remain unchanged
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append-if-not-empty',
|
||||||
|
'appendDelimiter' => ' ',
|
||||||
|
], ['A' => '', 'B' => 'existing']);
|
||||||
|
$this->assertSame('existing', $result['B']);
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAppendIfNotEmptyAppendsNonEmpty(): void
|
||||||
|
{
|
||||||
|
// Non-empty result → appended with delimiter
|
||||||
|
$result = $this->applyOne([
|
||||||
|
'sourceColumn' => 'A',
|
||||||
|
'outputColumn' => 'B',
|
||||||
|
'type' => 'map',
|
||||||
|
'outputAction' => 'append-if-not-empty',
|
||||||
|
'appendDelimiter' => ' ',
|
||||||
|
], ['A' => 'new', 'B' => 'existing']);
|
||||||
|
$this->assertSame('existing new', $result['B']);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
414
tests/FireflyImporterChunkStateTest.php
Normal file
414
tests/FireflyImporterChunkStateTest.php
Normal file
@ -0,0 +1,414 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
namespace UbsCsvTransformer\Tests;
|
||||||
|
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
use UbsCsvTransformer\FireflyImporter;
|
||||||
|
use UbsCsvTransformer\DebugLogger;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests for the chunked-import state file / resume feature.
|
||||||
|
*
|
||||||
|
* Strategy: subclass FireflyImporter and override import() so no real HTTP or
|
||||||
|
* CLI call is made. The override is configured per test via a callable queue.
|
||||||
|
*/
|
||||||
|
class FireflyImporterChunkStateTest extends TestCase
|
||||||
|
{
|
||||||
|
/** @var string Temporary directory for CSV and state files */
|
||||||
|
private string $tmpDir;
|
||||||
|
|
||||||
|
/** @var string Path to a throwaway JSON config file */
|
||||||
|
private string $jsonConfig;
|
||||||
|
|
||||||
|
protected function setUp(): void
|
||||||
|
{
|
||||||
|
DebugLogger::reset();
|
||||||
|
|
||||||
|
$this->tmpDir = sys_get_temp_dir() . '/ffi_state_test_' . uniqid('', true);
|
||||||
|
mkdir($this->tmpDir, 0700, true);
|
||||||
|
|
||||||
|
// Minimal Firefly importer config (format v3)
|
||||||
|
$configData = [
|
||||||
|
'version' => 3,
|
||||||
|
'flow' => 'csv',
|
||||||
|
'roles' => ['amount'],
|
||||||
|
'default_account' => 1,
|
||||||
|
];
|
||||||
|
$this->jsonConfig = $this->tmpDir . '/ff-config.json';
|
||||||
|
file_put_contents($this->jsonConfig, json_encode($configData));
|
||||||
|
}
|
||||||
|
|
||||||
|
protected function tearDown(): void
|
||||||
|
{
|
||||||
|
// Remove all temp files
|
||||||
|
foreach (glob($this->tmpDir . '/*') ?: [] as $f) {
|
||||||
|
@unlink($f);
|
||||||
|
}
|
||||||
|
@rmdir($this->tmpDir);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Helpers ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates an importer stub whose import() calls return results from $queue
|
||||||
|
* in order. Each element of the queue is either true (success) or false (failure).
|
||||||
|
*
|
||||||
|
* @param array<bool> $importResultQueue
|
||||||
|
* @param int $chunkSize
|
||||||
|
*/
|
||||||
|
private function makeImporter(array $importResultQueue, int $chunkSize): FireflyImporter
|
||||||
|
{
|
||||||
|
$config = [
|
||||||
|
'mode' => 'http',
|
||||||
|
'importerUrl' => 'https://example.com',
|
||||||
|
'accessToken' => 'test-secret-1234567',
|
||||||
|
'personalSecret' => 'test-pat',
|
||||||
|
'jsonConfig' => $this->jsonConfig,
|
||||||
|
'chunkSize' => $chunkSize,
|
||||||
|
];
|
||||||
|
|
||||||
|
$queue = $importResultQueue;
|
||||||
|
|
||||||
|
return new class ($config, $queue) extends FireflyImporter {
|
||||||
|
/** @var array<bool> */
|
||||||
|
private array $queue;
|
||||||
|
|
||||||
|
/** @param array<bool> $queue */
|
||||||
|
public function __construct(array $config, array $queue)
|
||||||
|
{
|
||||||
|
parent::__construct($config);
|
||||||
|
$this->queue = $queue;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function import(string $csvFile): array
|
||||||
|
{
|
||||||
|
$success = array_shift($this->queue) ?? true;
|
||||||
|
if ($success) {
|
||||||
|
return [
|
||||||
|
'success' => true,
|
||||||
|
'exit_code' => 200,
|
||||||
|
'output' => ['stdout' => '', 'stderr' => ''],
|
||||||
|
'duration' => 1.0,
|
||||||
|
'csv_file' => $csvFile,
|
||||||
|
'summary' => [
|
||||||
|
'completed' => true,
|
||||||
|
'created' => 1,
|
||||||
|
'by_type' => ['withdrawal' => 1],
|
||||||
|
'duplicates' => 0,
|
||||||
|
'errors' => [],
|
||||||
|
],
|
||||||
|
];
|
||||||
|
}
|
||||||
|
return [
|
||||||
|
'success' => false,
|
||||||
|
'error' => 'Simulated failure',
|
||||||
|
'output' => ['stdout' => '', 'stderr' => ''],
|
||||||
|
'exit_code' => 500,
|
||||||
|
];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Writes a CSV with $dataRows data rows (each row has two columns).
|
||||||
|
*/
|
||||||
|
private function writeCsv(string $path, int $dataRows): void
|
||||||
|
{
|
||||||
|
$fp = fopen($path, 'w');
|
||||||
|
assert($fp !== false);
|
||||||
|
fputcsv($fp, ['col_a', 'col_b'], ',', '"', '\\');
|
||||||
|
for ($i = 1; $i <= $dataRows; $i++) {
|
||||||
|
fputcsv($fp, ["val_a_{$i}", "val_b_{$i}"], ',', '"', '\\');
|
||||||
|
}
|
||||||
|
fclose($fp);
|
||||||
|
}
|
||||||
|
|
||||||
|
private function stateFile(string $csvPath): string
|
||||||
|
{
|
||||||
|
return $csvPath . '.ffi-state.json';
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Tests ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When chunkSize is 0, import() is used directly — no state file should appear.
|
||||||
|
*/
|
||||||
|
public function testNoStateFileWhenChunkingNotUsed(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 5);
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([true], 0);
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertTrue($result['success']);
|
||||||
|
$this->assertFileDoesNotExist($this->stateFile($csv));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When the file has fewer rows than chunkSize, no chunking occurs — no state file.
|
||||||
|
*/
|
||||||
|
public function testNoStateFileWhenRowsBelowChunkSize(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 3);
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([true], 10);
|
||||||
|
$importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertFileDoesNotExist($this->stateFile($csv));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* After chunk 1 of 3 fails, the state file must exist and record 0 completed chunks.
|
||||||
|
*/
|
||||||
|
public function testStateFileCreatedOnFirstChunkFailure(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9); // 3 chunks of 3
|
||||||
|
|
||||||
|
// Chunk 1 fails immediately
|
||||||
|
$importer = $this->makeImporter([false], 3);
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertFalse($result['success']);
|
||||||
|
$this->assertFileExists($this->stateFile($csv));
|
||||||
|
|
||||||
|
/** @var array<string, mixed> $state */
|
||||||
|
$state = json_decode((string) file_get_contents($this->stateFile($csv)), true);
|
||||||
|
$this->assertSame([], $state['completed_chunks']);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* After chunks 1 and 2 succeed but chunk 3 fails, the state file records [0, 1].
|
||||||
|
*/
|
||||||
|
public function testStateFileRecordsCompletedChunksOnPartialFailure(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9); // 3 chunks of 3
|
||||||
|
|
||||||
|
// Chunks 0, 1 succeed; chunk 2 fails
|
||||||
|
$importer = $this->makeImporter([true, true, false], 3);
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertFalse($result['success']);
|
||||||
|
$this->assertFileExists($this->stateFile($csv));
|
||||||
|
|
||||||
|
/** @var array<string, mixed> $state */
|
||||||
|
$state = json_decode((string) file_get_contents($this->stateFile($csv)), true);
|
||||||
|
$this->assertSame([0, 1], $state['completed_chunks']);
|
||||||
|
$this->assertArrayHasKey('0', $state['chunk_results']);
|
||||||
|
$this->assertArrayHasKey('1', $state['chunk_results']);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* After full success the state file is deleted automatically.
|
||||||
|
*/
|
||||||
|
public function testStateFileDeletedAfterFullSuccess(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 6); // 2 chunks of 3
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([true, true], 3);
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertTrue($result['success']);
|
||||||
|
$this->assertFileDoesNotExist($this->stateFile($csv));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* On a second run with an existing state showing [0, 1] done, only chunk 2
|
||||||
|
* (index 2) should call import() — i.e., exactly one call is made.
|
||||||
|
*/
|
||||||
|
public function testResumeSkipsAlreadyCompletedChunks(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9); // 3 chunks of 3
|
||||||
|
|
||||||
|
// ── First run: chunks 0+1 succeed, chunk 2 fails ────────────────────
|
||||||
|
$run1 = $this->makeImporter([true, true, false], 3);
|
||||||
|
$run1->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertFileExists($this->stateFile($csv));
|
||||||
|
|
||||||
|
// ── Second run: only chunk 2 should be attempted ────────────────────
|
||||||
|
// We record how many times import() is actually called via a counting wrapper
|
||||||
|
$counter = new \stdClass();
|
||||||
|
$counter->value = 0;
|
||||||
|
|
||||||
|
$config = [
|
||||||
|
'mode' => 'http',
|
||||||
|
'importerUrl' => 'https://example.com',
|
||||||
|
'accessToken' => 'test-secret-1234567',
|
||||||
|
'personalSecret' => 'test-pat',
|
||||||
|
'jsonConfig' => $this->jsonConfig,
|
||||||
|
'chunkSize' => 3,
|
||||||
|
];
|
||||||
|
|
||||||
|
$run2 = new class ($config, $counter) extends FireflyImporter {
|
||||||
|
private \stdClass $counter;
|
||||||
|
|
||||||
|
public function __construct(array $config, \stdClass $counter)
|
||||||
|
{
|
||||||
|
parent::__construct($config);
|
||||||
|
$this->counter = $counter;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function import(string $csvFile): array
|
||||||
|
{
|
||||||
|
$this->counter->value++;
|
||||||
|
return [
|
||||||
|
'success' => true,
|
||||||
|
'exit_code' => 200,
|
||||||
|
'output' => ['stdout' => '', 'stderr' => ''],
|
||||||
|
'duration' => 1.0,
|
||||||
|
'csv_file' => $csvFile,
|
||||||
|
'summary' => [
|
||||||
|
'completed' => true,
|
||||||
|
'created' => 1,
|
||||||
|
'by_type' => ['withdrawal' => 1],
|
||||||
|
'duplicates' => 0,
|
||||||
|
'errors' => [],
|
||||||
|
],
|
||||||
|
];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
$result2 = $run2->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertTrue($result2['success']);
|
||||||
|
$this->assertSame(1, $counter->value, 'Only the 1 remaining chunk (index 2) should be imported');
|
||||||
|
$this->assertFileDoesNotExist($this->stateFile($csv), 'State file must be deleted after full success');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A state file whose total_rows does not match the current CSV is silently
|
||||||
|
* discarded and a fresh import is started.
|
||||||
|
*/
|
||||||
|
public function testStaleMismatchedStateIsIgnored(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9); // 3 chunks of 3
|
||||||
|
|
||||||
|
// Plant a stale state file with a wrong total_rows
|
||||||
|
$staleState = [
|
||||||
|
'csv_file' => realpath($csv) ?: $csv,
|
||||||
|
'total_rows' => 99, // wrong
|
||||||
|
'chunk_size' => 3,
|
||||||
|
'total_chunks' => 3,
|
||||||
|
'completed_chunks' => [0, 1],
|
||||||
|
'chunk_results' => [],
|
||||||
|
'created_at' => '2020-01-01T00:00:00+00:00',
|
||||||
|
'updated_at' => '2020-01-01T00:00:00+00:00',
|
||||||
|
];
|
||||||
|
file_put_contents($this->stateFile($csv), json_encode($staleState));
|
||||||
|
|
||||||
|
// All 3 chunks should be called (fresh start despite stale state)
|
||||||
|
$counter = new \stdClass();
|
||||||
|
$counter->value = 0;
|
||||||
|
|
||||||
|
$config = [
|
||||||
|
'mode' => 'http',
|
||||||
|
'importerUrl' => 'https://example.com',
|
||||||
|
'accessToken' => 'test-secret-1234567',
|
||||||
|
'personalSecret' => 'test-pat',
|
||||||
|
'jsonConfig' => $this->jsonConfig,
|
||||||
|
'chunkSize' => 3,
|
||||||
|
];
|
||||||
|
|
||||||
|
$importer = new class ($config, $counter) extends FireflyImporter {
|
||||||
|
private \stdClass $counter;
|
||||||
|
|
||||||
|
public function __construct(array $config, \stdClass $counter)
|
||||||
|
{
|
||||||
|
parent::__construct($config);
|
||||||
|
$this->counter = $counter;
|
||||||
|
}
|
||||||
|
|
||||||
|
public function import(string $csvFile): array
|
||||||
|
{
|
||||||
|
$this->counter->value++;
|
||||||
|
return [
|
||||||
|
'success' => true,
|
||||||
|
'exit_code' => 200,
|
||||||
|
'output' => ['stdout' => '', 'stderr' => ''],
|
||||||
|
'duration' => 1.0,
|
||||||
|
'csv_file' => $csvFile,
|
||||||
|
'summary' => [
|
||||||
|
'completed' => true,
|
||||||
|
'created' => 1,
|
||||||
|
'by_type' => ['withdrawal' => 1],
|
||||||
|
'duplicates' => 0,
|
||||||
|
'errors' => [],
|
||||||
|
],
|
||||||
|
];
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertTrue($result['success']);
|
||||||
|
$this->assertSame(3, $counter->value, 'All 3 chunks must be imported when stale state is discarded');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A corrupt (non-JSON) state file is silently discarded; no exception is thrown.
|
||||||
|
*/
|
||||||
|
public function testCorruptStateFileIsIgnored(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 6); // 2 chunks of 3
|
||||||
|
|
||||||
|
file_put_contents($this->stateFile($csv), '{this is not valid json!!!}');
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([true, true], 3);
|
||||||
|
$result = $importer->importChunked($csv);
|
||||||
|
|
||||||
|
$this->assertTrue($result['success']);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* resetImportState() deletes an existing state file.
|
||||||
|
*/
|
||||||
|
public function testResetImportStateClearsStateFile(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9);
|
||||||
|
|
||||||
|
// Plant a state file
|
||||||
|
file_put_contents($this->stateFile($csv), '{}');
|
||||||
|
$this->assertFileExists($this->stateFile($csv));
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([], 3);
|
||||||
|
$importer->resetImportState($csv);
|
||||||
|
|
||||||
|
$this->assertFileDoesNotExist($this->stateFile($csv));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* hasResumeState() returns false when no state file is present.
|
||||||
|
*/
|
||||||
|
public function testHasResumeStateReturnsFalseWithoutStateFile(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9);
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([], 3);
|
||||||
|
$this->assertFalse($importer->hasResumeState($csv));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* hasResumeState() returns true after a partial failure creates a valid state file.
|
||||||
|
*/
|
||||||
|
public function testHasResumeStateReturnsTrueAfterPartialFailure(): void
|
||||||
|
{
|
||||||
|
$csv = $this->tmpDir . '/test.csv';
|
||||||
|
$this->writeCsv($csv, 9); // 3 chunks of 3
|
||||||
|
|
||||||
|
$importer = $this->makeImporter([true, false], 3); // chunk 2 (index 1) fails
|
||||||
|
$importer->importChunked($csv);
|
||||||
|
|
||||||
|
$importer2 = $this->makeImporter([], 3);
|
||||||
|
$this->assertTrue($importer2->hasResumeState($csv));
|
||||||
|
}
|
||||||
|
}
|
||||||
255
tests/RowFilterTest.php
Normal file
255
tests/RowFilterTest.php
Normal file
@ -0,0 +1,255 @@
|
|||||||
|
<?php
|
||||||
|
|
||||||
|
namespace UbsCsvTransformer\Tests;
|
||||||
|
|
||||||
|
use PHPUnit\Framework\TestCase;
|
||||||
|
use UbsCsvTransformer\RowFilter;
|
||||||
|
|
||||||
|
class RowFilterTest extends TestCase
|
||||||
|
{
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// Leaf-condition operators
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testEmptyOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'empty'],
|
||||||
|
['A' => '']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'empty'],
|
||||||
|
['A' => 'something']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNotEmptyOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-empty'],
|
||||||
|
['A' => 'value']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-empty'],
|
||||||
|
['A' => '']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testEqualsOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'equals', 'value' => 'hello'],
|
||||||
|
['A' => 'hello']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'equals', 'value' => 'hello'],
|
||||||
|
['A' => 'world']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNotEqualsOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-equals', 'value' => 'hello'],
|
||||||
|
['A' => 'world']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-equals', 'value' => 'hello'],
|
||||||
|
['A' => 'hello']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testContainsOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'contains', 'value' => 'foo'],
|
||||||
|
['A' => 'foobar']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'contains', 'value' => 'baz'],
|
||||||
|
['A' => 'foobar']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNotContainsOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-contains', 'value' => 'baz'],
|
||||||
|
['A' => 'foobar']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-contains', 'value' => 'foo'],
|
||||||
|
['A' => 'foobar']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testMatchesOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'matches', 'pattern' => '^\d{4}$'],
|
||||||
|
['A' => '1234']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'matches', 'pattern' => '^\d{4}$'],
|
||||||
|
['A' => 'abcd']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testNotMatchesOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-matches', 'pattern' => '^\d{4}$'],
|
||||||
|
['A' => 'abcd']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'not-matches', 'pattern' => '^\d{4}$'],
|
||||||
|
['A' => '1234']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testGtOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'gt', 'value' => '100'],
|
||||||
|
['Amount' => '150.50']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'gt', 'value' => '100'],
|
||||||
|
['Amount' => '50']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testGteOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'gte', 'value' => '100'],
|
||||||
|
['Amount' => '100']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'gte', 'value' => '100'],
|
||||||
|
['Amount' => '99.99']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testLtOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'lt', 'value' => '100'],
|
||||||
|
['Amount' => '50']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'lt', 'value' => '100'],
|
||||||
|
['Amount' => '200']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testLteOperator(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'lte', 'value' => '100'],
|
||||||
|
['Amount' => '100']
|
||||||
|
));
|
||||||
|
$this->assertFalse(RowFilter::evaluate(
|
||||||
|
['column' => 'Amount', 'operator' => 'lte', 'value' => '100'],
|
||||||
|
['Amount' => '100.01']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// Groups
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testAndGroupBothTrue(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate([
|
||||||
|
'and' => [
|
||||||
|
['column' => 'A', 'operator' => 'empty'],
|
||||||
|
['column' => 'B', 'operator' => 'empty'],
|
||||||
|
],
|
||||||
|
], ['A' => '', 'B' => '']));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testAndGroupOneFalse(): void
|
||||||
|
{
|
||||||
|
$this->assertFalse(RowFilter::evaluate([
|
||||||
|
'and' => [
|
||||||
|
['column' => 'A', 'operator' => 'empty'],
|
||||||
|
['column' => 'B', 'operator' => 'empty'],
|
||||||
|
],
|
||||||
|
], ['A' => '', 'B' => 'not-empty']));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOrGroupOneTrue(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate([
|
||||||
|
'or' => [
|
||||||
|
['column' => 'A', 'operator' => 'equals', 'value' => 'yes'],
|
||||||
|
['column' => 'B', 'operator' => 'equals', 'value' => 'yes'],
|
||||||
|
],
|
||||||
|
], ['A' => 'no', 'B' => 'yes']));
|
||||||
|
}
|
||||||
|
|
||||||
|
public function testOrGroupBothFalse(): void
|
||||||
|
{
|
||||||
|
$this->assertFalse(RowFilter::evaluate([
|
||||||
|
'or' => [
|
||||||
|
['column' => 'A', 'operator' => 'equals', 'value' => 'yes'],
|
||||||
|
['column' => 'B', 'operator' => 'equals', 'value' => 'yes'],
|
||||||
|
],
|
||||||
|
], ['A' => 'no', 'B' => 'no']));
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// Nested groups
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testNestedAndOrGroup(): void
|
||||||
|
{
|
||||||
|
// (A is empty) AND (B equals "foo" OR C not-empty)
|
||||||
|
$node = [
|
||||||
|
'and' => [
|
||||||
|
['column' => 'A', 'operator' => 'empty'],
|
||||||
|
[
|
||||||
|
'or' => [
|
||||||
|
['column' => 'B', 'operator' => 'equals', 'value' => 'foo'],
|
||||||
|
['column' => 'C', 'operator' => 'not-empty'],
|
||||||
|
],
|
||||||
|
],
|
||||||
|
],
|
||||||
|
];
|
||||||
|
|
||||||
|
// A empty, B matches → true
|
||||||
|
$this->assertTrue(RowFilter::evaluate($node, ['A' => '', 'B' => 'foo', 'C' => '']));
|
||||||
|
// A empty, C not-empty → true
|
||||||
|
$this->assertTrue(RowFilter::evaluate($node, ['A' => '', 'B' => 'bar', 'C' => 'value']));
|
||||||
|
// A empty, but neither B nor C match → false
|
||||||
|
$this->assertFalse(RowFilter::evaluate($node, ['A' => '', 'B' => 'bar', 'C' => '']));
|
||||||
|
// A not empty → false
|
||||||
|
$this->assertFalse(RowFilter::evaluate($node, ['A' => 'x', 'B' => 'foo', 'C' => '']));
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// Unknown operator
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testUnknownOperatorThrows(): void
|
||||||
|
{
|
||||||
|
$this->expectException(\InvalidArgumentException::class);
|
||||||
|
RowFilter::evaluate(
|
||||||
|
['column' => 'A', 'operator' => 'nonexistent'],
|
||||||
|
['A' => 'value']
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// Missing column (treats as empty string)
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
public function testMissingColumnTreatedAsEmpty(): void
|
||||||
|
{
|
||||||
|
$this->assertTrue(RowFilter::evaluate(
|
||||||
|
['column' => 'NonExistent', 'operator' => 'empty'],
|
||||||
|
['A' => 'something']
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
31
tests/fixtures/config-ubs-account/expected.csv
vendored
31
tests/fixtures/config-ubs-account/expected.csv
vendored
@ -1,17 +1,14 @@
|
|||||||
Belastung,Gutschrift,date,process_date,opposing_name,tags,description,opposing_account,notes,account_iban,account_currency
|
Belastung,Gutschrift,date,process_date,tags,opposing_iban,opposing_account,opposing_name,notes,description,account_iban,account_currency
|
||||||
-600.00,,2022-12-30,2022-12-30,"David Peter Reindl",Dauerauftrag,"Steuerrueckstellung
|
-600.00,,2022-12-30,2022-12-30,Dauerauftrag,"CH37 0026 7267 9314 35M2 P",,"David Peter Reindl","8906 Bonstetten
|
||||||
David Peter Reindl;8906 Bonstetten","CH37 0026 7267 9314 35M2 P","9967864LK2659211
|
9967864LK2659211","David Peter Reindl;8906 Bonstetten; STEUERRUECKSTELLUNG; Dauerauftrag","CH18 0026 7267 9314 3540 D",CHF
|
||||||
8906 Bonstetten","CH18 0026 7267 9314 3540 D",CHF
|
-46.35,,2022-12-30,2022-12-31,,,,"UBS AG",9900365AP6356307,"Saldo Zinsabschluss; Periode: 2022-10-01 - 2022-12-30","CH18 0026 7267 9314 3540 D",CHF
|
||||||
-46.35,,2022-12-30,2022-12-31,"UBS AG",,"Periode: 2022-10-01 - 2022-12-30
|
-39.90,,2022-12-30,2022-12-30,TWINT,,,"Swisscom Grossunternehme","Muellerstrasse 16 8004 Zuerich TWINT-Acc.:+41796305690
|
||||||
Zinsabschluss",,9900365AP6356307,"CH18 0026 7267 9314 3540 D",CHF
|
9967364GK5707142","SWISSCOM GROSSUNTERNEHME; Zahlung UBS TWINT; Muellerstrasse 16 na, 8004 Zuerich TWINT-Acc.:+41796305690","CH18 0026 7267 9314 3540 D",CHF
|
||||||
-39.90,,2022-12-30,2022-12-30,"Swisscom Grossunternehme",TWINT,"Swisscom Grossunternehme; Zahlung UBS TWINT",,"9967364GK5707142
|
-8.75,,2022-12-28,2022-12-27,"Abend Debitkarte",,,"Coop Pronto Chur","18279748-0 08/24
|
||||||
8004 Zuerich","CH18 0026 7267 9314 3540 D",CHF
|
7007 Chur
|
||||||
-8.75,,2022-12-28,2022-12-27,"Coop Pronto Chur",Debitkarte,"18279748-0 08/24
|
9930862BN7826808","Coop Pronto Chur;7007 Chur; Zahlung Debitkarte","CH18 0026 7267 9314 3540 D",CHF
|
||||||
Coop Pronto Chur;7007 Chur",,"9930862BN7826808
|
-1800.00,,2022-12-27,2022-12-27,e-banking,"CH63 0023 2232 5560 5988 0",,"Janine Geigele","8049 Zuerich
|
||||||
7007 Chur","CH18 0026 7267 9314 3540 D",CHF
|
9967361TI3188436","Janine Geigele;Am Wasser 36; 8049 Zuerich; CH; SKIFERIEN DOLOMITEN; e-banking-Vergütungsauftrag; Wohnung Dolomiten, 2 Personen","CH18 0026 7267 9314 3540 D",CHF
|
||||||
-1800.00,,2022-12-27,2022-12-27,"Janine Geigele",e-banking,"Skiferien Dolomiten
|
,9.00,2022-12-22,2022-12-22,TWINT,,,"Friis, Daniela Silvia",9930356GK0440989,"Friis, Daniela Silvia; Gutschrift UBS TWINT; +41796741245; TWINT-Acc.:+41796305690","CH18 0026 7267 9314 3540 D",CHF
|
||||||
Janine Geigele;Am Wasser 36; 8049 Zuerich; CH","CH63 0023 2232 5560 5988 0","9967361TI3188436
|
,19764.80,2022-11-25,2022-11-25,Gutschrift,,,SBB,9901820E67741531,"SBB;Corporate Treasury; Gutschrift; Lohn/Gehalt 00229537/202211","CH18 0026 7267 9314 3540 D",CHF
|
||||||
8049 Zuerich","CH18 0026 7267 9314 3540 D",CHF
|
-14.00,,2022-08-22,2022-08-21,TWINT,,,"Friis-Loop, Daniela",9967233GK1553933,"FRIIS-LOOP, DANIELA; Belastung UBS TWINT; +41796741245; TWINT-Acc.:+41796305690","CH18 0026 7267 9314 3540 D",CHF
|
||||||
,9.00,2022-12-22,2022-12-22,"Friis, Daniela Silvia",TWINT,"Friis, Daniela Silvia",,9930356GK0440989,"CH18 0026 7267 9314 3540 D",CHF
|
|
||||||
,19764.80,2022-11-25,2022-11-25,SBB,Gutschrift,"SBB;Corporate Treasury",,9901820E67741531,"CH18 0026 7267 9314 3540 D",CHF
|
|
||||||
-14.00,,2022-08-22,2022-08-21,"Friis-Loop, Daniela",TWINT,"Friis-Loop, Daniela; Belastung UBS TWINT",,9967233GK1553933,"CH18 0026 7267 9314 3540 D",CHF
|
|
||||||
|
|||||||
|
Loading…
Reference in New Issue
Block a user