2025-06-03_coordinate_management_workflow#

        graph TD
    subgraph Configuration
      Env[.env: DB_URL, other config]
      Config[config.py]
    end

    Env --> Config
    Config --> DB[SQLAlchemy Engine]
    DB --> CTable[Coordinates Table]
    CTable --> CoordMgr[coordinates_manager.py]
    CoordMgr -->|get_coordinates()| WF[weather_ingest.py]
    CoordMgr -->|get_coordinates()| ML[ml.py / other modules]

1. Rationale#

  • Single source of truth for all coordinate points.

  • Dynamic generation and seeding based on central coordinates.

  • Persistence and consistency across modules.

  • Clear detection and handling of coordinate changes.

2. High-Level Workflow#

  1. Read MY_LAT and MY_LONG from .env.

  2. Initialize DB and create coordinates table if missing.

  3. On first run (empty table), generate and seed 33 points.

  4. On subsequent runs, verify stored central coordinates match .env.

  5. If mismatch, exit with clear message and backup instructions.

  6. Provide set_coordinate and get_coordinates via coordinates_manager.py.

  7. Replace all static coordinate lists with get_coordinates().

  8. Remove hardcoded MY_LAT, MY_LONG usage elsewhere.

3. File-by-File Migration Steps#

Step

File(s)

Change

1

Models

Add class Coordinate(Base) with id, latitude, longitude.

2

Database

Call Base.metadata.create_all(bind=engine) after engine setup.

3

Coordinates Manager

Create module with seed_defaults(), set_coordinate(), get_coordinates().

4

Main Application

On startup event, seed defaults or verify central coords.

5

Weather Ingest

Remove static coordinates list; fetch via get_coordinates().

6

Coordinates Setup

Remove hardcoded logic; optionally use for generation helper.

7

Environment Config

Keep MY_LAT, MY_LONG and DB config; remove other coords config.

8

Coordinates Rationale

Update documentation.

4. Advanced Coordinate Change Detection#

  • Compare .env central coords with DB on startup.

  • On mismatch, exit and prompt user to backup:

    • Data in weather.db or other storage.

    • ML model artifacts.

    • Provide instructions to re-run seeding.

5. Next Steps#

  1. Switch to 💻 Code mode to implement the migration.

  2. Write coordinates_manager.py and update models.

  3. Refactor modules to use get_coordinates().

  4. Test end-to-end.

  5. Document release notes.

6. Data Ingestion & Storage Workflow#

        graph TD
  subgraph Configuration
    Env[.env: DATABASE_PATH, POLLING_INTERVAL_MINUTES, DATA_INGEST_INTERVAL_MINUTES, USER_AGENT]
    Config[config.py]
  end
  Env --> Config

  subgraph Database
    DBEngine[SQLite Engine] -->|WAL & busy timeout| DBFile[weather.db]
  end
  Config --> DBEngine

  subgraph Ingestion
    CLI[CLI script (coordinates_setup.py)] -->|calls| Ingest[ingest_weather_for_all_coordinates in weather_ingest.py]
    BG[FastAPI background job] -->|calls| Ingest
    Ingest --> HTTP[aiohttp.ClientSession]
    Ingest --> Conn[aiosqlite.Connection]
    Ingest --> DBEngine
  end

6.1 Unified Ingestion Logic#

The async function ingest_weather_for_all_coordinates in weather_ingest.py is the single entry point for fetching and storing weather data. It is invoked by both:

  • A CLI script that opens an aiosqlite.Connection and aiohttp.ClientSession. Example:

    python -m src.app.coordinates_setup

  • FastAPI background jobs via APScheduler job trigger_weather_ingestion_cycle, scheduled from main.py.

6.2 Configuration Centralization#

All operational settings live in config.py:

  • DATABASE_PATH (env var DATABASE_PATH, default weather.db)

  • POLLING_INTERVAL_MINUTES (env var POLLING_INTERVAL_MINUTES, default 10)

  • DATA_INGEST_INTERVAL_MINUTES (env var DATA_INGEST_INTERVAL_MINUTES, default 1)

  • USER_AGENT (env var USER_AGENT, default SkolDataApp/1.0 (kontakt@skoldata.se))

Environment variables override defaults, allowing customization without code changes.

6.3 SQLite Concurrency#

SQLite is configured in database.py:

  • PRAGMA journal_mode=WAL enables Write-Ahead Logging for better concurrency reads and writes.

  • PRAGMA busy_timeout=60000 (60 s) makes write attempts wait on locks before erroring.

Caveats:

  • Long-running transactions can block writers.

  • Frequent writes may cause busy timeouts; monitor logs for OperationalError: database is locked.

6.4 Operational Recommendations#

  • Manual Runs

    DATABASE_PATH=weather.db DATA_INGEST_INTERVAL_MINUTES=5 python -m src.app.coordinates_setup

  • Debugging

    LOG_LEVEL=DEBUG python -m src.app.coordinates_setup

    Or configure logging in main.py / __init__.py:

    import logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s %(levelname)s %(message)s")

  • Production

    • Use a dedicated background worker (e.g., systemd timer or Kubernetes CronJob).

    • Monitor for busy-locks and error logs.

    • Plan for migration to a client/server database (e.g., PostgreSQL) once data volume and concurrency demands exceed SQLite’s capabilities.

End of document.